Zephyrnet Logo

The Gensyn Protocol Trustlessly Trains Neural Networks at Hyperscale with Lower Order of Magnitude…

Date:


The Gensyn Protocol Trustlessly Trains Neural Networks at Hyperscale with Lower Order of Magnitude of Cost

Links: Gensyn website, Litepaper, CoinFund Portfolio, TechCrunch Article Link

Investment Thesis Summary

  • Secular Leverage to ML Growing Complexity and Value: The computational complexity of state of the art AI systems is doubling every 3 months, while the value of these models are continuing to increase quickly, while the former black-box-nature of these algorithms are now increasingly able to be fit with greater human-understandable illuminators.
  • Novel Coordination and Verification System Design: Gensyn is building a verification system (testnet v1 will be deployed later this year) which efficiently solves the state dependency problem in neural network training at any scale. The system combines model training checkpoints with probabilistic checks that terminate on-chain. It does all of this trustlessly and the overhead scales linearly with model size (keeping verification costs constant).
  • Thematic Focus on AI Decentralization: Most of the well-known examples of machine learning applications (Tesla self-driving cars, Google DeepMind) are produced by the same set of companies, that’s because the deep learning industry currently looks like a game of monopoly between Big Tech companies, as well as states like China and the United States. These forces are resulting in huge centralization forces that run contrary to web3 and even the historical origins of web1.

CoinFund is proud to support Gensyn Protocol’s recent fundraise and the team’s vision to enable trustlessly training neural networks at hyperscale and low cost through their novel verification system. Utilizing probabilistic checks that terminate on-chain while tapping into underutilized and underused compute sources ranging from presently underutilized gaming GPUs to sophisticated ETH1 mining pools about to detach from the Ethereum network as that network transitions to Proof of Stake, Gensyn protocol requires no administrative overseer or legal enforcement, rather facilitating task distribution and payments programmatically through smart contracts. Better still, the protocol’s decentralized nature means it will ultimately be majority community governed and cannot be ‘turned off’ without community consent; this makes it censorship resistant, unlike its web2 counterparts. Ultimately, we believe Gensyn is playing to become the foundational layer for web3-native ML compute, as third party participants eventually build rich user experiences and specific functionality in numerous niches.

Part 1: Introduction to Deep Learning’s Multi-decade Secular Growth

Every face you see on a video call and all the audio you hear is manipulated. To improve call quality, neural networks selectively adjust the resolution in Zoom and suppress background noise in Microsoft Teams. More recent advances even see lower resolution video ‘dreamed’ into a higher resolution. Neural networks are the models used in the deep learning branch of artificial intelligence. They are loosely based on the structure of the human brain and have myriad applications, perhaps ultimately creating human level artificial intelligence. Bigger models generally yield better results, and the hardware required for state-of-the-art development is doubling every three months. This explosion in development has made deep learning a fundamental part of the modern human experience. In 2020, a neural network operated the radar on a US spy plane, language models now write better scam emails than humans, and self-driving car algorithms outperform humans in many environments.

GPT-3 175B, the largest GPT-3 model proposed by OpenAI in Brown et al. (2020) used a cluster of 1,000 NVIDIA Tesla V100 GPUs for training — roughly equivalent to 355 years of training on a single device. DALL-E from Ramesh et al. (2021), another Transformer model from OpenAI, has 12 billion parameters and was trained on over 400 million captioned images. OpenAI bore the cost of training DALL-E but controversially refused to open source the model, meaning that perhaps one of the most important state-of-the-art multimodal deep learning models remains inaccessible to all but a select few. The huge resource requirements for building these foundation models create significant barriers to access, and, without a method to pool resources whilst still capturing value, will likely cause stagnation in AI advancement. Many believe that these generalized models are the key to unlocking Artificial General Intelligence (AGI), making the current method of training in isolated, artificial silos seem absurd.

Current solutions which provide access to compute supply are either oligopolistic and expensive or simply unworkable given the complexity of compute required for large-scale AI. Meeting the ballooning demand requires a system which cost-efficiently leverages all available compute (as opposed to today’s ~40% global processor utilization). Compounding this problem right now is the fact that the compute supply itself is hamstrung by asymptotic advances in microprocessor performance — alongside supply chain and geopolitical chip shortages.

Part 2: Why is Gensyn’s Coordination Needed?

The fundamental challenge in building this network is the verification of completed ML work. This is a highly complex problem that sits at the intersection of complexity theory, game theory, cryptography, and optimization. Besides human knowledge in model design, there are three fundamental problems slowing the progress of applied ML, 1) access to compute power; 2) access to data; and 3) access to knowledge (ground-truth labeling). Gensyn solves the first problem by providing on-demand access to globally scalable compute at its fair market price, while the Gensyn Foundation will seek to encourage solutions to two and three through research, funding, and collaborations with other protocols.

Specifically, access to superior processors enables increasingly large/complex
models to be trained. In the past decade, transistor density gains and advances in memory access speed/parallelization have dramatically reduced training times for large models. Virtual access to this hardware, via cloud giants like AWS and Alibaba, has simultaneously widened adoption. Accordingly, there is strong state interest in acquiring the means to produce state-of-the-art processors. Mainland China does not yet have the end-to-end capability to produce state-of-the-art semiconductors (namely, silicon wafers), an essential component in processors. They need to import these, particularly from TSMC (Taiwan Semiconductor Manufacturing Company). Chip vendors also attempt to block out other customers from accessing chip manufacturers by buying up supply. At the state level, the US has been aggressively blocking any move by Chinese companies to acquire this technology. Further up the tech stack, some companies have gone as far as creating their own deep learning specific hardware, like Google’s TPU clusters. These outperform standard GPUs at deep learning and aren’t available for sale, only for rent.

Vastly increasing the scale of accessible compute, whilst simultaneously reducing its unit cost, opens the door to a completely new paradigm for deep learning for both research and industrial communities. Improvements in scale and cost allow the protocol to build up a set of already-proven, pre-trained, base models–also known as Foundation Models–in a similar way to the model zoos of popular frameworks. This allows researchers and engineers to openly research and train superior models over huge open datasets, in a similar fashion to the Eleuther project. These models will solve some of humanity’s fundamental problems without centralized ownership or censorship. Cryptography, particularly Functional Encryption, will allow the protocol to be leveraged over private data on-demand. Huge foundation models can then be fine-tuned by anyone using a proprietary dataset, maintaining the value/privacy in that data but still sharing collective knowledge in model design and research.

High scale + low cost: the Gensyn protocol provides a cost similar to an owned GPU in a datacenter at a scale which can surpass AWS. (Prices as at Nov 2021).

Part 3: Gensyn Drives Web3-Native Data Centralization

The internet might have been born of the US Government in the 1960s, but by the 1990s it was an anarchic web of creativity, individualism, and opportunity. Well before Google was stockpiling TPUs, projects like SETI@home attempted to discover alien life by crowdsourcing decentralized compute power. By the year 2000, SETI@home had a processing rate of 17 teraflops, which is over double the performance of the best supercomputer at the time, the IBM ASCI White. This period of time is generally named ‘web1’, a moment before the hegemony of large platforms like Google or Amazon (web2), but decentralized compute faltered in scaling to meet the initial needs of the internet, due to several issues at the time.

However, the current centralization of web infrastructure into huge web2 platforms creates its own issues, such as cost (AWS’ gross margin is an estimated 61%, representing margin compression for most sub-scale researchers and data-driven businesses. At the same time, centralized compute instances also sacrifice control — AWS turned off the infrastructure of popular right-wing social media platform Parler with one day’s notice following the Jan 6th 2021 Capitol Riot. Many agreed with this decision, but the precedent is dangerous when AWS hosts 42% of the top 10,000 sites on the internet. However, training deep learning models across decentralized hardware is difficult due to the verification problem, which the Gensyn Protocol helps solve.

Building the marketplace as a Web3 protocol removes the centralized overhead expenses on scaling, and reduces the barriers-to-entry for new supply participants, allowing the network to potentially encompass every computing device in the world. Connecting all devices through a single decentralized network provides a level of scalability that is currently impossible to achieve through any existing provider, giving unprecedented on-demand access to the entirety of the world’s compute supply. For end-users, this completely dismantles the cost vs. scale dilemma and provides a transparent and low cost ML training compute for potentially infinite scalability (up to worldwide physical hardware limits) and for unit prices to be determined by market dynamics. This sidesteps the usual moats that large providers enjoy, significantly drives down prices, and facilitates truly global competition at the resource level, and even considers a case where existing cloud services providers also view Gensyn protocol as a distribution avenue that complements more centralized first-party bundled offerings.

Conclusion:

WIth AI almost as popular a buzzword as cryptocurrency and blockchains, our thesis for investing in Gensyn as previewed here must pass the tests of being easy to understand and evidence-backed, while being as ambitious in thinning about the opportunity set for the protocol’s ability to add value an initially targeted but generalizable resource network native to web3. With the Gensyn protocol, we believe that we are seeing the beginnings of a hyper-scalable, cost-efficient coordination network which paves the way for even more valuable insights that lay the groundwork for myriad applications in the future.

About CoinFund

CoinFund is a diverse, leading blockchain-focused investment firm founded in 2015, based in the U.S. Collectively, we have an extensive track record and experience in cryptocurrency, traditional equity, credit, private equity, and venture investing. The CoinFund strategies span both liquid and venture markets and benefit from our multidisciplinary approach that synchronizes technical cryptonative aptitude with traditional finance experience. With a “founders first” approach, CoinFund partners closely with its portfolio companies to drive innovation across the digital asset space.

Disclaimer

The content provided on this site is for informational and discussion purposes only and should not be relied upon in connection with a particular investment decision or be construed as an offer, recommendation or solicitation regarding any investment. The author is not endorsing any company, project, or token discussed in this article. All information is presented here “as is,” without warranty of any kind, whether express or implied, and any forward-looking statements may turn out to be wrong. CoinFund Management LLC and its affiliates may have long or short positions in the tokens or projects discussed in this article.


The Gensyn Protocol Trustlessly Trains Neural Networks at Hyperscale with Lower Order of Magnitude… was originally published in The CoinFund Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: https://blog.coinfund.io/the-gensyn-protocol-trustlessly-trains-neural-networks-at-hyperscale-with-lower-order-of-magnitude-227fe968fabf?source=rss—-f5f136d48fc3—4

spot_img

Latest Intelligence

spot_img