Compiled & edited by: Sunny, TechFlow
Blockchain AI computing protocol Gensyn announced on June 12 that it had completed a $43 million Series A funding round led by a16z.
Gensyn's mission is to provide users with access to computing scale equivalent to owning a private computing cluster, and it is crucial to achieve fair access and avoid being controlled or shut down by any central entity. At the same time, Gensyn is a decentralized computing protocol focused on training machine learning models.
Looking back at the podcast that Gensyn founders Harry and Ben did with Epicentre at the end of last year, you can find that they took an in-depth look at computing resources, including AWS, local infrastructure, and cloud infrastructure, to understand how to optimize and utilize these resources to support the development of AI applications.
At the same time, they also discussed in detail Gensyn's design philosophy, goals and market positioning, as well as the various constraints, assumptions and implementation strategies faced during the design process.
The podcast introduces the four main roles in the Gensyn off-chain network, explores the characteristics of the Gensyn on-chain network, and the importance of Gensyn tokens and governance.
In addition, Ben and Harry also shared some interesting AI popular science, allowing everyone to have a deeper understanding of the basic principles and applications of artificial intelligence.
Host: Dr. Friederike Ernst, Epicenter Podcast
Speakers: Ben Fielding & Harry Grieve, Co-founders of Gensyn
Original title: Ben Fielding & Harry Grieve: Gensyn – The Deep Learning Compute Protocol
Air Date: November 24, 2022
Blockchain as a trust layer for decentralized AI infrastructure
The host asked Ben and Harry why they combined their extensive experience in AI and deep learning with blockchain?
Ben said that their decision was not made overnight, but after a relatively long period of time. Gensyn's goal is to build a large-scale AI infrastructure, and when studying how to achieve the largest scale scalability, they realized that a trustless layer was needed.
They needed to be able to aggregate computing power without relying on centralized new vendors to come on board, or else they would run into administrative expansion limitations. To address this, they began to explore verifiable computing research, but they found that this always required a trusted third party or judge to check the computations.
This limitation led them to blockchain, which offers a way to break the need for a single decision-maker or arbitrator because it enables consensus among large groups.
Harry also shared his philosophy that he and Ben strongly supported free speech and were concerned about censorship.
Before they turned to blockchain, they were working on federated learning, which is an area of deep learning where you train multiple models on distributed data sources and then combine them to create a meta-model that can learn from all the data sources. They were working with banks on this approach. However, they soon realized that the bigger problem was getting the computing resources or processors that could train these models.
In order to maximize the integration of computing resources, they need a decentralized coordination method, which is where blockchain comes in.
Computing resources market survey: AWS, on-premises infrastructure and cloud infrastructure
Harry explained the different computing resource options for running AI models, depending on the size of the model.
Students might use AWS or local machines, and startups might choose AWS on-demand or the cheaper option of scheduled.
But for large-scale GPU needs, AWS may be limited by cost and scalability and usually chooses to build internal infrastructure.
Research shows that many organizations are struggling to scale, and some are even choosing to buy and manage GPUs themselves. In general, buying GPUs is more cost-effective in the long run than running them on AWS.
Choices for machine learning computing resources include cloud computing, running AI models locally, or building your own computing cluster. Gensyn’s goal is to provide access to computing scale equivalent to owning your own cluster, and critically, to achieve fair access that cannot be controlled or shut down by any central entity.
Table 1: All computing resource options currently on the market
Discussion on Gensyn's design concept, goals and market positioning
The moderator asked how Gensyn differs from previous blockchain computing projects, such as the Golem Network.
Harry explained that Gensyn’s design philosophy is thought of along two axes:
The sophistication of the protocol: Unlike general-purpose computing protocols like Golem, Gensyn is a sophisticated protocol focused on training machine learning models.
Scalability of verification: Early projects often relied on reputation or replication methods that were less tolerant to Byzantine failures, which did not provide sufficient confidence in the results of machine learning. Gensyn aims to leverage learnings from computational protocols in the crypto world and apply them specifically to machine learning to optimize speed and cost while ensuring a satisfactory level of verification.
Harry added that when thinking about the properties that the network must have, it needs to be targeted at machine learning engineers and researchers. It needs to have the validation part, but critically in terms of allowing anyone to participate, and it needs to be both censorship-resistant and hardware-neutral.
Constraints, assumptions and execution during design
In the process of designing the platform at Gensyn, Ben emphasized the importance of system constraints and assumptions. Their goal is to create a network that can transform the entire world into an AI supercomputer, and to do this, they need to find a balance between product assumptions, research assumptions, and technical assumptions.
As for why they built Gensyn as their first-layer blockchain, their reasoning was to maintain greater flexibility and decision-making freedom in key technical areas such as consensus mechanisms. They wanted to be able to future-proof their protocol and did not want to impose unnecessary restrictions at the early stages of the project. In addition, they believe that in the future, various chains will be able to interact through a widely accepted information protocol, so their decision is also in line with this vision.
Figure 2: Product assumptions, research assumptions and technical assumptions, limitations and implementation
Four main roles in the Gensyn off-chain network
In this Gensyn economic discussion, four main roles are introduced: submitters, workers, validators, and whistleblowers. Submitters can submit various problems to the Gensyn network, including generating specific images or developing AI models that can drive cars.
Submitter submits tasks
Harry explains how to use Gensyn to train a model. The user first defines the desired outcome, such as generating an image based on a text prompt, and then builds a model that takes a text prompt as input and generates the corresponding image. Training data is critical for the model to learn and improve. Once the model architecture and training data are prepared, the user submits them to the Gensyn network along with hyperparameters such as the learning rate schedule and training duration. The result of this training process is a trained model that the user can host and use.
When asked how to choose an untrained model, Harry suggested two methods.
The first is based on the currently popular concept of base models, where large companies such as OpenAI or Midjourney build a base model, and then users can train the base model on specific data.
The second option is to build a model from scratch, which is different from the base model approach.
In Gensyn, developers can submit various architectures for training and testing using a method similar to evolutionary optimization, and continuously optimize to build the desired model.
Ben provided an in-depth view of the underlying models from their perspective, which they believe is the future of the field.
As a protocol, we hope that Gensyn can be used by DApps that implement evolutionary optimization techniques or similar methods. These DApps can submit individual architectures to the Gensyn protocol for training and testing, and build the ideal model through iterative refinement.
Gensyn's goal is to provide a pure machine learning computing foundation and encourage the development of an ecosystem around it.
While pre-trained models can introduce bias because organizations may use proprietary datasets or withhold information about the training process, Gensyn's solution is to open up the training process rather than eliminating black boxes or relying on full determinism. By collectively designing and training base models, we can create global models that are not biased by any specific company's dataset.
Solver
For task allocation, one task corresponds to one server, but one model may be split into multiple tasks.
Large language models are designed to take advantage of the maximum hardware capacity available at the time. This concept can be extended to the network to take into account the heterogeneity of devices.
For specific tasks, such as validators or workers, there is an option to take over from the Mempool. A worker is randomly selected from those who have expressed their willingness to take over the task. If the model and data cannot be adapted to a specific device, but the owner of the device claims that it can, fines may be incurred due to system congestion.
Whether a task can run on a machine is determined by a verifiable random function that selects a worker from a subset of available workers.
Regarding the issue of verifying the worker's capabilities, if the worker does not have the claimed computing power, they will not be able to complete the computational task, which will be detected when submitting the proof.
However, the size of the task is an issue. If the task is set too large, it may cause system problems, such as denial of service attacks (DoS), in which workers claim to complete the task but never complete it, wasting time and resources.
Therefore, the decision of task size is crucial and needs to take into account factors such as parallelization and optimization of task structure. Researchers are actively studying and exploring the best approach based on various constraints.
Once the testnet is launched, practical scenarios will also be considered to see how the system performs in the real world.
Defining the perfect task size is challenging, and Gensyn is ready to adapt and improve based on real-world feedback and experience.
Verification mechanisms and checkpoints for large-scale computation on-chain
Harry and Ben showed that verifying the correctness of a computation is an important challenge because it is not deterministic like a hash function, so it is not possible to simply verify the hash to determine whether the computation was performed. To solve this problem, the ideal solution is to use the application of zero-knowledge proof of the entire computation process. Currently, Gensyn is still working on implementing this capability.
Currently, Gensyn introduces a hybrid approach using checkpoints to verify machine learning calculations through probabilistic mechanisms and checkpoints. By combining a random auditing scheme and gradient space paths, a relatively robust check can be established. In addition, zero-knowledge proofs are introduced to enhance the verification process and are applied on the global loss of the model.
Verifier and Whistleblower
The host and Harry discussed two additional roles involved in the verification process: Verifier and whistleblower. They elaborated on the specific responsibilities and functions of these two roles.
The task of the verifier is to ensure the correctness of the checkpoints, while the task of the whistleblower is to ensure the accuracy of the verifier in performing its duties. The whistleblower solves the verifier's dilemma problem and ensures that the verifier's work is correct and trustworthy. The verifier deliberately introduces errors in the work, and the role of the whistleblower is to identify and reveal these errors, thereby ensuring the integrity of the verification process.
Validators intentionally introduce errors to test the vigilance of whistleblowers and ensure the validity of the system. If there is an error in the work, the validator detects the error and notifies the whistleblower. The error is then recorded on the blockchain and verified on-chain. Periodically, and at a rate relevant to the security of the system, validators intentionally introduce errors to keep whistleblowers engaged. If whistleblowers find an issue, they participate in a game called the "pinpoint protocol" through which they can narrow the calculation to a specific point in the Merkle tree of a specific area of the neural network. This information is then submitted to the chain for arbitration. This is a simplified version of the process for validators and whistleblowers, which they will further develop and study after the seed round.
Gensyn On-Chain Network
Ben and Harry discussed in detail how the Gensyn coordination protocol works on-chain and the implementation details. They first mentioned the process of building a network block, which involves staking tokens as part of that staking network. Then, they explained how these components relate to the Gensyn protocol.
Ben explained that the Gensyn protocol is largely a vanilla substrate Polkadot network protocol. They use the Grandpa Babe consensus mechanism based on proof of stake, and validators operate in the usual way. However, all the machine learning components introduced previously are carried out off-chain, involving various off-chain participants to perform their respective tasks.
These participants are incentivized by staking, either through the staking section in Substrate or by submitting a specific amount of tokens in a smart contract. When their work is eventually verified, they will receive a reward.
The challenge mentioned by Ben and Harry is to ensure a balance between the amount staked, the amount of possible slashing, and the amount of rewards to prevent incentives for lazy or malicious behavior.
In addition, they discussed the complexity of adding whistleblowers, but their presence is essential to ensure the honesty of the verifier due to the need for scaled computing. Although they continue to explore ways to eliminate whistleblowers through zero-knowledge proof technology. They said that the current system is consistent with what is described in the lite paper, but they are actively working to simplify each aspect.
The moderator asked if they had some kind of solution for data availability, and Henry explained that they introduced a layer on top of substrate called proof of availability (POA). This layer utilizes techniques such as erasure coding to address the limitations they encounter in the broad storage layer market. They said they would be very interested in developers who have already implemented such a solution.
Ben added that their needs involve storing not only training data, but also intermediate proof data, which does not need to be stored long-term. For example, they may only need to keep it for about 20 seconds when they have completed a specific number of block releases. However, the storage fees they currently pay on Arweave cover hundreds of years, which is unnecessary for these short-term needs. They are looking for a solution that has both the guarantees and functions of Arweave, but can meet short-term storage needs at a lower cost.
Gensyn Tokens and Governance
Ben explained the importance of the gensyn token in the ecosystem, which plays a key role in staking, slashing, providing rewards, and maintaining consensus. Its main purpose is to ensure the financial soundness and integrity of the system. Ben also mentioned the careful use of inflation rates to pay validators and take advantage of game theory mechanisms.
He emphasized the purely technical use of Gensyn token and said that they would technically ensure the timing and necessity of introducing Gensyn token.
Harry said they are in the minority in the deep learning community, especially amidst the widespread skepticism surrounding cryptocurrencies among AI scholars. Nevertheless, they recognize the value of cryptocurrencies in terms of both technology and ideology.
However, at network launch, they expect most DL users will primarily transact in fiat currencies, and conversion to tokens will happen seamlessly behind the scenes.
On the supply side, workers and committers will be actively involved in token trading, and they have already received interest from Ethereum miners who have large GPU resources and are looking for new opportunities.
Here, it is important to ensure that the fear of cryptocurrency terms (such as tokens) among deep learning and machine learning practitioners is removed, separating it from the interface of the user experience. Gensyn said this is an exciting use case that connects the worlds of Web 2 and Web 3 because it has the economic rationale and the technology needed to support its existence.
Popular Science Artificial Intelligence
AI, Deep Learning, and Machine Learning
Ben shared his thoughts on the development of the field of AI in recent years. He believes that although the field of AI and machine learning has experienced a series of small bursts in the past seven years, the current progress seems to be creating real impact and valuable applications that resonate with a wider audience. Deep learning is the fundamental driver of these changes. Deep neural networks have demonstrated their ability to surpass the benchmarks set by traditional computer vision methods. In addition, models such as GPT-3 have also accelerated this progress.
Harry further explained the differences between AI, machine learning and deep learning. He believes that these three terms are often used interchangeably, but they have significant differences. He metaphorically said that AI, machine learning and deep learning are like Russian matryoshka dolls, with AI being the outermost layer.
In a broad sense, AI refers to programming machines to perform tasks.
Machine learning, which became popular in the 1990s and early 2000s, uses data to determine the probability of decisions rather than relying on expert systems with if-then rules.
Deep learning builds on machine learning but allows for more complex models.
Figure 3: Differences between artificial intelligence, machine learning, and deep learning
Artificial narrow intelligence, artificial general intelligence, artificial superintelligence
In this session, the host and guests took an in-depth look at the three key areas of artificial intelligence: artificial narrow intelligence (ANI), artificial general intelligence (AGI), and artificial super intelligence (ASI).
Artificial Narrow Intelligence (ANI): Current AI is largely at this stage, where machines are very good at performing specific tasks, such as detecting specific types of cancer from medical scans through pattern recognition.
Artificial General Intelligence (AGI): AGI refers to machines being able to perform tasks that are relatively simple for humans but are challenging to reflect in computing systems. For example, enabling a machine to successfully navigate a crowded environment while making discrete assumptions about all the inputs around it is an example of AGI. AGI refers to models or systems that are able to perform everyday tasks like humans do.
Artificial Super Intelligence (ASI): After reaching artificial general intelligence, machines may further develop into artificial super intelligence. This refers to machines that surpass human capabilities due to the complexity of their models, increased computing power, unlimited lifespan, and perfect memory. This concept is often explored in science fiction and horror films.
In addition, the guest also mentioned that the fusion of the human brain and machines, such as through brain-computer interfaces, may be a way to achieve AGI, but this also raises a series of moral and ethical issues.
Unlocking the Black Box of Deep Learning: Determinism and Probability
Ben explains that the black-box nature of deep learning models is due to their sheer size. You are still tracing a path through a series of decision points in the network. It's just that the path is so large that it's hard to connect the weights or parameters within the model to their specific values, because those values were derived after millions of examples were fed into it. You can do it with certainty, and you can track every update, but the amount of data you end up generating is going to be very large.
He saw two things happen:
As we understand more and more about the models that we're building, the black box nature is starting to fade away. Deep learning is a research field that's been going through an interesting rapid period of a lot of experimentation that wasn't driven by the fundamentals of the research. It was more about seeing what we can get out of it. So we throw more data at it, try new architectures, and just see what happens, rather than designing this thing from first principles and knowing exactly how it works. So there was this exciting period where everything was a black box. But he thinks that rapid growth is starting to slow down, and we're seeing people revisit these architectures and examine and say, "Why does this work so well? Let's dig in and prove it." So in a way, that curtain is coming back.
Another thing that's happening that's probably more controversial is a shift in people's views on whether computing systems need to be completely deterministic or whether we can live in a probabilistic world. We as humans live in a probabilistic world. The self-driving car example is probably the clearest, when we drive, we accept that random events will happen, there may be minor accidents, there may be problems with the self-driving car system. But we can't accept that at all, we say this has to be a completely deterministic process. One of the challenges of the self-driving car industry is that they assume that people will accept the probabilistic mechanisms applied to self-driving cars, but in fact people don't accept it. He thinks this will change, and what may be controversial is, will we as a society allow probabilistic computing systems to coexist with us? He's not sure if this path will be smooth, but he thinks it will happen.
Gradient optimization method: the core optimization method of deep learning
Gradient optimization is one of the core methods of deep learning and plays a key role in neural network training. In a neural network, a series of layer parameters are essentially real numbers. Network training involves setting these parameters to realistic values that allow data to pass correctly and trigger the desired output at the final stage of the network.
Gradient-based optimization has revolutionized the field of neural networks and deep learning. This method uses the gradient, which is the derivative of the error for each layer of the network. By applying the chain rule, the gradient can be back-propagated through the layers of the network. In this process, you can determine where you are on the error surface. The error can be modeled as a surface in Euclidean space, which looks like a lot of ups and downs. The goal of optimization is to find the region that minimizes the error.
The gradient for each layer shows you where you are on this surface and in which direction you should update your parameters. You can use the gradient to navigate this undulating surface, finding the direction that will reduce your error. The size of your step depends on how steep the surface is. If it's steep, you'll jump farther; if it's not steep, you'll jump smaller. Essentially, you're just navigating this surface, looking for a groove, and the gradient helps you determine both where you are and which direction you should go.
This approach was a huge breakthrough because the gradient provides a clear signal and useful direction that can guide you to know where on the surface you are and whether you are on a hilltop, in a gully, or in a flat area more effectively than random jumps in parameter space.
Although there are many techniques in deep learning to solve the problem of finding the optimal solution, the real world situation is usually more complicated. There are many regularization techniques used in deep learning training that make it more of an art than a science. And this is why the application of gradient-based optimization in the real world is more like art than an exact science.
Summarize
Gensyn's goal is to build the world's largest machine learning computing resource system, which can make full use of idle or underutilized computing resources, such as personal smartphones, computers and other devices.
In the context of machine learning and blockchain, the records saved in the ledger are usually the results of the calculation, that is, the state of the data that we have processed through machine learning. This state can be: "I have machine-learned this wave of data, it is valid, and the time of occurrence is X year X month". The main goal of this record is to express the result state, rather than to detail the calculation process.
In this framework, blockchain plays an important role:
Blockchain provides a way to record the status of data. Its design can ensure the authenticity of data and prevent tampering and denial.
There is an economic incentive mechanism within the blockchain that can coordinate the behavior of different roles in the computing network, such as the four roles mentioned: submitters, workers, validators, and whistleblowers.
Through the investigation of the current cloud computing market, we found that cloud computing is not completely undesirable, but all computing methods have their own specific problems. The decentralized computing method of blockchain can play a role in some scenarios, but it cannot completely replace traditional cloud computing. In other words, blockchain is not a universal solution.
Finally, AI can be seen as a kind of productivity, however, how to effectively organize and train AI falls into the category of production relations. This includes factors such as cooperation, crowdsourcing, and incentives. In this regard, Web 3.0 provides a large number of possible solutions and scenarios.
Therefore, we can understand that the combination of blockchain and AI, especially in terms of data and model sharing, coordination of computing resources and verification of results, provides new possibilities for solving some problems in the training and use of AI.
References
1. https://docs.gensyn.ai/litepaper/
2. https://a16zcrypto.com/posts/announcement/investing-in-gensyn/