Written by: Hakeen
1. Ethereum Upgrade Roadmap M2SVPS
The Merge
In the Merge phase, the POW consensus mechanism will transition to POS, and the beacon chains will be merged together. For ease of understanding, we simplify the Ethereum structure into the following diagram:
Let's first define what sharding is: simply put, it is the process of horizontally splitting the database to distribute the load
After switching to POS: the block proposer and block validator are separated, and the POS workflow is as follows (based on the above figure):
Submit transactions on Rollup Validators add transactions to shard blocks Beacon chain selects validators to propose new blocks The rest of the validators form random committees and validate proposals on the shards
Both proposing a block and proving a proposal need to be completed within one slot, which is usually 12 seconds. Every 32 slots constitute an epoch cycle, and each epoch will disrupt the order of validators and re-elect the committee.
After the merger, Ethereum will implement proposer-builder separation (PBS) for the consensus layer. Vitalik believes that the end game for all blockchains will be to have centralized block production and decentralized block validation. Since Ethereum blocks after sharding are very data-intensive, the centralization of block production is necessary due to the high requirements for data availability. At the same time, there must be a solution that can maintain a decentralized set of validators that can validate blocks and perform data availability sampling.
Miners and block verification are separated. Miners build blocks and then submit them to validators. They bid to validators to select their own blocks, and then validators vote to decide whether the blocks are valid.
Sharding is a partitioning method that can distribute computing tasks and storage workloads in a P2P network. In this way, each node is not responsible for handling the transaction load of the entire network, but only needs to maintain information related to its partition (or shard). Each shard has its own validator network or node network.
Security issues of sharding: For example, if the entire network has 10 shard chains, and it takes 51% of the computing power to destroy the entire network, then it only takes 5.1% of the computing power to destroy a single shard. Therefore, subsequent improvements include an SSF algorithm, which can effectively prevent 51% computing power attacks. According to Vitalik, switching to SSF is a multi-year roadmap. Even if a lot of work is done at present, it will be one of the major changes that Ethereum will implement later, and it will be long after Ethereum's PoS proof mechanism, sharding, and Verkle tree are fully launched.
The beacon chain is responsible for generating random numbers, assigning nodes to shards, capturing snapshots of individual shards and various other functions. It is responsible for completing communication between shards and coordinating network synchronization.
The execution steps of the beacon chain are as follows:
The block producer commits to the block header and bid together. The block producer (validator) on the beacon chain selects the winning block header and bid, and will unconditionally receive the winning bid fee regardless of whether the block packager finally generates the block body. The committee (randomly selected from the validators) votes to confirm the obtained block header. The block packager discloses the block body.
The Surge
The main goal of this route is to promote Rollup-centric expansion. Surge refers to the addition of Ethereum sharding, a scaling solution that the Ethereum Foundation claims will further enable low-gas-fee second-layer blockchains, reduce the cost of rollups or bundled transactions, and make it easier for users to operate nodes that protect the Ethereum network.
The diagram can still be understood by the following simplified diagram:
Take the operation principle of zkrollup as an example: in zkrollup, it is divided into a sequencer and an aggregator. The sequencer is responsible for sorting user transactions and packaging them into batches and sending them to the aggregator. The aggregator executes the transaction, generates the post state root from the prev state root, and then generates the proof. The aggregator finally sends the prev state root, post state root, transaction data, and proof to the contract on L1. The contract is responsible for verifying whether the proof is valid. The transaction data is stored in calldata. Zkrollup data availability allows anyone to restore the global state of the account based on the transaction data stored on the chain.
However, the cost of using calldata is very expensive, so the entire EIP-4844 protocol (which may change at any time) proposes to change the size of transaction blocks to 1-2MB, laying a solid foundation for future rollups and data sharding. Currently, the block size of Ethereum is about 60KB ~ 100KB. Taking EIP-4844 as an example, the block size limit can be increased by about 10-34x. This block format is called blob (also called data shard).
The Scourge
At this stage, Scourge is a supplement to the roadmap, mainly used to solve the problem of MEV. So what is MEV?
MEV stands for Miner Extractable Value / Maximal Extractable Value. This concept was first applied in the context of proof of work and was originally called "Miner Extractable Value". This is because in proof of work, miners have the role capabilities of inclusion, exclusion, and order of transactions. However, after the transition to proof of stake through merger, validators will be responsible for these roles and mining will no longer apply (the value extraction method introduced here will remain after this transition, so the name change is required). In order to continue using the same acronym to ensure continuity while maintaining the same basic meaning, "Maximal Extractable Value" is now used as a more inclusive alternative.
Arbitrage opportunities include:
By compressing the storage space, the price difference of gas fees is obtained; Referee front-running: Extensive search for transactions on mempool, the machine performs calculations locally to see if it is profitable, and if so, initiates the same transaction with its own address. And use higher gas fees; Find liquidation targets: Robots compete to parse the blockchain data as quickly as possible to determine which borrowers can be liquidated, and then become the first to submit liquidation transactions and collect liquidation fees themselves. Sandwich transactions: Searchers monitor large transactions on DEX in the mempool. For example, someone wants to buy 10,000 UNI using DAI on Uniswap. Such large transactions would have a significant impact on the UNI/DAI pair, potentially significantly increasing the price of UNI relative to DAI. The searcher can calculate the approximate price impact of this large transaction on the UNI/DAI pair, and execute the optimal buy order immediately before the large transaction to buy UNI at a low price, and then execute the sell order immediately after the large transaction, with the large order resulting in a higher selling price.
Disadvantages of MEV:
Some forms of MEV, such as sandwich transactions, can result in a significantly worse user experience. Users caught in the middle face higher slippage and poorer trade execution. At the network layer, general front-runners and the gas auctions they often participate in (when two or more first movers compete to get their transactions included in the next block by gradually increasing the gas fees of their own transactions) lead to network congestion and high gas fees for others trying to run normal transactions. In addition to what happens within a block, MEV can also have a detrimental effect between blocks. If the MEV available in a block greatly exceeds the standard block reward, miners may be incentivized to re-mine blocks and capture MEV for themselves, leading to a reorganization of the blockchain and instability in consensus.
Most of the MEV is extracted by independent network participants called "seekers." Seekers run complex algorithms on blockchain data to detect profitable MEV opportunities, and there are bots that automatically submit these profitable transactions to the network. The MEV problem on Ethereum involves the use of bots to exploit network transactions, causing congestion and high fees.
The Verge
Verge will implement "Verkle trees" (a type of mathematical proof) and "stateless clients". These technical upgrades will allow users to become network validators without storing large amounts of data on their machines. This is also one of the steps around rollup expansion. The simple working principle of zk rollup was mentioned earlier. The aggregator submitted the proof, and the verification contract on layer 1 only needs to verify the KZG commitment and the generated proof in the blob. Here is a brief introduction to the KZG commitment, which is to ensure that all transactions are included. Because rollup can submit partial transactions and generate proofs, if KZG is used, then all transactions will be ensured to be included in the generated proof.
The Verge ensures that verification is very simple. You only need to download N bytes of data and perform basic calculations to verify the proof submitted by the rollup.
It is worth mentioning that there are many schemes for ZK rollup, such as stark, snark or bulletproof. Each scheme has different ways of proof and verification, so there is a trade-off. SNARKs is currently easier to use than STARKs technology, and the technology is more complete, so many projects use SNARKs at the beginning, but with the iteration of STARKs technology, they will eventually gradually turn to STARKs that are resistant to quantum attacks. Although one of the main improvements of Ethereum EIP-4844 is the transaction format blob in order to adapt to rollup, which expands the block capacity, the main bottleneck of all current zero-knowledge proofs is still the proof algorithm itself. On the one hand, the proof problem is solved by improving the algorithm, and on the other hand, the proof efficiency is improved by stacking hardware, which also derives the ZK mining track. Those who are interested can go to this article.
The Purge
The Purge will reduce the amount of space required to store ETH on hard drives, attempting to simplify the Ethereum protocol and not require nodes to store history. This will greatly increase the bandwidth of the network.
EIP-4444：
Clients must stop providing historical headers, bodies, and recipients older than one year on the P2P layer. Clients can prune this historical data locally. Preserving Ethereum's history is fundamental, and I believe there are various out-of-band ways to achieve this. Historical data can be packaged and shared through torrent magnet links or networks such as IPFS. In addition, systems such as Portal Network or The Graph can be used to obtain historical data. Clients should allow importing and exporting historical data. Clients can provide scripts to obtain/verify data and automatically import them.
The Splurge
This route mainly includes some piecemeal optimization fixes, such as account abstraction, EVM optimization, and random number scheme VDF.
The Account Abstraction (AA) mentioned here has always been the first goal that ZK-based Layer 2 wants to achieve. So what is account abstraction? After implementing account abstraction, a smart contract account can also actively initiate transactions without relying on the "meta transaction" mechanism (which is proposed in EIP-4844).
In Ethereum, accounts are divided into contract accounts and external accounts. Currently, there is only one type of transaction in Ethereum, which must be initiated by an external address. Contract addresses cannot actively initiate transactions. Therefore, any change in the state of a contract must rely on a transaction initiated by an external address, whether it is a multi-signature account, a mixer, or any configuration change of a smart contract, it needs to be triggered by at least one external account.
No matter what application is used on Ethereum, users must hold Ethereum (and bear the risk of Ethereum price fluctuations). Secondly, users need to deal with complex fee logic, gas price, gas limit, transaction blocking, and these concepts are too complicated for users. Many blockchain wallets or applications try to improve user experience through product optimization, but the effect is minimal.
The goal of the account-centric solution is to create an account for the user that is managed based on a smart contract. The benefits of implementing account abstraction are:
The current contract can hold ETH and directly submit a transaction with all signatures. The user does not necessarily need to pay gas fees for the transaction, it depends entirely on the project. Due to the implementation of custom cryptography, it will not be mandatory to use ESCDA elliptic curves for signatures in the future. In the future, a mobile phone's fingerprint recognition, facial recognition, biometrics and other technologies can be used as signature methods. This significantly improves the user’s interaction experience with Ethereum.
2. Modularity of Ethereum
The entire Ethereum has now shown a modular trend, and the execution layer is the responsibility of Layer 2 (such as arbitrum, zksync, starknet, polygon zkevm, etc.). They are responsible for executing transactions of users on L2 and submitting proofs. Layer 2 generally uses OP technology/ZK technology. In theory, the TPS of ZK technology is much higher than that of OP. Currently, a large number of ecosystems are in the OP system, but in the future, as ZK technology improves, more and more applications will migrate to the ZK system. This section is a detailed description and supplement of the roadmap, why and how.
Currently, Ethereum only separates the execution layer. In fact, other layers are still mixed together. In Celestia's vision, the execution layer only does two things: for a single transaction, it executes the transaction and changes the state; for transactions in the same batch, it calculates the state root of the batch. Currently, part of the work of the Ethereum execution layer is assigned to Rollup, which is StarkNet, zkSync, Arbitrum and Optimism.
Now, whether it is optimism, polygon, starknet, zksync, etc., they are all exploring the path of modularization.
Optimism proposed the bedrock/op stack, polygon is also developing polygon avail as a data availability layer, and supernets are used to simplify the creation of chains and shared validator sets.
Settlement layer: It can be understood as the process by which the Rollup contract on the main chain verifies the validity of the previous state root, the later state root, the proof (zkRollup) or the fraud proof (Optimistic Rollup) mentioned above. Consensus layer: Regardless of whether PoW, PoS or other consensus algorithms are used, the consensus layer is to reach a consensus on something in the distributed system, that is, to reach a consensus on the validity of the state transition (the previous state root is converted into the later state root after calculation). In the context of modularization, the meanings of the settlement layer and the consensus layer are somewhat similar, so some researchers have unified the settlement layer and the consensus layer. Data availability layer: Ensure that the transaction data is fully uploaded to the data availability layer, and the verification node can reproduce all state changes through the data of this layer.
What needs to be distinguished here is the difference between data availability and data storage:
Data availability is distinct from data storage, in that the former is concerned with whether the data published in the latest block is available, while the latter is concerned with storing data securely and ensuring that it can be accessed when needed.
1. Various Rollups on the Settlement Layer
From the perspective of the settlement layer, the focus of rollup is currently believed to be on the ZK system. If the size, gas consumption, and cost of the ZK proof system are improved through the ZK rollup, combined with recursion and parallel processing, its TPS can be greatly expanded. So let's start with ZK rollup.
As Ethereum's expansion progresses, Zero Knowledge Proof (ZKP) technology is considered by Vitalik to be the solution that is expected to be the final solution in the expansion battle.
The essence of a ZKP is to allow someone to prove that they know or have something. For example, I can prove that I have the key to a door without having to take it out. Proving knowledge of the password to an account without having to enter it and risk exposure has implications for personal privacy, encryption, businesses, and even nuclear disarmament. Let’s take a look at a modified version of the Yao Millionaire Problem: This problem discusses two millionaires, Alice and Bob, who want to know which of them is richer without revealing their actual wealth.
Assuming the monthly rent of the apartment is $1,000, to meet the tenant's criteria, you have to pay at least 40 times the monthly rent. Then we (tenants) need to prove that our annual income is more than $40,000. But the landlord doesn't want us to find loopholes, so he chooses not to publish the specific rent. His purpose is to test whether we meet the criteria, and the answer is only yes or no, and he is not responsible for the specific amount.
There are now ten boxes, labeled $10-100k in increments of $10k. Each has a key and a slot. The homeowner walks into the room with the boxes, destroys the nine keys, and takes the key for the box labeled $40k.
The tenant's annual salary reaches $75,000. The bank agent supervises the issuance of a document proving the assets, without specifying the specific funds. The essence of this document is the bank's asset declaration and verifiable claim document. Then we put the document into a box of 10k to 70k. Then the landlord uses the 40k key to open the box and sees the verifiable claim document inside, then the tenant is determined to meet the criteria.
The points involved here include that the declarant (bank) issues a certificate of asset compliance, and the verifier (homeowner) verifies whether the tenant is qualified through the key. Once again, there are only two options for the verification result - qualified and unqualified, and there will not and cannot be any requirements for the specific amount of assets of the tenant.
We can still use the following figure as an understanding. Transactions are executed on layer 2 and submitted to shards. Layer 2 generally adopts the form of rollup, that is, multiple transactions are packaged into a batch on layer 2 to process transactions, and then submitted to the rollup smart contract of layer 1. This includes the old and new state roots. The contract on layer 1 will verify whether the two state roots match. If they match, the old state root on the main chain will be replaced with the new state root. How to verify that the state root obtained after batch processing is correct? Here, optimistic rollup and zk rollup are derived. Fraud proof and zk technology are used to confirm transactions and verify state roots respectively.
Layer 2 (rollup) here is equivalent to the declarant (bank) in the above example. Its packaging operation is this declaration operation. It does not declare the specific amount, but confirms whether it meets the standard. After packaging, it is submitted to layer 1 as this claimable declaration document. Verifying the old and new state roots is like the landlord using the key to verify whether the financial strength of the tenant he expects meets the standard. The state root verification problem is how to make the statement submitted by the bank credible.
For optimistic, or fraud-proof, rollups, the mainchain’s Rollup contract records a complete record of changes to the internal state root of the Rollup, as well as the hash value of each batch processing (that triggers the state root change). If someone finds that the new state root corresponding to a batch processing is wrong, they can publish a proof on the mainchain to prove that the new state root generated by the batch processing is wrong. The contract verifies the proof, and if the verification passes, all batch processing transactions after the batch processing are rolled back.
The verification method here is equivalent to the declarant (bank) submitting a verifiable asset declaration document, and then making all the asset documents public on the chain, and the data is also public on the chain. Other challengers calculate based on the original data to see if there are errors or forgeries in the verifiable asset documents. If there are problems, a challenge is raised, and if the challenge is successful, the bank is compensated. The most important issue here is that time needs to be reserved for challengers to collect data and verify the authenticity of the document.
For Rollups that use Zero Knowledge Proof (ZKP) technology, each batch contains a cryptographic proof called ZK-SNARK. The bank generates asset declaration documents through cryptographic proof technology. In this way, there is no need to reserve time for challengers, and thus the role of challengers does not exist.
2. The reason why ZK-based Rollup is not as good as expected
At present, hermez of the polygon system has been released, and the zksync dev mainnet and starknet mainnet have also been launched. However, their transaction speed seems to be too far from our theoretical speed, especially the users of starknet can clearly perceive that the speed of its mainnet is surprisingly slow. The reason is that the difficulty of generating proofs with zero-knowledge proof technology is still very large, the cost is still very high, and there is a need to balance the compatibility of Ethereum and the performance of zkevm. The Polygon team also admitted: "The testnet version of Polygon zkEVM also has limited throughput, which means that it is far from the final form as an optimized expansion machine."
3. Data availability layer
The abstract execution steps of Ethereum are as follows:
In the process of Ethereum's decentralization, we can also see it on The Merge roadmap - decentralized validators. The most important thing is to achieve client diversity and lower the entry threshold of machines, and increase the number of validators. Therefore, if some validators whose machines do not meet the standards want to participate in the network, they can use light clients. The operating principle of light nodes is to ask for block headers from nearby full nodes. Light nodes only need to download and verify block headers. If light nodes do not participate, all transactions require full nodes to perform verification. Therefore, full nodes need to download and verify every transaction in the block. At the same time, as the number of transactions increases, the pressure on full nodes is also increasing, so the node network gradually tends to be high-performance and centralized.
But the problem here is that a malicious full node can give a missing/invalid block header, but a light node cannot falsify it. There are two ways to solve this problem. At first, fraud proofs are used. A trusted full node is needed to monitor the validity of the block. After an invalid block is found, a fraud proof is constructed. If no fraud proof is received within a period of time, it is determined to be a valid block header. However, a trusted full node is obviously required here, that is, a trusted setting or honest assumption is required. However, if the block producer can hide some transactions, the fraud proof will obviously fail, because the honest node also relies on the data of the block producer. If the data itself is hidden, then the trusted node will think that the submitted data is all the data, and naturally no fraud proof will be generated.
In a paper co-authored by Mustarfa AI-Bassam and Vitalik, a new solution is proposed - erasure coding. Erasure codes are used to solve data availability problems. For example, celestia and polygon avail all use reed-solomon erasure codes. But how to ensure that the transmitted data is complete data can be combined with KZG commitment/fraud proof.
In the KZG commitment/fraud proof, it is possible to ensure that block producers publish complete data and do not hide transactions. The data is then encoded through erasure codes and then sampled for data availability, so that light nodes can correctly verify the data.
The data submitted by the aggregator in the Rollup is stored on the chain in the form of calldata. This is because calldata is cheaper than other storage areas.
Calldata cost in gas = Transaction size × 16 gas per byte
The main expense of each transaction is the calldata cost, because on-chain storage is extremely expensive, which accounts for 80% to 95% of the rollup cost.
Due to this problem, we proposed the new transaction format blob of EIP-4844 to expand the block capacity and reduce the gas fee required to submit to the chain.
4. On-chain and off-chain data availability layers
So how do we solve the problem of expensive on-chain data? There are several ways:
The first is to compress the size of calldata uploaded to L1. There have been many optimizations in this regard. The second is to reduce the cost of storing data on the chain, and provide "big blocks" for rollups through Ethereum's proto-danksharding and danksharding, with larger data availability space, and use erasure codes and KZG commitments to solve the problem of light nodes. Such as EIP-4844. The third is to put data availability off-chain. The general solutions for this part include celestia/polygon avail, etc.
According to the location where data availability is stored, we divide it into the following figure:
Validium's solution: put data availability off-chain, so the transaction data will be maintained by centralized operators, and users will need trusted settings, but the cost will be very low, but at the same time there is almost no security. Later, starkex and arbitrum nova both proposed to establish DAC to be responsible for the storage of transaction data. DAC members are well-known individuals or organizations within the legal jurisdiction, and the trust assumption is that they will not collude and do evil.
Zkporter proposes guardians (zksync token holders) to stake and maintain data availability. If a data availability failure occurs, the staked funds will be confiscated. Volition allows users to choose on-chain/off-chain data availability, and choose between security and cost based on their needs.
At this time, Celestia and Polygon Avail appeared. If Validium has the need for off-chain data availability, but is afraid of low decentralization, which may lead to private key attacks similar to cross-chain bridges, then a decentralized universal DA solution can solve this problem. Celestia and Polygon Avail provide off-chain DA solutions for Validium by becoming a separate chain. However, through a separate chain, although security is improved, the cost will be increased accordingly.
The expansion of Rollup actually consists of two parts. One is the execution speed of the aggregator, and the other requires the cooperation of the data availability layer. Currently, the aggregator is run by a centralized server. Assuming that the speed of transaction execution can reach an infinite degree, the main expansion dilemma is that it is affected by the data throughput of the underlying data availability solution. If Rollup wants to maximize its transaction throughput, it is crucial to maximize the data space throughput of the data availability solution.
Going back to the beginning, use KZG commitments or fraud proofs to ensure data integrity, and use erasure codes to expand transaction data to help light nodes perform data availability sampling, further ensuring that light nodes can correctly verify data.
Maybe you also want to ask, how does KZG promise to work to ensure the integrity of its data? Perhaps I can explain it briefly:
KZG commitment: proves that the value of a polynomial at a specific position is consistent with the specified value. KZG commitment is nothing more than a type of polynomial commitment that can verify a message without giving a specific message. The general process is as follows:
We convert the data into a polynomial through erasure coding and expand it. We use KZG commitment to ensure that our expansion is valid and the original data is valid. We can then use the expansion to reconstruct the data and finally perform data availability sampling.
The committer generates a commitment and binds it to a message. The bound message is sent to the verifier. The communication scheme here is related to the size of the proof. The verifier brings in multiple values of the finite field to verify whether they are still equal to a (this is the process of availability sampling). The basic principle is that the more times the verification is done, the higher the probability of correctness.
Celestia requires validators to download the entire block, while danksharding now utilizes data availability sampling technology.
Since blocks are partially available, we need to ensure synchronization whenever we reconstruct blocks. When blocks are indeed partially available, nodes communicate with each other to piece the blocks together.
Comparison of KZG commitment and data fraud proof:
It can be seen that KZG promises to ensure that the extension and data are correct, while fraud proofs introduce third parties for observation. The most obvious difference is that fraud proofs require a time interval for the observer to react and then report the fraud, which requires direct synchronization of nodes so that the entire network can receive the fraud proof in time. KZG is obviously faster than fraud proofs, and it uses mathematical methods to ensure the correctness of the data without a waiting time.
It can prove that the data and its extensions are correct. However, since one-dimensional KZG commitments require more resources, Ethereum chooses two-dimensional KZG commitments.
For example, 100 rows × 100 columns means 100,000 shares. But each sampling is not a one in ten thousand guarantee. So expanding by four times means that at least 1/4 of the shares must be unavailable before you can draw an unavailable share, which means it is truly unavailable because it cannot be recovered. Only when 1/4 of the shares are unavailable can it be unrecoverable, and only then can the error be effectively discovered, so the probability of drawing once is about 1/4. Drawing more than ten times, fifteen times, can achieve a 99% reliability guarantee. Now choose within the range of 15-20 times.
5、EIP-4844（Proto-Danksharding）
In the proto-danksharding implementation, all validators and users must still verify the availability of the full data directly.
The main feature introduced by Proto-danksharding is a new transaction type, which we call transactions with blobs. A transaction with blobs is similar to a regular transaction, except that it also carries an additional data called a blob. Blobs are very large (~125 kB) and are much cheaper than a similar amount of call data. However, these blobs are not accessible from the EVM (only commitments to the blob). And the blobs are stored by the consensus layer (beacon chain) instead of the execution layer. This is actually the beginning of the concept of data sharding.
Because validators and clients still need to download the full blob contents, the data bandwidth target in proto-danksharding is 1MB per slot, rather than the full 16MB. However, since this data does not compete with the gas usage of existing Ethereum transactions, there are still large scalability gains.
Although achieving full sharding (with data availability sampling, etc.) is a complex task and will remain a complex task after proto-danksharding, this complexity is contained in the consensus layer. Once proto-danksharding is launched, execution layer client teams, rollup developers, and users do not need to do further work to complete the transition to full sharding. Proto-danksharding also separates blob data from calldata, making it easier for clients to store blob data in a shorter time.
Notably, all work was done by consensus layer changes and no additional work was required by client teams, users, or Rollup developers.
Both EIP-4488 and proto-danksharding result in a long-term maximum usage of about 1 MB per slot (12 seconds). This equates to about 2.5 TB per year, which is much higher than the growth rate Ethereum needs today.
In the case of EIP-4488, solving this problem requires the history expiration proposal EIP-4444 (mentioned in the roadmap section), where clients are no longer required to store history beyond a certain period of time.
6. Data Sharding
Here, we will try to explain as much as possible from a novice's perspective the issues that everyone is discussing during the Ethereum expansion process. So let's go back to sharding and emphasize again the one-sided concept of sharding: in simple terms, it is the process of horizontally splitting the database to distribute the load.
Here, there is a very important problem with our data sharding. In PBS (separation of proposers and block builders, mentioned in the roadmap The Merge), each node group in the shard only processes transactions within the shard, and transactions between shards are relatively independent. So how should two users AB transfer money to each other if they are on different shards? This requires a good cross-shard communication capability.
The old way is to shard the data availability layer, and each shard has independent proposers and committees. In the validator set, each validator takes turns to verify the data of the shard, and they download all the data for verification.
weakness is:
Strict synchronization technology is required to ensure that validators can be synchronized within a slot. Validators need to collect votes from all committees, which can cause delays. It is also very stressful for validators to fully download data.
The second method is to abandon complete data verification and adopt the data availability sampling method (this method was implemented in the later stage of The Surge). There are two random sampling methods here: 1) Block random sampling, sampling some data shards, if the verification is passed, the verifier signs. But the problem here is that transactions may be missed. 2) Reinterpret the data as a polynomial through erasure coding, and then use the characteristics of polynomials that can restore data under specific conditions to ensure the complete availability of the data.
The key to "sharding" is that the validator is not responsible for downloading all the data, and this is why Proto-danksharding is not considered "sharding" (even though it has "sharding" in its name). Proto-danksharding requires each validator to download all shard blobs in full to verify their availability; Danksharding will later introduce sampling, and individual validators only need to download fragments of shard blobs.
3. The Future of Ethereum: Layer 3
ZK-based Layer 2, which is regarded as the future of Ethereum, such as zksync and starknet, have proposed the concept of Layer 3. In simple terms, it is Layer 2 of Layer 2.
The high transaction costs on Ethereum are pushing it (L3) to become the settlement layer for L2. I believe that in the near future, due to significantly lower transaction costs, increasing support for DeFi tools, and increased liquidity provided by L2, end users will conduct most of their activities on L2, and Ethereum will gradually become the settlement layer.
L2 improves scalability by reducing gas costs per transaction and increasing transaction rates. At the same time, L2s retain the benefits of decentralization, general logic, and composability. However, some applications require specific customizations that may be better served by a new independent layer: L3!
L3 is related to L2, just like L2 is related to L1. As long as L2 can support Verifier smart contracts, L3 can be implemented using validity proofs. When L2 also uses validity proofs submitted to L1, as StarkNet does, this becomes a very elegant recursive structure where the compression advantage of L2 proofs is multiplied by the compression advantage of L3 proofs. In theory, if each layer achieves, for example, 1000 times lower cost, then L3 can be 1,000,000 times cheaper than L1 - while still maintaining the security of L1. This is also a real use case for the recursive proofs that starknet is proud of.
Here we need to use some knowledge of "On-chain and Off-chain Data Availability Layer". The entire Layer 3 includes:
Rollup (on-chain data availability) and validium (off-chain data availability). The two correspond to different application requirements. Web2 companies that are sensitive to price and data can use validium to put data off-chain, which greatly reduces the gas fee on the chain and can achieve privacy without disclosing user data, allowing companies to control their own data and use custom data formats. The previous data business model of the company can still run smoothly.
L2 is for scaling and L3 is for customized features such as privacy.
In this vision, there is no attempt to provide "quadratic scalability"; instead, there is a layer in the stack that helps applications scale, and then separates the layers based on the customized functionality requirements of different use cases.
L2 is for general extensions and L3 is for custom extensions.
Custom extensions may come in different forms: specialized applications that use something other than the EVM for computation, rollups whose data compression is optimized for the data format of a specific application (including separating “data” from “proofs” and replacing proofs entirely with a single SNARK per block), etc.
L2 is used for trustless expansion (rollup), and L3 is used for weak trust expansion (validium).
Validium is a system that uses SNARKs to verify computations, but leaves data availability to a trusted third party or committee. In my opinion, Validium is severely underrated: in particular, many "enterprise blockchain" applications may actually be best served by a centralized server running a validium prover and periodically submitting hashes to the chain. Validium has a lower level of security than rollups, but can be much cheaper.
For dApp developers, there are several options for infrastructure:
Develop your own Rollup (ZK Rollups or Optimistic Rollups)
The advantage is that you can inherit Ethereum's ecosystem (users) and its security, but for a dApp team, the development cost of Rollup is obviously too high.
Choose Cosmos, Polkadot, or Avalanche
The development costs will be lower (for example, dydx chose Cosmos), but you will lose the Ethereum ecosystem (users) and security.
Develop a Layer 1 blockchain yourself
The development costs and difficulty are high, but it gives you the highest control.
Let’s compare three situations:
Difficulty/Fee: Alt-layer 1 > Rollup > Cosmos Security: Rollup > Cosmos > Alt-layer 1 Ecosystem/Users: Rollup > Cosmos > Alt-layer 1 Control: Alt-layer 1 > Cosmos > Rollup
As a dApp developer, if you want to inherit the security and traffic on Ethereum, you can't develop a new chain, so you can only choose rollup. However, developing a layer 2 rollup yourself is very expensive, so the appropriate solution becomes to use the layer 3 SDK to develop your own application-specific rollup (application-specific rollup), that is, Layer 3.
4. Future Development of Layer 2
Since Ethereum is designed based on the account model, all users are in the same state tree, so parallel processing is not possible. Therefore, the shackles of Ethereum itself require it to strip off the execution operation and combine multiple rollup transactions into one transaction as a settlement layer. Now all the problems are focused on improving the throughput of layer 2. Not only can Layer 3 improve the throughput of transactions, but also parallel processing on Layer 2 can greatly improve the throughput of the entire network.
Starknet is also actively exploring parallelization issues. Although the algorithm is still a hindrance at present, it is expected that it will not become an obstacle in the future. Potential bottlenecks include:
Sequencer tx processing: Some sequencer work seems to be inherently serial. Bandwidth: Interconnects between multiple sequencers will be limited. L2 state size
In the starknet community, members also proposed that the parallel processing method of aptos is very good. As for Starknet, it is also currently promoting the ability of parallel tx sorting within the sorter.
V. Conclusion
Ethereum is stripping off the execution layer, and all actions are moving towards its vision of a "global" settlement layer. Although the progress of the entire Ethereum is slow at present, that is, because it is too large as a whole, each update involves many interests and trade-offs. But it is undeniable that Ethereum is undergoing major changes. Ethereum's large amount of on-chain activities, economic mechanism improvements, and Ethereum 2.0 scalability, its leading innovations ICO, Defi, NFT and many other things are worthy of the excitement and expectations of the Ethereum community. I believe that with more and more countries deploying Ethereum nodes, such as the Argentine capital government plans to deploy Ethereum verification nodes in 2023, Ethereum will really be able to realize its grand vision in the near future.