Original author: Colby Serpa
Original translation: DAOrayaki
Nostr 2.0 may be able to be built on top of Bitcoin as a Layer 2, providing secure off-chain data storage, just as the Lightning Network provides instant off-chain payments as a Layer 2.
This article will explain how Nostr relays can synchronize their data while maintaining lightweight characteristics, allowing users to selectively delete data, which is not available in Layer 1 blockchains. At the same time, compared to storing large amounts of data in the Bitcoin blockchain, storing data using Nostr relays may be cheaper due to the limited capacity and speed of Bitcoin blocks.
The following simple computer science design improves the distribution properties of the Nostr network under the standardized CAP theorem criteria. CAP stands for Consistency, Availability, and Partition Tolerance
Nostr relays do not know when a configuration file is incomplete, and relays lack consistency (the C in the CAP theorem)
Relays lack consistency (the C in the CAP theorem)
Consistency means that the database synchronized on each computer is the same. Nostr relays cannot be synchronized in a trust-minimized manner in the same way that a blockchain synchronizes its data block by block. Unlike a Bitcoin full node, the database stored by Nostr relays is often incomplete. Nostr relays have no way of discovering missing data except by blindly requesting all posts signed by a specific user.
Nostr's consistency/synchronization issues:
If two users upload their respective posts to different Nostr relays, then the two users may not be able to see each other's posts because Nostr is not like a blockchain. In a blockchain, every time there is a new record, all full nodes synchronize the blockchain. All full nodes add this data to their blockchain at the same time as a block. Every full node on the Bitcoin blockchain has the exact same blockchain.
If we want Nostr users to always be able to see each other's posts, then all Nostr relays need a way to identify missing data in user profiles so that they can request the missing pieces from other Nostr relays or users.
Sync Nostr relays with weekly on-chain Merkle root and full tree hashes
About once a week, users can organize all of their posts into a Merkle tree.
Each leaf in the Merkle tree contains the hash of a post, just like each leaf in Bitcoin contains the hash of a transaction.
Once the user has organized the entire configuration file into a Merkle tree, they will publish the Merkle root in an OP_RETURN on-chain, below a regular Bitcoin transaction. This is why Nostr 2.0 does not require a hard fork of the blockchain to work. OP_RETURN is a section below a Bitcoin transaction that allows small comments to be appended before the sender signs.
Additionally, the user will hash the entire tree and upload it on-chain along with the Merkle root (in an OP_RETURN). The Merkle root is just the hash of the top branch, not the hash of the entire tree. The hash of the entire tree is critical for users and relays to be able to detect if profile data is missing.
To get the hash of the entire tree, place the Merkle root at the root of the text file. Then, place the Merkle branches on the lines below the root. Then, place the Merkle leaves on the lines below the branches. Once the tree is arranged as described, hash it all at once. Below is an example of a whole tree hash of what the whole tree hash of the Merkle tree shown above would look like. Whole Tree Hash (Hashing All Merkle Tree Data at Once)
The Merkle root and the entire tree hash provide two key functions:
Merkle roots allow users and relays to download a portion of a configuration file at a time, just like being able to download a transaction without having to download an entire block.
The whole tree hash lets users and relays know if their stored configuration files are incomplete. Unlike the Merkle root, the whole tree hash will only match if you have all the bits of data in the Merkle tree.
This cheap method can be used to update entire profiles weekly or at a user-defined frequency. Nostr will still work as it does now, but if users want all users to see their posts, they can occasionally pay a few sats to sync their data between Nostr relays.
Users and relays can download posts one branch at a time. After each branch, they hash that branch with another branch that is closest to the Merkle root to check if that matches the Merkle root on-chain (similar to SPV). If the branches hash together to match the Merkle root, then they know that branch is part of the user profile, even if they don't have the full user profile yet. Users can download different branches of the same profile from many different Nostr relays, while verifying the validity of each branch and ensuring that the downloaded profile is complete.
Downloading branches one by one prevents latency attacks that could cripple many distributed networks, which is why the Bitcoin whitepaper uses Merkle roots and branches to protect SPV light wallets.
Why can't a Merkle root function like the entire tree hash?
If Nostr relays relied only on the Merkle root, they would have no way of knowing when the Merkle tree was complete, because every pair of branches closest to the Merkle root would hash to the same Merkle root.
To ensure that the user's profile is complete, the relay or user will hash their updated entire Merkle tree and verify that it matches the entire tree hash on chain. If the entire tree hash matches, then the user data is complete. If the entire tree hash does not match, then the relay or user can tell other relays their latest leaf number and request the missing branch until the entire tree hash matches. In order to keep track of all the new Merkle roots that are added every week or so, Nostr relays must become Bitcoin full nodes. Nostr 2.0 relays are indirectly paid to store the Bitcoin blockchain, while enhancing the security of both Bitcoin and Nostr.
Nostr Storage Limits: User Rules of Thumb
Since relays have the power to choose what to store, unlike Bitcoin full nodes, Nostr relays may lose some user data. Therefore, users should only store data on Nostr relays if they can back it up locally. Web5 self-hosted services can allow users to sync backups to all local devices, which will reduce the risk for users who are concerned about using Nostr. Ultimately, the blockchain is the only place where data is truly immutable. Despite this, Nostr is a fairly secure hybrid solution that still works well for many applications. The trade-offs are listed below:
Three-layer trust minimization
Layer 1: Immutable and expensive data storage that is extremely difficult to censor. (On-chain blocks are synchronized with all Bitcoin full nodes)
Layer 2: Mutable and cheap data storage, moderately censorship-resistant. (Off-chain Merkle trees and on-chain hashing, synced Nostr relays on demand)
Local data storage is synchronized to all local devices and is easily auditable. (Local Centralization)
The fundamental trade-off between Nakamoto consensus-based blockchains and Nostr
The more Nostr relays that store data for a particular address, the harder it is to censor that data. This means that popular data hosted by many Nostr relays may be harder to censor than unpopular data that is rarely downloaded.
On the other hand, Nakamoto consensus blockchains can prevent censorship based on the age of the data. The longer the data exists in the blockchain, the more difficult it is to delete it using a 51% attack.
ZKCSP with Proof-of-Retrievability and Lightning Payment Nostr Relay
Since we can verify that certain branches belong to a specific user, it is possible to pay Nostr relays every time they pass on a small piece of data to a user. To achieve this, the user needs to download the header of the blockchain (just like in SPV) to be able to perform the typical functions of a light wallet. The user will utilize the SPV functions of the light wallet to fetch a specific transaction from the chain, which will contain the Merkle root of the user's profile and the entire tree hash in its OP_RETURN. Now, the user can pay the relay to download the profile of that user branch by branch and verify each branch by hashing them to match the Merkle root on the chain.
To send sats (the smallest unit of Bitcoin) to the Nostr relay in exchange for providing data, we use an evolution of Gregory Maxwell’s (famous Bitcoin Core developer) ZKCP design (Zero-Knowledge Conditional Payments) [ 1 ], namely ZKCSP: Retrievability Proofs [ 2 ] combined with the Lightning Network.
According to the ZKCSP white paper:
“…no trusted third party is required…we also implement the ZKCSP protocol for the proof-of-retrievability case, where the client pays the server to prove that the client’s data is correctly stored on the server.” [ 2 ]
The user launches a lightning smart contract with several financiers.
The user sends a request to the financiers around him, who sign the request.
Users send requests signed by funders directly to Nostr relays connected to these funders.
Users now initiate the ZKCSP construction to ensure that the Nostr relay will only receive payment from the funder after the correctly requested data has been delivered.
Once step 3 has occurred, the user will make modifications on top of the original request signed by the financier before initiating the ZKCSP construction in step 4. The user will add additional content on top of the original request, specifying the amount to be deducted from the user’s and the financier’s balances (they must be the same amount, plus the financier’s fee), and then the user will sign their additions to the original message.
If a user specifies to send more sats than they have, or more than the financier has frozen on that Nostr relay, then the Nostr relay will reject the request as the relay will not be able to get paid.
In this way, users can connect with many Nostr relays while blocking their funds with only a few financiers. A similar approach can be adopted for the Lightning Network, where trust-minimized financiers are permissionless middlemen between users and merchants. Normal P2P lightning hops can also be used in Nostr 2.0, but this approach may be useful if routing and channel balancing often fail.
Whitelist Unlock Paid Nostr Relays
If Nostr relays wish to store all of this user-viewed data, they can whitelist certain keys. If Nostr relays cannot whitelist users who wish to store data, then they will store any data sent to them. If users could always send data to relays for free, then users would never have to pay Nostr relays. Nostr can only offer a paid option if relays have the option to refuse to store data that cannot be paid for. Free relays still exist, but the option for paid relays does not currently exist.
Rather than trying to store all Nostr data for free, paid Nostr relays could use whitelists to selectively store all the data viewed daily by their paying users. Some Nostr relays will continue to adopt a free model, but at the largest scale this is unsustainable as most free relays are just enthusiastic hobbyists. Whitelisting (even if we could securely assign a public key to each Nostr profile) giving Nostr relays the ability to decide which data is stored would be impossible.
One public key per profile unlocks whitelisting functionality: Bitcoin address becomes your Nostr public key
This system ultimately enables us to assign a public key to each profile.
There is no benefit for users to create a new public key for each post... because they are all associated with their profile! This is different from a Bitcoin address. Unlike Bitcoin, letting users have multiple public keys in the same application does not improve privacy.
The public key of the Nostr profile must match the public key of the Bitcoin transaction containing the weekly hash (the Merkle root and the entire tree hash of all the user's posts). Unlike Nostr users using Schnorr signatures, they will need to sign using a Bitcoin wallet (mobile/light wallet or full node).
The beauty of this is that each Nostr account will be represented by its Bitcoin address, which means that users can send payments directly to a Nostr account without requesting two different public keys. This reduces the cognitive cost of new users in the system. Users still need to send each other public keys or DNS to find each other on Nostr, rather than using usernames.
If other Nostr applications use different public keys, they can still be attached to the same decentralized identity (DID) - this way, the way to identify your account remains consistent across applications. However, this Nostr consensus rule will limit the use of only one public key per profile on each Nostr application.
DHT acts as a peer discovery leaderboard.
Relays can use a distributed hash table (DHT) to find other relays. Relays can share their whitelist in the distributed hash table by listing the public keys where data is stored. This way, relays with missing branches for a certain public key can scan the DHT and connect directly to the IP addresses of other relays that claim to store those missing branches. Relays can then download the missing branches directly from these Nostr relays.
Relays will also be able to find the most active relays by checking how many previous ZKCSP transactions — both recently and over time — those relays have solved on-chain. In this system, all Nostr relays become full nodes, so auditing other relays’ previous transactions will be trivial. This will make faking trust expensive, since on-chain transactions always require transaction fees. If a Nostr relay opens many channels to build trust with himself in order to gain the trust of other relays, he will have to pay a lot of transaction fees to maintain his fake reputation every week. After the attacker fails to provide the missing branch, a timeout will cause the relay to disconnect — so this is only a temporary, costly attack (just like a 51% attack was a temporary, costly attack).
The DHT is not as anonymous as mining, as the public key of each Nostr relay will be listed next to its IP address in the DHT, but it will avoid the need for relays to blindly send requests across the network - at a large enough scale, this would cause the network to overload. Nostr relays who wish to achieve greater privacy can use a VPN or other IP protection service.
Users will not have access to this same trust system because they are not full nodes. However, users can rely on it.
Financiers indirectly connected to hundreds of Nostr relays
Since users automatically store all block headers in their light wallets, users can see who the most active miners are. Miners becoming financiers will allow users to filter out the most popular miners so that they don’t have to blindly tie up funds with random financiers who have no connection to the viability of the network.
Financiers (miners) only need to lock their sats with Nostr relays, without passing the data itself between users and relays. Financiers (miners) only need to sign the user's request so that the user can directly interact with all Nostr relays connected to the financier - the 4 steps of ZKCSP+Lightning as described above.
in conclusion
Without Bitcoin’s Nakamoto consensus blockchain, the Lightning Network would not be able to exist as users would have nowhere to store bundled proofs of their off-chain transactions.
Just like users bundle all those Lightning Network transactions together and put a small proof into the blockchain, we will bundle all the Nostr data and put a small proof into the blockchain. In the same way that the Lightning Network provides instant payments on the second layer, Nostr may be able to provide data storage on the second layer without the risk of insecure sidechains.
It uses the same approach as the Lightning Network, with Bitcoin’s Nakamoto consensus blockchain on layer one and Nostr+ZKCSP Lightning on layer two.
