Written by: Feng & Carol, DFG
Why is decentralized storage critical to encryption and Web3?
Public Layer 1 blockchains such as Bitcoin and Ethereum have long relied on the development of external storage technology because blockchains are inherently unsuitable for storing large amounts of data for several reasons:
The blockchain itself is designed for value transfer, not for storing large amounts of data, and storing data in blocks is very expensive.
If a large amount of data is stored in blocks, network congestion will become very serious, which will trigger a gas war and cause fees to rise.
Even if the blockchain can accommodate the storage of large files by expanding its capacity, the data structure of the blockchain determines that block information is mostly stored in an encrypted and compressed form, which is not suitable for retrieval and analysis of file information.
Obviously, these cost and efficiency issues cannot be solved by centralized storage service providers, whose data is susceptible to censorship and is mutable, which goes against the spirit of blockchain and Web3.
Only by achieving decentralized storage can data rights confirmation and sovereign network be truly realized. Otherwise, it is meaningless to develop blockchain network at the expense of centralized efficiency. Therefore, decentralized storage network (DSN) has developed into a mainstream solution. This is a storage business model that uses blockchain as the underlying technology and stores files or file collections in shards on the storage space provided by the supplier through distributed storage.
Ideally, a DSN must be able to store, retrieve, and maintain data while ensuring that all participants in the network are incentivized for their contributions while maintaining the trustlessness of decentralized systems. However, in practice, each DSN project has made its own trade-offs. For example, Filecoin pursues decentralization, but its time-space proof algorithm has a large amount of computation and slow write speed, which affects the storage experience and performance; Storj sacrifices some decentralized features while pursuing high performance; Arweave, which takes a different approach, has achieved permanent storage well, but it also brings higher storage costs and is more suitable for NFT and DApp.
At present, the overall market size of the decentralized storage track is relatively small. According to a report released by Messari, the total revenue of the four leading storage protocols (Filecoin, Arweave, Storj and Sia) in 2022 is only about 14.5 million US dollars, which is in sharp contrast to the current tens of billions of US dollars in cloud storage market demand. According to a research report by Vantage Market Research, the global cloud storage market is expected to grow from US$90 billion in 2022 to US$330 billion in 2030, with a compound annual growth rate (CAGR) of 17.7% from 2023 to 2029.
The adoption of new technologies such as the Internet of Things (IoT), artificial intelligence (AI), and automation is driving exponential growth in data creation and storage demand. While centralized data storage service providers occupy a huge market share, they have exposed many drawbacks of data and market monopoly. The decentralized storage field urgently needs more game-changing startups to join in order to promote the realization of more Web3 use cases with high requirements for data sovereignty.
Global cloud storage market size forecast/Source: Vantage Market Research
Flostream Network: Decentralized Web3 infrastructure service provider
Flostream Network is trying to bring some changes to the decentralized cloud storage market. Flostream is a decentralized physical infrastructure (DePIN) service provider that is building a new decentralized storage and distribution solution for Web 3.
Flostream Network not only provides content storage services through Amazon S3 compatible APIs, but also can quickly retrieve content, realize on-demand content distribution, and support applications such as social media or video on demand streaming, and its performance is no different from similar Web 2 products.
Flostream is designed to provide data persistence, security, and reliability to users or applications in its ecosystem. Any data loaded into Flostream is completely owned by the user, ensuring digital property ownership of the data. Flostream has a local verification service that can verify any loaded data through digital signatures and timestamps. The Flostream team hopes to protect the integrity of the Internet through permanent storage and native authentication capabilities, allowing users to back up copies of their social media profiles, creator content, or the entire Internet to the Flostream network, thereby protecting the truth in the era of deepfakes and AI.
Floststream Storage and Content Availability
David Rhodus, founder of Flostream, worked at Consensys and developed many Ethereum-based POC products for some large companies, many of which eventually suffered from scalability issues. David and his team spent a lot of time experimenting with how to connect distributed systems with blockchains and how to control these systems using blockchains.
In the process of developing an open, decentralized wallet-to-wallet messaging meta-protocol, the team encountered a scalability problem. A large amount of communication will generate a huge amount of data, and the Flostream Network team realized that the existing decentralized storage solutions (IPFS, Filecoin, Crust, etc.) have many limitations, which prompted Flostream to start looking for its own decentralized storage solution.
Flostream has launched its own Amazon S3 compatible storage service, Flostream Storage, which has made a lot of innovations from the underlying technology to the storage architecture design to achieve differentiated competition. By introducing innovative technologies such as a proprietary metadata layer, Flostream Data Transfer Protocol (FDTP), and a cutting-edge hashing algorithm based on BLAKE3, Flostream Storage is expected to solve the scalability and low performance problems currently faced by the decentralized storage market. The purpose of developing an autonomous storage protocol is to optimize content distribution and eventually become the content availability layer of Web3, enabling data to be easily stored, moved, retrieved, and distributed.
Technical Overview
Flostream Storage consists of multiple cells, each containing a storage cluster and associated control and monitoring infrastructure. These cells provide customers with Amazon S3-compatible storage services, ensuring isolation and security of customer data. Cells consist of storage nodes and gateway nodes, which are the entry points to the cell storage cluster. Nodes in a cluster can be used as storage nodes or gateway nodes. Both types of nodes can communicate with all other nodes in the cluster, maintain a metadata repository, and provide S3 API endpoints.
Flostream storage architecture consisting of storage and gateway nodes / Source: Flostream documentation
Competitive Landscape
At present, whether it is mainstream centralized storage service providers such as AWS, Google Cloud or decentralized storage service providers Filecoin and Arweave, there are obvious shortcomings. Although the centralized solution has higher storage performance, the price is not cheap, and there is also the possibility of single point failure. The decentralized solution is relatively cheap (Arweave is a one-time payment, so it is slightly different), but there are many performance limitations in both storage and throughput, and it cannot be applied to large-scale, high-concurrency Web3 application scenarios.
The pain points of existing distributed storage protocols such as IPFS are that data is stored on an ever-expanding branch of nodes and relies on a linear file retrieval method, which limits scalability and speed. Flostream's goal is to achieve the performance advantages of centralized storage service providers while being low-priced and with lower network latency than the leading decentralized storage projects on the market, seeking a more balanced storage solution.
To address this series of bottlenecks, Flostream storage uses a proprietary metadata layer that enables fast data retrieval and low-cost operation. In addition, Flostream uses a self-developed cutting-edge hashing algorithm based on BLAKE3 for data compression and network management, which is 14 times faster than the SHA-256 currently used by AWS. At the same time, the high-performance data transmission protocol FDTP was introduced, which was optimized based on the existing UDP protocol. Flostream also uses intelligent data placement and data migration. Artificial intelligence data location migration can reduce hotspot deployment, and edge storage can process data closer to the data source, thereby achieving faster access and providing faster throughput under high load. The self-created intelligent deduplication algorithm minimizes storage redundancy and optimizes capacity usage, thereby reducing costs and increasing data transmission speeds. Through a series of technical optimizations, Flostream storage has achieved significant improvements in multiple dimensions:
The storage and traffic prices are lower than those of centralized AWS and Google Cloud. Although still slightly higher than decentralized projects such as Filecoin and Storj, it can provide superior storage and bandwidth performance at the same cost.
The read/write throughput can reach the level of current centralized storage service providers, and under the centralized operation mode, the throughput capacity will far exceed the leading decentralized storage projects such as Filecoin, Arweave, Storj, etc.
Still maintains high scalability and low latency similar to centralized storage service providers.
Flostream currently provides services to thousands of individual users and is conducting pilots with multiple enterprises to transfer storage workloads from AWS and Google to Flostream in the future.
Solid and experienced team
Flostream team members have extensive experience in blockchain, software engineering, cloud computing, data storage and project management, and are committed to using their skills and resources to provide the most advanced Web3 solutions in the crypto industry. Founder David is a serial entrepreneur with more than 25 years of engineering experience. As early as 2010, he established a streaming platform that supports more than 100,000 live sports events per year, which was successfully acquired by Amazon Web Services. Such cases are rare among Web3 entrepreneurs. In 2015, he joined Consensys and focused on the development of large-scale enterprise blockchain solutions. David's background in serial entrepreneurship not only provides a strong endorsement for the project, but also has attracted the attention of many well-known institutions. Flostream completed a $3 million financing in February this year, with DFG, Fuse, Scytale, D1 Ventures and TRGC participating in the investment. It is one of the few decentralized storage projects that can still obtain external funds in the crypto bear market.
Summarize
At present, most of the cloud storage market share is still monopolized by centralized solutions provided by giants such as Amazon and Google. Decentralized storage project products have not shown obvious advantages in terms of user experience, performance, cost, etc. These decentralized projects mainly operate in a mining manner and try to attract users through token incentives, which has led to a large number of users leaving after the token incentives were reduced in the crypto bear market. Lacking the native application attributes of Web3, it is inevitable that it will not be able to attract more users and enterprises in the long run and achieve large-scale growth.
Although decentralized storage can solve many problems, such as data privacy protection, anti-censorship, and avoiding single point failures, the development of decentralized systems is still immature, and there are still many problems to be solved. The scale of the decentralized storage industry is far from reaching its peak. Due to the overall sluggish market in the previous bear market, lack of content distribution capabilities, and limited application scenarios, even the leading decentralized projects such as Filecoin, Arweave, and Storj cannot compete with centralized storage giants in terms of market size.
As Layer2 technology matures, DApp runs faster and faster, and the cost is getting lower and lower. Web3 application narratives have returned to the public's attention. The accumulation of a large amount of data and market demand will once again turn the industry's heat to the decentralized storage track. In the future, the entire DSN market will need more middleware and aggregators to carry more user traffic on the one hand, and on the other hand, it will need to continuously adjust and optimize the DSN application threshold and cost to attract more users. Flostream's storage and data availability products are currently the ideal solution for seamlessly combining DApp and decentralized storage. As an early investor in Flostream, DFG is pleased to see that Flostream's products are bringing more practical and feasible application solutions to the broader Web3 market. As a veteran in the cloud service industry, David's previous entrepreneurial project was acquired by AWS, which has proven his outstanding leadership and creativity. We firmly believe that David can lead the team to the next success and enable Flostream to successfully lead a new wave of innovation in decentralized storage and Web3 applications.
Original article: https://dfg-official.medium.com/an-emerging-decentralized-storage-solution-revolutionizes-content-storage-and-availability-e9464472d5b8
