The reason I have come to deeply understand the true importance of implementing fully decentralized computation is rooted in a rather special experience behind the scenes. This story is worth telling—so please, let me take you through it slowly.
As early as 2015, I had already thrown myself into the early cryptocurrency space, but I also continued to run the “Fight My Monster” that I had founded previously—a venture-backed MMO (massively multiplayer online) game and social networking platform with 3 million users that is financially self-sustaining.
Although I’ve found my long-term career home in the field of decentralized computing, I can’t let down the community of players who have continued to support this game.
“Fight My Monster” was developed between 2010 and 2012. It used an architecture that was extremely innovative at the time. That architecture enabled the platform to scale up with low cost and very little operational workload (i.e., accommodating huge numbers of users).
At its core, the system consists of three main backend components. The first is a simple web hosting service, responsible for distributing the frontend game code and assets to the user’s web browser (at the time, I used a mainstream content delivery network, i.e., a CDN).
Like the web hosting service, the other two backend components must also support “horizontal scaling.” This means that to support more users and their associated compute and data needs, I only need to add hardware resources—I don’t need to rewrite backend software code, and I don’t need tedious system administration work. This feature allowed me to achieve rapid scaling up early in the project with only a tiny team and limited funding.
The second component is a horizontally scalable game server framework called “Starburst,” the result of painstaking work that I personally poured my heart into. It’s built on top of an open-source server that supports the RTMP protocol—RTMP is a network protocol developed by Adobe that allows Flash media assets running in web browsers (remember Flash?) to communicate directly with the server. Starburst carries the logic for the game and social network, and routes the huge number of events generated in the game to users in near real time.
The third component is a horizontally scalable NoSQL database called Cassandra (if you’re paying attention to databases, you probably know a modern derivative of it, ScyllaDB—which is currently excellent at meeting real-time AI needs).
It’s worth mentioning that in my team, besides the original company that developed Cassandra, there was only one other core committer of Cassandra. Also, I was among the earliest users to put Cassandra’s beta into intense production use. When the user count reached 800,000, the database became corrupted. The next 24 hours were nerve-wracking. I had to recover the data and bring the system back online with all possible speed (in that moment, I felt like I was sprinting toward the cloud, and the next moment I felt like everything was already beyond saving—but thankfully it ended up being fine).
The best thing about Cassandra is that you can scale storage capacity, read throughput, and—most importantly—write throughput simply by adding nodes (i.e., servers with high-capacity disks). This kind of feature is rare, but it’s crucial for my application, because “Fight My Monster” generates massive amounts of data.
Of course, Cassandra’s other important advantage is that it has fault tolerance. I set the replication factor to 5, which means everyone’s data is very safe (at least that’s what I believed at the time…). With a replication factor of 5, for any specific “quorum,” as long as 3 nodes are online, the system can write data successfully.
Unlike a blockchain, Cassandra doesn’t provide Byzantine fault tolerance. That means I have to manage infrastructure security very carefully to prevent hackers or malware from getting in. Back then, the hackers weren’t interested in a MMO game and social network whose main audience was children—and it wasn’t the kind of general-purpose platform others could build on.
The main technical pressure I faced was that, because the volume of data generated was enormous, the nodes that carried the backing database were under extreme load, causing them to “crash” constantly! This meant that I had to log into the infrastructure from time to time, take the faulty nodes offline, and add new ones. The good news was that the work wasn’t difficult: thanks to the fault-tolerance mechanisms, it didn’t have to be done urgently, and these “bare metal” servers could also be scheduled uniformly through my cloud service provider…
“Fight My Monster” uses an architecture that was ahead of its time, allowing me to scale effortlessly. In addition, its fault-tolerance and streamlined backend architecture with only three core system components mean the administrative workload and pressure required to keep the system running are minimized. So what lesson was so important that it still deeply shapes the way I think today and the development of the ICP? You can probably feel that something extraordinary happened back then—now I’ll tell this experience for the first time publicly.
At the time, I was traveling around Asia. As part of an early Ethereum core community “roadshow team,” I visited relevant parties (people often overlook the important support role that Asia—especially China—played in the development of early cryptocurrencies), and attended all kinds of Ethereum-related events. At these events, I mainly discussed some complex computer science technology I’d developed myself, hoping to improve the scalability and performance of networks like Bitcoin and Ethereum.
I don’t want to ramble about “the old days in the cryptocurrency world,” but I have to say it was a magical time. The early participants left me with so many wonderful memories, for which I’m deeply grateful. That experience was so captivating that I even didn’t have time to check my emails.
As a result, on my trip I missed countless emails—each one warning me: the credit card used to pay for the Fight My Monster project’s Cassandra node fees had expired…
So what did the cloud provider do when I didn’t respond within a few weeks (I can’t remember the exact timeframe, but it wasn’t long)? Very simple: they just deleted all the nodes!!!
Hard to describe what it felt like when I got back to Palo Alto, California and found all of this. It was a mix of shock and fear: an empty stomach, adrenaline surging, and dread spreading continuously. I desperately tried to save things, but ultimately confirmed there was no way back. Anyone who has experienced being hacked in the cryptocurrency space would understand the feeling.
And that’s how it ended. Fight My Monster completely vanished. In the community, even someone started a petition on Change.org, pleading for it to be brought back online.
Even though I lost the value it could have created, I still felt guilty—but in a way, it was also a kind of relief. I no longer had to take on the responsibility of maintaining it (otherwise I might have kept maintaining it for the community endlessly). Plus, it became a wonderful memory for a generation of kids (and some adults) who played the game, and it had already wrapped up before it became stale and boring—allowing me to focus on the decentralized computing network I was obsessed with back then.
However, I learned a very deep lesson from it: you can build decentralized computing infrastructure with scalability and fault tolerance, but as long as there is a single point of failure at any time, in any part of the system, the entire system can collapse completely.
For that reason, whenever I hear that some blockchains rely on admin keys held by core developers, or I hear that large numbers of nodes go offline because cloud providers shut them down or due to technical failures (even getting close to complete paralysis), I can’t help but worry. This extreme fragility is far more common in our industry than you might think.
The painful lesson from Fight My Monster deeply influenced the design philosophy of the Internet Computer. It’s not just the number of nodes—there are other, even more critical factors. If the nodes are anonymous, the actual number of independent entities running most of them could be far lower than you expect. Also, even if the node count is huge, they often run on the platforms of only a few cloud service providers—even if they’re owned and operated by different entities, those providers may suddenly decide to stop supporting them and shut them down overnight (as Hetzner did to a certain well-known blockchain project).
In practice, what truly matters is the number of independent entities running the nodes, and the independence of the nodes in terms of physical location and jurisdiction. This simple insight is the foundation of the “deterministic decentralization” concept, which will show up in a brand-new form in the upcoming Internet Computer expansion feature, the “ICP cloud engines.”
ICP Cloud Engines allow enterprises (in most cases) to create Internet Computer subnets that are actually theirs, and to control their own configuration. Enterprises can compose these subnets by selecting nodes from a node pool that acts like a “market.” At present, more than 1,500 nodes are on standby, and the number is still growing.
“Engine” provides a serverless cloud platform with outstanding features. It’s an ideal place to build AI applications and services because its technology stack is designed specifically to support AI that can easily generate advanced AIware software—software that’s secure and highly resilient.
These applications have a range of features: tamper resistance (immune to infrastructure-layer hacks), always-on (and thus highly resilient), optional autonomous operation (meaning there are no backdoors), and native support for digital assets, among other things. Engines can also run AIware applications such as the Open SaaS suite. This enables enterprises to carry out end-to-end business operations, and makes installing and maintaining complex applications like Open CRM or Open Email as easy as using mobile apps.
Nodes support hot swapping. This means you can adjust the compute replication factor or replace nodes without interrupting the services hosted by the cloud engine—freeing you from reliance on a specific compute provider. Additionally, you can choose to use new node types deployed on hyperscalers, as well as traditional Internet Computer nodes, turning it into a general-purpose marketplace for compute resources.
The Cloud Engine framework guides users to compose nodes with independence, while also providing flexibility that shared Internet Computer hosting doesn’t have. For example, a German company might choose nodes located only within the EU to meet GDPR (the General Data Protection Regulation) requirements.
However, the autonomous cloud engine console—hosted by the Internet Computer network and updated by the NNS (Network Nervous System, the world’s most advanced DAO)—will provide guidance. If a user combines nodes from the same node provider, nodes located in the same data center, or nodes operated by the same cloud operator, the console will issue strong warnings.
Therefore, even in the context of the ICP Cloud Engines, the lessons from the “Fight My Monster” project remain very relevant.
People might wonder why the framework strongly pushes to combine independent nodes. There are solid reasons behind it—reasons that are just as important for today’s broader blockchain community as they were back in 2015.
I’m looking forward to sharing more about the cloud engines with you soon.
The IC content you care about
Technology updates | Project information | Global events
Follow and watch IC • Binance channel
Get the latest updates