As everyone knows by now, AWS fell over for about a day this month, resulting in what’s likely to be billions of dollars in losses. Countless high availability applications and platforms blew out their annual SLA requirements in a single downtime incident. You’re only allowed 52.6 minutes of downtime per year to achieve four nines, and most affected systems were down for hours or more. And most disappointing of all, the much vaunted decentralization of blockchain and defi was shown once again to be an emperor with no clothes, as dapps and wallets everywhere went down hard.
And to that I say, shame on you.
It’s been five years since Pocket Network came online, created literally as a response to this kind of downtime breaking the entire point of decentralization. It’s been almost exactly five years to the day that the original Infura outage took out defi for hours. Five years is a long time to get to understand why decentralized applications and chains need decentralized infrastructure, and yet for some reason, technical and business leaders in this space can’t shake free from their own legacy thinking enough to understand what should be a simple truism:
If ANY part of your infra is single point-of-failure, your whole system is single point-of-failure.
Despite there being decentralized options for every layer in the stack, CTO’s and CEO’s, in an effort to be fast and cheap, almost invariably seem to default to building consensus based systems that only require one global CEO to shut the whole thing down. Given what we saw in our own network data, few few of those systems even have decentralized fallbacks in their load balancing stack, even when explicitly offered it, which is an unconscionable oversight for technical leaders of high availability applications. What the hell were you thinking?
And the thing is, it’s not just AWS outages. It’s network congestion. It’s permissioned systems. It’s single parsers or interfaces or dispatchers or access points. It’s chains and services that can’t be accessed without putting your email in to sign up for an API key. Even among those of us who have decentralization and permissionlessness as core mission and values, our systems may still require deeper evaluation and ongoing architectural improvements to meet our own standards. But flat out building a “decentralized” system on fully centralized architecture is about as “bait and switch” as it gets.
The entire purpose of consensus based systems is that they are resilient. That is literally their number one critical feature. That means resiliency from downtime, resiliency in access, resiliency against censorship, and network congestion, and resiliency against authoritarian fuckery.
When something as trivial as a standard service provider downtime shows basic resiliency to be a lie, why should users trust that all of these other aspects that they rely on to use these systems actually work as promised? Let’s be honest, the learning curve and PITA factor of using many web3 services is high; it is the promise of the benefits of decentralization, both explicit and implicit, that encourage the user to stay in spite of the extra steps. Your decentralized app getting caught with its pants down because you didn’t take the steps to build it on decentralized infra is a betrayal to your users. You owe them better than that.
A Call To Arms
For those of us who have been in the space a while, there’s a well known “decentralization tax” that is paid on both the provider and user side. Building out the coordination layer required to create a sustainable decentralized network is both expensive, and labor intensive. In the Proof of Stake world, it’s often driven by initial high inflation, which tends to put the project on an incredibly predictable cycle of high to low in market value over time until the project can reach critical mass, and shift to a more conservative tokenomics model. Many don’t make it.
One major failing of this space is a lack of metacoordination among truly decentralized providers. I’ve spoken many times about how building from top to bottom on a decentralized stack is critical to building actually unstoppable applications, and despite there being a number of DePIN focused orgs, very few of them seem to be working on the most obvious way to make fully decentralized stacks sustainable: using each other both to secure our own stacks, and to be part of a greater scope of offering which streamlines full stack infra in the same way that services like AWS do.
Decentralized Better Together
There are a handful of services that are required to build any app or chain, and if we are all serving each other in a meaningful way through strategic partnership, we act as core customers to each other, and demonstrate the value inherent to each of our offerings. Decentralized compute, storage, data views and transformation, automation, and syndication/access, along with a handful of special purpose tools for security, indexing, oracles, etc., can cluster together in partnerships to fill in all those needs for each other, and in doing so, become intimately familiar with the needs and hardships of engineers working with decentralized stacks. And, any provider can be part of multiple stacks, building greater expertise and connections.
We contribute real world value to each other which helps break the pump to flatline market cycle with consistent revenue from far earlier in the cycle, and we become collective experts on all the reasons that engineers tend to just go easy mode and spin up a US-EAST-1 container. And we work together to fix those reasons, improving our service for each other along the way. And we can talk with our external customers, and recommend additional decentralized solutions for them.
Existing programs like DAO grants and bounties, token swap partnerships, and other mechanisms that have been used by DAOs for the last five years can become fundamental economy stabilizers for all participating projects. Locking in value to a slowly vested structure during the highest activity phases of market excitement mean that the value carries forward over time for the project, instead of immediately turning into a massive market exit, and guarantees high level service from partnering projects for the lifetime of that partnership. Service level and the economic health of the market are both improved for the long term.
One Stop Shopping
One of the strengths of all-in-one providers is that you just go to one place and click the button, and you have the whole stack ready to roll. (Exaggeration, of course, but that’s the perception). If you’re already working with a decentralized collective, it makes sense to have that all in one stack ready to access in one place. Each of the providers in your stack can link to this from their own docs/website. My goal is to build an index of stacks based on decentralized providers that supports as close to one-click automation of fully decentralized stacks as possible. In my opinion, this is one of the final missing pieces in driving far greater adoption of web3 services. If you’re interested in joining that initiative, shoot me an email to jinx at pokt dot foundation.
It’s Time We Demand Better
Yes, it can be cheaper and easier to use a centralized provider. But the total cost of ownership for an application is not just the server bill. It’s the cost of downtime, the cost of platform risk, the cost of being de-platformed, and the cost of losing user trust. When you build on the unstoppable stack, you are architecting those costs out of your system from day one. Stacks that are built from top to bottom with decentralized services become largely immune to most of the common reasons that apps go offline. There is no central server to crash, no single company to send a shut off order to, no single point of failure to exploit. It’s just a resilient, self-sustaining application.
THIS is what we signed up for. It’s what we expect in our decentralized finance, storage, compute, AI, and communications. And at the bare minimum, we expect decentralized solutions as load balanced backups to systems you feel you absolutely must use a custom purpose centralized provider as your primary source. It is lazy to the point of negligence to not take the rudimentary steps required to incorporate unstoppable failover systems that keep data flowing when everything else is crashing.
The choices we make about these “boring” infrastructure layers are the choices that will define the next decade of the internet. Will we build a world of resilient, open, and permissionless services? Or will we repeat the mistakes of the last twenty years and end up with a new set of centralized gatekeepers, just with a “Web3” label on them? This is the decision we are all making with every line of code we write, every service we choose to use, and every project we fund.
It’s time we vote with our dollars and feet. If your “defi” services went down on the 19th, put them on notice. Decentralize, or be traded for a project who will.