Cover photo

Ronin (2022) - When the Guards Were Too Few

We walked.

Moolah had picked a route - north out of Ueno, up through Yanaka, around the cemetery, down through the small streets that smell of incense and old wood - and we walked it at her pace, which was slower than I had expected. She had told us to wear walking shoes, and Dev had taken her literally, and so he was wearing what looked like a pair of brand-new running sneakers that he kept glancing down at with embarrassed pride.

The petals from last week were mostly gone now. Only a few still clung to the trees. The ground was carpeted with them, browning at the edges, beginning to mat into the sidewalks. The end of the bloom, that long sad week that hanami-goers do not gather to celebrate.

"Today's story," Moolah said, "is shorter than last week's. It needs to be. But it will hurt in a different way. Last week was about a system that failed because nobody understood how it worked. Today is about a system that failed because of a phone call."

"A phone call?"

"More or less. We will get there. First - context. Have you heard of Axie Infinity?"

"The game?" Dev said. "With the little creatures."

"The game with the little creatures. In 2021, Axie Infinity became one of the biggest things in this entire industry. It was a video game where players could collect, breed, and battle small monsters called Axies, each of which was an NFT. The game had real economic value attached. Players in countries with weaker currencies - particularly the Philippines and parts of Southeast Asia - discovered that they could earn real money playing the game. Some of them earned more from Axie than from their day jobs. There were entire households whose income came from playing this game. There were guilds - coordinating groups - that loaned Axies to players who could not afford their own. At its peak, Axie Infinity had over two million daily users. The game's developer, a Vietnamese company called Sky Mavis, became one of the most prominent gaming studios in the entire blockchain industry."

"That's a lot of people."

"That is a great many people, many of whom were depending on the game for their actual livelihood. This matters. It will matter more in a moment."

She paused at a corner, looked left, looked right, and continued.

"Now - the game ran on Ethereum, originally, but Ethereum was too slow and too expensive for the scale of activity Axie generated. So Sky Mavis built their own blockchain - a sidechain called Ronin - that was specifically designed for the game. Players could move their tokens from Ethereum to Ronin via a bridge, play the game cheaply on Ronin, and then move their winnings back to Ethereum when they wanted to cash out. The bridge held the original assets on the Ethereum side, and Ronin held the matching wrapped versions. The standard lock-and-mint pattern we discussed several weeks ago."

"And the bridge held a lot."

"At the time of the hack, the Ronin bridge held approximately six hundred and twenty-five million dollars in user funds. Ether and stablecoins, mostly. The savings of millions of players. The connective tissue between the in-game economy and the outside world."

"And the validators?"

"This is the critical detail. The Ronin bridge was secured by nine validator nodes. Of those nine, five were required to sign off on any transaction crossing the bridge. The validators were operated by a small number of trusted parties - Sky Mavis itself ran four of them, and another four were run by partners and community institutions. There was a ninth, run by a non-profit gaming organization called Axie DAO. To move funds out of the bridge, an attacker would need to gain control of at least five of these nine validator keys."

"Nine seems low."

"It is. It is, in fact, much lower than nearly any other major bridge of comparable size. By comparison, the Polygon bridge had over a hundred validators. The Cosmos hub had over a hundred and fifty. Nine is, in retrospect, unconscionable for a bridge holding three quarters of a billion dollars. But nine was what they had built. And nine, as it would turn out, were also more validators than they were properly maintaining."

She paused outside a small temple. A woman was sweeping the steps.

"In November of 2021, around the height of Axie's popularity, the game's transaction volume surged enormously. Sky Mavis's own validators were having trouble keeping up with the load of signing transactions. To help, the team asked Axie DAO - the non-profit running the ninth validator - to delegate signing authority to Sky Mavis's infrastructure on a temporary basis. Sky Mavis would handle Axie DAO's signing duties for a few weeks, just to ease the load. This was, at the time, a sensible operational decision. The traffic was overwhelming. Axie DAO's volunteers could not keep up. Sky Mavis could. The arrangement worked."

"And then?"

"And then it was forgotten. After the surge passed, the temporary delegation was never undone. Months went by. Nobody on either side noticed. Sky Mavis's infrastructure continued to hold signing authority for Axie DAO's validator. Which meant that, in practice, four of the nine validators were under Sky Mavis's direct control, plus one more through the forgotten delegation. Five validators. The exact number needed to authorize a transfer."

I felt cold. "So if anyone got into Sky Mavis's systems -"

"They would have keys to all five of the doors that mattered. The four they were supposed to control, and the one they were not."

"And someone got in."

"Someone got in."

We had reached the cemetery wall. The crows were loud somewhere inside. Moolah stopped and looked at the wall for a long moment before continuing.

"On November twenty-third, 2021 - about four months before the hack - a Sky Mavis senior engineer received a job offer on LinkedIn. The offer was elaborate. It came from a recruiter for a fictitious company. The recruiter and the engineer corresponded for weeks. They had multiple interviews. The engineer was offered an extraordinarily generous compensation package. As the final step, the engineer was sent a PDF document containing the formal job offer. The engineer, to do due diligence on the offer, downloaded and opened the PDF on his work laptop."

"It had malware."

"The PDF contained a piece of malware specifically designed to install a remote access trojan on the engineer's machine. The trojan, once installed, gave the attackers access to Sky Mavis's internal systems. From there, over a period of several months, the attackers - who are now widely believed to have been the Lazarus Group, a North Korean state-sponsored hacking organization - moved laterally through the network, escalating privileges, mapping the infrastructure, and identifying the validator keys. By March of 2022, they had control of four Sky Mavis validators. They needed one more."

"And they had the fifth."

"The fifth was the Axie DAO validator. Whose signing had been silently delegated to Sky Mavis four months earlier and never undone. By gaining control of Sky Mavis's infrastructure, the attackers had inadvertently - or perhaps very, very deliberately, depending on what they had been able to discover during their reconnaissance - gained the fifth key as well. They had everything they needed."

"When?"

"On March twenty-third, 2022, the attackers used the five keys to authorize two large withdrawals from the Ronin bridge. One of one hundred and seventy-three thousand ether. Another of twenty-five and a half million USDC. Total value at the time was approximately six hundred and twenty-five million dollars. The transactions executed normally. The bridge sent the funds. The attackers sent them through several anonymizing protocols and disappeared."

"How long until anyone noticed?"

"Six days."

"Six days?"

"Six days. The bridge's transactions did not trigger any monitoring alerts because, from the chain's perspective, they appeared to be legitimate withdrawals authorized by the required threshold of validators. There was nothing irregular to see. The funds had simply moved. Nobody noticed they had moved out of the bridge instead of through it. The hack was discovered when a single user attempted to withdraw funds and found there was no liquidity left in the bridge to honor their withdrawal. They reported the issue. Sky Mavis investigated. The investigation revealed everything."

I wrote in the notebook. Ronin Bridge - March 2022 - $625M. Five of nine validators compromised. Four held by Sky Mavis directly. Fifth via forgotten delegation from Axie DAO. Attackers entered through a fake job offer / spear-phishing PDF in November 2021. Hack went undetected for six days.

Underneath, I wrote: the bridge had nine guards. The thieves had keys to the only five that mattered. They got the keys through HR.

"What happened to Axie?"

"Sky Mavis, to its credit, raised a hundred and fifty million dollars in emergency financing within a few weeks and committed to making all affected users whole. Most of the players who had funds on the bridge eventually got their balances back, some of them after a long delay. The team rebuilt the bridge with substantially more validators - the new system has dozens, distributed across many parties, with much stricter security requirements. The game itself never quite recovered. Daily users had been declining for several months before the hack, as the in-game economy had its own deflationary problems, and the hack accelerated the decline. By the end of 2022, Axie Infinity had a small fraction of the users it had had at its peak. Many of the players in the Philippines who had been depending on it for income were forced back into traditional work, or into other games, or into nothing. The story of Axie is, in some sense, the story of an entire economic experiment that briefly worked, lifted thousands of people, and then collapsed because of an engineer who opened a PDF."

"And the attackers?"

"Were never caught. The funds were laundered through various mixing services. A portion was eventually frozen by exchanges and recovered, but the majority disappeared into the North Korean state apparatus, which has been implicated in many similar attacks before and since. The hack of the Ronin bridge is, by some estimates, the largest single cryptocurrency theft in history at the time it occurred. It has since been surpassed, but the methods used - the patient social engineering, the lateral movement through corporate infrastructure, the exploitation of weak validator architectures - have become a template that other state-sponsored attackers have copied. The Ronin hack was, in many ways, the moment when this industry stopped being threatened only by lone hackers and clever code-readers, and began being threatened by professional state intelligence services with deep resources and unlimited patience. We are still living in that change."

The temple bell rang somewhere behind us. The afternoon was getting long.

"What are the lessons?" Dev asked, more quietly than usual.

"Many. Validator counts matter, and a small validator set is a single point of failure even when each individual validator is secure. Operational security - the procedures and people that surround the technical infrastructure - is often the weakest link in any system, and most users never look at it. Forgotten configurations are perhaps the most dangerous category of vulnerability there is - a temporary measure that is never undone becomes a permanent vulnerability that is never reviewed. Social engineering is the most reliable attack vector in modern security, more reliable than any code exploit, because it targets the part of the system that cannot be patched: the human at the keyboard. And, finally - and this is the lesson I want you to carry - the threat model in DeFi has changed. The early years assumed clever hobbyists and amateur thieves. The current era includes nation-state intelligence services with budgets, training, and patience that no individual project can match. The thieves are, increasingly, professionals. The defenders are, increasingly, outmatched."

I underlined that last line in the notebook. The thieves are professionals now.

"And the engineer?"

Moolah was quiet for a long time. We had reached a small wooden gate at the far side of the cemetery, and she had stopped beside it. The crows were very loud somewhere up in the trees.

"He kept his job for some time. He was not, in the strict sense, at fault - the malware was extremely sophisticated, and he had not done anything unusual or careless. The systems around him should have been more resilient. The delegation should have been undone. The validator architecture should have been more decentralized. The attack succeeded because of many failures, not just one. But he carried the weight of it anyway, in the way one always does when the chain of cause runs through one's own hands. He is, last I heard, no longer in the industry. He works in a different field now, doing something quieter. I do not know if he sleeps well. I imagine he does not always."

She looked at the cemetery wall.

"Be careful what you click, Priya. Be careful what you download. Most of the worst things that happen in this industry begin with a single careless moment by a single tired person. Most of those people are no different from you. They just had a worse afternoon than they realized."

We walked the rest of the way home mostly in silence. Dev was looking at the sidewalk.

The petals on the ground were almost all brown now. The hanami was over. The air smelled, faintly, of rain.

"Same place next week?" Dev asked, eventually.

"No," Moolah said. "Somewhere I do not want to go. But I have to. We are going to talk about FTX. Bring the notebook. Bring kindness. Some of these stories require kindness."

She slipped down a side alley. We did not see where she went.


Next: FTX (2022) - The Exchange Wasn't DeFi After All