Cover photo

The Re-entrancy Weekend (Curve, 2023) - A Bug in the Translator

The wisteria was out.

Along the long wall that runs beside the path on the east side of the park, the wisteria had begun, that week, to drop its long pale-purple racemes from its trellis - the second-favorite bloom in the Tokyo year, after the cherries, and the one that nobody crowds to see in the same way. The petals smelled faintly sweet, faintly like soap. The bees were back. The bench, when we arrived, was warm.

Moolah was in a more settled mood. She had a latte. She seemed, after the heaviness of the last several weeks, slightly recovered - like someone who had finished telling the worst of the stories and could now allow herself to breathe.

"Today," she said, "is the last of these. The last disaster in our list. I have chosen to end here, on this one, for a specific reason that I will tell you at the end. The story is more technical than the last few. It does not involve fraud. It does not involve a state intelligence service. It does not, in the strictest sense, involve any single bad actor at all. It involves a bug in a tool, and the tool was trusted by a great many protocols, and the bug, when it was discovered, took down a meaningful slice of the entire DeFi ecosystem in a single weekend in the summer of 2023. The story is a quieter one than the others. But its lesson is, in some ways, the most important."

She arranged herself on the bench.

"Have you ever heard of Curve?"

"It came up in passing," I said. "An exchange?"

"A specific kind of exchange. Curve is a decentralized exchange - like the one you used in Episode Eight - but optimized for a particular use case: trading between assets that should be worth roughly the same amount as each other. Stablecoin to stablecoin. Different versions of staked ether. Different wrapped versions of the same underlying asset. Trades where the prices should, in theory, be very close to one-to-one. Curve's mathematical curve - the function that determines its prices - is specifically designed to handle these trades with very low slippage, which makes it the dominant venue for moving between similar-valued assets. By 2023, Curve was one of the most important pieces of infrastructure in all of DeFi. Many other protocols depended on its pools for their own functioning. The Curve token, CRV, was held by a wide range of institutions. Curve was, by any measure, a foundational piece of plumbing."

"And there was a bug."

"The bug was not in Curve itself, exactly. The bug was in the language Curve was written in."

"What does that mean?"

"Most smart contracts on Ethereum are written in a language called Solidity. Solidity is mature, widely used, and has a huge ecosystem of tooling and security analysis around it. Curve, however, was written in a different language called Vyper. Vyper is intentionally simpler than Solidity. It was designed by some of Ethereum's earliest developers as a more security-focused alternative - a language with fewer features, fewer ways to introduce subtle bugs, and easier to audit. The trade-off was that Vyper's user base was smaller, its ecosystem less mature, and its compiler - the program that translates the Vyper source code into the bytecode that actually runs on the blockchain - had been written by a small team."

"And the compiler had a bug."

"The compiler had a bug. Specifically, in versions of Vyper from 2021 onward, a particular feature of the language - a feature called re-entrancy locks - was implemented incorrectly. Do you remember what re-entrancy is? From three weeks ago?"

"The DAO bug. Withdrawing money before the ledger gets updated."

"Exactly. Re-entrancy was the bug that nearly killed The DAO in 2016. It is one of the most famous and well-understood vulnerabilities in smart contract programming. Every modern language has built-in protections against it. Re-entrancy locks are the standard defense - they prevent a function from being called recursively in a way that would allow the kind of double-spend that drained The DAO. Vyper had this feature. Developers used it. Many protocols, including several Curve pools, relied on it. The protection worked the way the developers expected it to work."

"Except -"

"Except in a few specific versions of the Vyper compiler, the implementation of re-entrancy locks had a quiet flaw. The lock, in those versions, did not actually prevent re-entrancy in all the ways the language specification claimed it did. There was a particular pattern of function calls that could slip past the lock. This was, technically, the compiler's bug - not the protocols' bug. The protocols had written their code correctly, according to the documentation. The documentation was right. The compiler, in implementing what the documentation described, had made a small mistake."

"And nobody noticed."

"Nobody noticed for over a year. The bug had been introduced in late 2021. It sat in the compiler, undisturbed, for nearly two years. Every protocol written in those compiler versions, that relied on re-entrancy locks, was vulnerable. None of them knew. The audits had been performed. The audits had not found the bug, because the audits had been looking at the protocols, not at the compiler. The assumption - reasonable, almost universal - was that the compiler did what the documentation said. That assumption was, for nearly two years, wrong."

"How was it found?"

"A security researcher discovered the flaw and quietly notified the Vyper team in early 2023. They began working on a fix. They considered, in the manner of all responsible disclosures, how to handle the situation. They decided to patch the compiler and notify affected protocols privately, so that contracts could be migrated before the bug became publicly known. This is standard practice. It usually works. It did not, in this case, work in time."

"Someone else found it."

"Someone else found it. We do not know whether they discovered it independently or whether the disclosure leaked through some channel that the responsible-disclosure process had not anticipated. What we know is that on the morning of July thirtieth, 2023 - a Sunday - an attacker began exploiting the bug across multiple Curve pools simultaneously. They drained four pools in rapid succession. Approximately seventy million dollars was taken from the affected pools in the span of a few hours."

"On a Sunday."

"On a Sunday. Which, in DeFi, is a specifically dangerous day. Many of the people who would normally respond to a crisis - security teams, protocol developers, white-hat hackers - are not at their desks. The first response was slow. By the time the affected protocols realized what was happening, much of the damage was done. The CRV token, sensing crisis, began to fall. Curve's other pools - even ones not directly affected by the bug - saw enormous outflows as users panicked and withdrew. The price of CRV dropped over thirty percent in hours."

She paused.

"And here is where it became more than a single hack."

"What do you mean?"

"The founder of Curve - a developer named Michael Egorov - had, for some time before the hack, been using his personal CRV holdings as collateral for very large loans on multiple lending protocols. He had borrowed something approaching a hundred million dollars against his CRV. The borrowing had been visible on-chain. Other observers had been concerned about it for months - if CRV ever fell sharply, his loans would be liquidated, and the resulting forced selling could trigger a much larger collapse. The hack created exactly the conditions that the worried observers had been worried about. CRV was crashing. Egorov's positions were heading toward liquidation. If they liquidated, the resulting cascade could have wiped out Curve entirely, and possibly several adjacent protocols, and possibly more than that. The bug had become a systemic risk."

I wrote rapidly. Curve, July 30, 2023. Vyper compiler bug - re-entrancy locks broken. Four pools drained, ~$70M. CRV crashed. Founder's collateralized loans threatened to cascade. Systemic risk to broader DeFi.

"What happened?"

"Several things at once. First, white-hat hackers - security researchers acting in good faith - exploited the same bug defensively, draining funds from vulnerable pools before the malicious attacker could reach them. Some of these white-hat actors had not coordinated with the protocols beforehand. They simply saw the attack happening, recognized that they could pull funds out faster than the attacker could, and did so. They then publicly identified themselves and returned the funds. Several million dollars were saved this way. The image of strangers racing the attacker to drain the pools first - to prevent worse theft - is one of the more peculiar things this industry has ever produced. There is no equivalent in traditional finance. Imagine a bank robbery in progress, and a passerby pulling out the vault first, in order to give it back to the bank later, more safely. That is, more or less, what happened."

"And the founder?"

"Egorov, working with several large investors and a few rival protocols who recognized that his collapse would hurt them too, arranged emergency loans and asset sales to reduce his leverage and prevent liquidation. He sold large amounts of CRV in over-the-counter deals - at a discount - to avoid having to sell into the public market and crash the price further. Several of his positions were closed. He was, by the end of the weekend, no longer in immediate danger of liquidation. The systemic cascade did not happen. The market stabilized. CRV, eventually, recovered some of its losses. The protocols that had been drained worked with the white-hat hackers to recover most of the stolen funds. The malicious attacker - whose identity remains unconfirmed - kept some of what they took, but a smaller amount than they had originally drained."

"That sounds almost like a happy ending."

"It is, by the standards of these stories, the closest thing to one. Total losses were in the low tens of millions, recovery was substantial, no users lost everything, the systemic risk did not propagate. Compare that to Terra, FTX, or Ronin. By contrast, the Curve weekend was a manageable crisis that the industry, working roughly together, weathered. The lesson is not that the system was unscathed. The lesson is that, when the system has built up reflexes - public disclosure norms, white-hat traditions, coordinated emergency response, deep liquidity to absorb shocks - even significant attacks can be partially contained. The infrastructure of resilience matters, and it had been built up, slowly, over the years between The DAO and Curve."

"What did the industry learn?"

She considered.

"Several things. Tooling matters as much as code. A bug in a compiler is, in some sense, a bug in every contract written using it. The trust relationships that hold this whole stack together - the trust in compilers, in libraries, in standard implementations - are usually invisible, until they fail. After Curve, there was a major effort to formally verify the Vyper compiler, to audit its critical sections, and to encourage more diversity in the languages used by major protocols. The single point of failure was identified. The community moved, slowly, to reduce its concentration."

"Second."

"Personal leverage by core developers is dangerous. Egorov's loans had been a known concern long before the hack. The fact that one developer's leveraged position could threaten the protocol he had founded was a structural weakness that the community, for various reasons, had failed to address. After the weekend, there was much louder discussion of governance constraints on insider behavior, and several major DeFi projects adopted clearer rules about how core team members could use the project's tokens as collateral. The discussion is ongoing. The constraints are imperfect. But the issue, at least, is taken more seriously now."

"Third."

"The white-hat tradition is real and matters. The fact that a small community of security researchers, with no formal coordination, will act in defense of protocols they do not own - racing attackers, returning funds, taking on legal risk to do the right thing - is a peculiar feature of this industry, and one of its few genuinely admirable cultural traits. It cannot be relied upon. It is not a substitute for good security. But it has, in many cases, saved enormous amounts of money. The Curve weekend is one of the clearest examples. It deserves to be remembered."

The wisteria racemes overhead were very still in the warm air. A bee was working its way along one of the lower clusters with focused, professional attention. Somewhere across the park, a school choir was practicing - a small thread of high voices, just barely audible over the breeze.

"Why did you choose to end on this one?" I asked.

Moolah was quiet for a moment.

"Because the others are stories of disaster. This one is, partly, a story of competence. Imperfect, partial, late, but real. Most of the disasters I have told you about over the last six weeks ended badly for many people, and the lessons came at terrible cost. The Curve weekend, by contrast, was a serious crisis that the industry partially handled. People made good decisions in real time. The reflexes existed. The community responded. The damage was contained. I wanted to end on this because I do not want you to come out of this section of our lessons believing that DeFi is only a series of disasters. It is also the people who, when disasters come, sometimes do the right thing. Most of those people are anonymous. And most of them are not paid. They show up on weekends, in their pajamas, on Discord, and they help. That tradition, fragile and real, is worth remembering. It is, in some ways, the best thing this industry has."

I wrote that down. The reflexes matter. The white-hats are real. The community sometimes does the right thing.

"That's the end of Part Four," Moolah said.

"The end of the disasters?"

"The end of the disasters. There will be more, in the future. There always are. But the ones I needed to teach you, to make sense of where this industry is, you have now heard. The DAO. Black Thursday. Terra. Ronin. FTX. Curve. Six stories. Six categories of failure. Reentrancy bugs. Liquidation cascades. Algorithmic stablecoin design. Validator centralization. Custodial fraud. Tooling vulnerabilities. The whole topology of how this industry has hurt itself, in the order it learned to hurt itself, with the lessons that came from each. You have it now. You can read any future incident through the lens of one of these six, or some combination of them. The shapes repeat."

"What's left?"

"Living. We started in autumn with how this all works. We spent the winter learning to use it. We spent early spring looking at its plumbing. We spent late spring telling its ghost stories. What is left is summer, and the part of the year where you decide what you are going to do with what you know. How you are going to keep yourself safe. How regulation is going to change all of this. Where the industry is going. And, eventually, the year coming around to where it began."

She finished her latte. The wisteria petals continued, faintly, to drift.

"I have liked telling you these stories," she said, after a long pause. "More than I expected to. I have been carrying some of them for a long time."

"Thank you for telling them."

"Thank you for listening." Her voice was quiet. "Most people don't, in this industry. They do not want to know about the disasters. They want to talk about the next thing, the future, the upside. The history is unfashionable. You have listened patiently to several months of unfashionable history. I am grateful. Many of the people I have lost, over the years, to the events I have just described - would, I think, be glad that someone is still telling these stories carefully. I would like to think, at least, that they would be."

She slipped down off the bench. She paused at the path, and turned back, and the afternoon light caught her in a way that made me, for one moment, see something I had been failing to see for the entire series.

She was smaller than I had remembered.

Or, more precisely - she had been carrying something heavy for a long time, and she had just set part of it down.

"Same place next week," she said. "We start the last part. Four episodes left. I am going to teach you how to live in DeFi without getting hurt by it. The defensive arts. The boring wisdom. You will not enjoy all of it. But it will keep you out of every story I have just told you."

She turned and walked off down the path. The wisteria above her trembled slightly as she passed under it.

I closed the notebook. The pressed cherry petal from Episode Twenty-Two was still inside. I had been bringing it with me, accidentally, every week. It had not faded. It had only grown more transparent.

We walked home in the warm spring air. Dev did not say anything for a long time. When he finally spoke, all he said was: "I am glad we did Part Four."

I was, too.


End of Part IV. Next: Part V begins with Episode 27 - Security for the Rest of Us.