Money Is Not Eventually Correct: What We Learned Simulating Our Own Ledger

We wrote a fairly pleased-with-ourselves post a while back about our wallet system - how we went from a race condition that printed money to a plpgsql function that debits a balance in under a millisecond. We stand by every word. The procedure is correct. We can prove the procedure is correct.

Then one evening somebody asked the obvious follow-up and the room went quiet: is the wallet correct?

Not the same question. A stored procedure is one call. A wallet is a thing that has been called forty thousand times, in an order nobody planned, interleaved with cron sweeps and vendor timeouts and a refund that arrived twice because a marketplace retried a webhook. Every individual call can be provably right and the balance can still be wrong, because the bug was never in a call. It was in a sequence.

We had tested the primitive. We had not tested the history.

"Eventually" Is Not a Thing Money Does

There's a class of system where being briefly wrong is fine. Your follower count, your recommendations, the number next to a notification bell. Converge in a few seconds and nobody is harmed. None of that applies here. When a client loads £100,000 into their wallet, there is no window in which that number is allowed to be £103,000 while the system "settles".

What makes it dangerous is that the failure is silent. A double credit doesn't panic. It returns 200 OK, the customer is happy, the dashboard is green, and the wallet is quietly wrong until somebody reconciles at month end and finds three thousand pounds that came from nowhere.

Bugs that throw exceptions are a gift. Bugs that get the arithmetic wrong and then say "thank you, have a nice day" are the ones that keep us up.

We already had a money-leak suite covering every scenario we'd thought of, plus the chaos and fuzz layers hammering the vendor boundary. The limitation is right there in the sentence: every scenario we'd thought of. A refund is correct. A refund after another refund of the same order is where money appears from nowhere, and nobody writes that test, because nobody thinks "what if it happens twice" until it has already happened twice.

So we stopped asserting scenarios and started asserting a property:

final balance − opening balance  ==  Σ CREDIT − Σ DEBIT

If those two sides ever disagree, money was created or destroyed without a row to account for it. That's true of every sequence that has ever run or ever will. Generate long random sequences of real money movements, run them for real, check that equation after every single operation.

A Seed Is the Whole Test Case

The harness takes one int64 and turns it into everything - which operations run, their arguments, which faults fire, how far the clock moves, and every identifier written to the database.

ops := sim.Generate(84213)       // 60 operations, always the same 60
ops := sim.GenerateN(84213, 25)  // exactly the first 25 of those same 60

Generation is pure. No clock, no database, no ambient state. Which means "seed 84213 fails" is a complete, portable bug report that fits in a Slack message. Nobody has to attach a 40MB trace file.

The operations are ordinary things the system does on a Tuesday - create an order, replay a client reference, refund, tick a cron, advance the clock, drain queued webhooks, arm a fault on the vendor mock - arranged in orders no human would think to arrange.

Deciding whether the result is right is the harder half, and that component has a name: the oracle. Ours is ten rules, checked after every operation:

ledger-sum-eq-delta looks redundant next to conservation and isn't. One rebuilds the balance from the transaction table, the other from the ledger. If only one of them works, the two halves of our double entry have quietly gone their separate ways.

Determinism Is Earned, Not Declared

"Same seed, same result" sounds like something you announce. It is not. It's something you go and earn, by hunting down every impure input in the code path and either giving it a seam or removing it. We measured it the only way that means anything: run one seed twice against databases restored to identical state, hash every column of every row the run touched, compare.

We found nine sources of nondeterminism, over four rounds, each round exposing causes the previous one had been hiding underneath a louder failure.

Two of those are worth the detour.

The one that was a real bug

Round two closed the procedure's clock. orders.created_at still diverged. Round three closed that, and orders.updated_at still diverged, a third distinct cause sitting underneath the other two like a Russian doll made of timestamps. We went looking, and found this in three of the four repository functions that update an order:

Set("updated_at", "NOW()")

Squirrel binds that as a parameter, not as SQL. What we were actually sending, in production, for months:

UPDATE orders SET updated_at = $1 WHERE id = $2   -- args: ["NOW()", 1]

The literal seven-character string NOW(), shipped to PostgreSQL as an argument. It worked purely because PostgreSQL's datetime parser is lenient enough to accept the string 'NOW()' as its special value now. We tested that directly, because we refused to believe it:

cast "NOW()" -> "2026-05-29 14:51:18.503745+00"

So the SQL function was never called at any of those three sites. A fourth site, nine lines away, used squirrel.Expr("NOW()"), which genuinely is the function. The codebase carried both spellings with no visible difference between them, and Postgres cheerfully accepted both. No ordinary test could have caught it, because both spellings are right enough. Only a check demanding byte-identical state across two runs cared.

The one that hid as slowness

You may remember that we built a chaos library on purpose so our tests could simulate vendors having a bad day. It turns out it also works beautifully for giving your own test harness a bad day, which is not a use case we designed for.

Our vendor mock ships that library as a shared fixture, and it was quietly switched on for the entire determinism sweep. All 201 of its rules are probabilistic:

enabled: 201
by mode:    probabilistic: 201
by failure: status 111, malformed 39, delay_then_pass 21,
            jitter 15, drop 8, delay_then_fail 7

Every one of those rolls its dice with Math.random() inside the mock server. Not our seeded generator. Not anything our int64 can reach. A separate process, in a different language, quietly deciding whether this particular request gets to be a 500 today.

So every hash the sweep produced was contingent on none of those 201 rules firing. Eleven seeds passed before we noticed. They were right. They were also lucky, and the harness had no way of telling those apart, which is the single worst property a measuring instrument can have.

It hid so well because the individual probabilities are tiny - around 0.005 each - so an actual divergence was rare. What the 43 delay and jitter rules reliably produced was real wall-clock seconds on every single upstream call. The symptom wasn't a red test. The symptom was a sweep taking four minutes per seed instead of thirty seconds, and nobody investigates "a bit slow today" as a correctness problem. We spent a while assuming Postgres was the bottleneck, which in hindsight is a bit like blaming the engine when someone is standing on the brake.

The fix is one line at the top of a run, and its shape matters: WithQuietUpstream disables the library for the duration of the test and restores it on cleanup, because it's shared fixture state that other suites legitimately depend on. Silencing it globally would have fixed our sweep by breaking somebody else's chaos coverage, which is not a trade we're allowed to make.

That seed went from about four minutes to 38 seconds for both runs.

The general lesson is the one we keep relearning: resetting the database restores PostgreSQL and nothing else. Anything living outside it - a mock's rule set, its consumed counters, a cache, an env var somebody exported last Tuesday - has to be reset by the harness itself. That class of hidden state is exceptionally good at hiding, precisely because the reset step looks like it covers everything.

Where we landed: 50 out of 50 seeds deterministic, zero failures.

We left exactly one column unfixed. orders.time_taken is a latency measurement, and two runs recorded 419ms and 316ms. We could have closed it in one line with the clock seam we'd already built - and then every order would have recorded a latency of exactly zero, because the start and the end would be read at the same virtual instant. The hash would go green and the column would become worthless. Excluding it says "this is not reproducible". Faking it would have said "this took no time at all", which is a much worse thing for a test suite to say.

Shrinking: 120 Operations Down to Three

A run that breaks an invariant is typically 120 operations long and tells you nothing. So the harness shrinks it, repeatedly dropping operations and re-running, keeping any subsequence that still breaks the same named rule.

We tried it by injecting a one-line refund bug on purpose. It reduced 120 operations to three, in 69 runs:

[  0] CreateVoucherOrder{wallet:2 product:0 qty:3 denom:0 ref:""}
[  1] Refund{order:8}
[  2] Refund{order:8}

Create an order, refund it, refund it again, money appears from nowhere. It reads as a sentence. You don't debug that, you just look at it and know. And it doesn't claim minimality, it demonstrates it - the trace shows the search trying every remaining two-operation subset and finding all of them clean before it gives up:

shrunk to 3 op(s) preserving "refund-le-debit" in 69 run(s) [minimal]

Each of those is committed as a JSON artifact, so it replays on a clean checkout and the bug can't come back unnoticed.

The Honest Scoreboard

Here's the part where a post like this lists the terrifying bugs the shiny new harness found lurking in production.

Ours found one. It was the updated_at binding. We found it while chasing a timestamp rather than by breaking an invariant, and its entire production impact was that a column said 2026-05-29 14:51:18 when it should have said 2026-05-29 14:51:18.

Every other failure the harness produced was its own fault, or its environment's:

Weeks of work. Nine sources of nondeterminism. A template database, four clock seams, a seeded ID generator, a delta-debugging shrinker and 50 green seeds. All that effort, to fix updated_at.

But hey. Nothing will ever break again.

We're including that table rather than quietly deleting it, because a harness that reports its own bugs as system bugs is worse than no harness at all - and the run log has a classification field specifically so nobody can lose that distinction while writing it up six months from now. Also because the result is genuinely the one we wanted. We built an instrument to find money bugs, pointed it at the money, and the money was fine. The boring stored procedure is doing exactly what we claimed.

Not Eventually. Just Correct.

None of this is novel. Delta debugging is from the nineties, property-based testing has been mainstream for a decade, and deterministic simulation is what the database people have been doing to each other for years. We learned it by watching them.

What's ours is the discipline of pointing it at the one thing that cannot be wrong. Not the API surface, not the rendering, not the cache. The ledger. That number is either exactly right, always, or we don't have a business.

So we generate sequences nobody thought of, run them against a real database, check the books after every operation, and when something breaks we get three lines instead of a stack trace. Same seed, same money, every time.

And if it turns out the next thing we find is another timestamp, we'll take it. Boring outcomes are the entire point.

This is part of our engineering series. Read The Wallet That Couldn't Count for the stored procedure this harness interrogates, Breaking Our Own System on Purpose for the chaos and fuzz layers around it, or Around the World in 80 Days for the boring stack underneath all of it.