The case of the slow festival app

Every mystery novel makes you the same promise: the answer exists, the clues are in front of you, and if you look carefully enough, you'll get there.

Software makes that promise too — and unlike the novel, it never cheats. A computer does nothing without a reason. Every strange behavior has a mechanism, every mechanism leaves evidence, and the evidence sits there waiting for anyone curious enough to go look. Debugging is the one detective genre where the reader is guaranteed a solvable case.

Here's the best one we've had all summer.

The victim

Pickathon is a music festival in the woods outside Portland. Its schedule lives in a picker app built on Vibes DIY — browse the sets, favorite the ones you want, see which sets your friends are going to. We've written about it before. As the festival approached, people started actually using it, which is when the reports came in:

It takes twenty or thirty seconds to load. Sometimes it doesn't load at all.

Days before the gates opened. Cue the rain-streaked window and the slow saxophone.

Clue one: the crash that wasn't random

Start with the loudest symptom. Sometimes the app didn't load — after fifteen to thirty seconds you got a bare error page: error code: 1102. That's Cloudflare's way of saying a server program was killed for using too many resources. Not a random outage. Something specific was too heavy.

A detective's first move is elimination. The app's front door responded fine. A sibling app on the same platform, loaded the same way, worked perfectly — so the platform wasn't down; something about this app was fatal. Then the telling detail: even a request for just the page's headers — no content at all — died the same death. Whatever was exploding ran before the page was even drawn. That pins it to one early step: gathering the app's public documents to tuck into the page, so the schedule can appear instantly.

So we looked at the app's database — the schedule, plus everything festival-goers had saved: favorites, notes, friends — and found this: about 2,500 current documents, half a megabyte of actual information. But the table holding them had 396,673 rows — 145 megabytes.

The explanation is that the database keeps history. Every time a document is saved, a new revision is written — like a notebook that keeps every crossed-out draft. Across the whole database, this app had accumulated an average of 161 drafts per document — most of them piled onto a suspiciously small group of pages we'll meet again in clue three. And the code that answered "what does this data say now?" did it by reading every draft ever written and keeping the newest of each — hauling 145 MB through a small server to deliver 0.5 MB of answer. That's fine when the notebook is thin, and this notebook was thin for months. The app that finally broke it was simply the first one popular enough, and edited enough, to get thick.

One fix later (ask the database for just the newest drafts — it's good at that), the crash was gone. But a good detective doesn't stop at what happened. Two questions remained. Why was it still slow? And who wrote half a million drafts?

Clue two: the question asked 551,551 times

Still fifteen to twenty seconds to load. Time to interrogate the database directly — databases will show their work if you ask (the tool is literally called EXPLAIN). The confession was immediate: the query that found "the newest revision of each document" was phrased so that the database answered it once per draft instead of once per document. 551,551 times, for one page load.

Rephrasing the question — same answer, different grammar — took it from 8.3 seconds to 0.28 seconds. Nothing had been "broken." Every single step the database took was correct. The shape of the question was wrong, and the cost of a badly shaped question grows with the size of the notebook.

Which brings us back to the real mystery. The schedule only changes when the festival edits it. Where were half a million drafts coming from?

Clue three: the helper that went blind

The app has a background helper with a careful, sensible design: once a minute, fetch the festival's official schedule, compare it with what's already stored, and write only what changed. On a normal day it should write nothing at all.

It had been rewriting all 330 schedule entries. Every minute. For weeks.

The mechanism is the best twist in the case. The read the helper used to check its own past work has a quiet limit: it returns at most 2,000 documents, alphabetically, and when it hits the limit it simply stops — no error, no "there's more," nothing. Nobody notices a ceiling until they're over it. As festival-goers poured in and saved favorites, the database crossed 2,000 documents — and because the schedule entries happened to sort last alphabetically, they were the first thing pushed off the edge. The helper looked at the database, saw no schedule entries at all, concluded it had never written any, and wrote all 330. Sixty seconds later: same blindness, same conclusion, same 330 writes.

Here's the cruel part: nothing ever looked wrong. The schedule stayed perfectly correct the entire time — writing the same thing again is invisible. But the app's backend does one thing at a time, so about 55 seconds of every minute went to pointless rewriting, and every first-time visitor's page load waited in line behind it. And every rewrite added drafts to the notebook that clues one and two were already choking on. In a second store, the running change-log had reached 661,704 entries describing what happened to roughly 40 live documents — a diary two thousand times longer than the life it described, and heavy enough that loading it killed a server with 128 MB of working memory outright. The app was getting slower because people loved it, through a mechanism nowhere near any code you'd think to inspect.

The fixes read like the moral at the end of a fable. A helper that needs to know what it did last time should remember (it now keeps a little bookmark document, deliberately named to sort to the very front, where the ceiling can never push it off) — not re-derive it from a read that can silently lie. A limit that cuts things off should say so — a plausible-looking wrong answer is so much worse than an error. And the platform now discards identical rewrites at the one gate every write passes through, so no app can churn its own history like that again. Before: ~240 backend operations a minute. After: 2.4.

Clue four: the twist

Case closed? We reloaded the app. First visit, signed out, fresh browser… still slow.

So we ran the experiment that should have been step one: we deployed a completely empty app — one component, no data, no backend — and loaded it side by side with the festival app. The empty app took 15.2 seconds. The festival app took 16.3.

Read that again, because it inverts the whole case: the entire cost of the app's 2,500-document database, its favorites, its sync — everything we'd been heroically optimizing — was about one second. The other fifteen were the platform's own floor, paid by every app, including one with nothing in it.

And the floor had a secret: it was two different numbers. A warm load (the page cached at the edge near you) took a quarter of a second. A cold one took 13–14 seconds. We had only ever measured warm — because we'd loaded the page a dozen times while investigating. Nobody on a team ever experiences their own product cold; you've always already visited. The users reporting "twenty to thirty seconds" weren't exaggerating. They were the only ones seeing the truth.

That number lit the fuse on a whole second investigation — into bundles, caches, and a compiler being shipped to people who would never compile anything — but that's another episode.

What the case teaches

Every clue in this story was available to anyone. The crash code was printed on the error page. The database will happily tell you it has 396,673 rows if you ask. EXPLAIN is free. Deploying an empty app to compare against takes a minute. The detective work wasn't genius — it was the discipline of the genre: believe there's a mechanism, isolate one variable at a time, and let the evidence overrule your favorite theory (we were sure it was the sync engine; it was innocent).

If you want the transferable questions, take these four:

• Which of your questions get more expensive with history, rather than with state? "What's current" should cost what the current stuff costs — never what the diary costs.
• Does anything in your system return a plausible answer instead of an error when it hits a limit? That's where a mystery is being written right now.
• Does any job re-derive what it did last time instead of remembering? Memory is cheap; re-derivation goes blind.
• Have you measured cold, or only warm? Build the empty version of the thing. It tells you what fraction of the problem is even yours.

The festival app was built by describing it in plain language — and most of this investigation happened the same way, questions asked in plain English by people and AI agents working together, answers read off the actual system. That's the part that should excite you: the mystery genre of software is open to anyone now. The case is always solvable. You're allowed to solve it.