Cattle not pets

People who run servers have an old saying: treat your machines like cattle, not pets. A pet has a name, gets hand care, and is irreplaceable. A herd is made of animals that are all set up the same way, so a new one can join without anybody having to remember anything special about it.

We run a herd. Every change anyone proposes to Vibes DIY gets its own complete copy of the platform, named after the change, live within minutes and gone when the change lands. Alongside those are the copies you use every day. This week a security review turned up two places where we were still treating that herd like pets, and both are now fixed in the way the saying recommends: the safe setting comes built in.

Why we went looking

The review started with someone else's news. PHP, the language behind a large share of the web, pre-announced an emergency release this week for two critical bugs, and CERT-EU's advisory described them as possible remote code execution. Vibes DIY doesn't use PHP, but it raised the right question: where in our own system does a request carry an identity, and how sure are we that it gets checked?

The part that reads identities held up. Sign-in tokens are verified with a well-tested library and checked field by field. What we found instead was two settings.

A tag only part of the herd was wearing

When you sign in with the vibes-diy command-line tool, your computer gets a certificate that our signing authority vouches for, and every request it makes carries that certificate. Since the end of June, the tool also sends the authority's signature, so the server can check that the certificate is genuine and not something a computer made for itself.

Checking that signature was switched on by a setting. In July we turned it on for the two copies that serve real traffic, and left the test copies alone, because at the time they looked like a separate world with their own sign-in and their own data.

In September we measured that every copy, test ones included, reads the same database. Nobody went back to the setting. So on a test copy, a home-made certificate would still have passed, against the same data the real copies serve.

The fix is to stop treating the check as a setting at all. It is now on in the code, in every copy, with nothing to switch. A setting still exists, but it can only turn the check off, and only as an emergency lever. Every new copy is born checking, including the ones that don't exist yet. Our own team's sign-ins on the test copies kept working, because certificates aren't tied to a web address: every test copy trusts the same authority, so one sign-in works on all of them.

A secret whose absence meant "come in"

The second setting guards a door inside the platform. When the platform itself needs to talk to the part of a vibe that runs on our servers, to fire its scheduled tasks or pass along a request, it presents a shared secret. A vibe's own code never has that secret, so it can't knock on the same door pretending to be the platform.

When that check was first written, it let everything through if no secret was configured, so the code could ship before every copy had the secret installed. That was a reasonable way to roll it out. The rollout finished months ago, and the "let everything through" branch stayed, so a missing secret still read as an open door rather than as a mistake.

Now a missing secret means the door stays shut, and deploying a copy without the secret fails before anything ships. Before switching it, we read the deploy records for every kind of copy, rather than trusting our memory, and confirmed each one already carried the secret. Afterwards we watched a real vibe's scheduled task keep firing every fifteen minutes on the new code.

Our tests were pets too

The most useful moment of the week came from our own test suite. When the certificate check became the default, 338 of our 855 test files failed at once, nearly all with the same error: authentication required.

The tests had been creating a fresh certificate authority for each test, while the test server only ever trusted the first one it saw. The old, looser check compared authorities by name, and every test authority had the same name, so it never noticed. The tests had been passing on exactly the gap we were closing. We changed them to work the way production does: one authority, and certificates it actually signed. The secret change turned up the same pattern on a smaller scale, in tests that built the protected parts of the platform with no secret at all. Now they hold one, the way every real copy does.

A test that only passes with the safety off is a pet. It needs special care that nothing in production gives it.

The robot with the keys

One more thing turned up in the same review. Dependabot, the GitHub bot that proposes library updates, gets help from one of our automated jobs that tidies its changes, and that job holds write access to the code. It now checks who actually opened the change, not who last touched it. It sets itself up from our main branch, not from the proposed change, and it installs packages without running any of their install scripts. A helper with keys does its job and nothing else.

The rules we keep

• The safe setting is the default, in the code. A setting can loosen a protection in an emergency, and nothing else.
• What every copy needs, a deploy refuses to go out without. A deploy missing its secret fails before anything ships, not after.
• Measure before switching. Read the deploy records and run the real path before and after.
• When the map changes, revisit what depended on it. The certificate setting was right for the world as we understood it in July. The database measurement in September changed that world, and the setting should have come along.

What this means for you

You don't see any of the copies we run, and that's how it should be. The apps you make on Vibes DIY belong to you, and that holds however many copies of the platform we're running on a given afternoon. Every new copy now starts out protected, without anyone having to remember to set it up.