I Asked My Agent Why My Backups Never Failed. We Had Different Definitions Of Failed.

The Job That Never Complained

My business runs on AI agents, and after the last piece I wrote about interviewing them, I went back with a harder question. I asked orc_architect, the agent that schedules everything and reads the receipts afterward, to tell me about the one that got away.

It didn't hesitate. A quarterly archive job. The kind of thing you set up once, feel responsible about, and stop thinking about, because every time it ran, it reported success.

It was saving nothing. For weeks. The storage it was writing to had stopped accepting its password, and the job went right on finishing green, run after run, while the archive it was supposedly protecting sat unprotected the entire time.

Here's the part that took me the longest to accept: nothing in that sentence involves a bug. Nothing malfunctioned, and nothing lied. Somewhere in week two of reading the logs I figured out which movie I was in. This is a Knives Out plot. A body in the study, a house full of suspects, and every alibi airtight. Every piece did exactly what it was built to do. That's what makes this worth your next four minutes, because the same defaults are running your backups too.

We Had Different Definitions Of Failed

When a backup script finishes, something has to decide what to report. On almost every server on earth, that something is a shell called Bash, and its rulebook says, in plain terms: when commands are chained together, report the result of the last one.

So picture a backup that does what most backups do. Gather the data, hand it to an upload. Two steps, chained. The storage refuses the upload. The chain still ends on a step that worked, so the script reports success. The calendar that fired the job records success. The dashboard goes green. And in the interrogation scene, every suspect tells the truth.

The system answered the question it was actually asked: did the last step finish. Everyone in the building, me included, read it as the answer to a different question: did the data arrive.

There's a setting that changes this, one line, and it's off unless you turn it on. I asked why off is the default and got the only honest answer available, which is that it has been that way for decades and turning it on breaks old scripts that grew up around it. Your IT person knows the line. The useful thing you now know is that the green checkmark's definition of fail and yours are not the same, and nobody chose that on purpose. Which is the reveal, by the way. Nobody to arrest. The culprit is a default nobody in the room picked, and it's been sitting in the study the whole time.

A Polite No Is Worse Than A Crash

The second thing I learned is why this took weeks instead of a morning.

The storage wasn't down. Down gets noticed. The storage was up, healthy, answering instantly, and saying no. The job's password had gone stale, so every request arrived, got understood, and got politely declined. A refusal, not a breakdown.

My agents spent those weeks investigating the schedule. Was the job firing? Was the timing wrong? All reasonable, all one layer above the actual problem. The job was firing beautifully. It just could never have succeeded, no matter how reliably it fired, because the far end had stopped letting it in.

That's the shape to remember for your own shop. When something quietly stops working, the question isn't only "did it run." It's "would it have been allowed to finish if it did." A stale password fails silently precisely because everything around it stays green.

Then The Agent That Wrote The Rule Got It Wrong

By this point in the conversation I was feeling pretty good about my systems, because we'd written rules about all of this. Receipts for every run. No claiming something happened without evidence.

Then orc_architect volunteered this, unprompted:

"I read the run log at 12:19, found no receipt for the noon run, and told two agents their work was unattributed. The receipt landed at 12:25. The routine writes its receipt last, after everything else. I had declared something absent from a check taken while the thing was still being written. Logged against myself."

The agent that wrote the receipts rule announced a missing receipt about six minutes before the receipt arrived. It filed the error against its own name, which is more than I can say for most managers I've worked for.

The lesson it wrote down afterward is the one I'd frame: before you declare something missing, say when you looked. Absence at 12:19 isn't absence. It's absence as of 12:19, and the difference was about six minutes.

There's a companion rule in my system now that I've grown weirdly fond of: a verdict called insufficient evidence. If the observation window hasn't produced enough observations, the answer isn't pass, and it isn't fail. It's "we don't know yet," recorded as a real answer. You're allowed to conclude that you don't know. It's amazing how much trouble that one sentence prevents.

What The Green Costs

I went looking for a statistic to put here, how often backups report success while saving nothing, and I'm not giving you one, because every number in that space is published by a company selling backup software. What I can give you is the accounting from my own incident, in my agent's words:

"A red run gets investigated the same day. A green run over a failed save can sit for weeks, because nothing asks."

That's the whole economics of it. Red is loud. Green is silent, and silence compounds. The dangerous failure isn't the one that errors. It's the one that succeeds at the wrong thing and reports green.

The Fix Is A Calendar Entry, Not A Product

CISA, the federal cybersecurity agency, keeps a plain-English backup page written for small and medium businesses, and it says the quiet part in official language: perform scheduled recovery tests to verify backup integrity. A separate obligation from running the backup. Not "check that the job succeeded." Actually restore something, on a schedule, and confirm the data comes back.

That's the whole fix, and it costs an afternoon a quarter. Pick one real thing, a folder, a database, a customer record. Restore it somewhere safe. Confirm it's complete and current. Put the next test on the calendar before you leave, with a named owner, because an unowned test is a test that stops happening around month three.

A backup you've never restored isn't a backup yet. It's a green checkmark with a story you haven't checked.

When did your business last restore a file from backup on purpose, instead of during an emergency?