Knowing when an automation fails

An automation that has been running for months without anyone saying a word allows two readings: it works, or it stopped working and nobody noticed. From the outside, the two are indistinguishable, because an automated process doesn't complain: it doesn't warn you it's overwhelmed, doesn't ask when something looks off, doesn't call for help. The decision this article deals with is what minimum level of control to demand from any automated process — whether it's built by a vendor, your team or yourself — before you sign it off as finished. You don't need sophisticated monitoring infrastructure; you need to be able to answer one question: if this fails tomorrow, how do I find out, and how long does it take?

Silence is not success

When a person runs a process, failure has symptoms: someone asks a question, something gets left half-done, it shows up in the day-to-day conversation. When you automate, you also remove the witness. The work happens on its own, and silence — usually a good sign in a team — becomes ambiguous: it can mean "all fine" or "nobody is looking".

That's why many automations get signed off on the worst possible day: the day they start working. In reality, that's the day their life begins, and the question shifts from "does it work?" to "how will we know it's still working?". That second question almost never makes it into the handover, and it's what separates an operable automation from a polite time bomb.

The three ways to fail (only one is noisy)

An automated process can fail in three ways, and they are not equally visible.

It stops. It simply doesn't run: invoices don't go out, orders don't get recorded. This is the most dramatic failure and, paradoxically, the least dangerous, because sooner or later somebody misses the output.

It keeps running, but wrongly. The source file changed format, a column got renamed, and now the process puts amounts in the wrong field or assigns customers to the wrong account manager. Everything looks like it's running; the damage accumulates quietly and surfaces weeks later, once it has spread everywhere.

It works partially. It processes some records and silently drops the rest: the cases that don't fit the rule fall through the cracks. This is the slowest failure to detect, because the visible output exists — it's just incomplete.

Most of the expensive surprises come from the last two. A control setup that only detects outright stoppages is watching for the failure mode least likely to hurt you.

The minimum controls: alerts, a log, and an owner

None of this requires new infrastructure. It requires three things, and all three fit inside any project.

Alerts that reach someone

The process must raise an alert when something goes wrong — and also when nothing happens although something should. If it's supposed to process orders every morning and today it processed none, that's not a quiet day: it's an alert. Absence-based alerts are what catch silent stoppages, and they almost never come as standard. And one nuance that seems obvious until it fails: an alert that lands in a mailbox nobody opens is not an alert, it's an archive.

A log of what it did

There has to be a record of what was processed, when, and with what outcome, somewhere your own people can consult. Not for bureaucracy's sake: because the day a customer asks "where's my invoice?", the difference between answering in minutes and opening an investigation is that log. It's also what lets you catch partial failure: comparing what went in with what came out.

An owner with a name

A specific person watches the alerts and knows it's their job to watch them. Without an owner, alerts degrade into noise within weeks: everyone sees them, nobody acts on them. It doesn't have to be the person who built the automation; it has to be someone who knows what normal looks like, what doesn't, and who to call when something isn't.

The question that organises everything else

If you keep only one criterion, keep this one: for each way this can fail, who finds out first — you or your customer? If the honest answer is "the customer will call us", the controls are insufficient, because you're using your customers as your alarm system. It's the most expensive alarm system there is: every alert costs trust.

For internal processes the question has its own version: do I find out about the failure before the wrong number reaches a decision? An automated report doesn't do damage when it breaks — it does damage when someone decides based on it.

Five questions before signing off an automation

1. If it stops tonight, who finds out, through what channel, and when?
2. If it keeps running but with wrong data, what would catch it?
3. Does it also alert when nothing happens and something should?
4. Can someone in-house reconstruct what it did — and didn't do — last week?
5. Is there a named owner who watches the alerts and knows it's their job?

If any answer is "I don't know", that's the pending work, and it's worth doing before the first scare rather than after. None of this is an optional extra to add if there's budget left over: it's part of the automation, the way brakes are part of a car. When you commission one, ask for it in the handover; when you already have one running without any of it, treat this as the first improvement, not the last. Keeping an automated process alive is ongoing work — and knowing before anyone else that something broke is part of how we think automation should be run.