The scariest automation failure isn't the one that crashes. It's the one that fails silently — the Zap that hit a 429, gave up, and never told anyone. Three weeks later someone asks why the CRM has a hole in it.
Every automation I ship has to pass one test: what happens when this runs at 3am and something upstream is down?
1. Retries, with backoff and a ceiling
Most APIs fail transiently. Rate limits, timeouts, a bad gateway during someone else's deploy. A single attempt turns a two-second blip into permanent data loss.
Retry with exponential backoff — 1s, 4s, 15s, 60s — and stop at a fixed ceiling. Retrying forever is its own outage: you'll hammer a recovering service back into the ground.
And retry only what's safe to retry, which brings us to the important one.
2. Idempotency, or you'll double-charge someone
If a webhook can be retried, it will be delivered twice. Stripe does it. Meta does it. Your own retry logic does it.
Every side-effecting step needs a key that makes a repeat a no-op:
const key = `lead:${payload.id}:v1`;
if (await seen(key)) return { skipped: true };
await createCrmContact(payload);
await mark(key);
Cheap to build. Impossible to retrofit after you've sent the same invoice twice.
Idempotency is the difference between "we retried it" and "we retried it and everything was fine."
3. Dead letters — a place for the ones that didn't make it
When retries are exhausted, the record does not disappear. It goes to a dead-letter store — a table, a queue, an Airtable base, I don't care — with the payload, the error, and the timestamp.
That gives you two things you can't get any other way: you can replay the failure once the upstream is healthy, and you can count failures, which is the only honest measure of whether your system works.
4. Alerting that a human will actually see
An error log nobody reads is a diary, not an alert.
The rule I use: alert on a rate, not an event. One failure at 3am is noise. Five failures in ten minutes is a page. Zero executions in an hour on a workflow that normally runs every five minutes is a silent death — and it's the alert most people forget to build.
Every system I hand over ships with a heartbeat: if the workflow doesn't check in, someone gets a message.
The dashboard is not decoration
Last thing. Every system gets a small dashboard: executions, success rate, dead letters, last run. Not because it's pretty — because it's the thing that lets the client stop thinking about it.
Trust in automation isn't built by the automation working. It's built by the client being able to see that it worked, at a glance, on a Tuesday morning, without asking you.
That's when it's finished.