So-so automation is the real risk

Everybody fears the automation that breaks. Nobody fears the automation that almost works — and that's backwards.

The tool that fails loudly is safe. It throws an error, somebody swears, and a human does the work the old way. Annoying, cheap, self-correcting. The tool that's right most of the time is the one that costs you, because "right most of the time" is exactly the accuracy level at which people stop checking.

Think about what a 95% correct automation does to a team over a quarter. For the first two weeks, everyone verifies everything and catches the misses. By week four, checking feels like busywork — it was right the last forty times. By week eight, nobody remembers whose job checking was. Now the 5% flows straight through with the confidence of the 95%: the invoice billed twice that got paid twice, the quote carrying a stale surcharge, the "complete" status on work still sitting half-done. Each error arrives pre-trusted.

I watched this firsthand in transportation and logistics, and it applies to every industry. A confidently wrong answer about a warranty, a maintenance interval, or what a repair should cost doesn't look like an error — it looks like an answer. A new hire can't tell the difference. That's why "the AI said so" is the most expensive sentence in a modern operation.

The fix isn't better AI — no model gets to 100 and stays there. The fix is structural: every automation ships with a stop condition, a defined reviewer, and a rule about which direction trust flows. The rule I hold to is simple: the system flags, the human confirms, and it never runs the other way round. Speed without review is just faster mistakes.

And run the new thing in shadow first — watching, logging, touching nothing — until its misses are boring and its catches are proven. Autonomy is earned in stages, not granted at install.

Don't ask "how accurate is it?" Ask "what happens on the day it's wrong?" If the answer is "nothing, a human catches it," ship it. If the answer is a shrug, you've built a liability with a dashboard.