Last month I watched a support team demo their new AI deflection wall with pride. A customer asked where their invoice was, and the bot answered — confidently, politely, instantly — with a ticket number that didn’t exist. Nobody caught it for three days. The bot wasn’t lying. It was doing exactly what probabilistic models do: producing the most likely next tokens. The team had just mistaken fluency for truth, and their customers paid for the confusion.

I’ve seen the same mistake in my own terminal. You’ll pull a generated script that passes CI, then watch it break in production because the logic missed an edge case the model never saw. That gap between plausible output and reliable behavior is where the real work lives — and it’s work only a human can do.

The deflection wall: fluency is not truth

A customer service bot that answers instantly feels like progress. Your ticket queue shrinks, your response-time dashboard turns green, and management declares victory. But a language model doesn’t check your CRM — it predicts what a helpful answer sounds like. When it doesn’t know the invoice status, it doesn’t say “I don’t know.” It invents a ticket number with total confidence.

That’s not a bug you patch with a better prompt. It’s the nature of the tool. A model has no notion of your system state, your security policy, or the particular quirks of your hardware. It might suggest a command that works in a container but assumes a different PATH on your bare-metal host. It’ll introduce off-by-one loops, missing error handling, or wrong assumptions about return values — and every one of them will look perfectly reasonable until it runs.

Faked evidence is the same failure with higher stakes

Now push that failure one step further. A generated photo, a cloned voice note, a “screenshot” of a conversation that never happened — probabilistic output presented as ground truth. I’ve watched people forward AI-generated images as proof of things that never occurred, not out of malice but because the output looked authoritative.

This is why I don’t trust model output as evidence of anything. A model can’t swear an affidavit. It can’t be fired for getting it wrong. When faked evidence enters an incident report, a support thread, or a compliance audit, the only thing standing between you and a bad decision is a human who checks the primary source — the logs, the database, the actual system state.

What the model can’t give you: accountability and empathy

Here’s the part the tooling discourse skips. Even if the model were always right, two things would still be missing.

First, accountability. When a generated migration script drops a table at 3 AM, “the model suggested it” isn’t an explanation your boss accepts. You ran it. You own it. The model can’t take the pager, can’t write the postmortem, can’t look a customer in the eye. Responsibility doesn’t transfer to a probability distribution — it stays with the human who pressed enter.

Second, empathy. A support agent reading “we’re looking into it” for the fifth time isn’t facing a technical problem — they’re facing a trust problem. A human can hear frustration, adjust tone, and say “I understand this is blocking your payroll run, here’s exactly what happens next.” A deflection wall says the words without the understanding, and customers can tell. Every time.

Where probabilistic output genuinely helps

None of that means the tool is useless — it means the tool has a lane. For boilerplate, repetitive scaffolding, or translating a clear comment into a function signature, the model saves real typing. A well-prompted model produces a decent first draft of a cron wrapper, a systemd unit file, or a simple parsing loop. I’ve used AI for IT troubleshooting the same way: fast first pass, human verdict at the end.

In those cases your effort shifts from writing to reviewing. That’s still valuable work — arguably the more skilled half. Don’t let anyone tell you reviewing is “just” checking.

Ground truth means running it on the real system

Ground truth comes from observing the actual outcome on the target system, not from trusting a confidence score. You verify by running the code, checking logs, and confirming the intended state change really happened. That step can’t be outsourced to the model because it requires touching the real environment, which the model never sees.

My rule is simple: nothing generated runs on production without a staging run first. A sandbox where you can watch destructive commands fail safely builds the same reflex — you learn what “verify first” feels like before the stakes are real. And when output needs explaining rather than executing, I reach for plain-English explanations of commands and errors instead of trusting the first plausible-sounding answer.

Here’s the smallest version of the workflow — a wrapper I actually use before letting generated scripts near anything I care about:

#!/bin/bash
# verify-first.sh — run a generated script against staging, diff the
# result against the known-good state, and refuse to proceed on mismatch.
# Why a wrapper: it makes "did it change what I expected?" a habit
# instead of a judgment call at 3 AM.
set -euo pipefail
STAGING_HOST="${1:?usage: verify-first.sh <staging-host> <script>}"
SCRIPT="${2:?usage: verify-first.sh <staging-host> <script>}"
BEFORE=$(ssh "$STAGING_HOST" 'systemctl list-units --state=failed --no-legend | wc -l')
scp "$SCRIPT" "$STAGING_HOST:/tmp/candidate.sh"
ssh "$STAGING_HOST" 'bash -n /tmp/candidate.sh && bash /tmp/candidate.sh --dry-run'
AFTER=$(ssh "$STAGING_HOST" 'systemctl list-units --state=failed --no-legend | wc -l')
[ "$BEFORE" = "$AFTER" ] || { echo "❌ staging state changed: investigate before prod"; exit 1; }
echo "✅ staging clean — review the dry-run output, then decide like a human"

A short checklist for reviewing generated code

  • Run it in staging that mirrors production — never trust the model’s word for it
  • Check error paths: missing files, bad input, non-zero exit codes
  • Compare output against a known-good reference or a manual calculation
  • Look for hard-coded values that should be parameters (paths, ports, credentials)
  • Verify privileged operations sit inside the right sudo or capability boundaries

Keep the human in the loop

The fix isn’t less AI — it’s AI in the right seat. A high-bandwidth copilot that drafts, suggests, and explains, with a human who verifies, decides, and answers for the outcome. The model gives you speed. You supply the truth.

Next time a generated answer looks perfect, ask yourself one question: what would prove it wrong? Then go check.