Here’s the blunt version of what I actually watched happen: when the logic you’re teaching an AI contradicts a pattern it saw a billion times during training, the training wins. Not once — every single time, in the same conversation, minutes after it agreed to stop.
A system where every rule has to earn its confidence
I hit this building a scoring system for snapWONDERS’ forensic analysis, where every rule has to carry its own confidence — not “this looks suspicious,” but a specific, defensible weight, with a recorded reason that weight is what it is. Genuinely new logic, nothing like the standard patterns a model has already seen a million times.
The paradox: perfect diagnosis, zero prevention
Ask an AI assistant to write against rules like that and it makes a mistake, gets corrected, agrees, and quietly makes the same mistake again — three or four times across one long session, in my case. Ask it instead to check its own last output against the rules, and it finds the mistake immediately, and can even tell you it’s happened before. Same model, same rules, same session. Diagnosis: perfect. Prevention: not happening.
Why “check yourself first” doesn’t hold
The obvious fix is a better prompt — “check yourself before you answer.” It helps a little. It doesn’t hold, because it’s still the model policing itself inside the exact context that keeps pulling it back toward the wrong pattern. A correction that lives only in the conversation is competing with a pattern the model learned from billions of examples, in the same place, using the same weights. That’s not a fair fight, and no amount of politely-worded instruction changes which side is favoured.
What actually held: the check moves outside the conversation
Take the check out of the conversation entirely. Instead of asking the model to grade itself, there’s now something outside the chat — a mechanical check that runs no matter how confident the model sounded getting there — and it blocks the work going any further if it doesn’t match what was actually specified. It doesn’t care how sure of itself the model was five minutes earlier in the same thread.
The harder discipline: evidence, not conviction
The part that made it trustworthy rather than just tidy: nobody — including me — gets to decide a number “feels right” and move on. Any change has to be justified by something that actually happened, not a hunch about what should happen. That standard applies to the AI’s output and to my own judgement calls equally, which is the uncomfortable part. It’s a lot easier to demand rigour from a model than to hold your own threshold changes to the same bar.
Net effect: the AI is still going to make that mistake sometimes — that hasn’t changed. What changed is that “wrong” is now something a machine catches before it ships, and “right” is something I can point to evidence for, not just assert.
If you’re building anything rule-heavy with AI assistance — pricing logic, compliance, fraud scoring — and you’ve hit the same “it can diagnose it but can’t stop doing it” pattern, the fix probably isn’t a better prompt. It’s moving the check somewhere the model doesn’t get a vote.
Kenneth Springer is the founder of snapWONDERS, a digital forensic analysis platform for images and video. The confidence-scoring discipline described here underpins snapWONDERS’ forensic checks. snapWONDERS forensic analysis — no account required.