A Pipeline Where a Second AI Tries to Disprove the First One's Fix

Anyone who has had an AI agent do code review knows this. It's pretty good at other people's code. But have it review the code it just wrote and it turns strangely generous. You get "the implementation matches the intent" back, and when you actually run the thing, the bug is still there.

Which makes sense. Within the same model and the same context, whatever misconception it had while writing the code, it has again while reviewing it. It's a structure where it re-approves its own conclusions, so it's less a review than an echo.

So I moved the judging outside. I hand the code Claude fixed to Codex, and I ask for exactly one thing: "Assume this fix is wrong, and try to refute it."

Three rounds in a row, each found a deeper bug

The first time I really felt the value of this loop was a logout bug. The symptom was simple: pressing the logout button sometimes didn't log you out. Claude found a cause and fixed it, and reading the code, it looked plausible. I almost merged it.

I had Codex attempt a refutation, and round one came back with this: the fix itself is correct, but a redirect on the auth page intercepts the flow before that code path is ever reached, so the fix never gets a chance to run. The fixed code was dead code.

I fixed the redirect too and asked for another refutation. Round two found something else. This time it was an in-progress flag that had been added to prevent duplicate execution. At certain timings, that guard turns the logout request itself into a no-op. Only in round three did I get "I couldn't find anything to refute," and only then did I merge.

All three rounds involved code that had passed self-review. And in all three rounds, the refuter caught the problem because it wasn't asking "does the code look good" but "is there a scenario where this claim collapses."

The pattern kept repeating after that. On a server fix that had been through self-review twice, cross-checking still caught one missing value in a DB constraint, and a migration got added. On a retry-logic fix, the refuter sent back a blocking verdict over an edge case that misclassified a particular timeout response as a permanent failure. Every one of them was the kind of thing that was plausible enough to almost slip through.

Ask for refutation, not review

The most important part of the prompt design is framing. Say "review this" and the model gives you a safe answer, half praise and half minor nitpicks. Say "assume this fix is wrong and refute it; pass it only if you fail to refute it" and the posture changes. Passing stops being the default and becomes the consequence of a failed refutation.

On top of that, three more conditions.

1. Enforce an evidence format. Every finding must carry a file, a line number, and a reproduction scenario where the defect actually fires. "This part looks risky" is not accepted. If it can't construct a scenario, that suspicion isn't a finding, it's dismissed. In practice the refuter fairly often disproves its own suspicion and drops it, and that's the key mechanism filtering out false positives.
2. Get the verdict as a grade. Pass / pass after minor fixes / block - three levels are enough. With grades, "there are findings but it's mergeable" and "do not merge" don't get mixed together.
3. Run it in an independent context. The refuter gets only the diff and the relevant code. It does not get the conversation from the work session, or the narrative of "why it was fixed this way." The moment you share the narrative, the refuter catches the same misconceptions.

One operational tip as well. Run the refuter read-only. Give it permission to change code and it will try to fix things itself instead of pointing them out, and then you're back to the question of who verifies that fix. Keep the role pinned to judging, and send fixes back to the original worker.

When to run it

I don't run this loop on every change. Calling in a refuter for a typo fix is a waste. What always gets the loop is changes that are hard to undo. DB migrations, data deletion, payment and auth paths all live here. Changes that carry a "fixed it" claim are also targets. A bug fix needs a matched pair, the bug reproduced before and gone after, and the refuter is good at finding the gaps in that pair. The last category is changes I'm not going to read in full. Having a human read every line of a large AI-generated diff realistically breaks down. If I'm not going to read it, the least I can do is have a different model read it adversarially.

The cost question has to be addressed, and here it is: one round of refutation costs a few model calls. Compared with the cost of a human chasing down one bug that shipped to production, it wasn't worth agonizing over. If that logout bug above had made it to a release, I would have had to dig through both layers myself, the redirect and the flag guard, armed with nothing but reports that "sometimes logout doesn't work."

Why it has to be a different model

A fresh session of the same model gets you part of the way. Just separating the context makes the narrative-contamination problem go away. But after months of running this, my conclusion is that a genuinely different model clearly catches more.

Different models attend to different things. One is sensitive to timing problems in state management while the other is sensitive to contract violations and boundary values, that sort of thing. Two sessions of the same model share the same blind spots, but different models have blind spots that don't line up. The value of cross-checking comes precisely from that misalignment.

This structure should feel familiar. It's the same reason human teams assign code review to someone other than the author. Working with AI doesn't make that principle disappear; it just became cheap enough to apply all the time. A judge who isn't the author, eyes not steeped in the narrative, and an approval that only arrives once refutation has failed. What human organizations did only occasionally because it was expensive can now stand guard in front of every merge.