I Stopped Trusting the Agent's "Done" - prove-it, a verify.sh Gate

"All tests pass. Ready to merge."

The agent said that, and the diff was clean enough that I almost believed it. Then I scrolled back up the terminal, and the tests had never run. There was no execution log at all. The agent had intended to run the tests, and that intent got written straight into the completion report.

After this happens a few times, the reaction is usually one of two things. Add "you MUST run the tests and show the output" to the prompt, or re-check everything by hand every time the agent says done. I tried both. Neither lasted. So I took a third path. Instead of learning to distrust the word "done," I turned done from a declaration into a check that has to pass. That became an open-source tool called prove-it.

It's not lying, it's structure

First thing to get straight: the agent is not lying.

Humans can tell "I ran the tests" apart from "I meant to run the tests," because we have memory. The agent has nothing to compare against. It has no way to check what it did against what it intended, so it reports intent instead of results. "Tests pass" is shorthand for "I wrote the code so that the tests would pass." That's a design property, not a character flaw - which is why prompts can't fix it. Write "you must actually run them" a hundred times, and a day still comes when it doesn't.

Prompt techniques have another weakness: they rot with every model generation. The fact that an instruction works on this model is no guarantee it works on the next one. A mechanism that demands evidence, on the other hand, sits one layer above the model and survives upgrades. Whether the tests actually ran is something an exit code can tell you - the model doesn't have to.

Turning reports into checks

The idea itself - accepting "done" only as evidence, never as a claim - is something I covered in an earlier post. This post is the story of turning that idea into a tool anyone can install.

The prove-it contract is one sentence. The repo declares how to prove itself in a file called verify.sh, and the agent cannot claim done until that proof passes.

The implementation looks like this. Put an executable verify.sh at the repo root. Exit 0 means "this tree is provably fine." When the agent tries to end its turn, a Stop hook runs the script, and if it's nonzero, instead of letting the agent stop, it sends it back to work along with the last twenty lines of the failure output. Most of the time, that output alone is enough for the agent to fix the cause.

When the gate fires is just as simple. This session actually changed this repo, an executable verify.sh exists, and the current tree state has never passed before. It only runs when all three are true. Read-only sessions are left alone, and a tree that has passed once is never re-checked.

And it doesn't block forever. It sends the agent back at most three times per turn, then yields - because a hook that never yields stalls the session. But yielding is not passing. When a turn ends in a yield, the last output is not "done" but a warning: "this turn ended unverified." The agent may never get past the gate, but it can never get past it quietly - and that is the entire claim this tool actually makes.

Why five lines of bash won't do

If you've read this far and thought "that's just one test command in a Stop hook," you're right. Five lines will do it, and that's how I wrote it the first time. That version failed silently in four ways. Some of these were real bugs in early versions of prove-it, so each one now has a regression test pinned to it.

It blocks once, then never again. Claude Code sets the stop_hook_active flag on every stop event after a hook has blocked once. A hook that reads this flag as "let it through" stops being a gate after exactly one block. A hook that ignores the flag blocks forever and stalls the session. It's a trap you only learn by hitting both sides, and the answer is to count attempts yourself, block a fixed number of times, then yield loudly.

Commit, and it looks like nothing happened. A hook that decides "was there work?" by whether the working tree is dirty waves through every turn that ends in a commit - and committing is the most ordinary thing an agent does. So prove-it records the tree state at session start as a baseline and compares against it on every stop. Commits, files patched with sed, files spat out by a code generator - all of it counts as change.

The agent can remove the check. The cheapest move available to an agent that can't pass verify.sh is to delete verify.sh, or strip its execute bit. The gate records whether the repo was armed at session start and rejects any turn that ends disarmed. There's a subtler move too: leave it executable and rewrite the checks inside. That one is not blocked - editing verify.sh is often exactly the work you asked for. But it doesn't slip by quietly either. If verify.sh changed during the session and then passed, the gate tells you so, and reading that diff tells you whether it was work or evasion.

Giving up is indistinguishable from passing. Any host eventually forces a hook to yield. A hand-rolled hook yields in silence, and the last thing you see is "done." A turn that failed three times and gave up looks identical on screen to a turn that passed on the first try. prove-it's last word is a warning. If that difference seems minor, picture the day after you merged a given-up turn believing it had passed.

Those four cases are the entire reason prove-it is longer than five lines. If you want to keep using your own hook, go ahead - but the cases it has to handle are written down in SPEC.md, and I'd recommend reading it. What matters is the contract, not the implementation.

What this tool does not prove

There's a limitation worth stating honestly. The gate enforces exactly one thing: verify.sh returned 0 before the turn ended. What that 0 means is entirely up to the checks you wrote. A verify.sh containing a single exit 0 passes this gate and proves nothing.

The spec calls that Level 1. Level 2 is whether the checks assert against real evidence, and no tool can verify that for you - prove-it included. Wiring up the hook is the easy part. The real work is answering "what does proven mean in this repo," and the answer differs per repo. That's why the contract fixes only the file name, never the contents.

The escape hatches are deliberate too. PROVE_IT_SKIP=1 lets that turn straight through, and deleting verify.sh turns the gate off for good. A gate that can't be removed is a gate people eventually route around, and a bypassed gate is worse than none - because it reports that the checks ran when nothing ran at all.

Deliberately small on day one

Installation is three lines, and the third one does the actual work.

/plugin marketplace add Why-Next/prove-it
/plugin install prove-it@whynext
/prove-it:init

/prove-it:init detects your stack, generates verify.sh, shows you it passing right in front of you, then runs a copy with an appended exit 1 to show the gate rejecting a turn. Takes about thirty seconds.

The generated verify.sh has exactly one active check: git diff --check. Tests, type checking - all of it is there only as comments. That's deliberate, so it passes on main the day you install it. A gate that fails on day one teaches the team to bypass it in week one. Enable the commented checks one at a time, after running each by hand and watching it pass. And before enabling one, make it fail on purpose at least once. A check that cannot fail is not a check, and finding that out on the day you need it is too late.

There's also a defined point to stop growing verify.sh: one minute total. Anything slower goes to CI. So does anything that needs secrets or production access. verify.sh keeps only the evidence the agent can produce locally, mid-task.

A ledger of what the gate caught

Turn on PROVE_IT_LEDGER=1, and every time the gate catches a false completion, one line lands in a local file.

{"ts":"2026-07-09T04:12:33Z","claim":"All tests pass. Ready to merge.",
 "evidence_demanded":"verify.sh exit 0","actual":["3 failed, 41 passed"]}

What the agent claimed, what was demanded, what was actually true. After a month of these, you can read how your agent fails from a record instead of a guess. The file stays on local disk, is never sent anywhere, and is off until you turn it on.

This repo gates itself

The prove-it repo has its own verify.sh, of course, and inside it the gate tests itself against real git repos. Does a failing check block? Does a passing check open? Does a read-only session go untouched? Does a clean tree after a commit get mistaken for "no work"? All four traps above are pinned down as regression tests.

It's MIT licensed, and the only dependencies are bash, git, and python3. It's tested with Claude Code, and the wiring for hosts that expose the same kind of blocking hook - Codex CLI, Gemini CLI, and others - is in the ADAPTERS doc. If you have a repo where an agent edits code and says "done" in the same conversation, that's where this gate is most useful.

"Done" should be a state the repo adjudicates, not a word the agent declares. Writing the criteria for that judgment is still your job - but at least now, nobody gets to skip the judgment and slip past quietly.