How I verify AI-written code
On this page
Two things kept going wrong. Agents said tests had passed when they hadn't run them, or had run them on old code. And sometimes the tests really did pass, but the page was broken or ugly, which only a screenshot showed.
So I don't take an agent's word for it any more. A change that needs verifying has to show its evidence on the pull request. I wrote about this idea back in May, in how I build software with agent loops. This is how it works now.
This is the second article in a short series on checking AI-written code. The first covers how I review AI-written pull requests.
Verification runs on the same pull requests that get a reviewer (see the first article). Low-risk changes skip it and rely on their tests, the pre-push checks, the end-to-end tests and CI.
Like a review, the evidence only counts for the commit it ran on. A push to the branch makes it stale, and it has to be earned again.
Tests come first
The rule is that a build starts with a test that fails. The building agent shows the test's output on the old code, where it fails, and on the new code, where it passes.
I insist on that for two reasons. Agents used to write tests after the code, and those tests passed whatever the code did, so they could never fail. And a test that fails on the old code proves the change does something.
Some code also gets a property test: new or changed pure functions for money, pricing, parsing, serialisation and data migrations. A property test states a rule that should always hold. It might check:
- a round trip, where formatting then parsing gives back the original;
- an invariant, such as "a balance never goes negative";
- that running something twice gives the same result as running it once;
- that a fast version gives the same answer as a simple one.
Evidence is tests, plus screenshots for UI
The evidence for a change is its tests, run at one pinned commit. The pull request gets the test names, the command and the passing output.
What the tests look like depends on the change:
- API work needs integration or contract tests that call the endpoint the way a consumer would. They check the agreed shapes, status codes and error bodies.
- UI work needs component tests for each state and end-to-end tests for each journey. It also needs a screenshot of each scenario at desktop width and at about 500px wide.
The screenshots sit beside the tests, never in place of them.
Every regression found along the way becomes an automated test. The rule is never to weaken a test or the expected behaviour to get a pass. A check that was skipped, blocked or couldn't run isn't a pass either.
A cheap model drives, the lead judges
Running test suites and clicking through an app is mechanical work. I didn't want to pay a strong model for it, so a cheap model does it.
The cheap model, the verifier, gets a brief. It has:
- the commit to test;
- the test commands and how to start the app;
- each scenario, with its expected result;
- the views to screenshot.
It checks it is on the right commit first. Then it runs everything, walks each UI scenario in a real browser, and saves the screenshots, browser console, network requests and server log. It describes what it saw. It doesn't diagnose or fix anything, and it doesn't mark anything as passed.
The lead agent, the one running the change, judges. It reads each output far enough to confirm the right command ran. It compares the screenshots with the design I agreed. Any error, failed request or error-level line in the server log that the scenario doesn't expect fails it, whatever the page shows. A gap or an unclear screenshot isn't a pass.
Anything that fails goes back for repair. Then the change is reviewed and verified again, at the new commit.
That isn't the last check for customer-facing changes. I try UI and copy changes myself before they ship.
Flaky tests are bugs
A flaky test passes sometimes and fails sometimes on the same code. I treat it as a bug. It is never rerun until it passes, and the reviewer blocks any change that adds a test runner retry or raises one.
When one shows up, the agent:
- opens an issue with the test, the commit and both outputs;
- quarantines the test, moving it out of the main suite, so a red run there means something again;
- if the test covers one of the pull request's own scenarios, leaves that scenario unverified until the test is fixed;
- fixes the cause, such as shared state, timing or a real clock, as a separate change.
Two things led to this. Agents kept rerunning tests, or adding retries, until they went green, which hid real bugs. And once the end-to-end tests flaked, a red run stopped telling me anything.
What I'm missing
The Claude Code team's post on how they think about verification matched most of what I do, including treating flaky tests as defects. It also says running the app end to end matters more than tests. It suggests keeping the steps to start and try the app in the repository, close to the code they cover.
I agree, and it's a gap in my setup. My verifier is told how to start the app in each brief, and that gets worked out by hand every time. For UI work it does drive the app in a browser. For an API, a background worker or a command-line change, the tests run, but nothing starts the app and tries the change for real. Each project needs its own written steps for starting the app and trying a change, so verification can rely on them every time. I've noted that for later.
The same post runs a new commit gate in an observe mode first. It logs what it would have blocked, so false alarms show up before it blocks anyone. My new merge checks have gone straight to blocking. I've noted that one for later as well.