How I review AI-written pull requests
On this page
I used to review AI-written code the traditional way. An agent opened a pull request, and I read the code. There were too many pull requests across all my projects for me to keep up with reading the code.
Now each pull request gets a risk grade, the risky ones get one fresh reviewer, and each review only counts for the commit it read. When I checked this against Anthropic's post on verification, I found a hole: a pull request could weaken the rules its own reviewer reads.
This is the first of a short series on checking AI-written code. The next one covers verification: proving a change works.
Risk decides what gets a reviewer
Every pull request gets a risk grade. A script sets it from what the pull request changes. I first described risk-gated pull requests in May, and the grades have changed since:
- Low risk merges once its tests, the pre-push checks, the end-to-end tests and CI pass. No reviewer looks at it.
- Medium risk is a big change or one that changes dependencies. It gets a reviewer and verification.
- High risk gets a reviewer and verification, and then waits for me. This covers sensitive areas: authentication, payments, migrations, CI and the guards that enforce this whole process.
A pull request labelled customer-facing, such as a UI or copy change, gets a reviewer and verification whatever its grade.
I drew the line there because I only have attention for the risky ones. The grade decides what reaches me.
The cheap checks still run on every change. A pre-push hook runs formatting, lint, the type check and the unit tests the change affects. CI then runs the full set.
One fresh reviewer
In June I wrote about running several models side by side on a review. Now it is one reviewer per pull request, on a strong model, and it starts fresh.
The reviewer is a new agent with no conversation behind it. It is never the agent that built the change, and never a copy of that agent's session. I wanted to be sure the review isn't biased by the session that wrote the code.
It gets a review packet and nothing else:
- the original request and the decisions I confirmed before the build;
- the acceptance scenarios, each with where it is tested;
- the repository's written rules;
- the change between the base and the head of the branch;
- each test's output on the base and on the head.
It never sees the build chat, the builder's own review, a suggested verdict or what earlier reviews concluded. It has read-only tools, so it can't edit, commit or push.
Two parts of the report do most of the work:
- Spec checks the change against what I asked for. Is anything missing, done wrong, or added that nobody asked for?
- Standards checks the change against the repository's written rules, then a few design checks, such as two ways of doing one task.
Every bug has to come with the path that triggers it: the caller, the input and the line. A guess like "this could be nil" doesn't count. Each finding is marked blocking or non-blocking, and the report is posted on the pull request word for word.
Checking findings before fixing them
The lead agent, the one running the change, checks each finding before anything is fixed. It dismisses findings of these kinds, each with a one-line reason:
- inputs no caller can produce;
- preferences with no real problem behind them;
- old issues the change doesn't touch;
- patterns that match the rest of the repository.
A review with only nitpicks is a pass.
A confirmed blocker gets fixed. For a behaviour bug, that means a failing test first.
Later rounds only check the fix
After a fix, the reviewer checks again with a fresh context. It gets the list of blockers, but not the earlier report or its reasoning. From the second round, it only looks at the fix and whether it closed those blockers. A new finding on code that hasn't changed only blocks if it's a correctness or security bug.
I made that change because rounds never ended. A full re-review kept finding new nitpicks each time, so pull requests never finished.
A review only counts for the commit it read
Each review is pinned to two commits: the head of the branch and the merge base it was compared with. A push, a rebase, a change of target branch or a merge into the branch makes the review stale. The merge only goes through at the exact commit that was reviewed.
I added this after a stale review got merged. A pull request was reviewed, then changed, and the change nobody had reviewed was merged.
What I still do
With the reviewer in place, I still:
- answer the clarifying questions, and confirm the restated request and the design before anything is built;
- try customer-facing changes myself before they ship;
- read the reviewer's reports, not the diffs;
- approve the merge of every high-risk pull request.
The hole: a pull request could weaken its own rules
Addy Osmani at Anthropic wrote up how the Claude Code team thinks about verification, and I checked my setup against it. Most of it matched: one reviewer over several rounds, a review that only counts for the commit it read, and cheap checks before CI.
One gap was a real bug. The reviewer reads the repository's rules from the pull request's own branch. So a pull request that weakens a rule is reviewed against the weakened rule. I'd never considered it. It was clearly a hole once I read it. Their reviewer reads its rules from the base branch, never from the pull request. Mine needs to do the same, and that fix is still open.
A few smaller ideas I've noted for later:
- a checklist of past incidents, where the reviewer only gets the items the diff triggers;
- marking findings in unchanged code as pre-existing;
- measuring how often pull requests that were reviewed get reverted, compared with ones that skipped review;
- letting a new merge gate warn for a while before it starts blocking.
Turning missed bugs into new rules is its own subject. I'll write about that separately.