A production AI code reviewer in 3 weeks

§ 01

The problem: review is where delivery slows down

In an engineering organization shipping production software, code review is one of the few places where every change has to pass through a small group of senior engineers. That makes it a natural bottleneck: reviews queue up, context-switching eats senior time, and under deadline pressure the depth of review is the first thing that quietly degrades.

The request was straightforward: build an AI reviewer that takes the first pass on every pull request — catching defects early and leaving human reviewers to spend their attention where judgment is actually required.

The real risk in this project was not whether an LLM can comment on code. It obviously can. The risk was trust:

An AI reviewer that floods pull requests with plausible-sounding but empty criticism gets muted within a week — and once developers stop reading it, no later improvement brings them back.

So the project was framed around a single quality gate from day one: every finding the system publishes must be worth a developer's time.

§ 02

The core design decision: evidence, not plausible criticism

Language models are good at producing text that sounds like review. Left unconstrained, they produce generic advice, style opinions, and hypothetical concerns — each one individually defensible, collectively noise.

The system was built around the opposite requirement. A finding is only published if it can carry evidence:

• a concrete location in the changed code, not a general observation about the file;
• an explainable failure scenario — what input or state makes this code do the wrong thing;
• enough supporting context that the developer can verify the claim in place, without re-deriving it.

Findings that could not meet that bar were dropped before anyone saw them. That trade-off was deliberate: fewer findings, each one checkable. The validation gate — not the model's own confidence — decides what reaches the pull request.

This is the same principle I used in the Construction AI system: the model proposes, a separate gate decides what the user is shown.

§ 03

Evaluation before rollout: 100 historical PRs

Before the reviewer touched a single live pull request, it was run against 100 historical PRs — changes whose real fate was already known: which of them shipped cleanly, which came back with defects, what human reviewers had actually flagged.

That gave the project something most AI-reviewer rollouts skip: a measured baseline before launch. The historical set answered the questions that determine whether the tool survives contact with developers — how often the system's findings are real, what it misses, and where the publication threshold should sit so that the signal-to-noise ratio holds up.

It also turned tuning into an engineering loop instead of a debate. A change to prompts, context assembly, or the validation gate could be replayed against the same 100 PRs and compared on the same numbers.

§ 04

Shipping in three weeks

The system went from first commit to production in three weeks. Not because corners were cut, but because the scope was cut correctly:

• one measurable process — reviewing pull requests — rather than an open-ended "AI developer platform";
• the evaluation set and the validation gate built first, before integration polish;
• the riskiest assumption — "will developers trust the findings?" — tested on historical data before rollout, not discovered in production.

By the time the reviewer was attached to the live PR flow, the open question was not whether it worked — that had been measured — but how it would hold up at volume.

§ 05

The first month in numbers

The number that mattered most was not the volume of reviews — it was the word confirmed. Findings were counted only when the developers who owned the code accepted them as real. That is the direct consequence of the evidence gate: the system's output stayed at a density developers kept reading.

§ 06

The main takeaway

An AI code reviewer is not primarily a prompting problem. It is a trust-calibration problem.

The parts that made this one production-ready were the parts around the model: an evaluation set of 100 historical PRs that made quality measurable before launch, and a validation gate that required evidence instead of plausible criticism.

The model generated candidate findings. The gate decided which of them earned a developer's attention. That separation is what let the system complete 578 reviews in its first month — and still be read.