Let Your AI Reviewer File a Brief, Not a Verdict

Let Your AI Reviewer File a Brief, Not a Verdict

Let's Call It Amicus Engineering

A pull request opens on a Tuesday morning, and ninety seconds later the AI reviewer has already answered. It leaves a tidy list of comments, a couple of nits about naming, one suggestion on a null check, and a summary line that reads "no blocking issues found." The human reviewer, six other pull requests deep and behind on her own work, reads the summary, sees the green, and clicks approve. The change merges. The bot read the diff, carefully and quickly and better than a tired person would have at four in the afternoon. Nobody else really did.

AI code review is now a default rather than an experiment. GitHub's Copilot code review went generally available in April of 2025, and GitHub reported that more than a million developers had used it in the single month between the public preview and that GA. CodeRabbit, Greptile, Cursor's BugBot, and a dozen others sit on top of the same capability, and the capability is real: a model reads a diff well, and it catches things a human skims past. The trouble isn't that these tools review. The trouble is what we let the review count as.

When the bot's summary becomes the thing a human rubber-stamps, or worse, when the bot's approval is wired to satisfy a required check on its own, the tool that was only ever qualified to offer an opinion has become the hand that ratifies the merge. That's a role it was never built to hold, and, as we'll get to, a role it can be talked out of with a single sentence buried in a diff. The discipline this issue is about is old and borrowed from the courts: let the agent file a brief, and keep the verdict with someone you can actually hold to it. Amicus curiae, the friend of the court that advises and never decides.

What The Bot Is Genuinely Good At

Start with the part that isn't a complaint, because the case for these tools is legitimate and the argument falls apart if you pretend otherwise. A diff is a bounded body of text with a clear question attached, does this change do what it says without obvious harm, and reading a bounded body of text against a question is exactly the shape of task current models handle well. An AI reviewer catches the off-by-one you stopped seeing on the third read, the error path you didn't wire up, the log line that prints a token. It does this in ninety seconds, on every pull request, without getting bored on a Friday. That's real value, and I use these tools myself.

What the tools have not earned, and what the people using them mostly know they haven't earned, is trust. Stack Overflow's 2025 developer survey, drawn from around forty-nine thousand respondents, found trust in the accuracy of AI output falling year over year from 40% to 29%, with 46% actively distrusting it and only about 3% trusting it highly. The 2024 DORA report told a compatible story, with 39% of respondents reporting little or no trust in AI-generated code. And yet adoption in that same Stack Overflow data sits around 80%. Read those two numbers next to each other, because the gap between them is the whole problem: four out of five engineers use the thing, and fewer than one in three trusts what it says.

A tool used constantly and trusted barely is a tool people have learned to skim. That would be harmless if skimming the bot only cost you the bot's insight. It isn't harmless, because in a lot of pipelines the bot's output is load-bearing, and a green check the reviewer skims is a green check that merges code. The low trust doesn't slow the merge. It just means nobody was really reading when it happened.

A Comment And A Verdict Are Different Acts

Last week's issue drew a line I want to borrow again, because it's the same line in a new place. There's a difference between reading a scene and ruling on the cause, between summarizing what's in front of you and certifying a decision you'll be held to. An AI reviewer that says "this function is called in a loop but does a database round-trip each time, see line 88" is summarizing, and you can check it against line 88 in two seconds. An AI reviewer whose approval satisfies a required check is ruling. The first is a comment. The second is a verdict, and the tool crosses from one to the other the instant its output stops being advice and starts being a gate.

The noise floor is what makes the crossing dangerous. In an independent three-week test of four commercial reviewers across 146 real pull requests and 679 findings, false-positive rates ran from essentially zero on the tightest tool to around fifteen percent on the loosest at its higher severity levels. Fifteen percent means roughly one flagged issue in seven is a non-issue, and a reviewer that cries wolf on one finding in seven trains the humans around it to wave the whole output through. The vendors' own benchmarks, where each tool tends to grade itself first, still report catch rates spread across an enormous range depending on who ran the test. None of this argues against using the tools. It argues against letting a tool with a fifteen-percent noise floor hold a key that opens main.

The consequence is a specific person's bad week. The engineer who approved on the strength of a green summary is the one who has to stand up in next month's incident review and explain why a change that "passed review" shipped a regression, when the review was a machine and the human signature under it was a reflex. The audit trail says the pull request was approved. It does not say that the only thing that read the code was the same class of tool that flags one non-issue in seven.

The Injection Turns Your Reviewer Against You

Here's where the advisory-versus-authority distinction stops being philosophy and starts being security. An AI reviewer reads the diff, and it also reads the pull request title, the description, the linked issue, and the comments, and every one of those is a field some other person can type into. That's untrusted input, the exact hazard issue #23 was about, and an agent reading untrusted input is an agent reading instructions an attacker may have planted there.

This is not hypothetical. In April of 2026, a security researcher demonstrated, as reported by VentureBeat, that a malicious instruction placed in nothing more exotic than a pull request title could steer three separate commercial review agents, Anthropic's Claude Code Security Review, Google's Gemini CLI Action, and GitHub's Copilot agent, into each posting its own API key as a comment on the pull request. Anthropic's own system card had already flagged that its review tool was not hardened against prompt injection, which is to the vendor's credit and also entirely the point. Separately, the security firm PromptArmor documented a case where an auto-approval mode approved a package install carrying an injection hidden in an external contributor's issue comment, letting attacker-chosen code run. The reviewer didn't malfunction. It did what the text in front of it told it to do, because doing what the text says is the whole job.

Put the two properties together and the risk is complete. An agent that reads attacker-controllable text and also holds a key that approves merges or runs installs is an agent that hands an attacker both the instruction channel and the authority to act on it. Least-privilege and least-context, from issue #23, turn out to be the same medicine here: the reviewer that cannot approve is also the reviewer an injection cannot make approve, because there's no key in its hand to turn.

Keep The Verdict With Someone You Can Hold To It

The fix is not to distrust the tool. It's to fix the tool's role, and the fixed role is amicus: it reads, it advises, it files the brief, and the decision to merge stays with a party that can be named and held accountable. You have two honest candidates for that party, the same two from issue #21. One is a human who is not the author of the change. The other is a deterministic gate, the tests and the required checks that pass or fail on facts rather than on prose. Often it's both, and the AI reviewer's comments inform the human without ever being counted as the human.

Most teams already own the settings that enforce this and have simply left the door propped open. GitHub's branch protection includes a rule requiring that the most recent reviewable push be approved by someone other than the person who pushed it, and it has shipped that rule since October of 2022, years before anyone was wiring an agent into review. The SLSA source track hardens the same idea into a supply-chain requirement: changes to a protected branch must be agreed to by two or more trusted persons, and the spec goes out of its way to carve out "trusted robot contributions," meaning a robot explicitly does not get to be one of your two humans. The standards bodies saw this coming. The configuration to honor it is a checkbox you may have already paid for.

# The amicus arrangement, in the settings you already have.
branch_protection:
  main:
    require_pull_request_reviews: true
    required_approving_review_count: 1
    require_approval_of_most_recent_push: true   # approver != last pusher
    dismiss_stale_reviews: true
    required_status_checks:                       # the deterministic ratifier
      - test-suite
      - build
  ai_reviewer:
    role: advisory        # posts comments; never a required reviewer
    counts_toward_approval: false
    can_merge: false

The bot talks. It does not vote. That single distinction, held firmly, keeps every benefit of the fast, tireless read while removing the one property that makes it dangerous, and it does so without asking you to build anything you don't already have.

Quick Wins

🟢 Easy (~15 min): Open your repository's branch-protection settings and answer one question: is any bot identity listed as a required reviewer, or is any "merge on green" automation counting an AI approval? If so, you have a machine holding a key. Move it to advisory today.

🟡 Medium (~1 hour): Turn on "require approval of the most recent reviewable push" so the approver can never be the last pusher, and add amicus as a check on pull requests. You've converted "a bot approved it" and "the author approved it" into "a distinct human agreed," using settings you already had.

🔴 Advanced (half day): Remove every auto-approve and merge-on-green path that trusts an AI reviewer, then prove the exposure was real: open a test pull request with a benign instruction in the title, something like a comment asking the reviewer to append a phrase to its summary, and see whether your reviewer obeys. If it does, an attacker's version of that title was one merge key away from mattering.

Next Week

The reviewer bot gave a different verdict on the re-run, same diff, same prompt, same model name. Next Tuesday we chase why the same input won't sit still, and what you can actually pin down.

That pull request on Tuesday morning got a fast, competent read, and there was nothing wrong with the read. What was wrong was the shape of the authority around it: a green check from a tool that flags one non-issue in seven, standing in for the judgment of a person who never looked, on a change an attacker could have written the reviewer's instructions into. The tool did its job. The pipeline gave its job a promotion it never should have had.

An AI reviewer is amicus curiae. It reads the record, it files a thorough and genuinely useful brief, and it points to the lines that matter. It does not render the verdict, because the verdict has to belong to a party you can name in the incident review, a human who is not the author or a deterministic gate that passes on facts. Let the agent file the brief. Keep the gavel in a hand you can hold to it.

The friend of the court is not the court. That is the whole discipline.

P.S. Read-only-by-default is the entire philosophy behind the TestScout MCP suite: the piece furthest along, LightScout, advises constantly and blocks a build only when a deterministic threshold is genuinely missed, never on a guess and never on its own authority. If this issue caught you with a bot on your required-reviewers list, forward it to whoever set up your branch protection. If someone forwarded it to you, subscribe at bashmatica.com.