AI code review: what GitHub's ReviewBench measures

AI code review agents now comment on pull requests in a lot of teams before any human opens them. The uncomfortable question is how anyone knows whether those comments are good. On 5 October 2026, GitHub published ReviewBench, an open benchmark built to answer exactly that, using real pull requests rather than toy examples.

If you run an AI code review agent in production, this matters more than another leaderboard. It is the first public, reproducible way to compare two review configurations before your developers see them, and to check that a model swap or a prompt tweak actually helps rather than just producing more comments.

Background

A code review agent reads a pull request diff and leaves comments: a likely bug, a security hole, a missing test, code that will be painful to maintain. The trap is that an agent can look helpful while costing the team time. If it flags twenty issues and fifteen are wrong, developers quickly learn to skim past it, including the five comments that mattered.

Two familiar measures capture that tradeoff. Precision asks: of the issues the reviewer raises, what share are valid? Recall asks: of the real issues in the change, what share does the reviewer find? A cautious agent scores well on precision and misses bugs. A chatty one scores well on recall and buries the team in noise.

GitHub argues that existing benchmarks trade off label quality, coverage and how faithfully they represent real-world review. ReviewBench is an attempt to get all three in a single methodology.

A dataset built from real pull requests

ReviewBench contains 219 pull requests from 187 public open source repositories across 19 languages. To pick the sample, the team analysed the distribution of 103.9 million GitHub pull requests, so that the benchmark looks like real work rather than a hand-picked set of easy cases.

Weighted towards changes worth reviewing

Most real pull requests are tiny: one line, one file, a typo fix. A benchmark that copied that distribution faithfully would mostly test trivial cases. GitHub instead weighted pull request size towards what it calls the reviewable middle and tail, cutting down single-file micro-changes and keeping more substantive multi-file work. That is exactly where AI code review can add value, and exactly where it gets things wrong most often.

Ground truth from several sources

The most interesting part is how the "right answers" were built. The golden set, the list of known issues for each pull request, merges four sources:

• human reviewers;
• frontier LLMs;
• static analysis tools;
• issues inferred from the author's follow-up commits.

Findings are deduplicated semantically, then validated by an LLM grader (Claude Sonnet 5) applying the same rubric to every submission. Each finding carries a severity (Critical, Medium, Low) and a category: correctness, security, reliability, maintainability, testing.

Before release, senior engineers independently re-labelled every ground-truth finding. Their true-positive and false-positive calls agreed with ReviewBench 96.6% of the time. That number is what gives the automated grader credibility. Without that check, an LLM-graded benchmark is only as good as the LLM doing the grading.

Six metrics for AI code review

ReviewBench reports six metrics in two families.

Grounded metrics. Grounded precision, recall and F1 use only the golden-set labels. This is the strict, like-for-like comparison: of the issues we already know about, how many did the agent find, and what share of its findings matched a known issue? These are the numbers to use when comparing two agents head to head.

Augmented metrics. Augmented precision, recall and F1 also evaluate findings that match nothing in the golden set. An agent that surfaces a genuine issue no source had spotted gets credit instead of a penalty. That design choice matters: without it, a benchmark rewards agents that imitate existing reviewers rather than agents that actually find bugs.

Reading both families side by side is useful in practice. A large gap between grounded and augmented scores points either to an agent that finds new things or to one whose invented issues the grader wrongly accepted. Either way, it is worth a manual look.

From benchmark to production

The Copilot experiment

A benchmark is only worth something if it predicts what happens in front of real developers. GitHub tested that on its own product. The team introduced a multi-model ensemble review, which combines several independent model runs into a single review instead of relying on one run. The change was scored on ReviewBench first, then A/B tested in production.

The online results:

• addressed rate (the online counterpart to precision): +8.0%;
• recall: +13.6%;
• comment volume: +61%;
• cost per review: −8.0%;
• critical comments: 227% increase predicted, 262% observed.

The addressed rate deserves a definition. It is the percentage of Copilot code review comments that an LLM judges to have prompted the developer to make a corresponding code change. In other words, it is not a satisfaction score; it measures whether the comment changed the code.

GitHub is careful about what this proves: online experiments remain the ultimate measure of user impact. ReviewBench is a pre-production signal, not a replacement for an A/B test.

Benchmarking your own AI code review agent

The benchmark is not limited to Copilot. review-bench.ai offers a self-serve path:

1. sign in with GitHub;
2. register your agent with its container image, configuration and model key;
3. try it on a 25-pull-request test set;
4. run the full 219-pull-request benchmark, in three rounds;
5. submit the result for the leaderboard, after a maintainer review.

The evaluation methodology and the grader prompts are public, so you can see why a finding was accepted or rejected. GitHub also publishes its agreement measurements and the known threats to the benchmark's validity, so readers can judge where the uncertainty sits.

What this means for AI teams

For a DevOps or MLOps team that already runs an AI code review agent, ReviewBench mostly changes how that agent gets managed.

Measure before you swap models. Switching models, rewriting the system prompt or adding a static analyser to the chain were often gut-feel decisions. With a reproducible benchmark you can run both variants on the same 219 pull requests and compare precision, recall and cost before production ever sees the change.

Track precision, not just volume. The most telling number in GitHub's experiment is the 61% jump in comment volume. On a naive dashboard that looks like a win on its own. What makes it a real improvement is that precision and recall rose with it while cost fell. A team that only counts comments risks rewarding a chatty agent.

Add an addressed rate to your observability. GitHub's online metric is easy to copy: for each agent comment, did the developer change the code it pointed at? It plugs into pull request events and is far more honest than a satisfaction survey.

Calibrate your own grader. The 96.6% agreement between senior engineers and the automated grader is a reminder of a useful rule: any LLM-as-judge needs to be checked against humans on a sample. Teams that grade their agents with an LLM and skip this step may mostly be measuring their grader's biases.

Put cost on the same scorecard. An ensemble that combines several model runs sounds like it should cost more, yet GitHub reports cost per review down 8.0% alongside the quality gains. Whatever the reason in their setup, the lesson for your own pipeline is to record cost per review next to precision and recall on every benchmark run. A variant that wins on quality but doubles the inference bill is a different decision from one that wins on both, and you want that tradeoff visible before rollout, not on next month's invoice.

Keep the A/B test. There is a gap between the predicted rise in critical comments (227%) and the observed one (262%). The direction was right; the size was not. Use the benchmark to kill weak variants quickly, and make the final call on production data.

Key takeaways

• ReviewBench is an open AI code review benchmark released by GitHub on 5 October 2026: 219 pull requests, 187 repositories, 19 languages.
• Ground truth merges humans, LLMs, static analysis and follow-up commits, with 96.6% agreement between senior engineers and the automated grader.
• Six metrics: precision, recall and F1, in grounded (known issues) and augmented (rewards real discoveries) versions.
• On Copilot, the benchmark correctly predicted the direction of production gains: addressed rate +8.0%, recall +13.6%, cost −8.0%.
• For teams running their own AI code review agent, it is a chance to measure before swapping models and to track precision rather than volume.

Industrialising AI agents? SeedVision offers 3-5 day AI audits and 15-30 day production rollouts. See the packages or book a 30-min call.

Cover photo: Photo by Markus Spiske on Unsplash.