ReviewBench: An open benchmark for AI code review
We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post ReviewBench: An open benchmark for AI code review appeared first on The GitHub Blog.
ReviewBench is an open benchmark for AI code review agents, designed to evaluate and compare the performance of code review systems using a comprehensive and realistic dataset of GitHub pull requests.
What happened
GitHub has launched ReviewBench, an open benchmark for AI code review agents. The benchmark is built on a large dataset of GitHub pull requests and uses a multi-source golden set to evaluate the performance of code review systems. It includes metrics that measure both known and newly discovered issues, and it allows for configurable evaluation based on different review preferences.
Why it matters
ReviewBench provides a reliable and comprehensive way to evaluate AI code review agents, helping developers and teams choose the most suitable systems for their workflows. It enables objective comparisons and supports the continuous improvement of code review tools by offering an offline signal that anticipates production performance.
- ReviewBench offers a realistic and comprehensive benchmark for evaluating AI code review agents.
- The use of augmented metrics allows for more accurate assessment of code review systems, including newly discovered issues.
- Configurable evaluation options enable tailored comparisons based on specific review priorities and preferences.
The bigger picture
ReviewBench enhances the evaluation of AI code review systems by providing a standardized, reliable, and comprehensive benchmark. It helps teams make informed decisions about code review tools, improves the development of AI reviewers, and supports the integration of AI into software development workflows.
ReviewBench is a significant advancement in AI code review evaluation, providing a reliable, comprehensive, and configurable benchmark that helps developers and teams improve their code review processes and choose the most effective AI tools.
Technical details
- Multi-source golden set: Combines findings from human reviewers, frontier LLMs, and static analysis tools.
- Augmented evaluation metrics: Includes grounded and augmented precision, recall, and F1 scores to measure both known and newly discovered issues.
- Configurable evaluation: Allows slicing results by severity, category, and precision-recall preferences using the Fβ score.
- Auditable and reproducible: Features a published rubric, human-labeled dev set, and calibrated grader for transparency and consistency.
FAQ
What is ReviewBench?
ReviewBench is an open benchmark for AI code review agents, designed to evaluate and compare the performance of code review systems using a comprehensive dataset of GitHub pull requests.
How is ReviewBench constructed?
ReviewBench is built on 103.9 million GitHub pull requests, using a multi-source golden set that includes human reviewers, frontier LLMs, and static analysis tools. It includes metrics that measure both known and newly discovered issues in code reviews.
What are the key features of ReviewBench?
Key features of ReviewBench include a realistic dataset, augmented evaluation metrics, configurable evaluation based on severity and precision-recall preferences, and an auditable and reproducible framework with high agreement between human labels and the benchmark.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by GitHub AI Engineering Blog on Oct 5, 2026.
Comments
Sign in to join the discussion.