MSR 2026 Mining Challenge, Rio de Janeiro, Brazil
LGTM! Characteristics of Auto-Merged LLM-based Agentic PRs
MSR 2026 - 23rd International Conference on Mining Software Repositories, Mining Challenge.
Coding agents open pull requests faster than people can review them, so some repositories merge them without anyone commenting. We looked at ~33,000 agentic PRs from 2,800 repositories to see which ones get through.
Poster for MSR 2026
What counts as an auto-merge
A PR merged with no reviews and no comments from anyone but its author or bots. Switch events on and off.
- AIDev dataset: ~33k agentic PRs, 2,800 repos with 100+ stars
- 5 agents: Codex, Claude Code, Copilot, Cursor, Devin
- Two repos held 37.2% of the PRs, so each repo counts once
- Repos with fewer than 5 merged PRs are left out
All or nothing
Most repositories auto-merge every agentic PR or none. Agents get auto-merged more than humans in every task type. Select a task.
Table view (Figure 1, all six bands)
Small changes get through
Auto-merged PRs are smaller, and maintainers are wary of agents deleting code. Each dot is one repository.
Mature repositories review more
Repositories that never auto-merge agentic PRs are bigger, healthier and run more checks (Table 1, medians, all p < 0.001).
Not all agents are the same
Per agent, against human PRs in the same repositories. The all-or-nothing pattern fades, and only OpenAI Codex differs significantly from humans. Select an agent.
Table view (Figure 3, all six bands)
Cite
@inbook{10.1145/3793302.3793621,
author = {Branco, Ruben and Canelas, Paulo and Gamboa, Catarina and Fonseca, Alcides},
title = {LGTM! Characteristics of Auto-Merged LLM-based Agentic PRs},
year = {2026},
isbn = {9798400724749},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3793302.3793621},
booktitle = {Proceedings of the 23rd International Conference on Mining Software Repositories},
pages = {1019--1023},
numpages = {5}
}