LLM4Code 2025, co-located with ICSE 2025, Ottawa, Canada
Are Large Language Models Memorizing Bug Benchmarks?
LLM4Code 2025 - International Workshop on Large Language Models for Code. *Equal contribution.
Bug benchmarks like Defects4J have been public for years, so models may have seen the answers. We tested 9 open models on 5 bug benchmarks and on new code from 2024. Defects4J shows the strongest signs of memorization.
of Defects4J 5-grams that codegen-multi reproduces exactly
of Defects4J repositories are in TheStack, a common pretraining dataset
more familiar Defects4J is to codegen-multi than new Java code
A model that finishes the patch
We gave codegen-multi the start of the fixed file for Defects4J Lang bug 39 (Figure 1). Press the button to see what it wrote next.
- Prompt
- Generated by codegen-multi, identical to the Defects4J fix
- The patched lines
Three signals of leakage
Training data is rarely public, so no single test can prove a model saw a benchmark. We combine three signals. Pick one.
Is the benchmark's repository in TheStack, a pretraining dataset that models like StarCoder 2 report using?
Table view (Table III)
Negative log-likelihood measures how surprised a model is by each next token. Code it has seen many times scores close to zero.
int main
Next token?
Pick a next token (the example from Figure 2).
NLL depends on the tokenizer, so we only compare it within a model family.
5-gram accuracy: give the model the file up to a point and check whether its next 5 tokens match the real ones. 1.0 means it reproduces the file verbatim.
def __init__(self, *, qop: str = None,def __init__(\nself, \n*, \nqop: str = None \n, No match on __init__(self,, even though the code is the same. Only the line breaks differ (BugsInPy, fastapi bug 12).
That is why a low 5-gram score alone does not rule out memorization, and why we use all three signals.
Defects4J stands out
For all 9 models, Defects4J has the highest 5-gram accuracy (tied in 2). Darker means more familiar to that model. Select a cell.
Table view (Tables V and VI)
Newer models find Defects4J less special
How many times more surprising each dataset is than Defects4J, from the NLL ratios in Figure 4. Bars near 1× mean no special familiarity. Pick a model.
But not nothing
Bigger models still reproduce benchmark code. Both LLaMa 3.1 models got the 30 lines before the fastapi bug 12 patch from BugsInPy (Figure 6).
- Prompt
- Generated, not in the fix
- In the fix, not generated
What we recommend
- Pair Defects4J with recent benchmarks such as GitBug-Java.
- Re-check membership, NLL and 5-gram accuracy as pretraining data grows.
- Read the signals together: no single one gives the full picture.
Cite
@inproceedings{ramos2025memorizing,
title = {Are Large Language Models Memorizing Bug Benchmarks?},
author = {Ramos, Daniel and Mamede, Claudia and Jain, Kush and
Canelas, Paulo and Gamboa, Catarina and Le Goues, Claire},
booktitle = {LLM4Code 2025, co-located with ICSE 2025},
year = {2025},
eprint = {2411.13323},
archivePrefix = {arXiv},
primaryClass = {cs.SE},
url = {https://arxiv.org/abs/2411.13323}
}