Cross-Model Automated Program Repair Benchmark
Evaluating 7 state-of-the-art LLMs on a new benchmark that measures how well models fix human-written bugs, LLM-generated bugs, and code containing both, revealing self-repair bias and the limits of static APR benchmarks contaminated by training data.
About This Project
As LLMs become standard tools in software development, evaluating their ability to automatically fix bugs has gotten a lot more complicated. Existing benchmarks like QuixBugs and SWE-Bench suffer from data leakage, which means models might perform well just by memorizing training data rather than actually learning how to do program repair. This study introduces a new cross-model benchmark that evaluates 7 leading LLMs, including GPT-5.3 Codex, Claude Opus 4.6, Gemini 3.1 Pro, and others, on their ability to repair bugs across three categories: human-written bugs from established benchmarks, deceptive bugs generated by each LLM, and hybrid scenarios where LLM-generated faults are layered on top of already-buggy human code. Each model attempts to fix every other model's bugs, giving us insight into self-repair bias and how well models hold up against each other. Repair success is measured using the pass@k estimator across multiple values of k. Preliminary results on QuixBugs show near-perfect performance from all models, which highlights the need for more challenging, dynamically generated benchmarks that better reflect real-world program repair.