Making software faster or leaner is hard, manual work, and although large language models are increasingly used to refine code, existing benchmarks do not reflect how optimisation happens in real codebases. They tend to test isolated functions or a single metric, ignoring the trade-off between execution time and memory, measurement noise, and the effect of different inputs and conditions. This paper introduces SWE-Opt, a repository-level benchmark built from 102 expert-written optimisations in open-source projects. Each task comes with parameterised tests that measure runtime, peak memory and time-weighted memory usage across varied inputs, under noise-aware conditions. Current models struggle: their runtime gains are sparse and inconsistent, and memory improvements are almost absent. Letting an agent explore the repository step by step makes its patches correct far more often, but not faster. Experts, by contrast, achieved an aggregate speedup of 15.5 times and a 171.3-fold reduction in peak memory, improving runtime in 93.1% of tasks and peak memory in 70.6%. The gap between models and expert engineering remains wide.
Sarıkayak, E., Gu, W., Ghonim, H., & Chen, C. (2026). <i>SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization</i> (Version 2). arXiv. https://doi.org/10.48550/ARXIV.2606.25530
Sarıkayak, E., Gu, W., Ghonim, H., & Chen, C. (2026). <i>SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization</i> (Version 2). arXiv. https://doi.org/10.48550/ARXIV.2606.25530Sarıkayak, E. <i>et al.</i> (2026) ‘SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization’. arXiv. Available at: https://doi.org/10.48550/ARXIV.2606.25530.E. Sarıkayak, W. Gu, H. Ghonim, and C. Chen, ‘SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization’, 2026, <i>arXiv</i>. doi: 10.48550/ARXIV.2606.25530.Sarıkayak, Ezgi, Wenchao Gu, Hesham Ghonim, and Chunyang Chen. 2026. ‘SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization’. Version 2. Preprint, ArXiv. https://doi.org/10.48550/ARXIV.2606.25530.Sarıkayak E, Gu W, Ghonim H, Chen C. SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization [Internet]. arXiv; 2026. Available from: https://arxiv.org/abs/2606.25530@misc{https://doi.org/10.48550/arxiv.2606.25530, doi = {10.48550/ARXIV.2606.25530}, url = {https://arxiv.org/abs/2606.25530}, author = {Sarıkayak, Ezgi and Gu, Wenchao and Ghonim, Hesham and Chen, Chunyang}, keywords = {Software Engineering (cs.SE), Artificial Intelligence (cs.AI), Computation and Language (cs.CL), FOS: Computer and information sciences}, title = {SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization}, publisher = {arXiv}, year = {2026}, copyright = {Creative Commons Attribution 4.0 International} }