---
title: "SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization"
description: "Making software faster or leaner is hard, manual work, and although large language models are increasingly used to refine code, existing benchmarks do not…"
url: https://greencode.ai/publications/swe-opt-a-benchmark-for-evaluating-llms-on-performance-oriented-repository-level/
---

# SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization

- **Authors:** [Ezgi Sarıkayak](https://orcid.org/0009-0005-9373-3071 "ORCID profile of Ezgi Sarıkayak") , Wenchao Gu, [Hesham Ghonim](https://orcid.org/0000-0001-9625-642X "ORCID profile of Hesham Ghonim") , Chunyang Chen
- **Contributing partner:** [Siemens](https://greencode.ai/partners/siemens-ag/ "Siemens AG in GreenCode")
- **Published in:** arXiv preprint arXiv:2606.25530 (2026)
- **Published:** Jun 2026
- **Type:** Paper
- **DOI:** [10.48550/arXiv.2606.25530](https://doi.org/10.48550/arXiv.2606.25530)

### Synopsis

Making software faster or leaner is hard, manual work, and although large language models are increasingly used to refine code, existing benchmarks do not reflect how optimisation happens in real codebases. They tend to test isolated functions or a single metric, ignoring the trade-off between execution time and memory, measurement noise, and the effect of different inputs and conditions. This paper introduces SWE-Opt, a repository-level benchmark built from 102 expert-written optimisations in open-source projects. Each task comes with parameterised tests that measure runtime, peak memory and time-weighted memory usage across varied inputs, under noise-aware conditions. Current models struggle: their runtime gains are sparse and inconsistent, and memory improvements are almost absent. Letting an agent explore the repository step by step makes its patches correct far more often, but not faster. Experts, by contrast, achieved an aggregate speedup of 15.5 times and a 171.3-fold reduction in peak memory, improving runtime in 93.1% of tasks and peak memory in 70.6%. The gap between models and expert engineering remains wide.

[View at the publisher](https://doi.org/10.48550/arXiv.2606.25530) [All publications](https://greencode.ai/publications/)Cite this work

Sarıkayak, E., Gu, W., Ghonim, H., &amp; Chen, C. (2026). <i>SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization</i> (Version 2). arXiv. <https://doi.org/10.48550/ARXIV.2606.25530>
