---
title: "New paper: SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization"
description: "Siemens publishes \"SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization\" in arXiv preprint arXiv:2606.25530 (2026)."
url: https://greencode.ai/news/paper-swe-opt-a-benchmark-for-evaluating-llms-on-performance-orien/
---

# New paper: SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization

- 01 June, 2026
- GreenCode

### New paper: SWE-Opt: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization

Making software faster or leaner is hard, manual work, and although large language models are increasingly used to refine code, existing benchmarks do not reflect how optimisation happens in real codebases. They tend to test isolated functions or a single metric, ignoring the trade-off between execution time and memory, measurement noise, and the effect of different inputs and conditions. This paper introduces SWE-Opt, a repository-level benchmark built from 102 expert-written optimisations in open-source projects. Each task comes with parameterised tests that measure runtime, peak memory and time-weighted memory usage across varied inputs, under noise-aware conditions. Current models struggle: their runtime gains are sparse and inconsistent, and memory improvements are almost absent. Letting an agent explore the repository step by step makes its patches correct far more often, but not faster. Experts, by contrast, achieved an aggregate speedup of 15.5 times and a 171.3-fold reduction in peak memory, improving runtime in 93.1% of tasks and peak memory in 70.6%. The gap between models and expert engineering remains wide.

**Authors:** [Ezgi Sarıkayak](https://greencode.ai/researchers/ezgi-sarikayak/), [Wenchao Gu](https://greencode.ai/researchers/wenchao-gu/), [Hesham Ghonim](https://greencode.ai/researchers/hesham-ghonim/), [Chunyang Chen](https://greencode.ai/researchers/chunyang-chen/)

**Contributing partner:** [Siemens](https://greencode.ai/partners/siemens-ag/)

[Publication page and citation](https://greencode.ai/publications/swe-opt-a-benchmark-for-evaluating-llms-on-performance-oriented-repository-level/) [View at the publisher](https://doi.org/10.48550/arXiv.2606.25530)

#### Categories

- All news **(43)**
- Publications **(25)**
- Project news **(14)**
- Articles **(4)**

#### Recent Post

- #### [greenai.guide: plain-English help for using AI with less energy](https://greencode.ai/news/greenai-guide/)

  [Read more](https://greencode.ai/news/greenai-guide/)
- #### [New paper: IP-Preserving Enterprise System Modernization via LLM-Driven Specification Extraction and Multi-target Code Generation](https://greencode.ai/news/paper-ip-preserving-enterprise-system-modernization-via-llm-driven/)

  [Read more](https://greencode.ai/news/paper-ip-preserving-enterprise-system-modernization-via-llm-driven/)
- #### [New paper: (How) Do Large Language Models Understand High-Level Message Sequence Charts?](https://greencode.ai/news/paper-how-do-large-language-models-understand-high-level-message-s/)

  [Read more](https://greencode.ai/news/paper-how-do-large-language-models-understand-high-level-message-s/)
- #### [GreenCode is now on YouTube and TikTok](https://greencode.ai/news/greencode-on-youtube-and-tiktok/)

  [Read more](https://greencode.ai/news/greencode-on-youtube-and-tiktok/)
