The problem
Python loops over pandas rows (iterrows, apply with Python functions) run interpreted code for every row, when vectorised operations do the same job in optimised native code.
The fix you can use today
- Use vectorised column operations instead of iterrows().
- Use built-in methods (groupby, merge, str accessors) before custom apply().
- Profile the notebook step that takes longest first.
Tools that help
- GreenCode Constitution skill (GreenCode researchers): open-source energy principles and a catalogue of 37 energy anti-patterns that an AI coding agent can follow to find and fix waste like this
- CodeCarbon: a Python package that estimates the energy and CO2 of a run from hardware power and regional grid intensity
How GreenCode helps
GreenCode helps tackle this by building a load test from the system's own endpoints that records energy as well as speed and ranks the hotspots by energy, so you know which steps to fix first. More on this.
Tags
#python#loops#performance
Sources
All GreenCode Tech Tips