Tech Tip #67: Stop looping over DataFrame rows

Looping over a DataFrame row by row wastes your processor.

The problem

Python loops over pandas rows (iterrows, apply with Python functions) run interpreted code for every row, when vectorised operations do the same job in optimised native code.

The fix you can use today

  • Use vectorised column operations instead of iterrows().
  • Use built-in methods (groupby, merge, str accessors) before custom apply().
  • Profile the notebook step that takes longest first.

Tools that help

  • GreenCode Constitution skill (GreenCode researchers): open-source energy principles and a catalogue of 37 energy anti-patterns that an AI coding agent can follow to find and fix waste like this
  • CodeCarbon: a Python package that estimates the energy and CO2 of a run from hardware power and regional grid intensity

How GreenCode helps

GreenCode helps tackle this by building a load test from the system's own endpoints that records energy as well as speed and ranks the hotspots by energy, so you know which steps to fix first. More on this.

Tags

#python#loops#performance

Sources

All GreenCode Tech Tips

  • Tip #67
  • For Developers
  • Category Code efficiency
  • Format short video
GreenCode

Want this checked on your own system? Talk to the GreenCode team.

Get in touch