The problem

On a shared cluster, jobs fail part-way through on a bad input, a memory limit or an unhandled edge case, then get fixed and resubmitted. A study of a Dutch national cluster found about half its energy went on jobs that ended unsuccessfully.

The fix you can use today

• Run on a small input first.
• Set realistic memory and time limits, and validate inputs before the long run.
• Checkpoint long jobs so a failure doesn't restart from zero.

Researchers in Amsterdam and Vienna found that half of one cluster's energy went on jobs that failed. GreenCode is exploring checks like this for jobs on shared clusters.

Tools that help

• Slurm energy accounting: records the energy each job consumed, so users can look it up after the run
• Green Algorithms calculator: estimates the carbon footprint of a computation from its runtime, hardware and location
• CodeCarbon: a Python package that estimates the energy and CO2 of a run from hardware power and regional grid intensity

How GreenCode helps

GreenCode helps tackle this by establishing a repeatable energy baseline for research software and measuring again after each optimisation. A university research computing cluster is one of the project's use cases. More on this.

Tags

#hpc#research-software#testing

Sources

• Research software, shared supercomputers and the energy of a failed job (Chu et al., ICPADS 2024)
• Software Sustainability Institute, Green Data Practices: provide sample datasets for development and testing before full-scale runs

All GreenCode Tech Tips