Serving a model one request at a time leaves the accelerator underused while it still draws substantial power. Energy per answer depends heavily on utilisation.
The fix you can use today
Batch concurrent requests with a serving engine built for it.
Queue non-interactive inference and process it in batches.
Cap generated tokens to what the task needs.
GreenCode researchers found batching among the easiest ways to save energy when serving an AI model.
Tools that help
vLLM: an LLM serving engine that batches concurrent requests to keep the accelerator busy
nvidia-smi: reports the power draw of NVIDIA GPUs from the card's own sensors
CodeCarbon: a Python package that estimates the energy and CO2 of a run from hardware power and regional grid intensity
How GreenCode helps
GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, each request sent to the smallest part that can do it, and heavy jobs held for cleaner power. More on this.
