An idle GPU waiting for one request at a time wastes power.
Serving a model one request at a time leaves the accelerator underused while it still draws substantial power. Energy per answer depends heavily on utilisation.
GreenCode researchers found batching among the easiest ways to save energy when serving an AI model.
GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, each request sent to the smallest part that can do it, and heavy jobs held for cleaner power. More on this.
An AI chip waiting for one request at a time wastes power. Many teams serve a model one request at a time. The chip sits half empty. But it still draws a lot of power. So the energy per answer depends on how busy you keep it. Here is the fix you can use today. First, batch requests that arrive together. Use a serving engine built for that, such as vLLM. Second, queue work that nobody is waiting for, and run it in batches. Third, cap the length of answers to what the task needs. Fewer words generated means less work for the chip. Check the gain, using tools like nvidia-smi and CodeCarbon. GreenCode researchers found batching among the easiest energy savings for serving AI. GreenCode helps tackle this by keeping its own AI lean. Requests go to the smallest part that works, and heavy jobs wait for cleaner power. That was GreenCode Tech Tip number fifty-nine. Follow for the next tip, and find more on greencode.ai/tech-tips.
Share this
