The problem

Models are deployed in full precision by default, needing more memory, more GPUs and more energy per answer than the task requires.

The fix you can use today

• Test 8-bit or 4-bit quantised versions against your quality bar.
• Use the quantised model where results hold up.
• Measure energy and latency per request before and after.

Tim Dettmers and colleagues showed that very large models can run on 8-bit numbers without losing quality. GreenCode's writing on green AI draws on this work.

Tools that help

• llama.cpp: runs quantised open-weight language models on ordinary CPUs and GPUs
• nvidia-smi: reports the power draw of NVIDIA GPUs from the card's own sensors
• AI Energy Score (Hugging Face): comparable energy ratings for AI models on common tasks

How GreenCode helps

GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, retrieval so a model reads only what it needs, and each request routed to the smallest component that can handle it. More on this.

Tags

#llm#model-efficiency#gpu#inference

Sources

• Green AI is a mathematics problem (LLM.int8(), mixed precision, QLoRA)
• Dettmers et al., LLM.int8(): 8-bit matrix multiplication for transformers at scale (2022)
• UNESCO, model compression such as quantisation can save up to 44% of energy
• Hugging Face, Transformers: Optimizing inference, quantization (note: may add latency when not memory-bound)

All GreenCode Tech Tips