Tech Tip #32: Quantise before you buy more GPUs

8-bit weights ran 175-billion-parameter models without losing quality.

Video created by explainapaper.com

The problem

Models are deployed in full precision by default, needing more memory, more GPUs and more energy per answer than the task requires.

The fix you can use today

  • Test 8-bit or 4-bit quantised versions against your quality bar.
  • Use the quantised model where results hold up.
  • Measure energy and latency per request before and after.

Tim Dettmers and colleagues showed that very large models can run on 8-bit numbers without losing quality. GreenCode's writing on green AI draws on this work.

Tools that help

  • llama.cpp: runs quantised open-weight language models on ordinary CPUs and GPUs
  • nvidia-smi: reports the power draw of NVIDIA GPUs from the card's own sensors
  • AI Energy Score (Hugging Face): comparable energy ratings for AI models on common tasks

How GreenCode helps

GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, retrieval so a model reads only what it needs, and each request routed to the smallest component that can handle it. More on this.

Tags

#llm#model-efficiency#gpu#inference

Sources

Transcript

Eight-bit weights ran models of a hundred and seventy-five billion parameters, without losing quality. Most AI models are deployed at full precision, by default. Each number in the model takes more memory than the task needs. That means more graphics cards, and more energy per answer. Fewer bits per number means less memory traffic, and smaller hardware. Here is the fix you can use today. Test eight-bit or four-bit versions of your model against your quality bar. Use the smaller version wherever the results hold up. Measure energy and response time per request, before and after. Tim Dettmers and colleagues showed this for very large models. They ran on eight-bit numbers without losing quality. GreenCode's writing on green AI draws on this work. Run quantised open models on ordinary chips, using tools like llama.cpp. Then check power and compare models, using tools like nvidia-smi and AI Energy Score. GreenCode helps tackle this by keeping its own AI models small and fine-tuned. Each request goes to the smallest part that can do it. That was GreenCode Tech Tip number thirty-two. Follow for the next tip, and find more on greencode.ai/tech-tips.

All GreenCode Tech Tips · All GreenCode videos

  • Tip #32
  • For CTOs and technical leads
  • Length 1:16
  • Category AI and machine learning
GreenCode

Want this checked on your own system? Talk to the GreenCode team.

Get in touch