8-bit weights ran 175-billion-parameter models without losing quality.
Models are deployed in full precision by default, needing more memory, more GPUs and more energy per answer than the task requires.
Tim Dettmers and colleagues showed that very large models can run on 8-bit numbers without losing quality. GreenCode's writing on green AI draws on this work.
GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, retrieval so a model reads only what it needs, and each request routed to the smallest component that can handle it. More on this.
Eight-bit weights ran models of a hundred and seventy-five billion parameters, without losing quality. Most AI models are deployed at full precision, by default. Each number in the model takes more memory than the task needs. That means more graphics cards, and more energy per answer. Fewer bits per number means less memory traffic, and smaller hardware. Here is the fix you can use today. Test eight-bit or four-bit versions of your model against your quality bar. Use the smaller version wherever the results hold up. Measure energy and response time per request, before and after. Tim Dettmers and colleagues showed this for very large models. They ran on eight-bit numbers without losing quality. GreenCode's writing on green AI draws on this work. Run quantised open models on ordinary chips, using tools like llama.cpp. Then check power and compare models, using tools like nvidia-smi and AI Energy Score. GreenCode helps tackle this by keeping its own AI models small and fine-tuned. Each request goes to the smallest part that can do it. That was GreenCode Tech Tip number thirty-two. Follow for the next tip, and find more on greencode.ai/tech-tips.
Share this