---
title: "Tech Tip #68: Start pre-trained, fine-tune the adapter"
description: "LoRA trained 10,000 times fewer parameters with comparable quality."
url: https://greencode.ai/tech-tips/68-fine-tune-with-lora/
---

# Tech Tip #68: Start pre-trained, fine-tune the adapter

**LoRA trained 10,000 times fewer parameters with comparable quality.**

### The problem

Teams fine-tune whole models to adapt them to a domain, needing big GPU clusters and long runs.

### The fix you can use today

- Start from a pre-trained model instead of training from scratch.
- Fine-tune a small adapter with LoRA, or QLoRA with 4-bit weights to fit on one GPU.
- Compare quality with full fine-tuning on your own evaluation set.

Researchers at Microsoft showed that LoRA trains 10,000 times fewer parameters and still does as well. GreenCode's own AI uses small, tuned models.

### Tools that help

- [PEFT (Hugging Face)](https://huggingface.co/docs/peft?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): parameter-efficient fine-tuning methods such as LoRA, so only small adapter weights are trained
- [CodeCarbon](https://github.com/mlco2/codecarbon?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): a Python package that estimates the energy and CO2 of a run from hardware power and regional grid intensity

### How GreenCode helps

GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, retrieval so a model reads only what it needs, each request sent to the smallest part that can do it, and heavy jobs held for cleaner power. [More on this](https://greencode.ai/feature/green-optimised-generative-ai/).

### Tags

#llm#model-efficiency#gpu

### Sources

- [Green AI is a mathematics problem (LoRA and QLoRA)](https://greencode.ai/news/mathematics-and-green-ai/)
- [Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode)
- [Green Software Foundation, Green Software Patterns: Leverage pre-trained models and transfer learning](https://github.com/Green-Software-Foundation/patterns/blob/main/docs/development/pre-trained-transfer-learning.md?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode)

[All GreenCode Tech Tips](https://greencode.ai/tech-tips/)

#### Key Facts

- Tip **#68**
- For **CTOs and technical leads**
- Category **AI and machine learning**
- Format **short video**
![GreenCode](https://greencode.ai/wp-content/uploads/2024/12/greencode-logo-website-small-colours-2.webp)

Want this checked on your own system? Talk to the GreenCode team.

[Get in touch](https://greencode.ai/contact/)
