When should you adapt a small language model, and when should you not?

Adapting a model means changing how it behaves by training on examples, usually without touching most of its weights. LoRA does that by freezing the original weights and adding small trainable matrices. For the broader steps and tools, including QLoRA on larger models, see how to fine-tune a small open model. This project asks a narrower question, and puts one task that suits the method beside one that does not.

The model is SmolLM2-135M-Instruct: 135 million parameters, Apache 2.0 licence, about 540 MB in memory. It is tiny and often wrong, which is useful here, because you can see what training changes. The SmolLM2 paper describes the family. All data is synthetic, and the script is 09_adapt_small_model.py.

The adapter itself

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM")
model = get_peft_model(model, cfg)

r sets the size of the added matrices, and the target modules are the attention layers. This leaves 1,843,200 trainable parameters, 1.4% of the model. The adapter file is 7.4 MB, next to 538 MB of frozen weights in 32-bit form. The training loop masks the prompt, so the model is only graded on its answer. The file lm_utils.py has the whole loop in about 40 lines.

Each run below took 250 steps with batches of 8, at most about two minutes on an Apple M5 Max's built-in GPU (PyTorch 2.15, Transformers 5.22, PEFT 0.21), on a machine also running other work.

Task A: a good fit, a fixed format

Turn a shop note into one line that software can read:

Rewrite the note as one line: ITEM=<item>; ISSUE=<noise|broken|late|billing>; URGENT=<yes|no>

The snow blower fell apart after a day. It cannot wait, answer right now.

Expected: ITEM=snow blower; ISSUE=broken; URGENT=yes. The test uses 100 notes about items never seen in training. One test set also uses wordings never seen.

Three lessons. The untuned model could not do this at all, even with examples in the prompt. A larger model might, so test instructions first with the model you actually have. Second, 20 examples gave most of the gain, and going to 400 added nothing on new wordings. All 400 came from the same twelve phrasings, so this is project three's variety lesson again. Third, training loss reached 0.000 in all three runs, while exact accuracy on new wording sat near 55%. The loss would have told you that everything was fine.

A 55% model is not ready for use. Its value is in what it shows: format came easily, and generalizing to new wordings did not.

Task B: a bad fit, facts that change

Now teach 12 facts about a fictional community centre, such as the pool's closing time. Train on one phrasing of each question, then ask.

A perfect score on trained wording looks like success. Reword the question to "How late can I swim at night?" and the tuned model answers "24 hours". Asked what a grown-up member pays per year, it answers "75 dollars", a real value from another fact. It learned to emit values in response to a pattern and it delivers them with the same confidence when wrong.

Then three facts changed, as facts do. The tuned model kept giving all three old values, and correcting it means another training run. The lookup needed three edited lines and three new vectors, and still got 92% on the hard wording. The lookup works exactly as in project eight, with no language model writing anything.

The middle row deserves a note. A model this small, given the right line, still answered correctly only 58% of the time, and sometimes it made up details: it invented a "$25,000" income limit that was on no line. Handing a weak model the right text does not guarantee a right answer, so a retrieved line shown as-is can be the better product.

How do you decide in ten minutes?

Go through these questions in order and stop at the first yes.

Reasons to pick an adapter do exist: a model too small to follow instructions, as here; a need for speed, privacy or offline use; a shape of output that prompting cannot hold steady. The costs include examples to maintain, retraining when the base model changes, a licence to respect and a test set to keep running. Before you train, write the score you need to reach, and what you will do if it stays at 55%.