Why can a model fit on your machine and still be impossible to train?

The usual rule tells you if a model will fit: parameters times bytes per weight, as this calculator article explains. That number answers "can I run it?" Training is another bill entirely. People find out when a run that looked safe crashes on its first step.

This project measures that bill with one small model, SmolLM2-135M-Instruct, which has 134,515,008 parameters and an Apache 2.0 licence. Each row below ran in a new process on the CPU, using 32-bit numbers unless noted, and read the operating system's peak-memory figure. That makes it a real peak, not a guess. Peaks include some overhead from loading, and the 244 MB that Python and the libraries took before any model was already subtracted. The script is 07_memory_budget.py.

Where the memory goes

Running needs the first row and a smaller share of the last. Training needs all four. So before a single example is processed, training already costs four times the weights. Those three fixed rows come to 2,152 MB.

What it measured

Sequence length is 256 tokens unless the row says otherwise. Measured here on an Apple M5 Max with 64 GB, PyTorch 2.15 and Transformers 5.22.

Read it this way:

• Fitting says little. At batch 8, running needs 1,269 MB and training needs 6,484 MB, 5.1 times more. Using only the weights' 538 MB as the reference, training takes 12 times as much.
• At batch 1, you are paying the fixed costs. Even a single example costs 3,093 MB. Cutting batch size has a floor, and that floor is about four times the weights.
• Length and batch weigh the same. Batch 4 at length 512 (2,048 tokens) needs about what batch 8 at length 256 needs (2,048 tokens): 6,489 and 6,484 MB. Long documents are a memory decision, so truncating or splitting them saves as much as cutting batch size.
• The fixes work on different rows. Gradient checkpointing recomputes activations instead of keeping them, which saved 37% here at the price of extra computing. 16-bit numbers shrink everything by half in principle; measured, they saved 34% because peak also includes overhead. LoRA freezes the weights and trains only small added matrices, so gradients and optimizer state almost vanish: 460,800 trainable parameters, 0.34% of the model.

Those figures are from one small model on a CPU. Graphics cards keep other buffers and were not tested here, so treat these as a floor and leave a margin.

Turning it on yourself

Here is the core of one measurement. It needs to run in a new process, because the peak figure never goes back down.

import resource, sys, torch
from transformers import AutoModelForCausalLM

def peak_mb():
    v = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    return v / (1024 ** 2 if sys.platform == "darwin" else 1024)

model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct", dtype=torch.float32)
model.gradient_checkpointing_enable()        # remove this line to compare
model.config.use_cache = False
model.train()
opt = torch.optim.AdamW(model.parameters(), 1e-4)
x = torch.randint(0, 1000, (8, 256))
for _ in range(2):                           # Adam's state exists from the second step on
    model(input_ids=x, labels=x).loss.backward()
    opt.step()
    opt.zero_grad(set_to_none=False)
print("peak MB:", round(peak_mb()))

The second loop is not a detail. Adam allocates its state on the first update, so a test with one step reports a number that is too low and sends you into a crash on step two.

How do you estimate your budget before you start a run?

Add up the rows, then keep a margin. A quick formula, where P is parameters in billions and B is bytes per weight:

Calculated, for a model of 7 billion parameters at 2 bytes per weight, before any activations:

So a machine with 24 GB of graphics memory can comfortably hold this model to run, cannot train all of it, and can often train an adapter on it. That last case may need the lower-precision loading explained in the quantization article, since 14 GB leaves limited room for activations. None of those three lines was run here; they are arithmetic based on the small model's pattern, and no GPU of that size was tested.

To find the activation share, do what this project does: run two real training steps with your longest example at your intended batch size, read the peak, then double the batch and see how it grows. Two minutes of that tells you more than any table. It also tells you which fix to try first: if the fixed costs dominate, use an adapter or lower precision; if activations dominate, shorten sequences, reduce batch size or turn on checkpointing. Project 13 covers what to look at before deciding that memory is the actual problem.