How much memory does an AI model need, and will it fit on your machine?

Whether a model will run on your computer is mostly arithmetic. A model is a very long list of numbers called parameters, and the size of the list is known before you download anything.

The setup guide gives a short table for common sizes. This article gives you the method, so you can check any model.

The formula

def size_gb(params_billions, bits_per_weight):
    return params_billions * bits_per_weight / 8

print(size_gb(27, 4.9))   # about 16.5 GB
print(size_gb(70, 4.9))   # about 42.9 GB

There are 8 bits in a byte, which is where the division comes from. The bits per weight depend on the quantization, the compression level of the file. The llama.cpp project measures these on Llama 3.1 8B: about 16 bits for F16, 8.5 for Q8_0, 5.7 for Q5_K_M, 4.89 for Q4_K_M and 3.16 for Q2_K.

The formula holds up. It gives 4.9 GB for the 8B model at Q4_K_M, and llama.cpp reports 4.9 GB. For 70B it gives 42.9 GB against a real 43.1 GB, and for 405B about 248 GB against a real 249.1 GB.

Memory against model size

These are calculated from the formula, rounded, for the weights alone:

And here are real measured or published sizes to compare against:

Models that read pictures carry an extra encoder, so their downloads run a little larger than the sum suggests.

What you can really use

The installed number is rarely the usable number.

Community guides put the Mac default near 70% to 75%, a figure the MLX LM documentation does not state. That documentation does say a large model that fits in RAM can often be sped up by raising the limit with sudo sysctl iogpu.wired_limit_mb=N. Leave enough for macOS and your apps if you do.

Then add context

The context window is what the model keeps in view: your question, the conversation and any documents. It takes memory that the file size does not show, and how much depends on the model. Ollama notes that a larger context raises memory use, and it sets its default from your VRAM: about 4,000 tokens below 24 GiB, 32,000 up to 48 GiB and 256,000 above that. It recommends at least 64,000 for agents and coding tools. If several requests run at once, Ollama's memory need scales with the number of parallel requests times the context length.

Here is a planning sum for a Qwen3.8-27B coding setup on a 32 GB machine, where the 30% margin is our own rule of thumb rather than a published standard:

1. Weights: 18 GB, from the download page.
2. Margin for context and runtime: 30% of 18 GB, about 5.4 GB.
3. Total: about 23.4 GB.
4. Left for the system and apps out of 32 GB: about 8.6 GB. It should work, with little room for much else.

To check, load the model and run ollama ps, which shows the memory in use and how it splits between graphics processor and CPU. Anything other than 100% GPU means you are over budget.

Here is my hardware: [graphics card or Mac model, installed memory, operating system]. Here are the models I am considering, with their parameter counts and download sizes: [list]. For each one, calculate the weights in GB, add a 30% margin, and say whether it fits my usable memory. Show the arithmetic. If you do not know how much of my memory the graphics processor can use, say so instead of guessing.

Why does a model that should fit still run slowly?

Fitting and being fast are separate questions. Speed depends on memory bandwidth, because the processor has to read through the active parameters for every token it writes. So 128 GB of slower shared memory can load a large model and still crawl. LMSYS measured a DGX Spark, with 273 GB/s, at 2.7 tokens per second on Llama 3.1 70B. That is roughly two words per second, usable for a batch job but tiring in conversation.

A mixture-of-experts model, which wakes only part of itself for each token, changes the sum. gpt-oss-120b has 117 billion parameters but only 5.1 billion active at a time, so all of it must fit in memory while the speed is closer to a much smaller model's. The mixture-of-experts explainer covers the design.

The last cause is spill-over: when a model does not fully fit, the remainder runs on the CPU and holds everything back. If speed collapses, check that first, then lower the context or move to a smaller model.