How do you build your own language model, and which path fits your goal and budget?

"My own language model" is a phrase that hides very different projects. A student watching a toy learn spelling, a company teaching an open model its ticket format, and a university training a multilingual model on a national supercomputer are all building their own model. They differ by a factor of about a million in cost.

This page is the map for this series. It shows the levels, the stages every from-scratch model goes through, and the tools for each. A language model, or LLM (large language model), is a program that predicts the next piece of text. Pretraining teaches it that skill from raw text, starting from random numbers. Fine-tuning continues training on an existing model, usually with examples of the behavior you want.

The six levels

Dollar amounts are US dollars. The rented figures use prices read on November 2, 2026; the cost article shows the arithmetic.

Level 3 sits below level 4 on purpose: fine-tuning is cheaper and gives more practical results, while levels 1, 2 and 4 teach how models work. Most people should do level 1 or 2 for understanding and level 3 for use. Levels 1 and 3 already have step-by-step guides here: a tiny model from scratch and fine-tuning a small model.

The stages of a from-scratch model

The nanochat project by the researcher Andrej Karpathy is a useful reference because one short script runs all of these, in order:

1. Collect and clean text. The speedrun needs about 150 files holding some 250 million characters each. See how to prepare a dataset.
2. Train a tokenizer, the piece that cuts text into numbered tokens. See how to build one.
3. Pretrain the model on those tokens. This stage uses most of the money.
4. Fine-tune for chat, so that it answers instead of merely continuing your sentence.
5. Evaluate it against numbers that mean something. See how to test your model.
6. Publish, if you wish, with a licence that matches what you trained on. See publishing and licensing.

The rented-GPU walkthrough goes through stages 1 to 5 with commands.

Which tool does what

With Axolotl, a first run is two commands, assuming it is installed. The file names the model, the data and the method:

axolotl fetch examples
axolotl train examples/llama-3/lora-1b.yml

A worked plan

Say you work for a Quebec organization and want an assistant that sorts French-language requests into your categories. A sensible order:

1. Spend an afternoon on level 1 so that loss, tokens and overfitting stop being abstract.
2. Write 300 French examples of the sorting task and test a plain prompt on them. That score is your baseline.
3. If the prompt falls short, fine-tune a small open model that handles French (level 3), and compare on the same examples.
4. Skip levels 4 and 5. A GPT-2-grade model trained for $64 will not sort your requests reliably, and $880,000 buys nothing that the fine-tune needs.

Where should you stop climbing?

Stop at the first level that delivers your goal, and be honest about which goal it is. If you wanted to understand models, you are done when you can explain why your loss curve flattened. Level 2 can get you there for free, and level 4 adds the experience of paying for compute and seeing a chat model appear from nothing.

If you wanted a tool, stop as soon as a measured test says it beats the prompt-only baseline. Going further costs more without necessarily helping. Many tasks never need more than a prompt, as the guide to choosing between prompts, documents and fine-tuning explains.

Levels 5 and 6 are a different kind of decision, closer to funding a research project than to a weekend build. They make sense when control is the point: a dataset whose licences you can defend, a language the big models serve badly, or a country that wants its own. The comparison with fine-tuning and RAG walks through those cases, and the cost breakdown helps you check whether your budget matches.