What happens when a tiny language model learns from random weights?

The step-by-step article on nanoGPT and nanochat tells you which ready-made projects to run and what the larger path looks like. This project is different in one way: you read every line of a small model, then you check what happens against predictions you can work out on paper.

The text is L. M. Montgomery's 1908 novel Anne of Green Gables, which Project Gutenberg lists as public domain in the USA. After removing Gutenberg's header and footer, it has 559,087 characters and 68 different symbols. Check your own country's rules before you publish anything trained on a book.

Each character gets a number, so this model's "tokens" are single characters. Real models use larger pieces, as explained in how to build your own tokenizer.

The model, in about fifty lines

The companion file mini_gpt.py holds the entire model. This is the part that matters: the attention block, where each position looks at earlier positions and decides what to use, followed by a small network that processes the result.

class Block(nn.Module):
    def __init__(self, d, heads, drop):
        super().__init__()
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, heads, dropout=drop, batch_first=True)
        self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d), nn.Dropout(drop))

    def forward(self, x, mask):
        h = self.n1(x)
        x = x + self.attn(h, h, h, attn_mask=mask, need_weights=False)[0]
        return x + self.ff(self.n2(x))

The mask hides later characters, so the model cannot cheat by looking at the answer. Four of these blocks, with 128 numbers describing each character and a window of 128 characters, come to 818,432 parameters. Run python 06_tiny_language_model.py to download the text and train. It takes PyTorch and no account.

Predict the first number before you run it

With random weights, the model should give every symbol about the same probability. The loss for guessing evenly among 68 symbols is the natural logarithm of 68, which is 4.220. Measured at step 0: 4.218 on training text and 4.216 on validation text. This check at the start costs nothing. Had it printed 9, something would be broken already.

A second prediction: what could plain counting achieve? Tally which character follows which, with no neural network at all. That table scored a validation loss of 2.444. Your model must beat it, or the neural network has added nothing.

What it writes as the loss falls

Measured here on an Apple M5 Max's built-in GPU (PyTorch 2.15, batch of 64, learning rate 0.0003, one seed).

And what the same model produced, starting from the letter A each time:

Step 0, random weights:

A Dq”Lfe[C c‘yptbt?tvCU.aHsJXP”PBu

Step 200, about the level of the counting table. It has found spaces and common letter pairs:

Annnt fese cay tone tofor se aree has f avewonge. g Mre y, I theeree ame

Step 1,000, real words, with quotation-like rhythm:

Anne well to you, every. I’m see use so as with hard they,

Step 6,000, grammar-shaped text, a voice, and no meaning:

Anne was encouraging at all. And what did not give such a dismalet have been so many sorry.

Nobody told it that "Anne" is a name or what a sentence is. It lowered one number. It also invented "dismalet", which is a nice illustration: its pieces are English-like, but no dictionary was involved.

What the numbers say

Read the two loss columns together. At step 1,000 they are close. By step 6,000 training loss has fallen another 0.59, while validation loss fell only 0.34, and over the last 1,000 steps it moved by 0.012. The model is now mostly improving on the book it has seen. A larger network or more steps would widen this gap with so little text. More text is the remedy, which is what the jump from one novel to hundreds of millions of characters is for.

Validation loss of 1.339 is 1.93 bits per character. The counting table, at 2.444, is 3.53 bits. You can compare models with this measure as long as the tokens are the same.

The trained file takes 3.29 MB. Speed tells you something else. On this machine, 100 training steps took 1.2 seconds on the GPU and 30.7 seconds on the CPU using all its threads. The full run on the CPU would be about 31 minutes (calculated, not run). So this one works on a laptop with no GPU if you lower the settings or wait, and the job is no longer one you can keep restarting casually. For a CPU, nanoGPT's documented small settings are a good guide: fewer layers and a shorter window.

How do you tell learning from copying?

A model can look clever while it recites. So the script measures the longest stretch of a generated sample that appears word for word in the training text. For the final 221-character sample, that was 15 characters. Common phrases will always match, and 15 characters is about three short words. It is not a guarantee: one sample proves little, and with a bigger model or a smaller text, this number would rise.

The validation column is the other test. Text the model never saw has patterns it can use only if it learned something general. Validation loss fell from 4.216 to 1.339, far beyond the counting table, so it did learn structure: words, spelling and some grammar.

Neither test shows understanding. This model has no facts, and it has no way to know that it is writing nonsense. What it demonstrates is the mechanism of learning. Everything a large language model adds, such as better tokens, far more data, more layers and later fine-tuning, sits on this same loop. The next question is how much memory that loop needs once models grow, which project seven measures.