How do you build your own tokenizer for a language model?

A language model cannot read letters. It reads numbers, and the tokenizer is the small program that converts text to numbers and back. Our introduction to tokens explains why models read pieces of words. This article is about making that program yourself, which is the first thing you train in any from-scratch project, before the model.

Unlike a model, a tokenizer learns no meaning. It only learns which byte sequences are common in your text. That makes it fast to train: nanochat's reports about 34 seconds for 2 billion characters on a recent MacBook Pro.

The algorithm in plain words

Byte pair encoding (BPE) is the method behind the tokenizers of GPT-2, GPT-4, Llama and many others. Text is first turned into bytes, the 256 basic values a computer stores. Then:

1. Count every pair of neighboring tokens.
2. Give the most frequent pair a new number.
3. Replace that pair everywhere, and repeat.

After thousands of rounds, common words and word pieces have their own numbers, and rare words are still representable, in the worst case byte by byte. Your final vocabulary is the 256 bytes, plus one token per merge, plus any special tokens.

Build one in thirty lines

Karpathy's educational minbpe project is the best code to read next. This shorter version uses Python only, with nothing to install.

def pairs(ids):
    counts = {}
    for a, b in zip(ids, ids[1:]):
        counts[(a, b)] = counts.get((a, b), 0) + 1
    return counts

def merge(ids, pair, new_id):
    out, i = [], 0
    while i < len(ids):
        if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

text = "les élèves étudient; les élèves écrivent; les enseignantes étudient"
ids = list(text.encode("utf-8"))     # start from raw bytes: 256 possible tokens
print(len(text), "characters,", len(ids), "bytes")

merges = {}
for step in range(8):
    counts = pairs(ids)
    best = max(counts, key=counts.get)
    new_id = 256 + step
    ids = merge(ids, best, new_id)
    merges[best] = new_id

vocab = {i: bytes([i]) for i in range(256)}
for (a, b), new_id in merges.items():
    vocab[new_id] = vocab[a] + vocab[b]

print(len(ids), "tokens after 8 merges")
print(b"".join(vocab[i] for i in ids).decode("utf-8"))
print([vocab[i].decode("utf-8", "replace") for i in ids])

The output begins:

67 characters, 74 bytes
41 tokens after 8 merges
les élèves étudient; les élèves écrivent; les enseignantes étudient
['les él', '�', '�', 'v', 'es é', 't', 'u', 'd', 'i', 'ent', ...

There are three lessons here. The 7 extra bytes are the accents, since "é" and "è" each take 2 bytes in UTF-8, the standard text encoding. The line reassembled from tokens matches the original exactly, which is the test a tokenizer must always pass. And the "�" marks show the two bytes of "è" held in separate tokens. That is normal: a token can end in the middle of a character, and the text comes back whole only when the bytes are joined again. Print a token by itself and it may look broken.

Notice also that "les él" became a token, because this tiny text repeats it. Trained on too little text, a tokenizer memorizes phrases. Use a large and varied sample.

At real scale

The Python version above would take far too long on billions of characters. Hugging Face's tokenizers library is written in Rust and handles them quickly. Its quick tour uses a whitespace splitter and a character-based vocabulary, which turned an emoji into an unknown token in its own example. For a language model, a byte-level setup is safer because every possible text can be represented. This version needs pip install tokenizers and a text file of your own, here corpus.txt:

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.decoders import ByteLevel as ByteLevelDecoder

tok = Tokenizer(BPE())
tok.pre_tokenizer = ByteLevel(add_prefix_space=False)
tok.decoder = ByteLevelDecoder()

trainer = BpeTrainer(
    vocab_size=32768,
    initial_alphabet=ByteLevel.alphabet(),
    special_tokens=["<|endoftext|>", "<|user|>", "<|assistant|>"],
)
tok.train(["corpus.txt"], trainer)
tok.save("tokenizer.json")

sample = "Déjà, l’élève québécois écrit à l’hôpital."
out = tok.encode(sample)
print(out.tokens)
print(tok.decode(out.ids) == sample)

The printed tokens look garbled, for example é where you expect é. This is only how the library displays raw bytes as text. The last line should print True.

Special tokens mark things that are not text, such as the end of a document or the start of the assistant's turn. They are added on top of the merges. The minbpe documentation warns against turning user-typed text into special tokens by default, since someone could then fake the assistant's turn. Encode untrusted input so that those strings stay plain text.

Choosing a vocabulary size

A larger vocabulary means fewer tokens per sentence, so more text fits in each training batch. It also means a larger table for the model to learn, and rare tokens that it sees too seldom. For a small from-scratch model, 32,768 trained on about 2 billion characters is a tested starting point. For French, include real French text: a vocabulary trained on English leaves accented words in many small pieces, which uses up your context and your budget.

Should you train your own tokenizer or reuse an existing one?

If you are fine-tuning, always reuse. An open-weight model's numbers were learned for its own tokenizer, so token 5,000 means something specific to it. Using another tokenizer scrambles its input. The same goes for adding many new tokens: those would need training too.

If you are pretraining from scratch on ordinary English or a common language mix, either choice is reasonable, since training one takes minutes and reusing one takes none. Check the licence, and test the round trip.

Train your own when your text differs from what existing tokenizers saw: a language with little data online, such as many Indigenous languages, or a specialized field such as code or chemistry. Compare candidates with one number: how many tokens each needs for the same sample of your text. nanochat's tok_eval script reports this kind of compression ratio. Fewer tokens for the same text means cheaper training and more room in the context window. Then freeze it, because every later stage depends on that exact file.