What is Mixtral 8x7B, and how does a mixture of experts work?

Yesterday Mistral AI, a young French company based in Paris, officially released Mixtral 8x7B, a language model you can download, run on your own hardware and use commercially with very few restrictions. According to Mistral's own tests, it performs about as well as OpenAI's GPT-3.5 on most standard benchmarks.

That claim is getting attention because open models, ones whose files are published for anyone to use, have generally trailed the best closed systems. Developers are already running it on their own computers, and well-known researchers such as Andrej Karpathy have publicly said the results look like a real step forward.

Who made it, and what was released

Mistral AI was founded in Paris in May. Its three co-founders are Arthur Mensch, its CEO, along with Guillaume Lample and Timothée Lacroix. In September it released Mistral 7B, a smaller model it also gave away as a free download. Yesterday the company also closed a funding round of 385 million euros, about $415 million, led by the venture firm Andreessen Horowitz.

The release has three parts:

• Mixtral 8x7B, the base model, trained on text from the open web.
• Mixtral 8x7B Instruct, a version tuned to follow instructions, using supervised fine-tuning (training on examples of good answers) and direct preference optimization, or DPO (training on pairs of answers where one is preferred).
• A paid API, in beta, where Mixtral runs under the name mistral-small, alongside mistral-tiny (Mistral 7B) and a stronger mistral-medium, which is available only through the API.

The weights, the files of numbers that make up a trained neural network, are licensed under Apache 2.0, an open-source license that allows commercial use. They are available on the model-sharing site Hugging Face and through BitTorrent.

How a mixture of experts works

Most language models are "dense": every part of the network works on every word. Mixtral is sparse. Inside each of its layers, one component, called the feedforward block, comes in eight separate versions, the "experts." Before each token (a word or piece of a word) is processed, a small router network looks at it and picks two of the eight. Only those two do the work, and their results are added together.

Ars Technica offers a factory analogy: instead of every worker touching every product, a supervisor sends each job to the two workers best suited to it. The factory has many workers, but each job uses only a few.

The name can mislead. "8x7B" suggests eight complete 7-billion-parameter models, or 56 billion parameters. In fact only the feedforward blocks are multiplied by eight; the rest of the network is shared. A parameter is one of the adjustable numbers a model learns during training.

The payoff is efficiency. Because only about 13 billion parameters are active for each token, Mixtral runs at roughly the speed and cost of a 13-billion-parameter model while drawing on the knowledge stored in all 46.7 billion. Mistral says it answers about six times faster than Meta's Llama 2 70B while beating it on most benchmarks.

How well it performs

Mistral reports that Mixtral matches or outperforms both Llama 2 70B and GPT-3.5 on most of the benchmarks it tested, and is strong at writing code. The Instruct version scores 8.30 on MT-Bench, a test of how well chat models handle multi-turn conversations, which Mistral says makes it the best open-source model on that test and comparable to GPT-3.5. On TruthfulQA, a test of whether models repeat common falsehoods, the base model scored 73.9% against Llama 2's 50.2%.

These are the company's own numbers, so independent testing over the coming weeks will matter. Mistral also notes that without a guiding prompt, Mixtral will follow whatever instructions it is given, so builders who need content limits must add them.

Why it matters

Mixtral narrows the gap between open and closed models. For developers, that means a GPT-3.5-class model they can run on their own servers, inspect, fine-tune and build into products without sending data to an outside company. Early users report running it locally at around 30 tokens per second on an Apple M1 computer.

It also brings mixture-of-experts design into the open. The idea is not new, but Mixtral is a widely available, well-documented example that shows how a model can grow in total knowledge without growing equally in running cost. Mistral has contributed code to vLLM, an open-source serving system, so others can run it efficiently.

Finally, it shows how quickly a newcomer can reach this level. Mistral is about seven months old, and its CEO, Arthur Mensch, describes the goal as building a European leader in generative AI with an open approach.

How can you try Mixtral yourself?

If you are a developer, the simplest route is Mistral's own platform, which serves Mixtral as mistral-small; you can register there for early access to its API.

To run it on your own computer, desktop apps such as LM Studio let you download a version of the model and chat with it offline. Early users report good speeds on recent Apple laptops, though a large download and a well-equipped machine help.

Researchers and companies can download the full weights from Hugging Face and use them under Apache 2.0, including in commercial products. Start with the Instruct version if you want a chat assistant, and add a system-style instruction describing what the model should and should not do.