What is GPT-3, and why are researchers so excited about it?

Last Thursday, a team of 31 OpenAI researchers posted a long paper on arXiv titled Language Models are Few-Shot Learners. It introduces GPT-3, a language model with 175 billion parameters. That is roughly ten times the size of any earlier language model of its kind and more than a hundred times larger than GPT-2, which OpenAI showed in February 2019 with 1.5 billion parameters.

Size alone is not the headline. What has researchers talking is how GPT-3 is used. Instead of retraining the model for each job, the authors simply describe a task in plain text, sometimes add a handful of worked examples, and let the model continue. With no extra training, it translates, answers questions, fills in missing words, unscrambles letters and does simple arithmetic, often well, and sometimes close to systems built for one task only.

What a language model actually does

A language model is a program that has read a very large amount of text and learned to guess the next piece of text. A "parameter" is one of the adjustable numbers inside the model that training tunes; more parameters give the model more room to store patterns. GPT-3 was trained on about 300 billion tokens (a token is a word or a piece of a word), mostly from a filtered copy of Common Crawl, a large public crawl of the web, along with books and Wikipedia.

Because it only ever learned to continue text, everything you ask of it has to be framed as text to continue. That sounds limiting, yet it turns out to be very flexible: a question invites an answer, a list invites the next item, and a pattern invites more of the same pattern.

Why this paper stands out

Until now, the usual recipe in language research has been two steps. First you pretrain a big model on general text; then you fine-tune it, meaning you train it again on thousands of labeled examples for one specific task. Fine-tuning works, but every new task needs a new dataset.

GPT-3 skips the second step entirely in the paper's experiments. The model's weights never change during testing. A few results show how far that goes:

That last result is the one that gives many readers pause. When short news articles written by GPT-3 were mixed with real ones, human judges could hardly tell them apart.

What it cannot do yet

The authors are candid about weak points. Long passages can drift, repeat ideas or contradict themselves. Some reading and reasoning tests remain hard. The model is expensive to run, and because it learned from the open web, it can reflect the biases found there. The paper spends a full section on possible misuse, such as misinformation, spam and phishing.

OpenAI has not released the model itself. What is public is the paper and a GitHub repository with unfiltered samples, the synthetic arithmetic and word-scramble tests, and statistics about the training languages. The repository comes with a warning that samples may contain offensive language, since the training data came from across the web.

What is few-shot learning?

Few-shot learning means showing a model a small number of solved examples inside its input and letting it work out the pattern for a new case. The paper names three settings. In zero-shot, the model gets only an instruction. In one-shot, it gets an instruction plus one demonstration. In few-shot, it gets several demonstrations, typically between 10 and 100, as many as fit in its window of 2,048 tokens.

Here is what a few-shot input looks like for a translation task, in the same spirit as the paper's own examples:

Translate English to French.

sea otter => loutre de mer
cheese => fromage
good morning => bonjour
thank you =>

The model has never been told the rules of this format. It sees three pairs, recognizes the pattern, and continues with the French for the last phrase. No retraining happens; the "learning" lives entirely in the text you supply. In the paper, results usually get better as the number of examples goes from none to one to several, and the gap grows with model size, which is one reason the authors think scale matters so much.

For a teacher, the idea feels familiar. We rarely give students a rulebook first; we show two or three solved problems and say "now you try." GPT-3 suggests that a large enough language model can respond to the same approach, and many researchers expect this way of working with models to spread quickly.