What it means
Pre-training is where a model absorbs almost everything it knows. It runs on an enormous corpus with a simple objective — predict the next token — for months, across thousands of accelerators, at a cost typically measured in tens or hundreds of millions of dollars.
The output is a base model: fluent and knowledgeable, but not helpful in the conversational sense. A base model completes text rather than following instructions; ask it a question and it may continue with more questions, because that is a plausible continuation. Turning it into an assistant takes further stages — instruction tuning and RLHF.
The expense is why very few organizations pre-train frontier models, and why the rest of the industry builds on top of what those few release.
Why it matters
The cost structure explains the industry. A handful of labs can afford pre-training; everyone else fine-tunes, prompts, or calls an API. It also explains why knowledge is frozen at a cutoff — updating it means redoing the expensive part.
In practice
Pre-training is essentially never the right answer for an ordinary organization. If a model does not know your domain, retrieval is the first tool and fine-tuning the second; pre-training from scratch is neither.