The life of an LLM
The model you chat with is the last step of a six-stage assembly line. Here is each stage — what actually happens, the tools and hardware labs use, and where the money goes.
Step through the pipeline. Each stage shows a real before → after, plus what the lab does, the tools, the hardware, and the spend.
Gather data
Assemble a massive, mostly-text corpus.
Crawl, license and pool sources; strike data deals; fight the quality and legal battles. Increasingly the hardest, most contested stage.
Common Crawl, web crawlers, dataset pipelines (FineWeb, RedPajama, The Stack).
Large CPU clusters + petabytes of storage. Little or no GPU yet.
Storage, bandwidth, and data licensing — deals can run into the tens or hundreds of millions. Compute here is minor.
Clean & tokenize
Filter the junk, then cut text into tokens.
Quality-filter (dedupe, drop spam, boilerplate and PII), then train a BPE tokenizer and encode the whole corpus to integer IDs.
Dedup (MinHash), quality classifiers; tokenizers (tiktoken, SentencePiece, HF tokenizers).
CPU-heavy and distributed (Spark / Ray). GPU optional.
Mostly engineer time and CPU compute — cheap next to training, but it decides the final quality.
Pretrain
Predict the next token, trillions of times.
Launch the big training run for weeks to months; babysit loss curves, restarts and instabilities. The output is the base model.
PyTorch / JAX, Megatron / DeepSpeed / FSDP, distributed schedulers, experiment tracking.
Thousands to tens of thousands of H100 / TPU accelerators on fast interconnect (InfiniBand / NVLink).
The megabill: a frontier run is often $10M–$100M+ in GPU-time. This dominates the entire budget.
Fine-tune & align
Turn a text-completer into a helpful assistant.
Instruction-tune on curated examples, then RLHF / DPO on human preference data; red-team for safety.
SFT + RLHF / DPO (e.g. TRL), reward models, human annotation platforms.
Tens to hundreds of GPUs — far less than pretraining — plus a large human labeling workforce.
Compute is modest; the spend shifts to people — annotators, domain experts, red-teamers. Quality beats quantity.
Evaluate
Prove it works — and doesn't misbehave.
Run public and private benchmarks, capability and safety evals, regression tests; sometimes external audits before release.
Eval harnesses (lm-eval-harness), private eval sets, LLM-as-judge, red-team suites.
Modest inference GPUs — mostly running the model across test sets.
Little compute; the cost is eval design, private test sets, and human review time.
Host / serve
Make the weights answer requests.
Deploy on inference infra with batching, quantization and autoscaling behind an API — or publish open weights for others to self-host.
Serving: vLLM, TGI, TensorRT-LLM. Local: Ollama, llama.cpp. Quantization: GGUF, AWQ.
Fleets of inference GPUs (A100 / H100 / L40S) for an API; a single consumer GPU or a laptop for a quantized local model.
Ongoing and usage-based: GPU-hours per million tokens. Closed = rent forever; open weights self-hosted = pay only your own hardware.