Why fetch at all?
The web changes every minute — new headlines, prices, results, product launches. But a language model is frozen at its training cutoff: it can’t know what happened after. And you can’t just teach it the news — retraining a frontier model takes weeks and millions, and it’s stale again the next day.
You can’t bake today’s news into the weights
A trained model’s weights are frozen. Teaching it a fresh fact by retraining costs weeks and millions — and it’s stale again tomorrow. So the fact doesn’t go into the model at all: it rides in the prompt. That’s retrieval — the “R” in RAG.
- $0 — no training run
- 0 weights changed
- Always current
- ~$12.4M compute
- 405B weights rewritten
- Stale again tomorrow
So the fresh fact never goes into the model. It’s fetched at question time and dropped into the prompt — Retrieval-Augmented Generation (RAG) — and the answer is written from what was just pulled.
The pipeline, one stage at a time
Getting from your question to that cited answer takes five moves. Step through them — each stage lights up in the diagram.
Your one question becomes several. The system rewrites it into sub-queries, so it searches the way a curious person would — not just the literal words you typed.
The same pipeline, live: a small Python backend (bring-your-own-key) does real query fan-out, live retrieval, grounded citations and a groundedness eval — every step visible. Reproduces the ideas, not Google’s index.
View on GitHub →More in this series
Now take that same machinery into production — where the “web” is your own private corpus and every answer has to be auditable. That’s RAG in a regulated shop like insurance. Each post takes one part of the pipeline there:
The same machinery over your own private data: a write path that ingests change data from an S3 data lake and a six-node LangGraph read path that answers with citations. Clone it and run it, zero cloud.
- Chunking: split the document wrong and you lose the answerSoon↳ Chunk
How you cut a claim file decides whether the water-damage detail and its policy number land in the same passage or get orphaned. Chunking as a retrieval decision — measured.
- Retrieve & rerank: recall finds candidates, rerank decides the answerSoon↳ Retrieve · rerank
Cheap dense retrieval casts a wide net; a reranker sharpens precision so the causal passage lands in the top few that fit the budget. Dense vs hybrid, on the same question.
- Access control at retrieval: filter at retrieval, not afterSoon↳ Access control
Filter restricted content after the model answers and it leaks anyway. The permission check belongs inside the retrieval query, built from the token, with an audit trail.
- Grounding & citations: every answer cites the file it came fromSoon↳ Grounding
Tie every sentence to the source that backs it and score how faithful the answer really is — so an answer in an insurance workflow can be checked, not just believed. Or it abstains.
- Evals & guardrails: the gate between a demo and productionSoon↳ Evals
A groundedness eval wired into CI that blocks the release, plus prompt-injection and PII defenses — the line between a demo and a system you can run in a regulated shop.