A model can't check the weather or run your database — it only predicts text. So it does the next best thing: it writes a structured request for a tool, pauses, and lets something outside the model do the work. Here's that loop, step by step.
Sometimes the best next token isn't a word — it's a request for a tool. The model emits a structured block and stops writing.
A tool call is just tokens. The model was fine-tuned to emit a structured block — a tool name plus JSON arguments — whenever a task needs the outside world.
It's still the same loop. From the autoregressive loop you know: predict the next token, append, repeat. A call is that loop producing a special block instead of prose.
The model routes. A fact it already knows, it answers directly. Live data or exact computation, it asks for a tool instead of guessing.
Valid JSON is guaranteed when the server constrains decoding — locking the decoder to the tool's schema (a grammar — a rule set the sampler must follow) forces the arguments to always parse. Without that constraint, valid JSON is likely, not certain.
USER REQUEST → THE MODEL'S OUTPUT
↳ Try each request — the weather and the math become tool calls; the fact it just answers.
Live data the model can't know — today's weather. It emits `get_weather` and stops; the loop hands off to the runtime.
NODE 02 / 03
The runtime runs it — not the model
The model can't run code. It only wrote a request — so something outside the model has to actually do the work.
Generation pauses. The instant the call block ends, the runtime stops sampling. The model is frozen — it executes nothing itself.
The harness executes. Your server — the harness wrapped around the model — parses the JSON, calls the real function or API, and captures whatever comes back.
Anything can be a tool — a weather API, a calculator sandbox, a database query, even another model. The model only ever produced text asking for it.
Errors come back too. A 503, a timeout, a bad expression — the failure is handed back like any other result, and the model has to deal with it.
INSIDE THE MODEL vs OUTSIDE THE MODEL
↳ Switch tools, then flip the error toggle — the failure is returned to the model just like a normal result.
The runtime makes a real HTTP request to a weather service and gets JSON back — bytes the model never had in its weights.
NODE 03 / 03
The result comes back as a turn
The tool's output doesn't go to you — it goes back to the model, as if it had been in the prompt all along.
The result becomes a new turn. The runtime appends it to the conversation with a tool role, right after the call that asked for it.
Generation resumes. The model reads the whole thread — including the fresh result — and continues from there, exactly like reading a longer prompt.
It can loop. If the answer needs another tool, the model emits another call, waits, reads that result, and only then writes prose. This loop is what an agent is.
Nothing is memorised. The result lives in the context window for this chat only — the weights never change (that's retrieval, not retraining).
THE CONVERSATION, TURN BY TURN
↳ Step through the loop — each turn is appended to the context the model reads before it answers.
The conversation starts: the user asks something the model can't answer from memory alone.
EXPLAIN IT BACK
A tool call pauses the model, runs code, and feeds the result back. So where does the reasoning about that result happen — in the tool, or in the model?
NEXT: TEST-TIME SEARCH
One call, then an answer. But hard problems need many attempts — trying an approach, checking the result, backing up, trying another. Next, test-time search: spending extra compute at answer-time to explore several reasoning paths and keep the best one.