xavier-ramirez.com
STAGE 0 · EXPERT · 3 NODES

Constrained decoding

You need machine-readable JSON, every single time — but a language model only predicts likely next-tokens, and 'likely' isn't 'valid'. Constrained decoding closes the gap: at every step it forbids any token that would break the structure, so the output is valid by construction.

THE 3 STEPS
01Why free sampling breaks02Masking the bad tokens03Grammars decide what's allowed
NODE 01 / 03

Free sampling doesn't know the rules

In Sampling strategies you saw the model score every token and pick one it likes. Nothing in that loop knows what valid JSON is — so a fluent, confident model still hands you text a parser rejects.

  • *The model optimises for plausible, not parseable.* A markdown fence or a friendly 'Sure!' is very likely text — and instantly invalid JSON.
  • One wrong token poisons the whole string. An unquoted key or a trailing comma makes JSON.parse throw on the entire response.
  • At scale this is expensive. Even a 1% failure rate means retries, dropped records, and pipelines that wedge on the one bad row.
  • Prompt-begging doesn't guarantee it. 'Reply with only JSON' lowers the failure rate; it can't make it zero.
YOU ASKED FOR: valid JSON · {"age": <number>}
FOUR WAYS THE SAME REQUEST BREAKS

Tap through the failure modes — the red span is exactly where JSON.parse gives up.

Prose preamble — the model opens with 'Sure!' before the object. The very first character isn't JSON, so parsing fails at byte 0.

PARSE_RESULT
STATUSINVALID
AT BYTE0
CAUSEUnexpected token 'S', "Sure! {"ag"... is not valid JSON
NODE 02 / 03

Mask the tokens that would break it

The fix doesn't change the model — it edits the model's choices. At every decode step a validator lists which tokens keep the JSON well-formed, and masking sets the probability of every other token to zero.

  • Masking = force p = 0. Every invalid token's logit is set to −∞, so after softmax its probability is exactly zero — it can never be sampled.
  • The survivors renormalise. The model still picks by its own preference among the valid tokens — masking removes options, it doesn't invent them.
  • Even a strong first choice gets vetoed. When the top token is a fence or a trailing comma, masking simply passes it over for the best valid one.
  • Toggle the constraint and watch the same candidates flip between 'breaks JSON' and 'valid by construction'.
ONE DECODE STEP · MASK ON/OFF

Switch the moment, then flip the mask off — the greedy pick becomes an invalid token and the output breaks.

First token — the model's favourite is a markdown fence. The mask forbids it and the humble `{` wins.

MASK_STATE
KEPT1
MASKED3
CHOSEN P100%
How the mask reshaped this step: how many candidates survived, how many were zeroed, and the renormalised probability of the token that won.
NODE 03 / 03

A grammar decides what's allowed next

Masking needs a source of truth: which tokens are valid right now? That source is a grammar — a compact set of rules describing every legal string. The engine tracks where you are in the grammar and reads off the allowed set.

  • A grammar is a shape, written down. These rules say the output must be {, then the key "age", then a number, then } — nothing else.
  • Position decides the allowed set. Right after "age": the grammar permits only number tokens — a digit or a minus sign. A quote is off the table.
  • Free decoding has no such map, so at every position it would allow anything; the grammar narrows it to exactly what's legal.
  • This is what powers structured outputs. JSON Schema or a GBNF grammar compiles into the allowed-set the masker enforces every step.
GOOD TO KNOW · STRUCTURED OUTPUTS
From JSON Schema to a token mask
This is the machinery behind 'JSON mode' and 'structured outputs'. You hand the API a JSON Schema (or a GBNF grammar); the engine compiles it to a state machine, and at every decode step it computes the allowed tokens and masks the rest. The model's quality still decides what it says — the grammar only guarantees the shape it comes out in.
SAME POSITION · FREE vs GRAMMAR

Step through the positions — after the colon, the grammar allows only a number; free decoding still allows a string, a brace, anything.

the grammar (GBNF) — read-only
root   ::= "{" ws "\"age\"" ws ":" ws number ws "}"
number ::= "-"? [0-9]+
ws     ::= [ \t]*

Start — the grammar's first rule demands an opening brace. Only `{` is allowed; everything else is masked.

ALLOWED_SET
POSITIONStart
GRAMMAR{
FREE…anything
At the current position: what the grammar permits next versus what unconstrained decoding would permit.
EXPLAIN IT BACK
A model is already very good at emitting JSON. Why bolt on a grammar and mask tokens instead of just trusting a good prompt?
NEXT: REASONING TOKENS

Masking controls the form of the output. Next, reasoning tokens: hidden scratch-work the model generates before its answer — where the goal is the opposite, to let it wander freely so the final answer comes out better.

← Back: mixture of expertsContinue: reasoning tokens →
Language: English