Originally published as part of VerifyClaw's internal security research into LLM-facing attack surfaces — adapted here for a general audience.
Constrained decoding — a GBNF grammar, a JSON schema, a sampler chain with temperature and top-k stages — is a popular way to make model output reliably parseable. It's easy to over-read what it proves: a grammar restricts the syntax of the output, not the model's underlying preference. A constrained response that looks safe doesn't establish that the unconstrained distribution wasn't heavily weighted toward something else.
Why it matters for MCP agents
Tool-calling agents lean on structured output constantly, and it's tempting to treat "the output validated against the schema" as a safety property. It isn't one — schema validity and semantic safety are orthogonal. A grammar can force a value into one of two allowed strings while the model's raw preference, if unconstrained, would have generated something entirely different.
How we study it
Comparing the raw logit distribution against each stage of the sampler chain — grammar filtering, temperature, top-k, final selection — separates "what the model would say" from "what it's permitted to say." A useful diagnostic here is the amount of raw probability mass retained after grammar filtering: a low retained-mass figure means the constraint is doing a lot of the safety work, which is worth knowing before you rely on it.
Related reading
Structured tool-call output is necessary for reliability — it isn't sufficient for safety. Scan the underlying skill logic too.