Originally published as part of VerifyClaw's internal security research into LLM-facing attack surfaces — adapted here for a general audience.
Refusal and compliance behavior looks binary from the outside — the model either does the thing or it doesn't — but the surface text often isn't a clean signal either way. "I cannot" can appear inside a quotation, get immediately followed by a compliant continuation, or coexist with actionable content that answers the request anyway. Judging safety from the opening phrase alone is a category error.
Why it matters for MCP agents
Automated pipelines that gate on a keyword match ("if response contains 'I cannot', mark as refused") inherit this blind spot directly. An attacker doesn't need to eliminate refusal language to get a harmful action performed — they just need the actionable part to survive somewhere in the full output, which a first-token or first-phrase check will never see.
How we study it
The fix is measuring the complete generated continuation against the actual task outcome, not a registered phrase in isolation — reporting phrase occurrence and content interpretation as two separate, separately-justified findings rather than collapsing them into one "refusal probability." A phrase that shares its first token with a very different continuation is exactly the case where a shortcut proxy gives a confident wrong answer.
Related reading
Deep semantic scanning reads full skill behavior, not keyword surface — see how it differs from a grammar-only scan on our docs page.