LLM SECURITY TELEMETRY · SOURCE GUIDE

10. Grammar constraints and sampler effects

Where this concern meets the GGUF execution graph

The highlighted annotations locate attack entry, possible influence, constraints and observable effects for this concern. They are source-based hypotheses, not a trace of a demonstrated attack. GGUF supplies weights and metadata; llama.cpp builds and executes the computation graph.

Source snapshot: llama.cpp 5266f24da75dc449bd56cbed7addb9c8e4a6a73e. Concrete Llama architecture, ordinary generation path, non-fused attention as configured by this driver. Other architectures and fused kernels can use different graphs. All code below is embedded and expandable; no network or inference is required.

STAGE 01

GGUF → model weights, vocabulary and architecture

GGUF is a model file. The loader reads its metadata and tensors; C++ architecture code constructs the executable graph.

MODEL BASELINE: keep the GGUF identity and inference revision fixed. This concern does not demonstrate an attack on the GGUF loader or a modification of model weights.

llama.cpp implementation

llama.cpp/src/llama-model-loader.cpp:562–580View code

562    if (!fname.empty()) {563        // Load the main GGUF564        struct ggml_context * ctx = NULL;565        struct gguf_init_params params = {566            /*.no_alloc = */ true,567            /*.ctx      = */ &ctx,568        };569570        metadata_ptr.reset(gguf_init_from_file(fname.c_str(), params));571        metadata = metadata_ptr.get();572        if (metadata == nullptr) {573            throw std::runtime_error(format("%s: failed to load model from %s", __func__, fname.c_str()));574        }575576        get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false);577        llm_kv = LLM_KV(llm_arch_from_string(arch_name));578579        files.emplace_back(new llama_file(fname.c_str(), "rb", use_direct_io));580        contexts.emplace_back(ctx);

llama.cpp implementation

llama.cpp/src/llama.cpp:343–371View code

343344        model->hparams.vocab_only = params.vocab_only;345        model->hparams.no_alloc   = params.no_alloc;346347        try {348            model->load_hparams(ml);349        } catch(const std::exception & e) {350            throw std::runtime_error("error loading model hyperparameters: " + std::string(e.what()));351        }352        if (model->arch == LLM_ARCH_CLIP) {353            throw std::runtime_error("CLIP cannot be used as main model, use it with --mmproj instead");354        }355        try {356            model->load_vocab(ml);357        } catch(const std::exception & e) {358            throw std::runtime_error("error loading model vocabulary: " + std::string(e.what()));359        }360361        model->load_stats(ml);362        model->print_info();363364        if (params.vocab_only) {365            LLAMA_LOG_INFO("%s: vocab only - skipping tensors\n", __func__);366            return {0, model_ptr.release()};367        }368369        if (!model->load_tensors(ml)) {370            return {-2, nullptr};371        }

STAGE 02

Rendered text → token IDs

The caller supplies the final prompt. llama_tokenize delegates to the loaded vocabulary; tokenization itself does not execute the transformer.

EXPERIMENT CONTROL: keep rendered input fixed while changing one grammar or sampler stage. A malicious input is a separate variable if the experiment includes one.

llama.cpp implementation

llama.cpp/src/llama-vocab.cpp:4410–4419View code

4410int32_t llama_tokenize(4411    const struct llama_vocab * vocab,4412                  const char * text,4413                     int32_t   text_len,4414                 llama_token * tokens,4415                     int32_t   n_tokens_max,4416                        bool   add_special,4417                        bool   parse_special) {4418    return vocab->tokenize(text, text_len, tokens, n_tokens_max, add_special, parse_special);4419}

Driver: read the rendered prompt

src/cst_capture.cpp:83–91View code

83int main(int argc,char ** argv){84    if(argc!=2){std::cerr<<"Usage: cst-capture config.json\n";return 2;}85    try {86        json j=json::parse(read_file(argv[1]));87        std::string model_path=j.at("model"),prompt=read_file(j.at("prompt_file"));88        std::string log=j.value("log_path",std::string("trial.cstbin")),jsonl=j.value("jsonl_path",std::string("trial.telemetry.jsonl"));89        std::string run=j.value("run_id",std::string("local"));90        int max_tokens=j.value("max_tokens",32),nctx=j.value("context_size",4096),stride=j.value("score_stride",1);91        if(max_tokens<=0 || nctx<=0 || stride<=0)throw std::runtime_error("invalid token/context/score limits");

STAGE 03

Decode → build or reuse graph → set current inputs

process_ubatch applies memory state, builds or reuses the graph, sets new inputs and requests graph computation. Reusing topology still computes the new input.

EXECUTION ENTRY: the current prefix and context state reach the model here. Building or reusing topology is not itself an attack; the relevant influence is identified in the stages below.

llama.cpp implementation

llama.cpp/src/llama-context.cpp:1334–1403View code

1334llm_graph_result * llama_context::process_ubatch(const llama_ubatch & ubatch, llm_graph_type gtype, llama_memory_context_i * mctx, ggml_status & ret) {1335    if (mctx && !mctx->apply()) {1336        LLAMA_LOG_ERROR("%s: failed to apply memory context\n", __func__);1337        ret = GGML_STATUS_FAILED;1338        return nullptr;1339    }13401341    auto * res = gf_res_prev.get();1342    auto * gf  = res->get_gf();13431344    // the new graph parameters1345    // in order to correctly reuse a graph, it's full topology has to be uniquely determined by these parameters1346    const auto gparams = graph_params(res, ubatch, mctx, gtype);13471348    if (!graph_reuse_disable && res->can_reuse(gparams)) {1349        //LLAMA_LOG_DEBUG("%s: reusing previous graph\n", __func__);13501351        // with pipeline parallelism, the previous graph_compute_async may still be running1352        // on the GPU. we must synchronize before set_inputs to avoid overwriting input tensors1353        // that the previous compute is still reading.1354        if (cparams.pipeline_parallel) {1355            ggml_backend_sched_synchronize(sched.get());1356        }13571358        n_reused++;1359    } else {1360        res->reset();13611362        ggml_backend_sched_reset(sched.get());1363        ggml_backend_sched_set_eval_callback(sched.get(), cparams.cb_eval, cparams.cb_eval_user_data);13641365        //const auto t_start_us = ggml_time_us();13661367        gf = model.build_graph(gparams);13681369        //LLAMA_LOG_INFO("graph build time: %.3f ms\n", (ggml_time_us() - t_start_us)/1000.0);13701371        if (!gf) {1372            LLAMA_LOG_ERROR("%s: failed to initialize graph\n", __func__);1373            ret = GGML_STATUS_FAILED;1374            return nullptr;1375        }13761377        if (!ggml_backend_sched_alloc_graph(sched.get(), gf)) {1378            LLAMA_LOG_ERROR("%s: failed to allocate graph\n", __func__);1379            ret = GGML_STATUS_ALLOC_FAILED;1380            return nullptr;1381        }1382    }13831384    // set the input data for the input tensors1385    {1386        //const auto t_start_us = ggml_time_us();13871388        // FIXME this call causes a crash if any model inputs were not used in the graph and were therefore not allocated1389        res->set_inputs(&ubatch);13901391        //LLAMA_LOG_INFO("graph set inputs time: %.3f ms\n", (ggml_time_us() - t_start_us)/1000.0);1392    }13931394    const auto status = graph_compute(res->get_gf(), ubatch.n_tokens > 1);1395    if (status != GGML_STATUS_SUCCESS) {1396        LLAMA_LOG_ERROR("%s: failed to compute graph, compute status: %d\n", __func__, status);1397        ret = status;1398        return nullptr;1399    }14001401    ret = GGML_STATUS_SUCCESS;14021403    return res;

llama.cpp implementation

llama.cpp/src/llama-model.cpp:2673–2690View code

2673ggml_cgraph * llama_model::build_graph(const llm_graph_params & params) const {2674    std::unique_ptr<llm_graph_context> llm = build_arch_graph(params);26752676    // add on pooling layer2677    llm->build_pooling(cls, cls_b, cls_out, cls_out_b, cls_norm);26782679    // add backend sampling layers (if any)2680    llm->build_sampling();26812682    // if the gguf model was converted with --sentence-transformers-dense-modules2683    // there will be two additional dense projection layers2684    // dense linear projections are applied after pooling2685    // TODO: move reranking logic here and generalize2686    llm->build_dense_out(dense_2_out_layers, dense_2_out_layers_b, dense_3_out_layers);26872688    llm->res->set_outputs(params);26892690    return llm->res->get_gf();

Driver: decode each prompt token

src/cst_capture.cpp:176–182View code

176        cst_set_position(0,0,0,0,"prefill");cst_log_prompt(input.data(),(int)input.size(),prompt.c_str());177        auto trial_start=Clock::now(),start=trial_start;178        std::vector<int32_t> keys;179        for(size_t i=0;i<input.size();++i){180            cst_set_position(0,0,(int)i,0,"prefill");keys.push_back((int)i);cst_set_attention_key_positions(keys.data(),(int)keys.size());181            decode_one(ctx.get(),input[i]);182        }

Inside the Llama forward graph

Tensor dependency order: embeddings → per-layer attention and FFN → final logits. K/V, attention and FFN repeat for each layer. The backend executes these operations during decode; graph construction alone does not compute their values.

STAGE 04

Token embeddings + positions

Token IDs select rows of the GGUF embedding matrix. The Llama graph supplies positions and enters its transformer-layer loop.

CONTROLLED BASELINE: identical input tokens and positions enter the embedding stage. The deliberately changed grammar or sampler stage is downstream of this driver’s forward pass.

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2303–2328View code

2303ggml_tensor * llm_graph_context::build_inp_embd(ggml_tensor * tok_embd) const {2304    const int64_t n_embd_inp = hparams.n_embd_inp();2305    const int64_t n_embd     = hparams.n_embd;23062307    assert(n_embd_inp >= n_embd);23082309    auto inp = std::make_unique<llm_graph_input_embd>(n_embd_inp);23102311    inp->tokens = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, ubatch.n_tokens);2312    cb(inp->tokens, "inp_tokens", -1);2313    ggml_set_input(inp->tokens);2314    res->t_inp_tokens = inp->tokens;23152316    inp->embd = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_embd_inp, ubatch.n_tokens);2317    cb(inp->embd, "inp_embd", -1);2318    ggml_set_input(inp->embd);23192320    // select one of the 2 inputs, based on the batch contents2321    // ref: https://github.com/ggml-org/llama.cpp/pull/185502322    std::array<ggml_tensor *, 2> inps;23232324    // token embeddings path (ubatch.token != nullptr)2325    {2326        auto & cur = inps[0];23272328        cur = ggml_get_rows(ctx0, tok_embd, inp->tokens);

llama.cpp implementation

llama.cpp/src/models/llama.cpp:108–131View code

108    inpL = build_inp_embd(model.tok_embd);109110    // inp_pos - contains the positions111    ggml_tensor * inp_pos = build_inp_pos();112113    using inp_attn_type = std::conditional_t<embed, llm_graph_input_attn_no_cache, llm_graph_input_attn_kv>;114115    inp_attn_type * inp_attn = nullptr;116    if constexpr (embed) {117        inp_attn = build_attn_inp_no_cache();118    } else {119        inp_attn = build_attn_inp_kv();120    }121122    const float kq_scale = hparams.f_attention_scale == 0.0f ? 1.0f/sqrtf(float(n_embd_head)) : hparams.f_attention_scale;123124    ggml_tensor * inp_out_ids = build_inp_out_ids();125126    for (int il = 0; il < n_layer; ++il) {127        res->t_layer_inp[il] = inpL;128129        ggml_tensor * inpSA = inpL;130131        // norm

STAGE 05

Per layer: Q/K/V + position encoding → KV cache

The graph computes Q/K/V, applies RoPE and adds K/V cache writes. Attention reads the available cache with a position mask.

With the same prefix, compare the same context state. Divergent selected tokens will change subsequent K/V state.

llama.cpp implementation

llama.cpp/src/models/llama.cpp:139–161View code

139            // rope freq factors for llama3; may return nullptr for llama2 and other models140            ggml_tensor * rope_factors = model.get_rope_factors(cparams, il);141142            // compute Q and K and RoPE them143            auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur,144                    n_embd_head, n_head, n_head_kv, il);145146            Qcur = ggml_rope_ext(147                    ctx0, Qcur, inp_pos, rope_factors,148                    n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,149                    ext_factor, attn_factor, beta_fast, beta_slow150                    );151152            Kcur = ggml_rope_ext(153                    ctx0, Kcur, inp_pos, rope_factors,154                    n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,155                    ext_factor, attn_factor, beta_fast, beta_slow156                    );157158            cb(Qcur, "Qcur", il);159            cb(Kcur, "Kcur", il);160            cb(Vcur, "Vcur", il);161

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2815–2833View code

2815    const auto * mctx_cur = inp->mctx;28162817    // store to KV cache2818    {2819        const auto & k_idxs = inp->get_k_idxs();2820        const auto & v_idxs = inp->get_v_idxs();28212822        ggml_build_forward_expand(gf, mctx_cur->cpy_k(ctx0, k_cur, k_idxs, il));2823        ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));2824    }28252826    ggml_tensor * kq_mask = inp->get_kq_mask();28272828    ggml_tensor * q = q_cur;2829    ggml_tensor * k = mctx_cur->get_k(ctx0, il);2830    ggml_tensor * v = mctx_cur->get_v(ctx0, il);28312832    ggml_tensor * cur = build_attn_mha(q, k, v, kq_b, kq_mask, sinks, v_mla, 0, kq_scale, il);2833    cb(cur, "kqv_out", il);

STAGE 06

Per layer: QK scores → masked softmax → weighted V

On the non-fused path, kq_soft_max contains attention probabilities. Those weights mix value vectors into the attention output.

Attention belongs to the model’s forward pass. In this driver, grammar filtering happens after decode and does not directly edit these attention rows.

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2605–2610View code

2605        ggml_tensor * kq = ggml_mul_mat(ctx0, k, q);2606        cb(kq, "kq", il);26072608        // note: this op tends to require high floating point range2609        //       while for some models F16 is enough, for others it is not, so we default to F32 here2610        ggml_mul_mat_set_prec(kq, GGML_PREC_F32);

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2633–2650View code

26332634        if (kq_b) {2635            kq = ggml_add(ctx0, kq, kq_b);2636            cb(kq, "kq_plus_kq_b", il);2637        }26382639        kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, hparams.f_max_alibi_bias);2640        ggml_soft_max_add_sinks(kq, sinks);2641        cb(kq, "kq_soft_max", il);26422643        if (!v_trans) {2644            // note: avoid this branch2645            v = ggml_cont(ctx0, ggml_transpose(ctx0, v));2646            cb(v, "v_cont", il);2647        }26482649        ggml_tensor * kqv = ggml_mul_mat(ctx0, v, kq);2650        cb(kqv, "kqv", il);

STAGE 07

Per layer: attention residual → FFN → layer output

Attention output is added to the residual, transformed by the FFN, then added again. l_out feeds the next layer; the layer loop repeats.

Hidden states precede the driver’s sampler chain. Later differences can arise when a different sampled token re-enters the next decode.

llama.cpp implementation

llama.cpp/src/models/llama.cpp:173–195View code

173        }174        if (il == n_layer - 1 && inp_out_ids) {175            cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);176            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids);177        }178        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);179        cb(ffn_inp, "ffn_inp", il);180181        // feed-forward network (non-MoE)182        if (model.layers[il].ffn_gate_inp == nullptr) {183184            cur = build_norm(ffn_inp,185                    model.layers[il].ffn_norm, NULL,186                    LLM_NORM_RMS, il);187            cb(cur, "ffn_norm", il);188189            cur = build_ffn(cur,190                    model.layers[il].ffn_up,   model.layers[il].ffn_up_b,   model.layers[il].ffn_up_s,191                    model.layers[il].ffn_gate, model.layers[il].ffn_gate_b, model.layers[il].ffn_gate_s,192                    model.layers[il].ffn_down, model.layers[il].ffn_down_b, model.layers[il].ffn_down_s,193                    NULL,194                    LLM_FFN_SILU, LLM_FFN_PAR, il);195            cb(cur, "ffn_out", il);

llama.cpp implementation

llama.cpp/src/models/llama.cpp:220–228View code

220        cur = ggml_add(ctx0, cur, ffn_inp);221        cb(cur, "ffn_out", il);222223        cur = build_cvec(cur, il);224        cb(cur, "l_out", il);225226        // input for next layer227        inpL = cur;228    }

STAGE 08

Final normalization → vocabulary logits

After the final layer, output normalization and the GGUF output projection produce result_output / t_logits.

OBSERVE BEFORE CONSTRAINTS: record raw logits before grammar and sampling transformations; constrained output does not imply raw model preference.

llama.cpp implementation

llama.cpp/src/models/llama.cpp:230–246View code

230231    cur = build_norm(cur,232            model.output_norm, NULL,233            LLM_NORM_RMS, -1);234235    cb(cur, "result_norm", -1);236    res->t_embd = cur;237238    if constexpr (!embed) {239        // lm_head240        cur = build_lora_mm(model.output, cur, model.output_s);241242        cb(cur, "result_output", -1);243        res->t_logits = cur;244    }245246    ggml_build_forward_expand(gf, cur);

STAGE 09

Backend executes these graph operations; callbacks observe tensors

The preceding boxes describe tensor dependencies. Numerics execute through this scheduler, which computes graph ranges, synchronizes and invokes requested observation callbacks during execution.

The tensor graph produces logits; this driver applies sampler stages outside that forward pass. Per-stage telemetry distinguishes filtering from raw prediction.

llama.cpp implementation

llama.cpp/src/llama-context.cpp:2491–2518View code

2491ggml_status llama_context::graph_compute(2492            ggml_cgraph * gf,2493                   bool   batched) {2494    int n_threads        = batched ? cparams.n_threads_batch : cparams.n_threads;2495    ggml_threadpool_t tp = batched ? threadpool_batch        : threadpool;24962497    if (backend_cpu != nullptr) {2498        auto * reg = ggml_backend_dev_backend_reg(ggml_backend_get_device(backend_cpu));2499        auto * set_threadpool_fn = (decltype(ggml_backend_cpu_set_threadpool) *) ggml_backend_reg_get_proc_address(reg, "ggml_backend_cpu_set_threadpool");2500        if (set_threadpool_fn) {2501            set_threadpool_fn(backend_cpu, tp);2502        }2503    }25042505    // set the number of threads for all the backends2506    for (const auto & set_n_threads_fn : set_n_threads_fns) {2507        set_n_threads_fn.second(set_n_threads_fn.first, n_threads);2508    }25092510    auto status = ggml_backend_sched_graph_compute_async(sched.get(), gf);2511    if (status != GGML_STATUS_SUCCESS) {2512        LLAMA_LOG_ERROR("%s: ggml_backend_sched_graph_compute_async failed with error %d\n", __func__, status);2513    }25142515    // fprintf(stderr, "splits: %d\n", ggml_backend_sched_get_n_splits(sched));25162517    return status;2518}

llama.cpp implementation

llama.cpp/ggml/src/ggml-backend.cpp:1802–1838View code

1802        if (!sched->callback_eval) {1803            enum ggml_status ec = ggml_backend_graph_compute_async(split_backend, &split->graph);1804            if (ec != GGML_STATUS_SUCCESS) {1805                return ec;1806            }1807        } else {1808            // similar to ggml_backend_compare_graph_backend1809            for (int j0 = 0; j0 < split->graph.n_nodes; j0++) {1810                struct ggml_tensor * t = split->graph.nodes[j0];18111812                // check if the user needs data from this node1813                bool need = sched->callback_eval(t, true, sched->callback_eval_user_data);18141815                int j1 = j0;18161817                // determine the range [j0, j1] of nodes that can be computed together1818                while (!need && j1 < split->graph.n_nodes - 1) {1819                    t = split->graph.nodes[++j1];1820                    need = sched->callback_eval(t, true, sched->callback_eval_user_data);1821                }18221823                struct ggml_cgraph gv = ggml_graph_view(&split->graph, j0, j1 + 1);18241825                enum ggml_status ec = ggml_backend_graph_compute_async(split_backend, &gv);1826                if (ec != GGML_STATUS_SUCCESS) {1827                    return ec;1828                }18291830                // TODO: pass backend to the callback, then the user can decide if they want to synchronize1831                ggml_backend_synchronize(split_backend);18321833                if (need && !sched->callback_eval(t, false, sched->callback_eval_user_data)) {1834                    break;1835                }18361837                j0 = j1;1838            }

llama.cpp implementation

llama.cpp/ggml/src/ggml-backend.cpp:461–464View code

461enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph) {462    GGML_ASSERT(backend);463    return backend->iface.graph_compute(backend, cgraph);464}

STAGE 10

Raw logits → grammar / sampler stages → selected token

This driver applies its configured samplers after decode. llama_sampler_apply dispatches each stage; a grammar stage filters the candidate distribution.

PRIMARY CONSTRAINT LOCATION: grammar masks candidate tokens and other stages transform selection. Study syntactically allowed misuse here; grammar syntax does not establish semantic safety.

llama.cpp implementation

llama.cpp/src/llama-sampler.cpp:382–389View code

382void llama_sampler_apply(struct llama_sampler * smpl, struct llama_token_data_array * cur_p) {383    if (!smpl) {384        return;385    }386387    GGML_ASSERT(smpl->iface->apply);388    smpl->iface->apply(smpl, cur_p);389}

llama.cpp implementation

llama.cpp/src/llama-sampler.cpp:2680–2685View code

2680static void llama_sampler_grammar_apply(struct llama_sampler * smpl, llama_token_data_array * cur_p) {2681    auto * ctx = (llama_sampler_grammar *) smpl->ctx;2682    if (ctx->grammar) {2683        llama_grammar_apply_impl(*ctx->grammar, cur_p);2684    }2685}

STAGE 11

Selected token → output bytes → next decode ↺

llama_token_to_piece returns the token bytes. The caller records them and feeds the selected token into the next decode when another prediction is needed.

BEHAVIORAL EVIDENCE: inspect the selected output against both grammar and task requirements. Selection feeds back into the next model pass.

llama.cpp implementation

llama.cpp/src/llama-vocab.cpp:4421–4429View code

4421int32_t llama_token_to_piece(4422    const struct llama_vocab * vocab,4423                 llama_token   token,4424                        char * buf,4425                     int32_t   length,4426                     int32_t   lstrip,4427                        bool   special) {4428    return vocab->token_to_piece(token, buf, length, lstrip, special);4429}

Driver: record bytes and feed back the selected token

src/cst_capture.cpp:219–240View code

219            for(auto & s:samplers)llama_sampler_accept(s.get(),id);220            std::string text=piece(vocab,id);cst_log_token_bytes(text.data(),text.size());221            event("special_token",{{"token_id",id},{"is_eog",llama_vocab_is_eog(vocab,id)},222                  {"eos_raw_logprob",llama_vocab_eos(vocab)>=0?json(selected_logprob(raw,nv,llama_vocab_eos(vocab))):json(nullptr)}});223            if(t==0){double ttft=ms(trial_start);cst_log_timing(CST_TIMING_FIRST_TOKEN,ttft,1,ttft>0?1000.0/ttft:0);}224            cst_log_generated_token(id,raw,nv);++generated;prefix.push_back(id);225            if(llama_vocab_is_eog(vocab,id)){stop="eog";break;}226            std::cout<<text<<std::flush;227            if(t+1==max_tokens)break;228            cst_set_position(0,0,(int)prefix.size()-1,t+1,"decode");229            keys.push_back((int)prefix.size()-1);cst_set_attention_key_positions(keys.data(),(int)keys.size());230            start=Clock::now();decode_one(ctx.get(),id);elapsed=ms(start);231            cst_log_timing(CST_TIMING_DECODE,elapsed,1,elapsed>0?1000.0/elapsed:0);232            auto memory=llama_get_memory(ctx.get());233            event("cache_lifecycle",{{"operation","append"},{"min_position",llama_memory_seq_pos_min(memory,0)},234                  {"max_position",llama_memory_seq_pos_max(memory,0)},{"logical_tokens",keys.size()},235                  {"allocated_context_capacity",llama_n_ctx(ctx.get())}});236        }237        elapsed=ms(trial_start);cst_log_timing(CST_TIMING_TOTAL_TRIAL,elapsed,generated,elapsed>0?generated*1000.0/elapsed:0);238        event("generation_end",{{"stop_reason",stop},{"generated_tokens",generated},{"prompt_tokens",input.size()}});239        cst_close();std::cout<<"\n";return cst_last_error()?1:0;240    }catch(const std::exception & e){event("generation_end",{{"stop_reason","error"},{"message",e.what()}});cst_close();std::cerr<<e.what()<<"\n";return 1;}

Generation feedback: selected token → next decode → the same forward graph with updated context. The final EOG or token-limit token is recorded without another decode. Attack success must be established from the guide’s behavioral criterion; tensor differences alone do not establish it.

Guide index · Implementation and schema

LLM-facing idea

The model’s raw preference can differ from the distribution permitted by a grammar or sampling chain. A constrained output does not demonstrate that the unconstrained model preferred the same choice.

Controlled experiment

Run the same rendered prompt with a documented sampler chain. For a controlled comparison, change only the grammar or one sampler stage. Keep the seed fixed for paired stochastic runs, but do not assume identical seeds guarantee matching outputs after input changes.

Choose a small GBNF grammar with known allowable strings. Inspect both raw logits and each transformed stage so model behavior can be distinguished from enforced output syntax.

Relevant telemetry

Event	What to inspect
`sampler_chain`	Stage order, grammar text and configured seed.
`sampler_distribution`	Stage-local entropy, candidate list and selected probability when present.
`sampler_filter`	Raw-model mass of remaining finite-logit candidates.
`logit_distribution`, `sampling`	Raw preference versus terminal selection.

Capture with this codebase

The following is a configuration overlay, not a standalone configuration. Merge it into examples/capture.json, supply a real model and prompt, and compute any region spans from that prompt.

{
  "samplers": [
    {
      "type": "grammar",
      "grammar": "root ::= \"YES\" | \"NO\""
    },
    {
      "type": "temperature",
      "value": 0.7
    },
    {
      "type": "top_k",
      "value": 20
    },
    {
      "type": "dist",
      "seed": 42
    }
  ]
}

Reading the evidence

Compare a candidate’s raw probability with its availability after grammar filtering. The retained-mass statistic measures surviving finite-logit candidates in the raw distribution; it is not the amount of probability removed by temperature. Greedy terminal sampling is a point mass even when raw model entropy is high.

Output example

Illustrative event excerpt: omitted envelope and data fields are intentional. These values are not experiment results or detection thresholds. Complete event envelopes are documented in the implementation guide.

{
  "kind": "sampler_filter",
  "data": {
    "stage": "grammar",
    "raw_model_mass_retained": 0.08,
    "interpretation": "remaining finite candidates measured in raw model distribution"
  }
}

Implementation source

Expand a file-and-line reference to read the actual source. These excerpts are embedded snapshots from this checkout. They show the measurement implementation; the experiments and output examples above remain illustrative.

Construct and initialize the sampler stages

src/cst_capture.cpp:148–175View code

148        std::vector<Sampler> samplers;std::vector<std::string> stage_names;149        json stages=j.value("samplers",json::array({{{"type","greedy"}}}));150        if(stages.empty())throw std::runtime_error("empty sampler chain");151        for(size_t i=0;i<stages.size();++i){152            const auto & s=stages[i];std::string type=s.at("type");llama_sampler * p=nullptr;153            if(s.contains("value")) {154                double v=s.at("value");155                if(!std::isfinite(v) || (type=="temperature" && v<=0) ||156                   ((type=="top_p" || type=="min_p") && (v<=0 || v>1)) ||157                   (type=="top_k" && (v<1 || v>nv || std::floor(v)!=v)))throw std::runtime_error("invalid sampler value");158            }159            if(type=="penalties" && (s.value("last_n",64)<0 || s.value("repeat",1.0)<=0))throw std::runtime_error("invalid penalties");160            if(type=="greedy")p=llama_sampler_init_greedy();161            else if(type=="dist")p=llama_sampler_init_dist(s.value("seed",0u));162            else if(type=="temperature")p=llama_sampler_init_temp(s.at("value"));163            else if(type=="top_k")p=llama_sampler_init_top_k(s.at("value"));164            else if(type=="top_p")p=llama_sampler_init_top_p(s.at("value"),1);165            else if(type=="min_p")p=llama_sampler_init_min_p(s.at("value"),1);166            else if(type=="penalties")p=llama_sampler_init_penalties(nv,s.value("last_n",64),s.value("repeat",1.0f),s.value("frequency",0.0f),s.value("presence",0.0f));167            else if(type=="grammar")p=llama_sampler_init_grammar(vocab,s.at("grammar").get<std::string>().c_str(),s.value("root",std::string("root")).c_str());168            else throw std::runtime_error("unsupported sampler: "+type);169            if(!p)throw std::runtime_error("sampler init failed: "+type);170            samplers.emplace_back(p,llama_sampler_free);stage_names.push_back(type);171            if((type=="greedy" || type=="dist")!=(i+1==stages.size()))throw std::runtime_error("exactly one terminal greedy/dist sampler required");172        }173        event("sampler_chain",{{"stages",stages},{"rng_state","seed_and_accept_history; no opaque RNG snapshot"}});174        // Penalties see the prompt; grammar accepts generated tokens only.175        for(size_t s=0;s<samplers.size();++s)if(stage_names[s]=="penalties")for(auto id:input)llama_sampler_accept(samplers[s].get(),id);

Apply sampler stages and record retained raw probability mass

src/cst_capture.cpp:188–218View code

188            const float * raw=llama_get_logits_ith(ctx.get(),-1);if(!raw)throw std::runtime_error("missing logits");189            double raw_max=*std::max_element(raw,raw+nv),raw_z=0;190            for(int i=0;i<nv;++i)raw_z+=std::exp((double)raw[i]-raw_max);191            std::vector<llama_token_data> buffer(nv);for(int i=0;i<nv;++i)buffer[i]={i,raw[i],0};192            llama_token_data_array dist{buffer.data(),buffer.size(),-1,false};193            start=Clock::now();194            for(size_t s=0;s<samplers.size();++s){195                llama_sampler_apply(samplers[s].get(),&dist);196                if(!dist.size)throw std::runtime_error("sampler removed all candidates");197                bool any_finite=false;198                for(size_t i=0;i<dist.size;++i) {199                    if(std::isnan(dist.data[i].logit) || dist.data[i].logit==INFINITY)throw std::runtime_error("invalid sampler logits");200                    any_finite=any_finite || std::isfinite(dist.data[i].logit);201                }202                if(!any_finite)throw std::runtime_error("sampler masked all candidates");203                std::vector<int32_t> ids;std::vector<float> logits;204                for(size_t i=0;i<dist.size;++i){ids.push_back(dist.data[i].id);logits.push_back(dist.data[i].logit);}205                int selected=dist.selected>=0 && (size_t)dist.selected<dist.size?dist.data[dist.selected].id:-1;206                if(stage_names[s]=="greedy") {207                    int32_t only=selected;float zero=0;208                    cst_log_sampler_distribution("greedy_point_mass",&only,&zero,1,selected);209                } else cst_log_sampler_distribution(stage_names[s].c_str(),ids.data(),logits.data(),(int)ids.size(),selected);210                double retained_mass=0;211                for(size_t i=0;i<dist.size;++i)if(std::isfinite(dist.data[i].logit))retained_mass+=std::exp((double)raw[dist.data[i].id]-raw_max)/raw_z;212                event("sampler_filter",{{"stage",stage_names[s]},{"raw_model_mass_retained",retained_mass},213                      {"interpretation","remaining finite candidates measured in raw model distribution"}});214            }215            if(dist.selected<0 || (size_t)dist.selected>=dist.size)throw std::runtime_error("sampler did not select a token");216            auto id=dist.data[dist.selected].id;double sampling_ms=ms(start);217            event("sampling",{{"elapsed_ms",sampling_ms},{"selected_id",id},218                  {"selection_method",stage_names.back()},{"selected_probability",stage_names.back()=="greedy"?1.0:dist.data[dist.selected].p}});

Implementation status and limits

The documented sampler families are instrumented individually. Opaque RNG state serialization and arbitrary additional sampler families are not implemented. Grammar constrains syntax, not all semantic misuse. A sampler error or invalid configuration is an execution failure, not a refusal or successful defense.

All tensor-based interpretations depend on actual tensor_observed coverage and the tested model/backend. See validation results. There is no universal security score or calibrated threshold in this implementation.

Offline HTML with embedded source excerpts. Rebuild from the repository root with python3 scripts/build_security_docs.py after changing the Markdown or source files. Install scripts/requirements-docs.txt first; pass --llama-cpp-dir /path/to/llama.cpp for a different checkout location. The builder requires the pinned source revision. Review excerpt ranges if source lines move. No inference runs in this page.