LLM SECURITY TELEMETRY · SOURCE GUIDE

08. Cache state, retention and inference isolation

Where this concern meets the GGUF execution graph

The highlighted annotations locate attack entry, possible influence, constraints and observable effects for this concern. They are source-based hypotheses, not a trace of a demonstrated attack. GGUF supplies weights and metadata; llama.cpp builds and executes the computation graph.

Source snapshot: llama.cpp 5266f24da75dc449bd56cbed7addb9c8e4a6a73e. Concrete Llama architecture, ordinary generation path, non-fused attention as configured by this driver. Other architectures and fused kernels can use different graphs. All code below is embedded and expandable; no network or inference is required.

STAGE 01

GGUF → model weights, vocabulary and architecture

GGUF is a model file. The loader reads its metadata and tensors; C++ architecture code constructs the executable graph.

MODEL BASELINE: keep the GGUF identity and inference revision fixed. This concern does not demonstrate an attack on the GGUF loader or a modification of model weights.

llama.cpp implementation

llama.cpp/src/llama-model-loader.cpp:562–580View code
    if (!fname.empty()) {        // Load the main GGUF        struct ggml_context * ctx = NULL;        struct gguf_init_params params = {            /*.no_alloc = */ true,            /*.ctx      = */ &ctx,        };        metadata_ptr.reset(gguf_init_from_file(fname.c_str(), params));        metadata = metadata_ptr.get();        if (metadata == nullptr) {            throw std::runtime_error(format("%s: failed to load model from %s", __func__, fname.c_str()));        }        get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false);        llm_kv = LLM_KV(llm_arch_from_string(arch_name));        files.emplace_back(new llama_file(fname.c_str(), "rb", use_direct_io));        contexts.emplace_back(ctx);

llama.cpp implementation

llama.cpp/src/llama.cpp:343–371View code
        model->hparams.vocab_only = params.vocab_only;        model->hparams.no_alloc   = params.no_alloc;        try {            model->load_hparams(ml);        } catch(const std::exception & e) {            throw std::runtime_error("error loading model hyperparameters: " + std::string(e.what()));        }        if (model->arch == LLM_ARCH_CLIP) {            throw std::runtime_error("CLIP cannot be used as main model, use it with --mmproj instead");        }        try {            model->load_vocab(ml);        } catch(const std::exception & e) {            throw std::runtime_error("error loading model vocabulary: " + std::string(e.what()));        }        model->load_stats(ml);        model->print_info();        if (params.vocab_only) {            LLAMA_LOG_INFO("%s: vocab only - skipping tensors\n", __func__);            return {0, model_ptr.release()};        }        if (!model->load_tensors(ml)) {            return {-2, nullptr};        }
STAGE 02

Rendered text → token IDs

The caller supplies the final prompt. llama_tokenize delegates to the loaded vocabulary; tokenization itself does not execute the transformer.

POTENTIAL ENTRY: a separate integrating application could reuse or misassign inference state. The standalone driver supplies a fresh primary context and does not demonstrate this attack.

llama.cpp implementation

llama.cpp/src/llama-vocab.cpp:4410–4419View code
int32_t llama_tokenize(    const struct llama_vocab * vocab,                  const char * text,                     int32_t   text_len,                 llama_token * tokens,                     int32_t   n_tokens_max,                        bool   add_special,                        bool   parse_special) {    return vocab->tokenize(text, text_len, tokens, n_tokens_max, add_special, parse_special);}

Driver: read the rendered prompt

src/cst_capture.cpp:83–91View code
int main(int argc,char ** argv){    if(argc!=2){std::cerr<<"Usage: cst-capture config.json\n";return 2;}    try {        json j=json::parse(read_file(argv[1]));        std::string model_path=j.at("model"),prompt=read_file(j.at("prompt_file"));        std::string log=j.value("log_path",std::string("trial.cstbin")),jsonl=j.value("jsonl_path",std::string("trial.telemetry.jsonl"));        std::string run=j.value("run_id",std::string("local"));        int max_tokens=j.value("max_tokens",32),nctx=j.value("context_size",4096),stride=j.value("score_stride",1);        if(max_tokens<=0 || nctx<=0 || stride<=0)throw std::runtime_error("invalid token/context/score limits");
STAGE 03

Decode → build or reuse graph → set current inputs

process_ubatch applies memory state, builds or reuses the graph, sets new inputs and requests graph computation. Reusing topology still computes the new input.

EXECUTION ENTRY: the current prefix and context state reach the model here. Building or reusing topology is not itself an attack; the relevant influence is identified in the stages below.

llama.cpp implementation

llama.cpp/src/llama-context.cpp:1334–1403View code
llm_graph_result * llama_context::process_ubatch(const llama_ubatch & ubatch, llm_graph_type gtype, llama_memory_context_i * mctx, ggml_status & ret) {    if (mctx && !mctx->apply()) {        LLAMA_LOG_ERROR("%s: failed to apply memory context\n", __func__);        ret = GGML_STATUS_FAILED;        return nullptr;    }    auto * res = gf_res_prev.get();    auto * gf  = res->get_gf();    // the new graph parameters    // in order to correctly reuse a graph, it's full topology has to be uniquely determined by these parameters    const auto gparams = graph_params(res, ubatch, mctx, gtype);    if (!graph_reuse_disable && res->can_reuse(gparams)) {        //LLAMA_LOG_DEBUG("%s: reusing previous graph\n", __func__);        // with pipeline parallelism, the previous graph_compute_async may still be running        // on the GPU. we must synchronize before set_inputs to avoid overwriting input tensors        // that the previous compute is still reading.        if (cparams.pipeline_parallel) {            ggml_backend_sched_synchronize(sched.get());        }        n_reused++;    } else {        res->reset();        ggml_backend_sched_reset(sched.get());        ggml_backend_sched_set_eval_callback(sched.get(), cparams.cb_eval, cparams.cb_eval_user_data);        //const auto t_start_us = ggml_time_us();        gf = model.build_graph(gparams);        //LLAMA_LOG_INFO("graph build time: %.3f ms\n", (ggml_time_us() - t_start_us)/1000.0);        if (!gf) {            LLAMA_LOG_ERROR("%s: failed to initialize graph\n", __func__);            ret = GGML_STATUS_FAILED;            return nullptr;        }        if (!ggml_backend_sched_alloc_graph(sched.get(), gf)) {            LLAMA_LOG_ERROR("%s: failed to allocate graph\n", __func__);            ret = GGML_STATUS_ALLOC_FAILED;            return nullptr;        }    }    // set the input data for the input tensors    {        //const auto t_start_us = ggml_time_us();        // FIXME this call causes a crash if any model inputs were not used in the graph and were therefore not allocated        res->set_inputs(&ubatch);        //LLAMA_LOG_INFO("graph set inputs time: %.3f ms\n", (ggml_time_us() - t_start_us)/1000.0);    }    const auto status = graph_compute(res->get_gf(), ubatch.n_tokens > 1);    if (status != GGML_STATUS_SUCCESS) {        LLAMA_LOG_ERROR("%s: failed to compute graph, compute status: %d\n", __func__, status);        ret = status;        return nullptr;    }    ret = GGML_STATUS_SUCCESS;    return res;

llama.cpp implementation

llama.cpp/src/llama-model.cpp:2673–2690View code
ggml_cgraph * llama_model::build_graph(const llm_graph_params & params) const {    std::unique_ptr<llm_graph_context> llm = build_arch_graph(params);    // add on pooling layer    llm->build_pooling(cls, cls_b, cls_out, cls_out_b, cls_norm);    // add backend sampling layers (if any)    llm->build_sampling();    // if the gguf model was converted with --sentence-transformers-dense-modules    // there will be two additional dense projection layers    // dense linear projections are applied after pooling    // TODO: move reranking logic here and generalize    llm->build_dense_out(dense_2_out_layers, dense_2_out_layers_b, dense_3_out_layers);    llm->res->set_outputs(params);    return llm->res->get_gf();

Driver: decode each prompt token

src/cst_capture.cpp:176–182View code
        cst_set_position(0,0,0,0,"prefill");cst_log_prompt(input.data(),(int)input.size(),prompt.c_str());        auto trial_start=Clock::now(),start=trial_start;        std::vector<int32_t> keys;        for(size_t i=0;i<input.size();++i){            cst_set_position(0,0,(int)i,0,"prefill");keys.push_back((int)i);cst_set_attention_key_positions(keys.data(),(int)keys.size());            decode_one(ctx.get(),input[i]);        }

Inside the Llama forward graph

Tensor dependency order: embeddings → per-layer attention and FFN → final logits. K/V, attention and FFN repeat for each layer. The backend executes these operations during decode; graph construction alone does not compute their values.

STAGE 04

Token embeddings + positions

Token IDs select rows of the GGUF embedding matrix. The Llama graph supplies positions and enters its transformer-layer loop.

CONDITIONAL INPUT: a state or external-system attack need not add new prompt tokens. Embedding lookup alone cannot establish cache isolation, authorization or external effects.

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2303–2328View code
ggml_tensor * llm_graph_context::build_inp_embd(ggml_tensor * tok_embd) const {    const int64_t n_embd_inp = hparams.n_embd_inp();    const int64_t n_embd     = hparams.n_embd;    assert(n_embd_inp >= n_embd);    auto inp = std::make_unique<llm_graph_input_embd>(n_embd_inp);    inp->tokens = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, ubatch.n_tokens);    cb(inp->tokens, "inp_tokens", -1);    ggml_set_input(inp->tokens);    res->t_inp_tokens = inp->tokens;    inp->embd = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_embd_inp, ubatch.n_tokens);    cb(inp->embd, "inp_embd", -1);    ggml_set_input(inp->embd);    // select one of the 2 inputs, based on the batch contents    // ref: https://github.com/ggml-org/llama.cpp/pull/18550    std::array<ggml_tensor *, 2> inps;    // token embeddings path (ubatch.token != nullptr)    {        auto & cur = inps[0];        cur = ggml_get_rows(ctx0, tok_embd, inp->tokens);

llama.cpp implementation

llama.cpp/src/models/llama.cpp:108–131View code
    inpL = build_inp_embd(model.tok_embd);    // inp_pos - contains the positions    ggml_tensor * inp_pos = build_inp_pos();    using inp_attn_type = std::conditional_t<embed, llm_graph_input_attn_no_cache, llm_graph_input_attn_kv>;    inp_attn_type * inp_attn = nullptr;    if constexpr (embed) {        inp_attn = build_attn_inp_no_cache();    } else {        inp_attn = build_attn_inp_kv();    }    const float kq_scale = hparams.f_attention_scale == 0.0f ? 1.0f/sqrtf(float(n_embd_head)) : hparams.f_attention_scale;    ggml_tensor * inp_out_ids = build_inp_out_ids();    for (int il = 0; il < n_layer; ++il) {        res->t_layer_inp[il] = inpL;        ggml_tensor * inpSA = inpL;        // norm
STAGE 05

Per layer: Q/K/V + position encoding → KV cache

The graph computes Q/K/V, applies RoPE and adds K/V cache writes. Attention reads the available cache with a position mask.

PRIMARY STATE LOCATION: cache writes, reads and position mapping determine which retained K/V entries are available. Cross-sequence misuse would require a separate integration; this adapter is single-sequence.

llama.cpp implementation

llama.cpp/src/models/llama.cpp:139–161View code
            // rope freq factors for llama3; may return nullptr for llama2 and other models            ggml_tensor * rope_factors = model.get_rope_factors(cparams, il);            // compute Q and K and RoPE them            auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur,                    n_embd_head, n_head, n_head_kv, il);            Qcur = ggml_rope_ext(                    ctx0, Qcur, inp_pos, rope_factors,                    n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,                    ext_factor, attn_factor, beta_fast, beta_slow                    );            Kcur = ggml_rope_ext(                    ctx0, Kcur, inp_pos, rope_factors,                    n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,                    ext_factor, attn_factor, beta_fast, beta_slow                    );            cb(Qcur, "Qcur", il);            cb(Kcur, "Kcur", il);            cb(Vcur, "Vcur", il);

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2815–2833View code
    const auto * mctx_cur = inp->mctx;    // store to KV cache    {        const auto & k_idxs = inp->get_k_idxs();        const auto & v_idxs = inp->get_v_idxs();        ggml_build_forward_expand(gf, mctx_cur->cpy_k(ctx0, k_cur, k_idxs, il));        ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));    }    ggml_tensor * kq_mask = inp->get_kq_mask();    ggml_tensor * q = q_cur;    ggml_tensor * k = mctx_cur->get_k(ctx0, il);    ggml_tensor * v = mctx_cur->get_v(ctx0, il);    ggml_tensor * cur = build_attn_mha(q, k, v, kq_b, kq_mask, sinks, v_mla, 0, kq_scale, il);    cb(cur, "kqv_out", il);
STAGE 06

Per layer: QK scores → masked softmax → weighted V

On the non-fused path, kq_soft_max contains attention probabilities. Those weights mix value vectors into the attention output.

POSSIBLE INFLUENCE: an incorrectly retained key/value or mask could expose unwanted context to attention. The current driver’s linear mapping is not evidence of arbitrary mutation correctness.

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2605–2610View code
        ggml_tensor * kq = ggml_mul_mat(ctx0, k, q);        cb(kq, "kq", il);        // note: this op tends to require high floating point range        //       while for some models F16 is enough, for others it is not, so we default to F32 here        ggml_mul_mat_set_prec(kq, GGML_PREC_F32);

llama.cpp implementation

llama.cpp/src/llama-graph.cpp:2633–2650View code
        if (kq_b) {            kq = ggml_add(ctx0, kq, kq_b);            cb(kq, "kq_plus_kq_b", il);        }        kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, hparams.f_max_alibi_bias);        ggml_soft_max_add_sinks(kq, sinks);        cb(kq, "kq_soft_max", il);        if (!v_trans) {            // note: avoid this branch            v = ggml_cont(ctx0, ggml_transpose(ctx0, v));            cb(v, "v_cont", il);        }        ggml_tensor * kqv = ggml_mul_mat(ctx0, v, kq);        cb(kqv, "kqv", il);
STAGE 07

Per layer: attention residual → FFN → layer output

Attention output is added to the residual, transformed by the FFN, then added again. l_out feeds the next layer; the layer loop repeats.

POSSIBLE INFLUENCE: unwanted cache content, if made visible, would affect downstream representations. Projected K/V summaries are not stored cache-row dumps.

llama.cpp implementation

llama.cpp/src/models/llama.cpp:173–195View code
        }        if (il == n_layer - 1 && inp_out_ids) {            cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids);        }        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);        cb(ffn_inp, "ffn_inp", il);        // feed-forward network (non-MoE)        if (model.layers[il].ffn_gate_inp == nullptr) {            cur = build_norm(ffn_inp,                    model.layers[il].ffn_norm, NULL,                    LLM_NORM_RMS, il);            cb(cur, "ffn_norm", il);            cur = build_ffn(cur,                    model.layers[il].ffn_up,   model.layers[il].ffn_up_b,   model.layers[il].ffn_up_s,                    model.layers[il].ffn_gate, model.layers[il].ffn_gate_b, model.layers[il].ffn_gate_s,                    model.layers[il].ffn_down, model.layers[il].ffn_down_b, model.layers[il].ffn_down_s,                    NULL,                    LLM_FFN_SILU, LLM_FFN_PAR, il);            cb(cur, "ffn_out", il);

llama.cpp implementation

llama.cpp/src/models/llama.cpp:220–228View code
        cur = ggml_add(ctx0, cur, ffn_inp);        cb(cur, "ffn_out", il);        cur = build_cvec(cur, il);        cb(cur, "l_out", il);        // input for next layer        inpL = cur;    }
STAGE 08

Final normalization → vocabulary logits

After the final layer, output normalization and the GGUF output projection produce result_output / t_logits.

OBSERVE: compare primary distributions with auxiliary scoring enabled and disabled. The smoke test checks that candidate replay leaves primary predictions unchanged.

llama.cpp implementation

llama.cpp/src/models/llama.cpp:230–246View code
    cur = build_norm(cur,            model.output_norm, NULL,            LLM_NORM_RMS, -1);    cb(cur, "result_norm", -1);    res->t_embd = cur;    if constexpr (!embed) {        // lm_head        cur = build_lora_mm(model.output, cur, model.output_s);        cb(cur, "result_output", -1);        res->t_logits = cur;    }    ggml_build_forward_expand(gf, cur);
STAGE 09

Backend executes these graph operations; callbacks observe tensors

The preceding boxes describe tensor dependencies. Numerics execute through this scheduler, which computes graph ranges, synchronizes and invokes requested observation callbacks during execution.

Cache context application and backend execution are the relevant mechanisms. Logical append records and caller-supplied cache statistics do not prove isolation across unsupported integrations.

llama.cpp implementation

llama.cpp/src/llama-context.cpp:2491–2518View code
ggml_status llama_context::graph_compute(            ggml_cgraph * gf,                   bool   batched) {    int n_threads        = batched ? cparams.n_threads_batch : cparams.n_threads;    ggml_threadpool_t tp = batched ? threadpool_batch        : threadpool;    if (backend_cpu != nullptr) {        auto * reg = ggml_backend_dev_backend_reg(ggml_backend_get_device(backend_cpu));        auto * set_threadpool_fn = (decltype(ggml_backend_cpu_set_threadpool) *) ggml_backend_reg_get_proc_address(reg, "ggml_backend_cpu_set_threadpool");        if (set_threadpool_fn) {            set_threadpool_fn(backend_cpu, tp);        }    }    // set the number of threads for all the backends    for (const auto & set_n_threads_fn : set_n_threads_fns) {        set_n_threads_fn.second(set_n_threads_fn.first, n_threads);    }    auto status = ggml_backend_sched_graph_compute_async(sched.get(), gf);    if (status != GGML_STATUS_SUCCESS) {        LLAMA_LOG_ERROR("%s: ggml_backend_sched_graph_compute_async failed with error %d\n", __func__, status);    }    // fprintf(stderr, "splits: %d\n", ggml_backend_sched_get_n_splits(sched));    return status;}

llama.cpp implementation

llama.cpp/ggml/src/ggml-backend.cpp:1802–1838View code
        if (!sched->callback_eval) {            enum ggml_status ec = ggml_backend_graph_compute_async(split_backend, &split->graph);            if (ec != GGML_STATUS_SUCCESS) {                return ec;            }        } else {            // similar to ggml_backend_compare_graph_backend            for (int j0 = 0; j0 < split->graph.n_nodes; j0++) {                struct ggml_tensor * t = split->graph.nodes[j0];                // check if the user needs data from this node                bool need = sched->callback_eval(t, true, sched->callback_eval_user_data);                int j1 = j0;                // determine the range [j0, j1] of nodes that can be computed together                while (!need && j1 < split->graph.n_nodes - 1) {                    t = split->graph.nodes[++j1];                    need = sched->callback_eval(t, true, sched->callback_eval_user_data);                }                struct ggml_cgraph gv = ggml_graph_view(&split->graph, j0, j1 + 1);                enum ggml_status ec = ggml_backend_graph_compute_async(split_backend, &gv);                if (ec != GGML_STATUS_SUCCESS) {                    return ec;                }                // TODO: pass backend to the callback, then the user can decide if they want to synchronize                ggml_backend_synchronize(split_backend);                if (need && !sched->callback_eval(t, false, sched->callback_eval_user_data)) {                    break;                }                j0 = j1;            }

llama.cpp implementation

llama.cpp/ggml/src/ggml-backend.cpp:461–464View code
enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph) {    GGML_ASSERT(backend);    return backend->iface.graph_compute(backend, cgraph);}
STAGE 10

Raw logits → grammar / sampler stages → selected token

This driver applies its configured samplers after decode. llama_sampler_apply dispatches each stage; a grammar stage filters the candidate distribution.

Hold the sampler settings fixed during isolation comparisons. Primary sampler history is separate from the auxiliary candidate replay.

llama.cpp implementation

llama.cpp/src/llama-sampler.cpp:382–389View code
void llama_sampler_apply(struct llama_sampler * smpl, struct llama_token_data_array * cur_p) {    if (!smpl) {        return;    }    GGML_ASSERT(smpl->iface->apply);    smpl->iface->apply(smpl, cur_p);}

llama.cpp implementation

llama.cpp/src/llama-sampler.cpp:2680–2685View code
static void llama_sampler_grammar_apply(struct llama_sampler * smpl, llama_token_data_array * cur_p) {    auto * ctx = (llama_sampler_grammar *) smpl->ctx;    if (ctx->grammar) {        llama_grammar_apply_impl(*ctx->grammar, cur_p);    }}
STAGE 11

Selected token → output bytes → next decode ↺

llama_token_to_piece returns the token bytes. The caller records them and feeds the selected token into the next decode when another prediction is needed.

BEHAVIORAL EVIDENCE: unexpected output may motivate a state audit; output alone cannot identify a cache ownership or retention bug.

llama.cpp implementation

llama.cpp/src/llama-vocab.cpp:4421–4429View code
int32_t llama_token_to_piece(    const struct llama_vocab * vocab,                 llama_token   token,                        char * buf,                     int32_t   length,                     int32_t   lstrip,                        bool   special) {    return vocab->token_to_piece(token, buf, length, lstrip, special);}

Driver: record bytes and feed back the selected token

src/cst_capture.cpp:219–240View code
            for(auto & s:samplers)llama_sampler_accept(s.get(),id);            std::string text=piece(vocab,id);cst_log_token_bytes(text.data(),text.size());            event("special_token",{{"token_id",id},{"is_eog",llama_vocab_is_eog(vocab,id)},                  {"eos_raw_logprob",llama_vocab_eos(vocab)>=0?json(selected_logprob(raw,nv,llama_vocab_eos(vocab))):json(nullptr)}});            if(t==0){double ttft=ms(trial_start);cst_log_timing(CST_TIMING_FIRST_TOKEN,ttft,1,ttft>0?1000.0/ttft:0);}            cst_log_generated_token(id,raw,nv);++generated;prefix.push_back(id);            if(llama_vocab_is_eog(vocab,id)){stop="eog";break;}            std::cout<<text<<std::flush;            if(t+1==max_tokens)break;            cst_set_position(0,0,(int)prefix.size()-1,t+1,"decode");            keys.push_back((int)prefix.size()-1);cst_set_attention_key_positions(keys.data(),(int)keys.size());            start=Clock::now();decode_one(ctx.get(),id);elapsed=ms(start);            cst_log_timing(CST_TIMING_DECODE,elapsed,1,elapsed>0?1000.0/elapsed:0);            auto memory=llama_get_memory(ctx.get());            event("cache_lifecycle",{{"operation","append"},{"min_position",llama_memory_seq_pos_min(memory,0)},                  {"max_position",llama_memory_seq_pos_max(memory,0)},{"logical_tokens",keys.size()},                  {"allocated_context_capacity",llama_n_ctx(ctx.get())}});        }        elapsed=ms(trial_start);cst_log_timing(CST_TIMING_TOTAL_TRIAL,elapsed,generated,elapsed>0?generated*1000.0/elapsed:0);        event("generation_end",{{"stop_reason",stop},{"generated_tokens",generated},{"prompt_tokens",input.size()}});        cst_close();std::cout<<"\n";return cst_last_error()?1:0;    }catch(const std::exception & e){event("generation_end",{{"stop_reason","error"},{"message",e.what()}});cst_close();std::cerr<<e.what()<<"\n";return 1;}

Generation feedback: selected token → next decode → the same forward graph with updated context. The final EOG or token-limit token is recorded without another decode. Attack success must be established from the guide’s behavioral criterion; tensor differences alone do not establish it.

Guide index · Implementation and schema

LLM-facing idea

An integrating application may retain, reuse or misassign inference state. The LLM-facing questions are which positions remain in context and whether auxiliary computation alters the primary prediction.

Controlled experiment

For the standalone executable, inspect logical cache append records and compare primary distributions with and without candidate scoring. The supplied real-model smoke-test script performs this isolation comparison.

For a separate application that reuses contexts, record state mutations through the C API and update attention key mappings after every mutation. Such lifecycle instrumentation is additional integration work; the current driver does not perform cache reuse or context shifting.

Relevant telemetry

Event What to inspect
cache_lifecycle Logical sequence position range, append operation and context capacity.
kv_projection Projected K/V activation summaries observed in the graph.
logit_distribution Primary prediction comparison with auxiliary scoring enabled/disabled.
position, signal_unavailable Position metadata and explicit unsupported captures.

Capture with this codebase

The following is a configuration overlay, not a standalone configuration. Merge it into examples/capture.json, supply a real model and prompt, and compute any region spans from that prompt.

{
  "capture_kv_projections": true,
  "kv_stride": 1
}

Reading the evidence

Verify primary predictions using corresponding raw distributions, not timing or text alone. Treat logical token count separately from allocated context capacity. Projected K/V records may repeat for multiple graph operations with the same name; they do not count unique cache writes.

Output example

Illustrative event excerpt: omitted envelope and data fields are intentional. These values are not experiment results or detection thresholds. Complete event envelopes are documented in the implementation guide.

{
  "kind": "cache_lifecycle",
  "data": {
    "operation": "append",
    "min_position": 0,
    "max_position": 6,
    "logical_tokens": 7,
    "allocated_context_capacity": 256
  }
}

Implementation source

Expand a file-and-line reference to read the actual source. These excerpts are embedded snapshots from this checkout. They show the measurement implementation; the experiments and output examples above remain illustrative.

Register patterns and candidate strings; create auxiliary context

src/cst_capture.cpp:138–146View code
        for(const auto & p:j.value("patterns",json::array())) {std::string name=p.at("name"),cat=p.value("category",std::string("canary")),s=p.at("text");cst_register_pattern(name.c_str(),cat.c_str(),s.data(),s.size());}        std::vector<Candidate> candidates;        for(const auto & c:j.value("candidates",json::array())) {            Candidate item{c.at("name"),c.value("category",std::string("tool")),tokenize(vocab,c.at("text"),false),tokenize(vocab,c.value("anchor",std::string("")),false)};            if(item.tokens.empty() || input.size()+max_tokens+item.tokens.size()>(size_t)nctx)throw std::runtime_error("candidate empty or exceeds auxiliary context capacity");            candidates.push_back(std::move(item));        }        Context auxiliary(nullptr,llama_free);        if(!candidates.empty()){auto acp=cp;acp.cb_eval=nullptr;auxiliary.reset(llama_init_from_model(model.get(),acp));if(!auxiliary)throw std::runtime_error("auxiliary context init failed");}

Auxiliary candidate replay and relative scores

src/cst_capture.cpp:59–82View code
static void score_candidates(llama_context * auxiliary,int nv,const std::vector<llama_token>& prefix,const std::vector<Candidate>& cs){    json relative=json::array();double total=0;    for(const auto & c:cs){        if(!suffix_match(prefix,c.anchor))continue;        auto start=Clock::now();llama_memory_clear(llama_get_memory(auxiliary),true);        for(auto id:prefix)decode_one(auxiliary,id);        std::vector<double> lp;        for(size_t i=0;i<c.tokens.size();++i){            lp.push_back(selected_logprob(llama_get_logits_ith(auxiliary,-1),nv,c.tokens[i]));            if(i+1<c.tokens.size())decode_one(auxiliary,c.tokens[i]);        }        cst_log_candidate_likelihood(c.name.c_str(),c.tokens.data(),lp.data(),(int)lp.size());        double joint=std::accumulate(lp.begin(),lp.end(),0.0);        relative.push_back({{"name",c.name},{"category",c.category},{"sequence_logprob",joint}});        total+=ms(start);    }    // These are scores over the supplied list, not disjoint semantic event probabilities.    if(!relative.empty()){        double m=-INFINITY;for(auto & x:relative)m=std::max(m,x["sequence_logprob"].get<double>());        double z=0;for(auto & x:relative)z+=std::exp(x["sequence_logprob"].get<double>()-m);        if(std::isfinite(m))for(auto & x:relative)x["relative_score"]=std::exp(x["sequence_logprob"].get<double>()-m)/z;        event("candidate_relative_scores",{{"entries",relative},{"interpretation","normalized_over_supplied_strings"},{"auxiliary_ms",total}});    }}

K/V projections and caller-supplied cache statistics

src/cst_core.cpp:354–363View code
extern "C" void cst_log_kv_projection(uint16_t layer,const char * kind,const float * x,int n) {    if(!config.emit_kv_cache_state || position%(uint32_t)config.kv_cache_stride)return;    if(!valid_vector(x,n)){invalid("invalid KV projection");return;}    auto j=stats(x,n);j["layer"]=layer;j["tensor_kind"]=kind?kind:"";j["interpretation"]="projected_activation_not_stored_cache";emit("kv_projection",j);}extern "C" void cst_log_kv_cache_state(uint16_t layer,uint32_t len,float k,float v,float h) {    if(!config.emit_kv_cache_state || position%(uint32_t)config.kv_cache_stride)return;    emit("kv_cache_state",{{"layer",layer},{"cache_len_tokens",len},{"k_l2_last_row",k},{"v_l2_last_row",v},                           {"attention_entropy_bits",h},{"source","caller_supplied"}});}

Generated bytes, stopping, decode timing and logical cache appends

src/cst_capture.cpp:219–240View code
            for(auto & s:samplers)llama_sampler_accept(s.get(),id);            std::string text=piece(vocab,id);cst_log_token_bytes(text.data(),text.size());            event("special_token",{{"token_id",id},{"is_eog",llama_vocab_is_eog(vocab,id)},                  {"eos_raw_logprob",llama_vocab_eos(vocab)>=0?json(selected_logprob(raw,nv,llama_vocab_eos(vocab))):json(nullptr)}});            if(t==0){double ttft=ms(trial_start);cst_log_timing(CST_TIMING_FIRST_TOKEN,ttft,1,ttft>0?1000.0/ttft:0);}            cst_log_generated_token(id,raw,nv);++generated;prefix.push_back(id);            if(llama_vocab_is_eog(vocab,id)){stop="eog";break;}            std::cout<<text<<std::flush;            if(t+1==max_tokens)break;            cst_set_position(0,0,(int)prefix.size()-1,t+1,"decode");            keys.push_back((int)prefix.size()-1);cst_set_attention_key_positions(keys.data(),(int)keys.size());            start=Clock::now();decode_one(ctx.get(),id);elapsed=ms(start);            cst_log_timing(CST_TIMING_DECODE,elapsed,1,elapsed>0?1000.0/elapsed:0);            auto memory=llama_get_memory(ctx.get());            event("cache_lifecycle",{{"operation","append"},{"min_position",llama_memory_seq_pos_min(memory,0)},                  {"max_position",llama_memory_seq_pos_max(memory,0)},{"logical_tokens",keys.size()},                  {"allocated_context_capacity",llama_n_ctx(ctx.get())}});        }        elapsed=ms(trial_start);cst_log_timing(CST_TIMING_TOTAL_TRIAL,elapsed,generated,elapsed>0?generated*1000.0/elapsed:0);        event("generation_end",{{"stop_reason",stop},{"generated_tokens",generated},{"prompt_tokens",input.size()}});        cst_close();std::cout<<"\n";return cst_last_error()?1:0;    }catch(const std::exception & e){event("generation_end",{{"stop_reason","error"},{"message",e.what()}});cst_close();std::cerr<<e.what()<<"\n";return 1;}

Real-model smoke-test isolation comparison

scripts/smoke_test.py:1–33View code
"""Optional real-model integration test: python scripts/smoke_test.py build/cst-capture model.gguf."""import jsonfrom pathlib import Pathimport subprocessimport sysimport tempfileexe = str(Path(sys.argv[1]).resolve())model = str(Path(sys.argv[2]).resolve())with tempfile.TemporaryDirectory(prefix='cst-smoke-') as temp:    root = Path(temp)    prompt = root / 'prompt.txt'    prompt.write_text('The capital of France is')    base = {'model': model, 'prompt_file': str(prompt), 'context_size': 256,            'max_tokens': 2, 'hidden_layers': [0], 'projection_dims': 4,            'regions': [{'start_token': 0, 'end_token': 1, 'region_type': 1, 'trust_level': 1}]}    def run(name, extra):        config = dict(base, **extra, log_path=str(root / (name+'.bin')),                      jsonl_path=str(root / (name+'.jsonl')))        path = root / (name+'.json')        path.write_text(json.dumps(config))        subprocess.run([exe, str(path)], check=True, stdout=subprocess.DEVNULL)        return [json.loads(line) for line in (root / (name+'.jsonl')).read_text().splitlines()]    baseline = run('baseline', {})    scored = run('scored', {'candidates': [{'name': 'Paris', 'text': ' Paris'}], 'score_stride': 2})    def distributions(events):        return [r['data'] for r in events if r['kind'] == 'logit_distribution']    assert distributions(baseline) == distributions(scored), 'candidate scoring changed primary output'    assert any(r['kind'] == 'hidden_state' for r in scored)    assert any(r['kind'] == 'attention' for r in scored)    assert any(r['kind'] == 'candidate_likelihood' for r in scored)    assert not any(r['kind'] == 'capture_error' for r in scored)    print('PASS: real inference, hidden/attention capture, and candidate isolation')

Implementation status and limits

Stored or quantized K/V rows are not automatically extracted. cst_log_kv_cache_state accepts caller-computed statistics; calling it is not itself measurement. Concurrent sequences, recurrent state and arbitrary cache mutation mapping are unsupported by the current adapter. The driver’s auxiliary scoring context has no telemetry callback and is cleared/replayed per candidate.

All tensor-based interpretations depend on actual tensor_observed coverage and the tested model/backend. See validation results. There is no universal security score or calibrated threshold in this implementation.