Context is measured, not estimated: a pre-flight count, a trim that keeps turns whole, a cache plan by capability¶
Context and problem statement¶
A long conversation, or a prompt with a large document, can exceed the model's window. The wire
then refuses it (context_overflow), after the call was queued and traced, and a caller that
retried the same request got the same refusal. A characters-divided-by-four estimate is wrong
by tens of percent for code, JSON and non-English text, so a guard built on it refuses requests
that fit and passes requests that do not: a guard that looks armed and is not.
Prompt caching has the same shape: a hosted model caches a prefix up to a breakpoint, a
llama.cpp server caches by slot with cache_prompt, and a model with neither silently ignores a
hint, so a caller cannot tell whether the hint did anything.
Decision¶
- A
TokenCounterport returns an exact count or None.Counted(tokens, method, failure):provider(the provider's own count API),tokenizer(the model's tokenizer), orunsupported. It never returns an estimate.adapters.http.HttpTokenCounterspeaks LiteLLM's/utils/token_countershape; theopenai_compatiblefactory builds one for every model whose table entry statescount_tokens. Role.contextchooses the pre-flight.off(the default) counts nothing.checkrefuses an overflow before the wire:request_fault/context_overflow,provider_code="preflight", zero generation requests.trimcuts the history and counts again to confirm; when the first cut was not enough it cuts every turn the policy allows, and refuses what still does not fit. The budget is the table'scontext_windowless the role's reply ceiling (max_tokens, else the model'smax_output) and a 5 % margin. Bothcall.generateandcall.chatrun it.- Unknown sends. No counter, a model that cannot be counted, a failed count request or no
window in the table is an
unknownverdict: the request is sent, the result carries aCallWarning, andllm_call_warningfires once per model. A missing optimisation is never an outage. - Trimming keeps turns whole. Tool outputs over
TrimPolicy.tool_output_charsare cut in the middle first; then whole turns (a user message and everything to the next one) are dropped oldest-first betweenkeep_firstandkeep_last. A tool call is never separated from its result. - Every count is reported.
stamp["count_requests"],stamp["trimmed"], and the eventsllm_count_failed,llm_trimmed,llm_context_overflow(where=preflightorwire) andllm_count_optimistic(the pre-flight said it fits; the wire said it did not). - The cache hint follows the stated capability.
Role.cache_prefixon a model withoutcacheis dropped with aCallWarning; on a self-hosted model withcacheit becomesextra_body.cache_prompt; on a hosted model it becomes a pydantic-aiCachePointafter the system prompt and tools.context.volatile_systemlints a pack whose system text would defeat the breakpoint.
Considered options¶
- tiktoken locally. Rejected as the default: exact only for OpenAI's own encodings, and wrong
for the self-hosted models this package serves; a consumer can still wrap it in a
TokenCounterand statetokenizer. - Refuse on unknown. Rejected: the guard would turn every uncounted model into an outage.
Consequences¶
- A count request adds one round trip per call when the role opts in;
offremains the default. - Whether a given gateway forwards
/utils/token_counterfor a given model is the table's statement (count_tokens), checked bytesting.conformance.check_token_counter.