Skip to content

llm_ports.call

llm_ports.call

The call paths: generate, chat and stream, their hooks, roles, per-call graders, output schemas, retry and skip.

Submodules: context (budgets, trimming, the cache plan), roles, models (what a model can do, who serves it), providers (providers by name, build), graders, schema, and resilience (the retry and skip mechanics of a long run).

TurnOptions

Bases: TypedDict

The optional keywords stream and chat share; each one omitted takes the default named.

tools instance-attribute

tools: Sequence[Tool]

The tools offered to the model (default none).

hooks instance-attribute

hooks: Hooks | None

Trace, admit, graders, report and event hooks (default Hooks()).

system instance-attribute

system: str

The system prompt (default empty).

settings instance-attribute

settings: Mapping[str, Any] | None

Model settings merged over the role's (default the role's).

metadata instance-attribute

metadata: Mapping[str, Any] | None

Extra trace metadata (default none).

max_steps instance-attribute

max_steps: int | None

Overrides Role.max_steps for this call (default the role's).

counter instance-attribute

counter: TokenCounter | None

Measures the history for the pre-flight (Role.context); None counts nothing.

Hooks dataclass

Hooks(
    admit: AdmitHook | None = None,
    trace: TraceHook | None = None,
    graders: Mapping[str, CallGrader] | None = None,
    on_scores: (
        Callable[
            [Role, tuple[CallScore, ...]], Awaitable[Any]
        ]
        | None
    ) = None,
    on_ok: (
        Callable[[Role, str, Generated[Any]], Any] | None
    ) = None,
    on_event: EventHook | None = None,
    generator_for: Callable[[str], Generator] | None = None,
)

Optional consumer hooks for generate.

Attributes:

  • admit (AdmitHook | None) –

    Runs before any trace opens; a returned reason makes the outcome gated.

  • trace (TraceHook | None) –

    Opens the call's observation; without one, a sink-less tracer is used.

  • graders (Mapping[str, CallGrader] | None) –

    The grader table by name; None means call.graders.GRADERS.

  • on_scores (Callable[[Role, tuple[CallScore, ...]], Awaitable[Any]] | None) –

    Awaited with the role and its scores whenever a reply was graded.

  • on_ok (Callable[[Role, str, Generated[Any]], Any] | None) –

    Called with the role, prompt and result after an ok reply.

  • on_event (EventHook | None) –

    Receives llm_call_failed, llm_call_graded, llm_fallback and llm_deadline_expired; without one, they go to the stdlib logger llm_ports.call.

  • generator_for (Callable[[str], Generator] | None) –

    Tier name to the Generator that serves it (e.g. call.providers.Backends.generator_for), for a fallback tier on another provider. None runs every tier on the generator passed to generate (the model id is on the request).

chat async

chat(
    role: Role,
    *,
    messages: Sequence[Message],
    chat: Chat,
    output: type[T] = str,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None,
    **opts: Unpack[TurnOptions]
) -> Generated[T]

Run one conversation turn for role and return its result: stream drained.

Takes exactly stream's arguments.

Raises:

  • CancelledError –

    The caller cancelled the call.

chat_request_for

chat_request_for(
    role: Role,
    *,
    messages: Sequence[Message],
    system: str = "",
    tools: Sequence[Tool] = (),
    settings: Mapping[str, Any] | None = None,
    max_steps: int | None = None
) -> ChatRequest

Resolve the role into one conversation request. Pure.

settings override the role's model settings for this call; max_steps overrides the role's.

stream async

stream(
    role: Role,
    *,
    messages: Sequence[Message],
    chat: Chat,
    output: type[T] = str,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None,
    **opts: Unpack[TurnOptions]
) -> AsyncGenerator[StreamEvent]

Run one conversation turn for role through chat, yielding its events and Finished last.

The path is described in the module docstring. The caller appends Finished.result.messages to its own history; the port keeps none.

Parameters:

  • role (Role) –

    The role: its tier, timeout, context policy, cache prefix and graders.

  • messages (Sequence[Message]) –

    The history, oldest first; the caller owns it.

  • chat (Chat) –

    The Chat port the call runs on (Backends.chat_for).

  • output (type[T], default: str ) –

    str for prose, a pydantic model, or a JsonOutput subclass.

  • validate (Validator[T] | None, default: None ) –

    Lists the problems with a parsed value; a non-empty list re-asks.

  • retry_message (RetryMessage | None, default: None ) –

    Builds a re-ask's message from the problems.

  • **opts (Unpack[TurnOptions], default: {} ) –

    The TurnOptions.

Raises:

  • CancelledError –

    The caller cancelled the call.

generate async

generate(
    role: Role,
    *,
    prompt: str,
    output: type[T] = str,
    generator: Generator,
    hooks: Hooks | None = None,
    system: str = "",
    images: Sequence[tuple[bytes, str]] = (),
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None,
    settings: Mapping[str, Any] | None = None,
    metadata: Mapping[str, Any] | None = None,
    web: bool = False,
    counter: TokenCounter | None = None
) -> Generated[T]

Run one generation for role through generator.

The path and the hooks are described in the module docstring. web=True says the role's tier fetches pages the prompt never rendered: like an image, an input no text grader sees, so call.graders.TEXT_ONLY_GRADERS are dropped for this call. The role itself is unchanged. counter counts the prompt for the role's pre-flight (Role.context, ADR-0024); the role's cache_prefix is translated by the tier's cache capability.

Returns:

  • Generated[T] –

    The result, with a named outcome. A model, wire or gate failure is an outcome, never an

  • Generated[T] –

    exception; so is the deadline running out (timeout, phase="total").

Raises:

  • RuntimeError –

    Only when a trace hook swallowed a generator exception, leaving no result.

  • CancelledError –

    The caller cancelled the call.

json_output

json_output(
    schema: Mapping[str, Any], name: str
) -> type[dict[str, Any]]

Return a hand-written JSON schema as an output= type.

The schema goes to the model unchanged as the grammar, and the value is a plain dict. name is what a test stub dispatches on. The result is a port value type (ports.JsonOutput); the Generator adapter translates it.

request_for

request_for(
    role: Role,
    *,
    prompt: str,
    system: str = "",
    images: Sequence[tuple[bytes, str]] = (),
    settings: Mapping[str, Any] | None = None
) -> GenRequest

Resolve the role into one wire request.

Pure. settings override the role's model settings for this call (a seed for a vote, a tighter max_tokens); the role stays the default.

require_ok

require_ok(res: Generated[T]) -> T

Return the result's value, or raise its failure.

For a caller whose except Exception is its ledger: under the never-raise contract a dead model would otherwise become an unrecorded None.

Raises:

  • GenerationFailed –

    (outcome, reason) for any non-ok outcome, gated included, and for an ok result without a value.

tracer_trace

tracer_trace(
    tracer: Tracer, *, provider: str | None = None
) -> TraceHook

Return a trace hook over any Tracer port adapter (Langfuse, OTel, TeeTracer, a recorder).

The hook opens one generation per call (observability.observe), named by the role's category, and ends it in error on a non-ok outcome. The end carries the call's result contract (usage, cost, finish, who answered, failure) through the one end payload observability.record uses.

Parameters:

  • tracer (Tracer) –

    The sink.

  • provider (str | None, default: None ) –

    The provider named on the generation; None takes the tier's provider from a schema 2 table, else "unknown" (1.x defaulted to "openai").

llm_ports.call.roles

Roles: the one place a model call's settings are declared.

A Role names one generation surface (feature.action, the string a category switchboard, a trace and a work row group by) and resolves it to a Tier: a model group, whether that group is self-hosted, and a client timeout rung. The role adds the per-role budget and sampling. Feature code declares a role and calls llm_ports.call.generate; it carries no model literal, no timeout literal and no framework import.

The tier table is data, and it belongs to the consumer: a TOML file ([ladder] rungs in seconds, [models] groups, [tiers.<name>] naming a model key, a rung and self_hosted), parsed and checked by load_tiers. The same table can arrive as a mapping (parsed TOML, a dict in a test). A consumer binds its table to its roles once, by subclass, so two consumers never share a process-global table.

A schema_version = 2 table adds providers by name, what each model can do, wire policies and fallback order (call.models); a table without schema_version is version 1 and loads as it always did.

Example

TABLE = load_tiers(Path(file).with_name("llm_tiers.toml"))

class Role(roles.Role): TIERS = TABLE.tiers

CATEGORIZE = Role("categorize.batch", tier="gateway", temperature=0.0)

Invariants
  • A Role whose tier is not in its class's table raises at construction, at import time, never as a call that resolved to nothing.
  • A tier's box key reads as self_hosted, with a DeprecationWarning, through 1.x.
  • A schema 2 table that names an unknown model, rung, provider, policy, tier, capability or failure kind, or whose fallbacks form a cycle, is TableInvalid naming the path.
  • A model id is written once: a schema 2 tier names a model key, and [models.<key>] id is the only place the wire string lives.

Rationale: ADR-0022.

Tier dataclass

Tier(
    name: str,
    model: str,
    timeout_s: float,
    self_hosted: bool = True,
    info: ModelInfo | None = None,
    policy: object | None = None,
    fallback: tuple[str, ...] = (),
    fallback_on: frozenset[str] = DEFAULT_FALLBACK_ON,
    table_fingerprint: str | None = None,
)

One resolved tier: a model, a client timeout, and whether the model is self-hosted.

Attributes:

  • name (str) –

    The tier's key in the table.

  • model (str) –

    The model id its [models] key names.

  • timeout_s (float) –

    The client timeout its [ladder] rung names, in seconds.

  • self_hosted (bool) –

    True when the model group is self-hosted.

  • info (ModelInfo | None) –

    What the model is and can do; None for a schema 1 table, where nothing is checked.

  • policy (object | None) –

    The tier's wire policy as declared ([policies.<name>], read-only and hashable: nested tables read as mappings, arrays as tuples); None for none. wire_policy is the same policy as a WirePolicy.

  • fallback (tuple[str, ...]) –

    Tier names tried in order when this tier fails with a kind in fallback_on.

  • fallback_on (frozenset[str]) –

    The failure kinds that move a call to the next fallback tier.

  • table_fingerprint (str | None) –

    The fingerprint of the table the tier was loaded from; None for a tier built by hand. Not compared.

wire_policy property

wire_policy: WirePolicy | None

The declared policy as a WirePolicy; None when the tier declares none.

Raises:

  • ValueError –

    The declared policy is not a valid WirePolicy (load_tiers refuses such a table at load, so only a hand-built tier can raise here).

TierTable dataclass

TierTable(
    ladder: Mapping[str, float],
    models: Mapping[str, str],
    tiers: Mapping[str, Tier],
    model_info: Mapping[str, ModelInfo] = dict(),
    providers: Mapping[str, ProviderSpec] = dict(),
    policies: Mapping[str, Mapping[str, Any]] = dict(),
    schema_version: int = 1,
)

A parsed tier table.

Attributes:

  • ladder (Mapping[str, float]) –

    Rung name to timeout, in seconds.

  • models (Mapping[str, str]) –

    Model key to model id.

  • tiers (Mapping[str, Tier]) –

    Tier name to resolved Tier.

  • model_info (Mapping[str, ModelInfo]) –

    Model key to ModelInfo; empty for a schema 1 table.

  • providers (Mapping[str, ProviderSpec]) –

    Provider name to ProviderSpec; empty for a schema 1 table.

  • policies (Mapping[str, Mapping[str, Any]]) –

    Policy name to its declared section, read-only; empty for a schema 1 table.

  • schema_version (int) –

    1 or 2.

fingerprint

fingerprint() -> str

Return the first 16 hex digits of the sha256 of the parsed table's canonical JSON.

Two tables that parse to the same values have the same fingerprint, whatever their key order or source format.

Raises:

  • TypeError –

    A policy holds a value with no canonical JSON form.

require

require(role: Role, *caps: Capability) -> None

Prove that role's tier, and every tier it can fall back to, has each of caps.

Called at import time, once per role that needs a capability (vision for a role sent images), so a table edit that removes it fails the boot rather than a call. A schema 1 table states no capability and is not checked.

Raises:

  • TableInvalid –

    The role's tier is not in this table, or a tier on its fallback path lacks a capability; the path names the tier.

Role dataclass

Role(
    name: str,
    tier: str,
    category: str = "",
    model: str | None = None,
    pack: str | None = None,
    max_tokens: int | None = None,
    temperature: float | None = None,
    thinking: bool = False,
    output_retries: int = 1,
    timeout_s: float | None = None,
    graders: tuple[str, ...] = (),
    settings: Mapping[str, Any] = dict(),
    accept_truncated: bool = False,
    policy: WirePolicy | None = None,
    stream_internally: bool = False,
    max_steps: int | None = None,
    tool_output_chars: int | None = None,
    context: ContextMode = "off",
    trim: TrimPolicy | None = None,
    cache_prefix: bool = False,
)

One generation surface and the settings its calls use.

Attributes:

  • TIERS (Mapping[str, Tier]) –

    The class's tier table. Empty on the base class, so the base class cannot be instantiated; a consumer subclasses it with its own table (see the module docstring).

  • name (str) –

    feature.action.

  • tier (str) –

    A key of TIERS.

  • category (str) –

    The category a switchboard and a trace group by; defaults to name.

  • model (str | None) –

    Overrides the tier's model.

  • pack (str | None) –

    The prompt pack id.

  • max_tokens (int | None) –

    The token ceiling for the reply.

  • temperature (float | None) –

    The sampling temperature.

  • thinking (bool) –

    The thinking switch sent to a self-hosted tier; off by default. On a hybrid thinking model the grammar binds only the content, and reasoning otherwise spends the token budget.

  • output_retries (int) –

    Re-asks allowed when the validator finds problems.

  • timeout_s (float | None) –

    Overrides the tier's timeout, in seconds.

  • graders (tuple[str, ...]) –

    Names of call.graders entries that call.generate scores every ok reply with.

  • settings (Mapping[str, Any]) –

    Per-role sampling the tier does not decide (seed, top_p, temperature).

  • accept_truncated (bool) –

    A reply the token ceiling cut is ok (with a CallWarning) instead of truncated: the 1.x behaviour, for a role whose consumer reads a partial reply.

  • policy (WirePolicy | None) –

    How the role's wire requests are retried; None takes the tier's declared policy, else the generator's default.

  • stream_internally (bool) –

    The adapter requests a stream and assembles it, so a cancelled or timed-out call closes a connection the server can stop on. Off by default.

  • max_steps (int | None) –

    The most model requests in one call.chat, tool rounds included; None takes ChatRequest's default (8).

  • tool_output_chars (int | None) –

    The one tool-output budget of the role's chat calls, in characters; None for no cut. A Tool.max_output_chars overrides it for that tool.

  • context (ContextMode) –

    Before a call is sent, off (the default, costs nothing), check (count it and refuse an overflow as context_overflow with provider_code="preflight"), or trim (cut the history by trim to fit, then refuse what still does not). ADR-0024.

  • trim (TrimPolicy | None) –

    The trimming policy for context="trim"; None is TrimPolicy().

  • cache_prefix (bool) –

    Ask for a prompt-cache breakpoint after the stable prefix: the system text (and, in a chat, the tool definitions). Dropped with a warning on a model that does not state cache.

Raises:

  • ValueError –

    tier is not in TIERS.

resolved_tier property

resolved_tier: Tier

The Tier this role names.

resolved_model property

resolved_model: str

The role's model override, else the tier's model.

resolved_timeout_s property

resolved_timeout_s: float

The role's timeout override, else the tier's timeout, in seconds.

model_settings

model_settings() -> dict[str, Any]

Return the settings this role sends.

On a schema 2 tier the thinking switch is provider-neutral, thinking: bool; the provider's factory spells it the way the model's ModelInfo.thinking says. On a schema 1 tier a self-hosted model gets extra_body.chat_template_kwargs.enable_thinking, which llama.cpp and vLLM read, as in 1.22.

load_tiers

load_tiers(
    source: Path | Mapping[str, Any],
    *,
    check_ladder: LadderCheck | None = None
) -> TierTable

Parse and check a tier table.

Parameters:

  • source (Path | Mapping[str, Any]) –

    The TOML file, or the table itself as a mapping with the same keys.

  • check_ladder (LadderCheck | None, default: None ) –

    The consumer's own ordering rule over the rungs (e.g. a client timeout that must clear a server deadline). Called as check_ladder(ladder, path) before the tiers resolve, with MAPPING_SOURCE as the path of a mapping; it raises ValueError to refuse the table.

Raises:

  • ValueError –

    A tier names a [models] key or a [ladder] rung the table does not have, or sets both self_hosted and its deprecated alias box. Every refusal of a schema 2 table is the subclass call.models.TableInvalid, naming the path.

llm_ports.call.context

Context: whether a request fits its model's window, what is cut when it does not, and cache hints.

Pure functions, no I/O. The call path (call.chat, call.generate) counts a request with a TokenCounter before it is sent, reads the verdict from fit, and on Role.context="trim" cuts the history with trim.

  • A count is exact or absent (ports.Counted): no character-ratio estimate is ever a token count. An unknown fit sends the request; it never blocks.
  • Trimming is deterministic, structural and reported. Order: tool outputs over the policy's budget are cut in the middle (cut_middle); then whole turns are dropped oldest-first between keep_first and keep_last, never splitting a tool call from its result. The system prompt is never cut.
  • A cache hint is a request, not a guarantee (cache_plan); Usage.cache_read on the result is the proof.

Rationale: ADR-0024.

Budget dataclass

Budget(
    context_window: int | None,
    reserve_output: int,
    margin: float = 0.05,
)

The input tokens a call may send.

Attributes:

  • context_window (int | None) –

    The model's window, from its ModelInfo; None when unknown.

  • reserve_output (int) –

    The tokens held for the reply (the role's max_tokens).

  • margin (float) –

    The fraction of the rest held back for counting error.

input_limit property

input_limit: int | None

(context_window - reserve_output) * (1 - margin), rounded down; None when the window is unknown.

Fit dataclass

Fit(
    verdict: Literal["fits", "over", "unknown"],
    tokens: int | None,
    limit: int | None,
)

Whether a counted request fits its budget.

Attributes:

  • verdict (Literal['fits', 'over', 'unknown']) –

    fits, over, or unknown (no count, or no window).

  • tokens (int | None) –

    The count; None when there was none.

  • limit (int | None) –

    The budget's input limit; None when the window is unknown.

over_by property

over_by: int | None

Tokens over the limit; None unless the verdict is over.

TrimPolicy dataclass

TrimPolicy(
    keep_first: int = 1,
    keep_last: int = 4,
    tool_output_chars: int = 4000,
    drop: Literal["oldest_turns"] = "oldest_turns",
)

How a history is cut to fit.

Attributes:

  • keep_first (int) –

    Leading messages never dropped (the task statement).

  • keep_last (int) –

    Trailing messages never dropped.

  • tool_output_chars (int) –

    The longest tool result kept whole; a longer one is cut in the middle.

  • drop (Literal['oldest_turns']) –

    What is dropped after the tool outputs: whole turns, oldest first.

Trimmed dataclass

Trimmed(
    messages: tuple[Message, ...],
    dropped_messages: int,
    cut_tool_outputs: int,
    chars_removed: int,
)

What trim returned and what it removed.

Attributes:

  • messages (tuple[Message, ...]) –

    The cut history.

  • dropped_messages (int) –

    Whole messages dropped.

  • cut_tool_outputs (int) –

    Tool results cut in the middle.

  • chars_removed (int) –

    Characters removed, over both.

changed property

changed: bool

True when anything was removed.

as_stamp

as_stamp() -> dict[str, int]

The counts, as stamp["trimmed"] carries them.

fit

fit(counted: Counted, budget: Budget) -> Fit

Return whether counted fits budget. Pure; an absent count or window is unknown.

cut_middle

cut_middle(text: str, max_chars: int) -> tuple[str, int]

Cut text to at most max_chars, keeping its head and its tail. Pure.

The tail usually holds the answer or the error, so the middle goes; the marker [... N chars omitted] says how much.

Returns:

  • tuple[str, int] –

    The text, and the characters removed (0 when it already fit).

trim

trim(
    messages: Sequence[Message],
    policy: TrimPolicy,
    *,
    over_by_tokens: int | None
) -> Trimmed

Cut messages by policy. Pure and deterministic.

Tool results over policy.tool_output_chars are cut in the middle first. Then whole turns (a user message and every message up to the next one) are dropped oldest-first, while every message of the turn lies between the first keep_first and the last keep_last. A turn is never split, so a tool call is never separated from its result.

Parameters:

  • messages (Sequence[Message]) –

    The history.

  • policy (TrimPolicy) –

    What may be cut.

  • over_by_tokens (int | None) –

    How far over the budget the history is. Turns are dropped until at least that many characters are gone (a token is at least one character, so fewer cannot be enough; the caller counts again to confirm). None drops every turn the policy allows.

Returns:

  • Trimmed –

    The cut history and what was removed.

cache_plan

cache_plan(
    info: ModelInfo | None,
    *,
    wanted: bool,
    settings: Mapping[str, Any]
) -> tuple[bool, dict[str, Any], tuple[CallWarning, ...]]

Translate a cache hint by the model's stated capability. Pure.

Parameters:

  • info (ModelInfo | None) –

    The model; None (a schema 1 tier) states nothing, and the hint passes to the adapter.

  • wanted (bool) –

    Whether the call asked for a cache breakpoint.

  • settings (Mapping[str, Any]) –

    The call's settings.

Returns:

  • bool –

    Whether the adapter still places breakpoints, the settings to send, and warnings. A model

  • dict[str, Any] –

    without cache drops the hint with a CallWarning(kind="unsupported"). A self-hosted

  • tuple[CallWarning, ...] –

    model with cache caches prefixes by slot: it gets extra_body.cache_prompt and no

  • tuple[bool, dict[str, Any], tuple[CallWarning, ...]] –

    breakpoint. A hosted model with cache keeps the hint for the adapter.

volatile_system

volatile_system(
    pack: Pack,
    contexts: Sequence[Mapping[str, Any]],
    *,
    template: str = "system"
) -> bool

True when the pack's system template renders differently across contexts.

A system text that interpolates a per-call value defeats a cache breakpoint after it: lint a pack whose role sets cache_prefix with two representative contexts.

Parameters:

  • pack (Pack) –

    The pack.

  • contexts (Sequence[Mapping[str, Any]]) –

    Render contexts; two or more.

  • template (str, default: 'system' ) –

    The system template's name in the manifest.

Returns:

  • bool –

    False when the pack has no such template.

llm_ports.call.models

The model table's value types: what a model can do, who serves it, and the closed vocabularies.

A schema_version = 2 tier table (call.roles.load_tiers) adds [providers.<name>] and [models.<key>] sections. A model key names one wire id, the provider that serves it, and what it can do, stated by the consumer in a closed vocabulary (CAPABILITIES): nothing is inferred from a model's name and nothing comes from a bundled database. A provider names a registered kind (the factory that builds its client) and, by NAME only, the URL key and the secret it needs: the table is reviewed and fingerprinted, so it never holds a URL or a key.

Example

schema_version = 2

[providers.gateway] kind = "openai_compatible" base_url = "gateway" # a key the caller resolves to a URL secret = "gateway_key" # a name the caller resolves to a key

[models.chat] id = "qwen3.5-9b" provider = "gateway" supports = ["json_schema", "vision"] thinking = "chat_template"

Invariants
  • A capability word, thinking style or failure kind outside its vocabulary is TableInvalid at load, naming the path (models.chat.supports[1]).
  • An unknown limit is None, never a guessed number.
  • Pure: no I/O, no vendor import.

Rationale: ADR-0022.

TableInvalid

TableInvalid(path: str, message: str)

Bases: ValueError

A tier table that cannot be used, raised at load or at build, naming where.

Attributes:

  • path (str) –

    The dotted path of the offending value (tiers.box.fallback[0]).

ModelInfo dataclass

ModelInfo(
    key: str,
    id: str,
    provider: str,
    supports: frozenset[Capability],
    context_window: int | None = None,
    max_output: int | None = None,
    thinking: ThinkingStyle = "none",
    slots: int | None = None,
    group: str | None = None,
    self_hosted: bool = True,
    usd_per_request: Decimal | None = None,
)

One model of a schema 2 table: its wire id, its provider, and what it can do.

Attributes:

  • key (str) –

    The model's key in [models].

  • id (str) –

    The id sent on the wire (a gateway route is fine). The only place it is written.

  • provider (str) –

    The key of the [providers] entry that serves it.

  • supports (frozenset[Capability]) –

    The capabilities the consumer states for it.

  • context_window (int | None) –

    Tokens in its context window; None when unknown.

  • max_output (int | None) –

    The most output tokens it returns; None when unknown.

  • thinking (ThinkingStyle) –

    How its thinking switch is spelled.

  • slots (int | None) –

    Calls it takes at once in-process; None when not limited here.

  • group (str | None) –

    Models sharing one backend resource; None when it shares none.

  • self_hosted (bool) –

    True when the consumer runs the model itself.

  • usd_per_request (Decimal | None) –

    A flat price per request; None when the cost is unknown.

ProviderSpec dataclass

ProviderSpec(
    name: str,
    kind: str,
    base_url: str | None,
    secret: str | None,
    gateway: str | None = None,
    options: Mapping[str, Any] = dict(),
)

One provider of a schema 2 table: a registered kind, and the names of what it needs.

Attributes:

  • name (str) –

    The provider's key in [providers].

  • kind (str) –

    The registered provider kind that builds its client (openai_compatible).

  • base_url (str | None) –

    A topology key the caller resolves to a URL; None when the kind needs none.

  • secret (str | None) –

    A secret name the caller resolves to a key; None for a keyless endpoint.

  • gateway (str | None) –

    Which gateway's response headers the provider reads, when one stands in front.

  • options (Mapping[str, Any]) –

    Every other key of the section, validated by the kind's factory.

parse_providers

parse_providers(
    raw: Mapping[str, Any],
) -> dict[str, ProviderSpec]

Parse a [providers] section.

Parameters:

  • raw (Mapping[str, Any]) –

    Provider name to its section.

Returns:

  • dict[str, ProviderSpec] –

    Provider name to ProviderSpec, in the section's order.

Raises:

  • TableInvalid –

    A section is not a table, kind is missing, a value has the wrong type, or base_url holds a URL rather than a key.

parse_models

parse_models(
    raw: Mapping[str, Any],
    providers: Mapping[str, ProviderSpec],
) -> dict[str, ModelInfo]

Parse a schema 2 [models] section against its providers.

Parameters:

  • raw (Mapping[str, Any]) –

    Model key to its section.

  • providers (Mapping[str, ProviderSpec]) –

    The parsed [providers] section.

Returns:

  • dict[str, ModelInfo] –

    Model key to ModelInfo, in the section's order.

Raises:

  • TableInvalid –

    An unknown key, a missing id or provider, a provider the table does not declare, a word outside its vocabulary, or a value of the wrong type.

needs

needs(
    *, structured: bool, images: bool, web: bool
) -> frozenset[Capability]

Return the capabilities one request needs. Pure.

Parameters:

  • structured (bool) –

    The output is a schema, not prose.

  • images (bool) –

    The request carries images.

  • web (bool) –

    The model is asked to fetch pages.

Returns:

  • frozenset[Capability] –

    vision for images, web for a page fetch, and json_schema for structured output

  • frozenset[Capability] –

    (missing accepts json_object in its place).

missing

missing(
    info: ModelInfo, wanted: frozenset[Capability]
) -> tuple[Capability, ...]

Return the capabilities in wanted that info lacks, in vocabulary order. Pure.

json_object stands in for json_schema: either lets a model return structured output.

translate_settings

translate_settings(
    settings: Mapping[str, Any], info: ModelInfo | None
) -> tuple[dict[str, Any], tuple[str, ...]]

Spell a request's provider-neutral thinking switch the way its model needs. Pure.

Parameters:

  • settings (Mapping[str, Any]) –

    The request's settings; a schema 2 role puts thinking: bool in them.

  • info (ModelInfo | None) –

    The request's model, or None when the table does not know it (the settings pass through unchanged, so pydantic-ai's own unified thinking applies).

Returns:

  • dict[str, Any] –

    The settings to send, and the names of settings the model cannot take (dropped):

  • tuple[str, ...] –

    unified keeps thinking (pydantic-ai's ModelSettings["thinking"]); chat_template

  • tuple[dict[str, Any], tuple[str, ...]] –

    moves it to extra_body.chat_template_kwargs.enable_thinking; none drops it, and names

  • tuple[dict[str, Any], tuple[str, ...]] –

    it when the role asked to think.

llm_ports.call.providers

Providers by name: the factory a kind registers, and build, which turns a table into backends.

A schema 2 tier table names, for every model, the provider that serves it; a provider names a registered KIND (openai_compatible) and, by name only, the URL key and the secret it needs. build resolves those names through the caller's urls and secrets, hands each provider to its kind's ProviderFactory, and returns Backends, which a consumer's facade asks for the Generator of a tier:

providers = adapters.pluggy.load_providers(builtins=[builtin("openai_compatible", factory)], pin=pin)
backends = build(TABLE, urls=topology, secrets=vault.get, deps=BuildDeps(), factories=providers)
res = await call.generate(ROLE, prompt=text, generator=backends.generator_for(ROLE.tier))

Kinds arrive through the provider registry (adapters.pluggy.load_providers, entry-point group ENTRY_POINT_GROUP), or as a plain mapping in a test. Only the core types live here: no vendor library and no entry-point reading.

Invariants
  • Everything that can be wrong with a table and its environment is raised at build, at boot: an unknown kind, a URL key the caller cannot resolve, a secret that resolves to nothing or to a value no header can carry (whitespace, a control or a non-ASCII character). Nothing waits for the first call to find a missing key.
  • A secret value is never logged, stamped or put in an error; only its name is.
  • A kind's key is flat and global: two plugins registering one kind raise, naming both.
  • A registry is frozen once loaded: a later add raises.

Rationale: ADR-0022.

Loaded dataclass

Loaded(
    group: str,
    plugins: tuple[str, ...],
    origin: Mapping[str, str] = dict(),
)

Which plugins a process loaded into one registry, and from where.

Attributes:

  • group (str) –

    The entry-point group.

  • plugins (tuple[str, ...]) –

    Plugin names in load order: built-ins as given, then entry points sorted by name.

  • origin (Mapping[str, str]) –

    Plugin name to builtin, caller, or <distribution>==<version>.

BuildDeps dataclass

BuildDeps(
    policy: WirePolicy | None = None,
    limiter: Limiter | None = None,
    pricer: Pricer | None = None,
    transport: Any | None = None,
    on_event: EventHook | None = None,
)

What every factory may share: the call's planes, passed once to build.

Attributes:

  • policy (WirePolicy | None) –

    The wire policy every built client uses when a request carries none; None takes each adapter's own default.

  • limiter (Limiter | None) –

    The concurrency limiter every built generator takes its slots from; None for no in-process limit.

  • pricer (Pricer | None) –

    Prices a reply's usage the gateway did not price.

  • transport (Any | None) –

    An HTTP transport every client is built on (a test's mock, a counting proxy).

  • on_event (EventHook | None) –

    Receives the build's events and every built client's; without one, the stdlib logger llm_ports.call.

Built dataclass

Built(
    generator: Generator | None = None,
    chat: Chat | None = None,
    counter_for: (
        Callable[[str], TokenCounter | None] | None
    ) = None,
    embedder_for: Callable[[str], Embedder] | None = None,
    reranker_for: Callable[[str], Reranker] | None = None,
    probe: Probe | None = None,
)

What one factory built for one provider. A kind builds the ports it serves; the rest are None.

Attributes:

  • generator (Generator | None) –

    The Generator for every model of the provider (the model id is on the request).

  • chat (Chat | None) –

    The Chat port for every model of the provider.

  • counter_for (Callable[[str], TokenCounter | None] | None) –

    Model id to a TokenCounter, or None for a model that cannot be counted.

  • embedder_for (Callable[[str], Embedder] | None) –

    Model id to an Embedder.

  • reranker_for (Callable[[str], Reranker] | None) –

    Model id to a Reranker.

  • probe (Probe | None) –

    The provider's health probe.

ProviderFactory

Bases: Protocol

Builds the clients of one provider kind.

kind property

kind: str

The kind's registered name, the value of a provider's kind key.

build

build(
    spec: ProviderSpec,
    *,
    url: str | None,
    secret: str | None,
    models: Sequence[ModelInfo],
    deps: BuildDeps
) -> Built

Build the provider's clients.

Parameters:

  • spec (ProviderSpec) –

    The provider's section; the factory validates spec.options.

  • url (str | None) –

    The resolved URL of spec.base_url; None when the provider names none.

  • secret (str | None) –

    The resolved secret of spec.secret; None for a keyless provider.

  • models (Sequence[ModelInfo]) –

    The table's models this provider serves, in table order.

  • deps (BuildDeps) –

    The shared planes.

Raises:

  • TableInvalid –

    The provider's section asks for something the kind cannot do.

DuplicateProvider

Bases: ValueError

Two plugins registered one provider kind; the message names both.

ProviderRegistry

ProviderRegistry(factories: Sequence[ProviderFactory] = ())

Bases: Mapping[str, ProviderFactory]

Provider kind to its factory, with the registry rules: a duplicate raises, a frozen one is closed.

Parameters:

  • factories (Sequence[ProviderFactory], default: () ) –

    Factories to add now, as plugin caller.

frozen property

frozen: bool

Whether freeze has run.

loaded property

loaded: Loaded

What loaded the registry; derived from the plugins so far while it is open.

add

add(
    factory: ProviderFactory, *, plugin: str = "caller"
) -> None

Register factory under its kind, contributed by plugin.

Raises:

  • TypeError –

    factory is not a ProviderFactory.

  • DuplicateProvider –

    Another plugin registered the kind.

  • RuntimeError –

    The registry is frozen.

freeze

freeze(loaded: Loaded | None = None) -> None

Close the registry, recording what loaded it; a later add raises.

plugin_of

plugin_of(kind: str) -> str

The plugin that registered kind.

Backends

Backends(
    table: TierTable,
    built: Mapping[str, Built],
    loaded: Loaded,
)

The clients build made, looked up by tier, model key or provider.

Attributes:

  • table (TierTable) –

    The table the backends were built from.

  • loaded (Loaded) –

    Which provider kinds were registered, by which plugins, from where.

providers property

providers: Mapping[str, Built]

Provider name to what was built for it.

generator_for

generator_for(tier: str) -> Generator

The Generator that serves tier's model.

Raises:

  • TableInvalid –

    The tier is unknown.

  • LookupError –

    The provider's kind builds no Generator.

chat_for

chat_for(tier: str) -> Chat

The Chat port that serves tier's model.

Raises:

  • TableInvalid –

    The tier is unknown.

  • LookupError –

    The provider's kind builds no chat port.

counter_for

counter_for(tier: str) -> TokenCounter | None

The TokenCounter for tier's model; None when the model does not state count_tokens.

A None is the pre-flight's unknown: the call is sent unchecked, never refused.

Raises:

embedder_for

embedder_for(model_key: str) -> Embedder

The Embedder for the model under model_key.

Raises:

  • TableInvalid –

    The key is unknown, or the model does not state embed.

  • LookupError –

    The provider's kind builds no Embedder.

reranker_for

reranker_for(model_key: str) -> Reranker

The Reranker for the model under model_key.

Raises:

  • TableInvalid –

    The key is unknown, or the model does not state rerank.

  • LookupError –

    The provider's kind builds no Reranker.

probe_for

probe_for(provider: str) -> Probe

The health probe of provider.

Raises:

  • TableInvalid –

    The provider was not built (unknown, or no model uses it).

  • LookupError –

    Its kind builds no probe.

aclose async

aclose() -> None

Close every built client that holds a resource, each once (a kind may share one). Idempotent.

build

build(
    table: TierTable,
    *,
    urls: Mapping[str, str],
    secrets: Callable[[str], str | None],
    deps: BuildDeps,
    factories: Mapping[str, ProviderFactory] | None = None
) -> Backends

Build the clients of every provider the table's models use.

Parameters:

  • table (TierTable) –

    A schema 2 tier table.

  • urls (Mapping[str, str]) –

    Topology key to URL, for each provider's base_url.

  • secrets (Callable[[str], str | None]) –

    Secret name to its value, or None when the caller has none by that name. Called once per provider that names a secret; the value goes to the factory and nowhere else.

  • deps (BuildDeps) –

    The shared planes every factory gets.

  • factories (Mapping[str, ProviderFactory] | None, default: None ) –

    Kind to factory: a ProviderRegistry from adapters.pluggy.load_providers, or a plain mapping. None registers no kind, so any provider is refused.

Returns:

  • Backends –

    The Backends, after emitting llm_backends_built (the table's fingerprint, each

  • Backends –

    provider's kind, and which plugins registered the kinds).

Raises:

  • TableInvalid –

    The table is schema 1, a provider names an unregistered kind, a URL key the caller does not resolve, or a secret that resolves to nothing or to a value holding whitespace, a control or a non-ASCII character (the error names the secret, never its value); or a factory refuses its section.

llm_ports.call.graders

Deterministic graders over one generation: the prompt in, the value out.

Pure. A CallGrader is (prompt, value, role) -> CallScore: a numeric value (what a trace score and a dashboard can sum), a pass or fail against the grader's own bar, and a reason that names the violated property on failure. call.generate runs the graders a Role.graders names on every ok reply and writes each as a score on the call's trace observation. A golden eval runs the same functions over a pack's golden set (board_graders), so the live score and the offline verdict agree on what "complete" or "echoed" means.

Built-in graders, for structured outputs (a dict or pydantic value under a grammar):

  • schema_complete: the share of fields that carry a value. A strict grammar admits "" for every required key, so a parsed reply is not necessarily an answered one. An empty list is an answer; only None and blank text count as unanswered. On a pydantic value, None on a field whose annotation admits None (float | None, Optional[X]) is an answer too, while None on a required field fails. A dict value has no annotations to read, so every None in it is unanswered, unless the caller declares the answer paths: schema_complete(..., model=Out) reads them off a model, nullable={"items[].stop"} names them, and partial(schema_complete, nullable=...) binds them into a table entry.
  • numbers_from_inputs: figures in the reply's prose that no part of the prompt rendered, digit runs by value and spelled numbers alike (numbers.loose): the number law, per call.
  • echo: the share of the reply's sentences copied verbatim from the prompt.
  • cites_url: the distinct http(s) URLs the reply carries, in any string field. A web-fetching tier's reply that cites nothing cannot be checked against its pages. Web calls only.
  • no_refusal: the stock phrases of an ungrounded refusal ("I don't have real-time access", "as of my training data", "please consult ..."). An honest "the sources do not cover X" is not matched.

A consumer adds its own graders by passing a table ({**GRADERS, "mine": fn}) to score_all, board_graders or call.Hooks(graders=...), or a registry to prompts.resolve. GRADERS is a read-only mapping: assigning into it raises TypeError, so one consumer's grader cannot leak into another's table.

Invariants
  • score_one and score_all never raise. A grader's exception becomes a failed score of value CRASHED (-1, below every honest value, all of which are >= 0) whose reason names the exception, so a crashed grader never reads as a pass.

Rationale: ADR-0003.

CallScore dataclass

CallScore(
    name: str, value: float, passed: bool, reason: str = ""
)

One grader's verdict on one reply.

Attributes:

  • name (str) –

    The grader's name.

  • value (float) –

    The numeric score; CRASHED when the grader raised.

  • passed (bool) –

    Whether the value meets the grader's own bar.

  • reason (str) –

    Empty on a pass; names the violated property on a failure.

leaves

leaves(
    value: Any, path: str = ""
) -> Iterator[tuple[str, Any]]

Yield (path, scalar) for every leaf of a dict, pydantic or list value, in order.

Pure. An empty list is itself a leaf.

prose

prose(value: Any) -> list[str]

Return every string leaf of the reply: what the number law and the echo check read.

nullable_paths

nullable_paths(
    model: type[BaseModel],
    prefix: str = "",
    _seen: frozenset[type] = frozenset(),
) -> frozenset[str]

Return the index-free leaf paths of every field whose annotation admits None.

Pure. Paths (items[].price) run through nested models and lists of models. They are read off the class, so they never depend on which rows a reply carried. A self-referential model stops at its own repeat.

schema_complete

schema_complete(
    _prompt: str,
    value: Any,
    _role: Role,
    *,
    model: type[BaseModel] | None = None,
    nullable: Iterable[str] | None = None
) -> CallScore

Score the share of the reply's fields that carry a value.

Passes only when no field is unanswered. None and blank text are unanswered; an empty list is an answer, and so is None on a path declared nullable.

Parameters:

  • _prompt (str) –

    The rendered prompt (unused).

  • value (Any) –

    The reply: a dict, a list or a pydantic value.

  • _role (Role) –

    The call's role (unused).

  • model (type[BaseModel] | None, default: None ) –

    The declared output model; its nullable_paths are the answer paths for any value, a dict reply included.

  • nullable (Iterable[str] | None, default: None ) –

    Answer paths named directly, index-free (items[].stop). Taken with model, the two sets are joined.

Returns:

  • CallScore –

    A CallScore; never raises on a value leaves can walk.

Invariants
  • With neither model nor nullable, a pydantic value's own annotations decide and a dict's every None is unanswered: the grader the built-in table carries.
  • A declared path exempts None and blank text only; a non-empty value scores the same.

numbers_from_inputs

numbers_from_inputs(
    prompt: str, value: Any, _role: Role
) -> CallScore

Count the distinct figures in the reply's prose that the prompt did not render.

Digit runs compare by value, and spelled numbers count too (numbers.loose). The prompt's own text licenses a spelled word, and '%' licenses 'percent'. Passes only at zero.

Rationale: ADR-0003.

echo

echo(prompt: str, value: Any, _role: Role) -> CallScore

Score the share of the reply's sentences copied verbatim from the prompt.

The compare ignores case and whitespace, and a sentence shorter than ECHO_MIN_CHARS is not counted; a reply with no counted sentence passes at 0. Fails at or above ECHO_MAX_SHARE.

cites_url

cites_url(
    _prompt: str, value: Any, _role: Role
) -> CallScore

Count the distinct http(s) URLs in the reply's string fields. Passes at one or more.

A web call only (WEB_ONLY_GRADERS): a text call has no pages to cite.

no_refusal

no_refusal(
    _prompt: str, value: Any, _role: Role
) -> CallScore

Count the ungrounded-refusal phrases in the reply's prose. Passes only at zero.

graders_for_inputs

graders_for_inputs(
    graders: tuple[str, ...],
    *,
    has_images: bool,
    has_web: bool = False
) -> tuple[str, ...]

Return the graders that can see this call's inputs, in their given order.

Pure. An image beside the prompt or a web-fetching tier removes TEXT_ONLY_GRADERS; a call without a web-fetching tier removes WEB_ONLY_GRADERS.

score_one

score_one(
    name: str,
    prompt: str,
    value: Any,
    role: Role,
    table: Mapping[str, CallGrader] | None = None,
) -> CallScore

Run one grader by name from table (default GRADERS).

Never raises: an unknown name or a grader exception becomes a failed score of value CRASHED.

score_all

score_all(
    role: Role,
    prompt: str,
    value: Any,
    table: Mapping[str, CallGrader] | None = None,
) -> tuple[CallScore, ...]

Run every grader the role names, in the role's order; never raises.

board_graders

board_graders(
    role: Role,
    table: Mapping[str, CallGrader] | None = None,
) -> dict[
    str, Callable[[dict[str, Any]], tuple[bool, str]]
]

Return the same graders in the Evaluator row shape (ports.Grader).

A golden row carries prompt and value. A row whose model never answered (value is None) fails every grader with its reason, so the board names the outage rather than grading an absence.

llm_ports.call.schema

JSON-object extraction and the research grammar.

The strict-schema transform (strict_json_schema) wraps pydantic-ai's, so it lives with that adapter: llm_ports.adapters.pydantic_ai.

  • extract_json_object: the JSON-object substring of a model response that is fenced, or prefixed or suffixed by prose.
  • research_schema, research: a free-text analysis scratchpad first, every property required, and the scratchpad popped from the value afterwards.

research_schema

research_schema(
    schema: Mapping[str, Any],
) -> dict[str, Any]

Return the research grammar for schema.

Pure. A free-text analysis property (RESEARCH_ANALYSIS) is injected first, and required lists every property, including the ones pydantic marks optional because they have a default.

without_analysis

without_analysis(res: Generated[T]) -> Generated[T]

Return res with the analysis scratchpad popped from a dict value; other values pass unchanged.

research async

research(
    generate: Callable[..., Awaitable[Generated[Any]]],
    role: Role,
    *,
    prompt: str,
    schema: Mapping[str, Any],
    name: str,
    **kwargs: Any
) -> Generated[dict[str, Any]]

Run a research call that returns a grammar-enforced dict.

The consumer's generate (its hooks bound) runs with research_schema(schema) as the grammar (json_output(..., name)), and the scratchpad is popped from the value. kwargs pass through to generate unchanged.

extract_json_object

extract_json_object(text: str) -> str

Return the JSON-object substring of a model response.

Tolerates markdown fences and leading or trailing prose: some models fence their JSON even when asked not to, and some routes cannot enforce response_mime_type. A leading code fence is stripped, then the first { through the last } is taken, so json.loads succeeds regardless of surrounding text. Falls back to the stripped text when no object is found, so the caller's JSON error stays meaningful.