llm_ports.call¶
llm_ports.call
¶
The call paths: generate, chat and stream, their hooks, roles, per-call graders, output schemas, retry and skip.
Submodules: context (budgets, trimming, the cache plan), roles, models (what a model can do, who serves it), providers (providers by name,
build), graders, schema, and resilience (the retry and skip mechanics of a long run).
TurnOptions
¶
Bases: TypedDict
The optional keywords stream and chat share; each one omitted takes the default named.
hooks
instance-attribute
¶
hooks: Hooks | None
Trace, admit, graders, report and event hooks (default Hooks()).
settings
instance-attribute
¶
settings: Mapping[str, Any] | None
Model settings merged over the role's (default the role's).
metadata
instance-attribute
¶
metadata: Mapping[str, Any] | None
Extra trace metadata (default none).
max_steps
instance-attribute
¶
max_steps: int | None
Overrides Role.max_steps for this call (default the role's).
counter
instance-attribute
¶
counter: TokenCounter | None
Measures the history for the pre-flight (Role.context); None counts nothing.
Hooks
dataclass
¶
Hooks(
admit: AdmitHook | None = None,
trace: TraceHook | None = None,
graders: Mapping[str, CallGrader] | None = None,
on_scores: (
Callable[
[Role, tuple[CallScore, ...]], Awaitable[Any]
]
| None
) = None,
on_ok: (
Callable[[Role, str, Generated[Any]], Any] | None
) = None,
on_event: EventHook | None = None,
generator_for: Callable[[str], Generator] | None = None,
)
Optional consumer hooks for generate.
Attributes:
-
admit(AdmitHook | None) –Runs before any trace opens; a returned reason makes the outcome
gated. -
trace(TraceHook | None) –Opens the call's observation; without one, a sink-less tracer is used.
-
graders(Mapping[str, CallGrader] | None) –The grader table by name; None means
call.graders.GRADERS. -
on_scores(Callable[[Role, tuple[CallScore, ...]], Awaitable[Any]] | None) –Awaited with the role and its scores whenever a reply was graded.
-
on_ok(Callable[[Role, str, Generated[Any]], Any] | None) –Called with the role, prompt and result after an
okreply. -
on_event(EventHook | None) –Receives
llm_call_failed,llm_call_graded,llm_fallbackandllm_deadline_expired; without one, they go to the stdlib loggerllm_ports.call. -
generator_for(Callable[[str], Generator] | None) –Tier name to the
Generatorthat serves it (e.g.call.providers.Backends.generator_for), for a fallback tier on another provider. None runs every tier on thegeneratorpassed togenerate(the model id is on the request).
chat
async
¶
chat(
role: Role,
*,
messages: Sequence[Message],
chat: Chat,
output: type[T] = str,
validate: Validator[T] | None = None,
retry_message: RetryMessage | None = None,
**opts: Unpack[TurnOptions]
) -> Generated[T]
Run one conversation turn for role and return its result: stream drained.
Takes exactly stream's arguments.
Raises:
-
CancelledError–The caller cancelled the call.
chat_request_for
¶
chat_request_for(
role: Role,
*,
messages: Sequence[Message],
system: str = "",
tools: Sequence[Tool] = (),
settings: Mapping[str, Any] | None = None,
max_steps: int | None = None
) -> ChatRequest
Resolve the role into one conversation request. Pure.
settings override the role's model settings for this call; max_steps overrides the role's.
stream
async
¶
stream(
role: Role,
*,
messages: Sequence[Message],
chat: Chat,
output: type[T] = str,
validate: Validator[T] | None = None,
retry_message: RetryMessage | None = None,
**opts: Unpack[TurnOptions]
) -> AsyncGenerator[StreamEvent]
Run one conversation turn for role through chat, yielding its events and Finished last.
The path is described in the module docstring. The caller appends Finished.result.messages
to its own history; the port keeps none.
Parameters:
-
role(Role) –The role: its tier, timeout, context policy, cache prefix and graders.
-
messages(Sequence[Message]) –The history, oldest first; the caller owns it.
-
chat(Chat) –The
Chatport the call runs on (Backends.chat_for). -
output(type[T], default:str) –strfor prose, a pydantic model, or aJsonOutputsubclass. -
validate(Validator[T] | None, default:None) –Lists the problems with a parsed value; a non-empty list re-asks.
-
retry_message(RetryMessage | None, default:None) –Builds a re-ask's message from the problems.
-
**opts(Unpack[TurnOptions], default:{}) –The
TurnOptions.
Raises:
-
CancelledError–The caller cancelled the call.
generate
async
¶
generate(
role: Role,
*,
prompt: str,
output: type[T] = str,
generator: Generator,
hooks: Hooks | None = None,
system: str = "",
images: Sequence[tuple[bytes, str]] = (),
validate: Validator[T] | None = None,
retry_message: RetryMessage | None = None,
settings: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None,
web: bool = False,
counter: TokenCounter | None = None
) -> Generated[T]
Run one generation for role through generator.
The path and the hooks are described in the module docstring. web=True says the role's tier
fetches pages the prompt never rendered: like an image, an input no text grader sees, so
call.graders.TEXT_ONLY_GRADERS are dropped for this call. The role itself is unchanged.
counter counts the prompt for the role's pre-flight (Role.context, ADR-0024); the role's
cache_prefix is translated by the tier's cache capability.
Returns:
-
Generated[T]–The result, with a named outcome. A model, wire or gate failure is an outcome, never an
-
Generated[T]–exception; so is the deadline running out (
timeout,phase="total").
Raises:
-
RuntimeError–Only when a trace hook swallowed a generator exception, leaving no result.
-
CancelledError–The caller cancelled the call.
json_output
¶
json_output(
schema: Mapping[str, Any], name: str
) -> type[dict[str, Any]]
Return a hand-written JSON schema as an output= type.
The schema goes to the model unchanged as the grammar, and the value is a plain dict. name is
what a test stub dispatches on. The result is a port value type (ports.JsonOutput); the
Generator adapter translates it.
request_for
¶
request_for(
role: Role,
*,
prompt: str,
system: str = "",
images: Sequence[tuple[bytes, str]] = (),
settings: Mapping[str, Any] | None = None
) -> GenRequest
Resolve the role into one wire request.
Pure. settings override the role's model settings for this call (a seed for a vote, a tighter
max_tokens); the role stays the default.
require_ok
¶
require_ok(res: Generated[T]) -> T
Return the result's value, or raise its failure.
For a caller whose except Exception is its ledger: under the never-raise contract a dead model
would otherwise become an unrecorded None.
Raises:
-
GenerationFailed–(outcome, reason)for any non-okoutcome,gatedincluded, and for anokresult without a value.
tracer_trace
¶
tracer_trace(
tracer: Tracer, *, provider: str | None = None
) -> TraceHook
Return a trace hook over any Tracer port adapter (Langfuse, OTel, TeeTracer, a recorder).
The hook opens one generation per call (observability.observe), named by the role's category,
and ends it in error on a non-ok outcome. The end carries the call's result contract (usage,
cost, finish, who answered, failure) through the one end payload observability.record uses.
Parameters:
-
tracer(Tracer) –The sink.
-
provider(str | None, default:None) –The provider named on the generation; None takes the tier's provider from a schema 2 table, else
"unknown"(1.x defaulted to"openai").
llm_ports.call.roles¶
Roles: the one place a model call's settings are declared.
A Role names one generation surface (feature.action, the string a category switchboard, a
trace and a work row group by) and resolves it to a Tier: a model group, whether that group is
self-hosted, and a client timeout rung. The role adds the per-role budget and sampling. Feature
code declares a role and calls llm_ports.call.generate; it carries no model literal, no timeout
literal and no framework import.
The tier table is data, and it belongs to the consumer: a TOML file ([ladder] rungs in seconds,
[models] groups, [tiers.<name>] naming a model key, a rung and self_hosted), parsed and
checked by load_tiers. The same table can arrive as a mapping (parsed TOML, a dict in a test).
A consumer binds its table to its roles once, by subclass, so two consumers never share a
process-global table.
A schema_version = 2 table adds providers by name, what each model can do, wire policies and
fallback order (call.models); a table without schema_version is version 1 and loads as it
always did.
Example
TABLE = load_tiers(Path(file).with_name("llm_tiers.toml"))
class Role(roles.Role): TIERS = TABLE.tiers
CATEGORIZE = Role("categorize.batch", tier="gateway", temperature=0.0)
Invariants
- A
Rolewhose tier is not in its class's table raises at construction, at import time, never as a call that resolved to nothing. - A tier's
boxkey reads asself_hosted, with aDeprecationWarning, through 1.x. - A schema 2 table that names an unknown model, rung, provider, policy, tier, capability or
failure kind, or whose fallbacks form a cycle, is
TableInvalidnaming the path. - A model id is written once: a schema 2 tier names a model key, and
[models.<key>] idis the only place the wire string lives.
Rationale: ADR-0022.
Tier
dataclass
¶
Tier(
name: str,
model: str,
timeout_s: float,
self_hosted: bool = True,
info: ModelInfo | None = None,
policy: object | None = None,
fallback: tuple[str, ...] = (),
fallback_on: frozenset[str] = DEFAULT_FALLBACK_ON,
table_fingerprint: str | None = None,
)
One resolved tier: a model, a client timeout, and whether the model is self-hosted.
Attributes:
-
name(str) –The tier's key in the table.
-
model(str) –The model id its
[models]key names. -
timeout_s(float) –The client timeout its
[ladder]rung names, in seconds. -
self_hosted(bool) –True when the model group is self-hosted.
-
info(ModelInfo | None) –What the model is and can do; None for a schema 1 table, where nothing is checked.
-
policy(object | None) –The tier's wire policy as declared (
[policies.<name>], read-only and hashable: nested tables read as mappings, arrays as tuples); None for none.wire_policyis the same policy as aWirePolicy. -
fallback(tuple[str, ...]) –Tier names tried in order when this tier fails with a kind in
fallback_on. -
fallback_on(frozenset[str]) –The failure kinds that move a call to the next fallback tier.
-
table_fingerprint(str | None) –The fingerprint of the table the tier was loaded from; None for a tier built by hand. Not compared.
wire_policy
property
¶
wire_policy: WirePolicy | None
The declared policy as a WirePolicy; None when the tier declares none.
Raises:
-
ValueError–The declared policy is not a valid
WirePolicy(load_tiersrefuses such a table at load, so only a hand-built tier can raise here).
TierTable
dataclass
¶
TierTable(
ladder: Mapping[str, float],
models: Mapping[str, str],
tiers: Mapping[str, Tier],
model_info: Mapping[str, ModelInfo] = dict(),
providers: Mapping[str, ProviderSpec] = dict(),
policies: Mapping[str, Mapping[str, Any]] = dict(),
schema_version: int = 1,
)
A parsed tier table.
Attributes:
-
ladder(Mapping[str, float]) –Rung name to timeout, in seconds.
-
models(Mapping[str, str]) –Model key to model id.
-
tiers(Mapping[str, Tier]) –Tier name to resolved
Tier. -
model_info(Mapping[str, ModelInfo]) –Model key to
ModelInfo; empty for a schema 1 table. -
providers(Mapping[str, ProviderSpec]) –Provider name to
ProviderSpec; empty for a schema 1 table. -
policies(Mapping[str, Mapping[str, Any]]) –Policy name to its declared section, read-only; empty for a schema 1 table.
-
schema_version(int) –1 or 2.
fingerprint
¶
fingerprint() -> str
Return the first 16 hex digits of the sha256 of the parsed table's canonical JSON.
Two tables that parse to the same values have the same fingerprint, whatever their key order or source format.
Raises:
-
TypeError–A policy holds a value with no canonical JSON form.
require
¶
require(role: Role, *caps: Capability) -> None
Prove that role's tier, and every tier it can fall back to, has each of caps.
Called at import time, once per role that needs a capability (vision for a role sent
images), so a table edit that removes it fails the boot rather than a call. A schema 1
table states no capability and is not checked.
Raises:
-
TableInvalid–The role's tier is not in this table, or a tier on its fallback path lacks a capability; the path names the tier.
Role
dataclass
¶
Role(
name: str,
tier: str,
category: str = "",
model: str | None = None,
pack: str | None = None,
max_tokens: int | None = None,
temperature: float | None = None,
thinking: bool = False,
output_retries: int = 1,
timeout_s: float | None = None,
graders: tuple[str, ...] = (),
settings: Mapping[str, Any] = dict(),
accept_truncated: bool = False,
policy: WirePolicy | None = None,
stream_internally: bool = False,
max_steps: int | None = None,
tool_output_chars: int | None = None,
context: ContextMode = "off",
trim: TrimPolicy | None = None,
cache_prefix: bool = False,
)
One generation surface and the settings its calls use.
Attributes:
-
TIERS(Mapping[str, Tier]) –The class's tier table. Empty on the base class, so the base class cannot be instantiated; a consumer subclasses it with its own table (see the module docstring).
-
name(str) –feature.action. -
tier(str) –A key of
TIERS. -
category(str) –The category a switchboard and a trace group by; defaults to
name. -
model(str | None) –Overrides the tier's model.
-
pack(str | None) –The prompt pack id.
-
max_tokens(int | None) –The token ceiling for the reply.
-
temperature(float | None) –The sampling temperature.
-
thinking(bool) –The thinking switch sent to a self-hosted tier; off by default. On a hybrid thinking model the grammar binds only the content, and reasoning otherwise spends the token budget.
-
output_retries(int) –Re-asks allowed when the validator finds problems.
-
timeout_s(float | None) –Overrides the tier's timeout, in seconds.
-
graders(tuple[str, ...]) –Names of
call.gradersentries thatcall.generatescores everyokreply with. -
settings(Mapping[str, Any]) –Per-role sampling the tier does not decide (seed, top_p, temperature).
-
accept_truncated(bool) –A reply the token ceiling cut is
ok(with aCallWarning) instead oftruncated: the 1.x behaviour, for a role whose consumer reads a partial reply. -
policy(WirePolicy | None) –How the role's wire requests are retried; None takes the tier's declared policy, else the generator's default.
-
stream_internally(bool) –The adapter requests a stream and assembles it, so a cancelled or timed-out call closes a connection the server can stop on. Off by default.
-
max_steps(int | None) –The most model requests in one
call.chat, tool rounds included; None takesChatRequest's default (8). -
tool_output_chars(int | None) –The one tool-output budget of the role's chat calls, in characters; None for no cut. A
Tool.max_output_charsoverrides it for that tool. -
context(ContextMode) –Before a call is sent,
off(the default, costs nothing),check(count it and refuse an overflow ascontext_overflowwithprovider_code="preflight"), ortrim(cut the history bytrimto fit, then refuse what still does not). ADR-0024. -
trim(TrimPolicy | None) –The trimming policy for
context="trim"; None isTrimPolicy(). -
cache_prefix(bool) –Ask for a prompt-cache breakpoint after the stable prefix: the system text (and, in a chat, the tool definitions). Dropped with a warning on a model that does not state
cache.
Raises:
-
ValueError–tieris not inTIERS.
resolved_timeout_s
property
¶
resolved_timeout_s: float
The role's timeout override, else the tier's timeout, in seconds.
model_settings
¶
model_settings() -> dict[str, Any]
Return the settings this role sends.
On a schema 2 tier the thinking switch is provider-neutral, thinking: bool; the
provider's factory spells it the way the model's ModelInfo.thinking says. On a schema 1
tier a self-hosted model gets extra_body.chat_template_kwargs.enable_thinking, which
llama.cpp and vLLM read, as in 1.22.
load_tiers
¶
load_tiers(
source: Path | Mapping[str, Any],
*,
check_ladder: LadderCheck | None = None
) -> TierTable
Parse and check a tier table.
Parameters:
-
source(Path | Mapping[str, Any]) –The TOML file, or the table itself as a mapping with the same keys.
-
check_ladder(LadderCheck | None, default:None) –The consumer's own ordering rule over the rungs (e.g. a client timeout that must clear a server deadline). Called as
check_ladder(ladder, path)before the tiers resolve, withMAPPING_SOURCEas the path of a mapping; it raisesValueErrorto refuse the table.
Raises:
-
ValueError–A tier names a
[models]key or a[ladder]rung the table does not have, or sets bothself_hostedand its deprecated aliasbox. Every refusal of a schema 2 table is the subclasscall.models.TableInvalid, naming the path.
llm_ports.call.context¶
Context: whether a request fits its model's window, what is cut when it does not, and cache hints.
Pure functions, no I/O. The call path (call.chat, call.generate) counts a request with a
TokenCounter before it is sent, reads the verdict from fit, and on Role.context="trim" cuts
the history with trim.
- A count is exact or absent (
ports.Counted): no character-ratio estimate is ever a token count. An unknown fit sends the request; it never blocks. - Trimming is deterministic, structural and reported. Order: tool outputs over the policy's
budget are cut in the middle (
cut_middle); then whole turns are dropped oldest-first betweenkeep_firstandkeep_last, never splitting a tool call from its result. The system prompt is never cut. - A cache hint is a request, not a guarantee (
cache_plan);Usage.cache_readon the result is the proof.
Rationale: ADR-0024.
Budget
dataclass
¶
Budget(
context_window: int | None,
reserve_output: int,
margin: float = 0.05,
)
The input tokens a call may send.
Attributes:
-
context_window(int | None) –The model's window, from its
ModelInfo; None when unknown. -
reserve_output(int) –The tokens held for the reply (the role's
max_tokens). -
margin(float) –The fraction of the rest held back for counting error.
input_limit
property
¶
input_limit: int | None
(context_window - reserve_output) * (1 - margin), rounded down; None when the window is unknown.
Fit
dataclass
¶
Fit(
verdict: Literal["fits", "over", "unknown"],
tokens: int | None,
limit: int | None,
)
Whether a counted request fits its budget.
Attributes:
-
verdict(Literal['fits', 'over', 'unknown']) –fits,over, orunknown(no count, or no window). -
tokens(int | None) –The count; None when there was none.
-
limit(int | None) –The budget's input limit; None when the window is unknown.
TrimPolicy
dataclass
¶
TrimPolicy(
keep_first: int = 1,
keep_last: int = 4,
tool_output_chars: int = 4000,
drop: Literal["oldest_turns"] = "oldest_turns",
)
How a history is cut to fit.
Attributes:
-
keep_first(int) –Leading messages never dropped (the task statement).
-
keep_last(int) –Trailing messages never dropped.
-
tool_output_chars(int) –The longest tool result kept whole; a longer one is cut in the middle.
-
drop(Literal['oldest_turns']) –What is dropped after the tool outputs: whole turns, oldest first.
Trimmed
dataclass
¶
Trimmed(
messages: tuple[Message, ...],
dropped_messages: int,
cut_tool_outputs: int,
chars_removed: int,
)
What trim returned and what it removed.
Attributes:
-
messages(tuple[Message, ...]) –The cut history.
-
dropped_messages(int) –Whole messages dropped.
-
cut_tool_outputs(int) –Tool results cut in the middle.
-
chars_removed(int) –Characters removed, over both.
fit
¶
Return whether counted fits budget. Pure; an absent count or window is unknown.
cut_middle
¶
cut_middle(text: str, max_chars: int) -> tuple[str, int]
Cut text to at most max_chars, keeping its head and its tail. Pure.
The tail usually holds the answer or the error, so the middle goes; the marker
[... N chars omitted] says how much.
Returns:
-
tuple[str, int]–The text, and the characters removed (0 when it already fit).
trim
¶
trim(
messages: Sequence[Message],
policy: TrimPolicy,
*,
over_by_tokens: int | None
) -> Trimmed
Cut messages by policy. Pure and deterministic.
Tool results over policy.tool_output_chars are cut in the middle first. Then whole turns
(a user message and every message up to the next one) are dropped oldest-first, while every
message of the turn lies between the first keep_first and the last keep_last. A turn is
never split, so a tool call is never separated from its result.
Parameters:
-
messages(Sequence[Message]) –The history.
-
policy(TrimPolicy) –What may be cut.
-
over_by_tokens(int | None) –How far over the budget the history is. Turns are dropped until at least that many characters are gone (a token is at least one character, so fewer cannot be enough; the caller counts again to confirm). None drops every turn the policy allows.
Returns:
-
Trimmed–The cut history and what was removed.
cache_plan
¶
cache_plan(
info: ModelInfo | None,
*,
wanted: bool,
settings: Mapping[str, Any]
) -> tuple[bool, dict[str, Any], tuple[CallWarning, ...]]
Translate a cache hint by the model's stated capability. Pure.
Parameters:
-
info(ModelInfo | None) –The model; None (a schema 1 tier) states nothing, and the hint passes to the adapter.
-
wanted(bool) –Whether the call asked for a cache breakpoint.
-
settings(Mapping[str, Any]) –The call's settings.
Returns:
-
bool–Whether the adapter still places breakpoints, the settings to send, and warnings. A model
-
dict[str, Any]–without
cachedrops the hint with aCallWarning(kind="unsupported"). A self-hosted -
tuple[CallWarning, ...]–model with
cachecaches prefixes by slot: it getsextra_body.cache_promptand no -
tuple[bool, dict[str, Any], tuple[CallWarning, ...]]–breakpoint. A hosted model with
cachekeeps the hint for the adapter.
volatile_system
¶
volatile_system(
pack: Pack,
contexts: Sequence[Mapping[str, Any]],
*,
template: str = "system"
) -> bool
True when the pack's system template renders differently across contexts.
A system text that interpolates a per-call value defeats a cache breakpoint after it: lint a
pack whose role sets cache_prefix with two representative contexts.
Parameters:
-
pack(Pack) –The pack.
-
contexts(Sequence[Mapping[str, Any]]) –Render contexts; two or more.
-
template(str, default:'system') –The system template's name in the manifest.
Returns:
-
bool–False when the pack has no such template.
llm_ports.call.models¶
The model table's value types: what a model can do, who serves it, and the closed vocabularies.
A schema_version = 2 tier table (call.roles.load_tiers) adds [providers.<name>] and
[models.<key>] sections. A model key names one wire id, the provider that serves it, and what it
can do, stated by the consumer in a closed vocabulary (CAPABILITIES): nothing is inferred from a
model's name and nothing comes from a bundled database. A provider names a registered kind (the
factory that builds its client) and, by NAME only, the URL key and the secret it needs: the table
is reviewed and fingerprinted, so it never holds a URL or a key.
Example
schema_version = 2
[providers.gateway] kind = "openai_compatible" base_url = "gateway" # a key the caller resolves to a URL secret = "gateway_key" # a name the caller resolves to a key
[models.chat] id = "qwen3.5-9b" provider = "gateway" supports = ["json_schema", "vision"] thinking = "chat_template"
Invariants
- A capability word, thinking style or failure kind outside its vocabulary is
TableInvalidat load, naming the path (models.chat.supports[1]). - An unknown limit is None, never a guessed number.
- Pure: no I/O, no vendor import.
Rationale: ADR-0022.
TableInvalid
¶
TableInvalid(path: str, message: str)
Bases: ValueError
A tier table that cannot be used, raised at load or at build, naming where.
Attributes:
-
path(str) –The dotted path of the offending value (
tiers.box.fallback[0]).
ModelInfo
dataclass
¶
ModelInfo(
key: str,
id: str,
provider: str,
supports: frozenset[Capability],
context_window: int | None = None,
max_output: int | None = None,
thinking: ThinkingStyle = "none",
slots: int | None = None,
group: str | None = None,
self_hosted: bool = True,
usd_per_request: Decimal | None = None,
)
One model of a schema 2 table: its wire id, its provider, and what it can do.
Attributes:
-
key(str) –The model's key in
[models]. -
id(str) –The id sent on the wire (a gateway route is fine). The only place it is written.
-
provider(str) –The key of the
[providers]entry that serves it. -
supports(frozenset[Capability]) –The capabilities the consumer states for it.
-
context_window(int | None) –Tokens in its context window; None when unknown.
-
max_output(int | None) –The most output tokens it returns; None when unknown.
-
thinking(ThinkingStyle) –How its thinking switch is spelled.
-
slots(int | None) –Calls it takes at once in-process; None when not limited here.
-
group(str | None) –Models sharing one backend resource; None when it shares none.
-
self_hosted(bool) –True when the consumer runs the model itself.
-
usd_per_request(Decimal | None) –A flat price per request; None when the cost is unknown.
ProviderSpec
dataclass
¶
ProviderSpec(
name: str,
kind: str,
base_url: str | None,
secret: str | None,
gateway: str | None = None,
options: Mapping[str, Any] = dict(),
)
One provider of a schema 2 table: a registered kind, and the names of what it needs.
Attributes:
-
name(str) –The provider's key in
[providers]. -
kind(str) –The registered provider kind that builds its client (
openai_compatible). -
base_url(str | None) –A topology key the caller resolves to a URL; None when the kind needs none.
-
secret(str | None) –A secret name the caller resolves to a key; None for a keyless endpoint.
-
gateway(str | None) –Which gateway's response headers the provider reads, when one stands in front.
-
options(Mapping[str, Any]) –Every other key of the section, validated by the kind's factory.
parse_providers
¶
parse_providers(
raw: Mapping[str, Any],
) -> dict[str, ProviderSpec]
Parse a [providers] section.
Parameters:
-
raw(Mapping[str, Any]) –Provider name to its section.
Returns:
-
dict[str, ProviderSpec]–Provider name to
ProviderSpec, in the section's order.
Raises:
-
TableInvalid–A section is not a table,
kindis missing, a value has the wrong type, orbase_urlholds a URL rather than a key.
parse_models
¶
parse_models(
raw: Mapping[str, Any],
providers: Mapping[str, ProviderSpec],
) -> dict[str, ModelInfo]
Parse a schema 2 [models] section against its providers.
Parameters:
-
raw(Mapping[str, Any]) –Model key to its section.
-
providers(Mapping[str, ProviderSpec]) –The parsed
[providers]section.
Returns:
-
dict[str, ModelInfo]–Model key to
ModelInfo, in the section's order.
Raises:
-
TableInvalid–An unknown key, a missing
idorprovider, a provider the table does not declare, a word outside its vocabulary, or a value of the wrong type.
needs
¶
needs(
*, structured: bool, images: bool, web: bool
) -> frozenset[Capability]
Return the capabilities one request needs. Pure.
Parameters:
-
structured(bool) –The output is a schema, not prose.
-
images(bool) –The request carries images.
-
web(bool) –The model is asked to fetch pages.
Returns:
-
frozenset[Capability]–visionfor images,webfor a page fetch, andjson_schemafor structured output -
frozenset[Capability]–(
missingacceptsjson_objectin its place).
missing
¶
missing(
info: ModelInfo, wanted: frozenset[Capability]
) -> tuple[Capability, ...]
Return the capabilities in wanted that info lacks, in vocabulary order. Pure.
json_object stands in for json_schema: either lets a model return structured output.
translate_settings
¶
translate_settings(
settings: Mapping[str, Any], info: ModelInfo | None
) -> tuple[dict[str, Any], tuple[str, ...]]
Spell a request's provider-neutral thinking switch the way its model needs. Pure.
Parameters:
-
settings(Mapping[str, Any]) –The request's settings; a schema 2 role puts
thinking: boolin them. -
info(ModelInfo | None) –The request's model, or None when the table does not know it (the settings pass through unchanged, so pydantic-ai's own unified
thinkingapplies).
Returns:
-
dict[str, Any]–The settings to send, and the names of settings the model cannot take (dropped):
-
tuple[str, ...]–unifiedkeepsthinking(pydantic-ai'sModelSettings["thinking"]);chat_template -
tuple[dict[str, Any], tuple[str, ...]]–moves it to
extra_body.chat_template_kwargs.enable_thinking;nonedrops it, and names -
tuple[dict[str, Any], tuple[str, ...]]–it when the role asked to think.
llm_ports.call.providers¶
Providers by name: the factory a kind registers, and build, which turns a table into backends.
A schema 2 tier table names, for every model, the provider that serves it; a provider names a
registered KIND (openai_compatible) and, by name only, the URL key and the secret it needs.
build resolves those names through the caller's urls and secrets, hands each provider to its
kind's ProviderFactory, and returns Backends, which a consumer's facade asks for the
Generator of a tier:
providers = adapters.pluggy.load_providers(builtins=[builtin("openai_compatible", factory)], pin=pin)
backends = build(TABLE, urls=topology, secrets=vault.get, deps=BuildDeps(), factories=providers)
res = await call.generate(ROLE, prompt=text, generator=backends.generator_for(ROLE.tier))
Kinds arrive through the provider registry (adapters.pluggy.load_providers, entry-point group
ENTRY_POINT_GROUP), or as a plain mapping in a test. Only the core types live here: no vendor
library and no entry-point reading.
Invariants
- Everything that can be wrong with a table and its environment is raised at
build, at boot: an unknown kind, a URL key the caller cannot resolve, a secret that resolves to nothing or to a value no header can carry (whitespace, a control or a non-ASCII character). Nothing waits for the first call to find a missing key. - A secret value is never logged, stamped or put in an error; only its name is.
- A kind's key is flat and global: two plugins registering one kind raise, naming both.
- A registry is frozen once loaded: a later
addraises.
Rationale: ADR-0022.
Loaded
dataclass
¶
Loaded(
group: str,
plugins: tuple[str, ...],
origin: Mapping[str, str] = dict(),
)
Which plugins a process loaded into one registry, and from where.
Attributes:
-
group(str) –The entry-point group.
-
plugins(tuple[str, ...]) –Plugin names in load order: built-ins as given, then entry points sorted by name.
-
origin(Mapping[str, str]) –Plugin name to
builtin,caller, or<distribution>==<version>.
BuildDeps
dataclass
¶
BuildDeps(
policy: WirePolicy | None = None,
limiter: Limiter | None = None,
pricer: Pricer | None = None,
transport: Any | None = None,
on_event: EventHook | None = None,
)
What every factory may share: the call's planes, passed once to build.
Attributes:
-
policy(WirePolicy | None) –The wire policy every built client uses when a request carries none; None takes each adapter's own default.
-
limiter(Limiter | None) –The concurrency limiter every built generator takes its slots from; None for no in-process limit.
-
pricer(Pricer | None) –Prices a reply's usage the gateway did not price.
-
transport(Any | None) –An HTTP transport every client is built on (a test's mock, a counting proxy).
-
on_event(EventHook | None) –Receives the build's events and every built client's; without one, the stdlib logger
llm_ports.call.
Built
dataclass
¶
Built(
generator: Generator | None = None,
chat: Chat | None = None,
counter_for: (
Callable[[str], TokenCounter | None] | None
) = None,
embedder_for: Callable[[str], Embedder] | None = None,
reranker_for: Callable[[str], Reranker] | None = None,
probe: Probe | None = None,
)
What one factory built for one provider. A kind builds the ports it serves; the rest are None.
Attributes:
-
generator(Generator | None) –The
Generatorfor every model of the provider (the model id is on the request). -
chat(Chat | None) –The
Chatport for every model of the provider. -
counter_for(Callable[[str], TokenCounter | None] | None) –Model id to a
TokenCounter, or None for a model that cannot be counted. -
embedder_for(Callable[[str], Embedder] | None) –Model id to an
Embedder. -
reranker_for(Callable[[str], Reranker] | None) –Model id to a
Reranker. -
probe(Probe | None) –The provider's health probe.
ProviderFactory
¶
Bases: Protocol
Builds the clients of one provider kind.
build
¶
build(
spec: ProviderSpec,
*,
url: str | None,
secret: str | None,
models: Sequence[ModelInfo],
deps: BuildDeps
) -> Built
Build the provider's clients.
Parameters:
-
spec(ProviderSpec) –The provider's section; the factory validates
spec.options. -
url(str | None) –The resolved URL of
spec.base_url; None when the provider names none. -
secret(str | None) –The resolved secret of
spec.secret; None for a keyless provider. -
models(Sequence[ModelInfo]) –The table's models this provider serves, in table order.
-
deps(BuildDeps) –The shared planes.
Raises:
-
TableInvalid–The provider's section asks for something the kind cannot do.
DuplicateProvider
¶
Bases: ValueError
Two plugins registered one provider kind; the message names both.
ProviderRegistry
¶
ProviderRegistry(factories: Sequence[ProviderFactory] = ())
Bases: Mapping[str, ProviderFactory]
Provider kind to its factory, with the registry rules: a duplicate raises, a frozen one is closed.
Parameters:
-
factories(Sequence[ProviderFactory], default:()) –Factories to add now, as plugin
caller.
loaded
property
¶
loaded: Loaded
What loaded the registry; derived from the plugins so far while it is open.
add
¶
add(
factory: ProviderFactory, *, plugin: str = "caller"
) -> None
Register factory under its kind, contributed by plugin.
Raises:
-
TypeError–factoryis not aProviderFactory. -
DuplicateProvider–Another plugin registered the kind.
-
RuntimeError–The registry is frozen.
Backends
¶
The clients build made, looked up by tier, model key or provider.
Attributes:
-
table(TierTable) –The table the backends were built from.
-
loaded(Loaded) –Which provider kinds were registered, by which plugins, from where.
generator_for
¶
generator_for(tier: str) -> Generator
The Generator that serves tier's model.
Raises:
-
TableInvalid–The tier is unknown.
-
LookupError–The provider's kind builds no
Generator.
chat_for
¶
chat_for(tier: str) -> Chat
The Chat port that serves tier's model.
Raises:
-
TableInvalid–The tier is unknown.
-
LookupError–The provider's kind builds no chat port.
counter_for
¶
counter_for(tier: str) -> TokenCounter | None
The TokenCounter for tier's model; None when the model does not state count_tokens.
A None is the pre-flight's unknown: the call is sent unchecked, never refused.
Raises:
-
TableInvalid–The tier is unknown.
embedder_for
¶
embedder_for(model_key: str) -> Embedder
The Embedder for the model under model_key.
Raises:
-
TableInvalid–The key is unknown, or the model does not state
embed. -
LookupError–The provider's kind builds no
Embedder.
reranker_for
¶
reranker_for(model_key: str) -> Reranker
The Reranker for the model under model_key.
Raises:
-
TableInvalid–The key is unknown, or the model does not state
rerank. -
LookupError–The provider's kind builds no
Reranker.
probe_for
¶
probe_for(provider: str) -> Probe
The health probe of provider.
Raises:
-
TableInvalid–The provider was not built (unknown, or no model uses it).
-
LookupError–Its kind builds no probe.
aclose
async
¶
aclose() -> None
Close every built client that holds a resource, each once (a kind may share one). Idempotent.
build
¶
build(
table: TierTable,
*,
urls: Mapping[str, str],
secrets: Callable[[str], str | None],
deps: BuildDeps,
factories: Mapping[str, ProviderFactory] | None = None
) -> Backends
Build the clients of every provider the table's models use.
Parameters:
-
table(TierTable) –A schema 2 tier table.
-
urls(Mapping[str, str]) –Topology key to URL, for each provider's
base_url. -
secrets(Callable[[str], str | None]) –Secret name to its value, or None when the caller has none by that name. Called once per provider that names a secret; the value goes to the factory and nowhere else.
-
deps(BuildDeps) –The shared planes every factory gets.
-
factories(Mapping[str, ProviderFactory] | None, default:None) –Kind to factory: a
ProviderRegistryfromadapters.pluggy.load_providers, or a plain mapping. None registers no kind, so any provider is refused.
Returns:
-
Backends–The
Backends, after emittingllm_backends_built(the table's fingerprint, each -
Backends–provider's kind, and which plugins registered the kinds).
Raises:
-
TableInvalid–The table is schema 1, a provider names an unregistered kind, a URL key the caller does not resolve, or a secret that resolves to nothing or to a value holding whitespace, a control or a non-ASCII character (the error names the secret, never its value); or a factory refuses its section.
llm_ports.call.graders¶
Deterministic graders over one generation: the prompt in, the value out.
Pure. A CallGrader is (prompt, value, role) -> CallScore: a numeric value (what a trace score
and a dashboard can sum), a pass or fail against the grader's own bar, and a reason that names the
violated property on failure. call.generate runs the graders a Role.graders names on every
ok reply and writes each as a score on the call's trace observation. A golden eval runs the same
functions over a pack's golden set (board_graders), so the live score and the offline verdict
agree on what "complete" or "echoed" means.
Built-in graders, for structured outputs (a dict or pydantic value under a grammar):
schema_complete: the share of fields that carry a value. A strict grammar admits""for every required key, so a parsed reply is not necessarily an answered one. An empty list is an answer; only None and blank text count as unanswered. On a pydantic value, None on a field whose annotation admits None (float | None,Optional[X]) is an answer too, while None on a required field fails. A dict value has no annotations to read, so every None in it is unanswered, unless the caller declares the answer paths:schema_complete(..., model=Out)reads them off a model,nullable={"items[].stop"}names them, andpartial(schema_complete, nullable=...)binds them into a table entry.numbers_from_inputs: figures in the reply's prose that no part of the prompt rendered, digit runs by value and spelled numbers alike (numbers.loose): the number law, per call.echo: the share of the reply's sentences copied verbatim from the prompt.cites_url: the distinct http(s) URLs the reply carries, in any string field. A web-fetching tier's reply that cites nothing cannot be checked against its pages. Web calls only.no_refusal: the stock phrases of an ungrounded refusal ("I don't have real-time access", "as of my training data", "please consult ..."). An honest "the sources do not cover X" is not matched.
A consumer adds its own graders by passing a table ({**GRADERS, "mine": fn}) to score_all,
board_graders or call.Hooks(graders=...), or a registry to prompts.resolve. GRADERS is a
read-only mapping: assigning into it raises TypeError, so one consumer's grader cannot leak into
another's table.
Invariants
score_oneandscore_allnever raise. A grader's exception becomes a failed score of valueCRASHED(-1, below every honest value, all of which are >= 0) whose reason names the exception, so a crashed grader never reads as a pass.
Rationale: ADR-0003.
CallScore
dataclass
¶
CallScore(
name: str, value: float, passed: bool, reason: str = ""
)
One grader's verdict on one reply.
Attributes:
-
name(str) –The grader's name.
-
value(float) –The numeric score;
CRASHEDwhen the grader raised. -
passed(bool) –Whether the value meets the grader's own bar.
-
reason(str) –Empty on a pass; names the violated property on a failure.
leaves
¶
leaves(
value: Any, path: str = ""
) -> Iterator[tuple[str, Any]]
Yield (path, scalar) for every leaf of a dict, pydantic or list value, in order.
Pure. An empty list is itself a leaf.
prose
¶
prose(value: Any) -> list[str]
Return every string leaf of the reply: what the number law and the echo check read.
nullable_paths
¶
nullable_paths(
model: type[BaseModel],
prefix: str = "",
_seen: frozenset[type] = frozenset(),
) -> frozenset[str]
Return the index-free leaf paths of every field whose annotation admits None.
Pure. Paths (items[].price) run through nested models and lists of models. They are read off
the class, so they never depend on which rows a reply carried. A self-referential model stops at
its own repeat.
schema_complete
¶
schema_complete(
_prompt: str,
value: Any,
_role: Role,
*,
model: type[BaseModel] | None = None,
nullable: Iterable[str] | None = None
) -> CallScore
Score the share of the reply's fields that carry a value.
Passes only when no field is unanswered. None and blank text are unanswered; an empty list is an answer, and so is None on a path declared nullable.
Parameters:
-
_prompt(str) –The rendered prompt (unused).
-
value(Any) –The reply: a dict, a list or a pydantic value.
-
_role(Role) –The call's role (unused).
-
model(type[BaseModel] | None, default:None) –The declared output model; its
nullable_pathsare the answer paths for any value, a dict reply included. -
nullable(Iterable[str] | None, default:None) –Answer paths named directly, index-free (
items[].stop). Taken withmodel, the two sets are joined.
Returns:
-
CallScore–A
CallScore; never raises on a valueleavescan walk.
Invariants
- With neither
modelnornullable, a pydantic value's own annotations decide and a dict's every None is unanswered: the grader the built-in table carries. - A declared path exempts None and blank text only; a non-empty value scores the same.
numbers_from_inputs
¶
Count the distinct figures in the reply's prose that the prompt did not render.
Digit runs compare by value, and spelled numbers count too (numbers.loose). The prompt's own
text licenses a spelled word, and '%' licenses 'percent'. Passes only at zero.
Rationale: ADR-0003.
echo
¶
Score the share of the reply's sentences copied verbatim from the prompt.
The compare ignores case and whitespace, and a sentence shorter than ECHO_MIN_CHARS is not
counted; a reply with no counted sentence passes at 0. Fails at or above ECHO_MAX_SHARE.
cites_url
¶
Count the distinct http(s) URLs in the reply's string fields. Passes at one or more.
A web call only (WEB_ONLY_GRADERS): a text call has no pages to cite.
no_refusal
¶
Count the ungrounded-refusal phrases in the reply's prose. Passes only at zero.
graders_for_inputs
¶
graders_for_inputs(
graders: tuple[str, ...],
*,
has_images: bool,
has_web: bool = False
) -> tuple[str, ...]
Return the graders that can see this call's inputs, in their given order.
Pure. An image beside the prompt or a web-fetching tier removes TEXT_ONLY_GRADERS; a call
without a web-fetching tier removes WEB_ONLY_GRADERS.
score_one
¶
score_one(
name: str,
prompt: str,
value: Any,
role: Role,
table: Mapping[str, CallGrader] | None = None,
) -> CallScore
Run one grader by name from table (default GRADERS).
Never raises: an unknown name or a grader exception becomes a failed score of value CRASHED.
score_all
¶
score_all(
role: Role,
prompt: str,
value: Any,
table: Mapping[str, CallGrader] | None = None,
) -> tuple[CallScore, ...]
Run every grader the role names, in the role's order; never raises.
board_graders
¶
board_graders(
role: Role,
table: Mapping[str, CallGrader] | None = None,
) -> dict[
str, Callable[[dict[str, Any]], tuple[bool, str]]
]
Return the same graders in the Evaluator row shape (ports.Grader).
A golden row carries prompt and value. A row whose model never answered (value is None)
fails every grader with its reason, so the board names the outage rather than grading an
absence.
llm_ports.call.schema¶
JSON-object extraction and the research grammar.
The strict-schema transform (strict_json_schema) wraps pydantic-ai's, so it lives with that
adapter: llm_ports.adapters.pydantic_ai.
extract_json_object: the JSON-object substring of a model response that is fenced, or prefixed or suffixed by prose.research_schema,research: a free-textanalysisscratchpad first, every property required, and the scratchpad popped from the value afterwards.
research_schema
¶
research_schema(
schema: Mapping[str, Any],
) -> dict[str, Any]
Return the research grammar for schema.
Pure. A free-text analysis property (RESEARCH_ANALYSIS) is injected first, and required
lists every property, including the ones pydantic marks optional because they have a default.
without_analysis
¶
Return res with the analysis scratchpad popped from a dict value; other values pass unchanged.
research
async
¶
research(
generate: Callable[..., Awaitable[Generated[Any]]],
role: Role,
*,
prompt: str,
schema: Mapping[str, Any],
name: str,
**kwargs: Any
) -> Generated[dict[str, Any]]
Run a research call that returns a grammar-enforced dict.
The consumer's generate (its hooks bound) runs with research_schema(schema) as the grammar
(json_output(..., name)), and the scratchpad is popped from the value. kwargs pass through
to generate unchanged.
extract_json_object
¶
extract_json_object(text: str) -> str
Return the JSON-object substring of a model response.
Tolerates markdown fences and leading or trailing prose: some models fence their JSON even when
asked not to, and some routes cannot enforce response_mime_type. A leading code fence is
stripped, then the first { through the last } is taken, so json.loads succeeds regardless
of surrounding text. Falls back to the stripped text when no object is found, so the caller's
JSON error stays meaningful.