Skip to content

llm_ports.ports

llm_ports.ports

The ports: Protocols and the frozen value types that cross them. Zero I/O.

outcomes holds the named-outcome vocabulary: reasons, GenerationFailed, and the HTTP-status mapping every adapter shares.

ChatRequest dataclass

ChatRequest(
    model: str,
    messages: tuple[Message, ...],
    system: str = "",
    tools: tuple[Tool, ...] = (),
    tool_choice: str = "auto",
    max_steps: int = 8,
    parallel_tools: bool = True,
    settings: Mapping[str, Any] = dict(),
    deadline: Deadline | None = None,
    timeout_s: float = 120.0,
    concurrency_group: str | None = None,
    cache_system: bool = False,
    cache_tools: bool = False,
    label: str = "chat",
    self_hosted: bool = True,
    output_retries: int = 1,
    policy: WirePolicy | None = None,
    tool_output_chars: int | None = None,
    streamed: bool = True,
)

One chat call, already resolved from a role.

Attributes:

  • model (str) –

    The model id.

  • messages (tuple[Message, ...]) –

    The history, oldest first; the last is normally the user's turn.

  • system (str) –

    The system prompt; empty for none. Never trimmed.

  • tools (tuple[Tool, ...]) –

    The tools the model may call.

  • tool_choice (str) –

    auto, none, required, or one tool's name.

  • max_steps (int) –

    The most model requests in the call, tool rounds and re-asks included. Reaching it with a tool call still pending is truncated (provider_code="max_steps").

  • parallel_tools (bool) –

    Let the model ask for several tool calls in one step.

  • settings (Mapping[str, Any]) –

    Sent to the model verbatim (max_tokens, temperature, ...).

  • deadline (Deadline | None) –

    The call's clocks; None means Deadline(total_s=timeout_s).

  • timeout_s (float) –

    The call's bound when deadline is None, in seconds.

  • concurrency_group (str | None) –

    The Limiter group; None means the model id.

  • cache_system (bool) –

    Ask for a cache breakpoint after the system prompt (ADR-0024).

  • cache_tools (bool) –

    Ask for a cache breakpoint after the tool definitions.

  • label (str) –

    The call's name in an adapter's logs.

  • self_hosted (bool) –

    True for a self-hosted tier.

  • output_retries (int) –

    Re-asks for an output or a tool call the model got wrong.

  • policy (WirePolicy | None) –

    How the adapter retries a wire request; None is the adapter's default.

  • tool_output_chars (int | None) –

    The request's tool-output budget, for a tool that sets none; None for no cut.

  • streamed (bool) –

    Stream each model reply as it is generated. False (a model that cannot stream) sends each step's whole text as one TextDelta.

Raises:

  • ValueError –

    max_steps is below 1.

clocks property

clocks: Deadline

deadline, or Deadline(total_s=timeout_s) when none was given.

Finished dataclass

Finished(result: Generated[T])

The stream's last event, exactly once, in every case: success, failure, deadline.

Message dataclass

Message(
    role: ChatRole,
    parts: tuple[Part, ...],
    cache: bool = False,
)

One turn of a conversation.

Attributes:

  • role (ChatRole) –

    user, assistant (text, reasoning and tool calls) or tool (tool results).

  • parts (tuple[Part, ...]) –

    The turn's parts, in order.

  • cache (bool) –

    Ask for a prompt-cache breakpoint after this message (ADR-0024). A request, not a guarantee: Usage.cache_read on the result says whether it was served.

text property

text: str

The message's text parts, joined; reasoning and tool parts are not included.

Part dataclass

Part(
    kind: PartKind,
    text: str | None = None,
    data: bytes | None = None,
    media_type: str | None = None,
    call_id: str | None = None,
    name: str | None = None,
    args: Mapping[str, Any] | None = None,
    untrusted: bool = False,
)

One piece of a message.

Attributes:

  • kind (PartKind) –

    What the part holds.

  • text (str | None) –

    The text of a text, reasoning or tool_result part.

  • data (bytes | None) –

    An image's bytes.

  • media_type (str | None) –

    An image's MIME type.

  • call_id (str | None) –

    The id that pairs a tool_call with its tool_result.

  • name (str | None) –

    The tool's name, on a tool_call or a tool_result.

  • args (Mapping[str, Any] | None) –

    A tool_call's arguments.

  • untrusted (bool) –

    True for a tool_result whose content came from outside (the default for a tool's output); the adapter marks it as data when it sends it.

ReasoningDelta dataclass

ReasoningDelta(text: str)

A piece of the model's reasoning; never part of the value, never sent back to a provider.

StepFinished dataclass

StepFinished(
    n: int, finish: Finish | None, usage: Usage | None
)

One model request of the call ended.

Attributes:

  • n (int) –

    The step's number, from 1.

  • finish (Finish | None) –

    How the reply ended; None when the provider said nothing.

  • usage (Usage | None) –

    The step's own usage; None when unmeasured.

TextDelta dataclass

TextDelta(text: str)

A piece of the reply's text, as it was generated.

Tool dataclass

Tool(
    name: str,
    description: str,
    parameters: Mapping[str, Any],
    handler: ToolHandler,
    timeout_s: float = 30.0,
    untrusted_output: bool = True,
    max_output_chars: int | None = None,
)

A client-side tool the model may call.

Attributes:

  • name (str) –

    The name the model calls it by.

  • description (str) –

    When and why to call it, for the model.

  • parameters (Mapping[str, Any]) –

    The arguments' JSON Schema, of type object.

  • handler (ToolHandler) –

    Runs one call; a coroutine function.

  • timeout_s (float) –

    The most one call may take; past it the call is cancelled and the model is shown an error result.

  • untrusted_output (bool) –

    What the tool returns is data from outside (the default): it is wrapped in a <tool_output ... untrusted="true"> boundary marker. The marker is something a system prompt can name; it is not a defence on its own.

  • max_output_chars (int | None) –

    The longest output the model is shown; None leaves it to the request's tool_output_chars. A longer one is cut in the middle with a marker.

Raises:

  • TypeError –

    handler is not a coroutine function.

  • ValueError –

    parameters is not an object schema, or timeout_s is not positive.

ToolCallFinished dataclass

ToolCallFinished(
    call_id: str,
    name: str,
    output: ToolOutput,
    duration_s: float,
    truncated: bool = False,
)

A tool call ended.

Attributes:

  • call_id (str) –

    The call's id.

  • name (str) –

    The tool's name.

  • output (ToolOutput) –

    What the handler returned (before the untrusted marker); an error result when it raised, timed out, or the tool or its arguments were wrong.

  • duration_s (float) –

    How long the handler ran, in seconds; 0 when it never ran.

  • truncated (bool) –

    True when the output was cut to its budget.

ToolCallStarted dataclass

ToolCallStarted(
    call_id: str, name: str, args: Mapping[str, Any]
)

The model called a tool; the handler is about to run.

ToolOutput dataclass

ToolOutput(
    content: str,
    is_error: bool = False,
    meta: Mapping[str, Any] = dict(),
)

What a tool handler returns.

Attributes:

  • content (str) –

    What the model is shown.

  • is_error (bool) –

    True when the tool failed; the model sees the content and may recover.

  • meta (Mapping[str, Any]) –

    For the consumer's UI; never sent to the model.

Failure dataclass

Failure(
    kind: FailureKind,
    retry_after_s: float | None = None,
    status: int | None = None,
    provider_code: str | None = None,
    phase: Phase | None = None,
    in_stream: bool = False,
    reason: str = "",
)

One failed call's kind and the facts a retry or fallback decision reads.

Attributes:

  • kind (FailureKind) –

    What went wrong.

  • outcome (Outcome) –

    The coarse outcome; derived from kind by OUTCOME_FOR, never passed in.

  • retryable (bool) –

    Whether a retry can help; derived from kind by RETRYABLE, never passed in.

  • retry_after_s (float | None) –

    The server's own wait hint, in seconds, when it gave one.

  • status (int | None) –

    The HTTP status, when there was one.

  • provider_code (str | None) –

    The provider's error code or type string, verbatim.

  • phase (Phase | None) –

    Where a transport or deadline failure happened, when known.

  • in_stream (bool) –

    True when the failure came after output had started.

  • reason (str) –

    The one-line reason a caller persists (the 1.x reason).

Invariants
  • outcome == OUTCOME_FOR[kind] and retryable == (kind in RETRYABLE).
  • reason is at most outcomes.REASON_MAX characters.

Rationale: ADR-0020.

Health dataclass

Health(
    state: HealthState,
    models: tuple[str, ...] | None = None,
    reason: str | None = None,
    latency_s: float | None = None,
)

One probe's answer.

Attributes:

  • state (HealthState) –

    up, down, or unknown when the probe itself failed.

  • models (tuple[str, ...] | None) –

    The model ids the backend lists (empty when its list is empty); None when it was not asked or did not answer with a list.

  • reason (str | None) –

    One line naming why the state is not up; None when it is. Never an exception's message: an error can quote a request header, credential included.

  • latency_s (float | None) –

    Seconds the probe took; None when it never reached the backend.

Probe

Bases: Protocol

A health check on one backend.

probe async

probe() -> Health

Ask the backend once. Never raises.

Returns:

  • Health –

    up with what it serves, down when it could not be reached or answered with an

  • Health –

    error, unknown when the probe itself failed.

Chat

Bases: Protocol

A multi-turn, tool-using, streamed conversation: messages in, events and new messages out.

Invariants
  • stream yields Finished exactly once, last, in every case, and never raises for a model, wire or gate failure (ADR-0001 holds for streams). Closing the iterator or cancelling the task stops the call; asyncio.CancelledError propagates.
  • chat is stream drained: it returns Finished.result. An implementation that subclasses Chat inherits that chat and writes only stream.
  • The port is stateless: the caller appends Generated.messages to its own history.

Rationale: ADR-0023.

chat async

chat(
    req: ChatRequest,
    output: type[T] = str,
    *,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None
) -> Generated[T]

Run one call to its end and return the result Finished carries.

Parameters:

  • req (ChatRequest) –

    The resolved call.

  • output (type[T], default: str ) –

    str for prose, a pydantic model, or a JsonOutput subclass.

  • validate (Validator[T] | None, default: None ) –

    Lists the problems with a parsed value, as for Generator.generate.

  • retry_message (RetryMessage | None, default: None ) –

    Builds a re-ask's message from the problems.

stream

stream(
    req: ChatRequest,
    output: type[T] = str,
    *,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None
) -> AsyncIterator[StreamEvent]

Run one call, yielding its events as they happen and Finished last.

Parameters:

  • req (ChatRequest) –

    The resolved call.

  • output (type[T], default: str ) –

    str for prose, a pydantic model, or a JsonOutput subclass.

  • validate (Validator[T] | None, default: None ) –

    Lists the problems with a parsed value, as for Generator.generate.

  • retry_message (RetryMessage | None, default: None ) –

    Builds a re-ask's message from the problems.

Citer

Bases: Protocol

Citation-grounded generation: a question and chunks to cited claims.

answer async

answer(q: str, chunks: Sequence[Chunk]) -> CitedAnswer

Answer q from chunks, each claim citing the ids it rests on.

No chunks yield no claims, not an error.

Invariants
  • Every claim's evidence_ids is a subset of {c.id for c in chunks}, so CitedAnswer.unknown_ids() is empty. A citation is a chunk id, never a URL the model typed.

Counted dataclass

Counted(
    tokens: int | None,
    method: CountMethod,
    failure: Failure | None = None,
)

An input-token count: exact, or absent.

Attributes:

  • tokens (int | None) –

    The count; None when this model cannot be counted or the count failed. Never an estimate.

  • method (CountMethod) –

    How it was measured.

  • failure (Failure | None) –

    Why the count request itself failed; None when it did not.

Rationale: ADR-0024.

Embedder

Bases: Protocol

Text embeddings: texts to one vector each, all in one named space.

stamp property

stamp: str

The vector space this embedder writes: model id and dims (e.g. example-embed-v1@1024).

Constant for the embedder's life.

embed async

embed(
    texts: Sequence[str],
    *,
    input_type: EmbedInput = "document"
) -> Embedded

Embed texts, one vector per text, in input order.

Never raises for a model, wire or gate failure: the failure is the outcome. A wrong count or a wrong dimension is invalid, never a partial result.

Parameters:

  • texts (Sequence[str]) –

    The texts to embed.

  • input_type (EmbedInput, default: 'document' ) –

    "query" lets an asymmetric model treat a search query differently from a stored document; the adapter owns any instruction prefix.

Rationale: ADR-0001.

EndFields

Bases: TypedDict

Every keyword of Generation.end, for an implementation that takes **fields.

An implementation may spell the keywords out or take **fields: Unpack[EndFields]. An absent key is the port's default: level "DEFAULT", every other key None (ADR-0025).

Evaluator

Bases: Protocol

The eval harness: rows and graders to one board.

run

run(
    surface: str,
    rows: list[dict[str, Any]],
    graders: Mapping[str, Grader],
    *,
    id_field: str = "id",
    monitors: (
        Callable[[list[dict[str, Any]]], dict[str, Any]]
        | None
    ) = None
) -> dict[str, Any]

Grade every row with every grader and return one board.

Parameters:

  • surface (str) –

    The name of the surface under evaluation, echoed on the board.

  • rows (list[dict[str, Any]]) –

    The rows to grade.

  • graders (Mapping[str, Grader]) –

    Grader name to grader.

  • id_field (str, default: 'id' ) –

    The row key that names a row in examples.

  • monitors (Callable[[list[dict[str, Any]]], dict[str, Any]] | None, default: None ) –

    Computes the surface's own rates over all rows; None for none.

Returns:

  • dict[str, Any] –

    A board with the keys surface, n, rows_failed, by_grader and monitors.

  • dict[str, Any] –

    by_grader maps each grader name to pass, fail and examples (at most three

  • dict[str, Any] –

    {id, reason} failures). monitors holds the callable's result verbatim, or {}

  • dict[str, Any] –

    when none was passed.

Invariants
  • Every adapter emits exactly this shape, and every key is always present.

GatewayReader

Bases: Protocol

What a gateway in front of the model reported in a response's headers.

A module of two functions satisfies it (llm_ports.adapters.litellm_proxy).

cost_from_headers

cost_from_headers(
    headers: Mapping[str, str],
) -> Decimal | None

Return the cost the gateway reported for the response, or None when it reported none.

Raises:

  • ValueError –

    The cost header is present but not a number. A caller treats it as absent and reports the bad header.

answered_from_headers

answered_from_headers(
    headers: Mapping[str, str],
) -> tuple[str | None, str | None]

Return (deployment id, upstream api base) as the gateway reported them; None for each it did not.

Generation

Bases: Protocol

One open generation observation.

Invariants
  • Neither method raises.

Rationale: ADR-0007.

end

end(
    *,
    output: Any = None,
    usage: Mapping[str, int | None] | None = None,
    level: str = "DEFAULT",
    status_message: str | None = None,
    metadata: Mapping[str, Any] | None = None,
    usage_v2: Usage | None = None,
    cost: Cost | None = None,
    finish: Finish | None = None,
    answered: Answered | None = None,
    failure: Failure | None = None
) -> None

Close the observation.

Parameters:

  • output (Any, default: None ) –

    What the call produced.

  • usage (Mapping[str, int | None] | None, default: None ) –

    The raw pair {"input": int | None, "output": int | None} as the provider reported it, or None when it reported neither. The caller decides what unmeasured means; the adapter decides the sink's key names.

  • level (str, default: 'DEFAULT' ) –

    "DEFAULT"; "ERROR" for a failed call; "CANCELLED" for a call its caller cancelled (or interrupted).

  • status_message (str | None, default: None ) –

    The error line carried with a level other than "DEFAULT".

  • metadata (Mapping[str, Any] | None, default: None ) –

    Extra fields for the sink.

  • usage_v2 (Usage | None, default: None ) –

    The call's usage summed over every request (ADR-0019); None when unknown.

  • cost (Cost | None, default: None ) –

    What the call cost, with its source; None when not priced.

  • finish (Finish | None, default: None ) –

    How the last reply ended.

  • answered (Answered | None, default: None ) –

    Who answered.

  • failure (Failure | None, default: None ) –

    The failure's kind and detail; None for a call that succeeded.

A 1.x implementation without the five keyword-only arguments added in 1.24 keeps working: observability passes them only to an end that accepts them.

Invariants
  • A None field is omitted from what the sink receives, never sent as null.
  • An unmeasured count is never written as 0.

score

score(
    name: str,
    value: float,
    comment: str | None = None,
    metadata: Mapping[str, Any] | None = None,
) -> None

Attach a numeric score to this observation. Never raises.

Parameters:

  • name (str) –

    The score's name.

  • value (float) –

    Its value.

  • comment (str | None, default: None ) –

    A one-line reason, shown beside the score.

  • metadata (Mapping[str, Any] | None, default: None ) –

    Extra fields stored on the score itself (who gave it, for a label); omitted from what the sink receives when None.

Generator

Bases: Protocol

Typed structured generation: a prompt and an output type to a value or a named outcome.

generate async

generate(
    req: GenRequest,
    output: type[T],
    *,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None
) -> Generated[T]

Run one call and parse the reply into output.

Never raises for a model, wire or gate failure: the failure is the outcome, with a one-line reason.

Parameters:

  • req (GenRequest) –

    The resolved call.

  • output (type[T]) –

    A pydantic model, a JsonOutput subclass, or str for prose.

  • validate (Validator[T] | None, default: None ) –

    Lists the problems with a parsed value. A non-empty list earns req.output_retries re-asks, after which the value is kept with its problems.

  • retry_message (RetryMessage | None, default: None ) –

    Builds a re-ask's message from the problems.

Rationale: ADR-0001.

Grounder

Bases: Protocol

Groundedness: one claim checked against the spans it cites.

verdict

verdict(
    claim: Claim, spans: Sequence[Chunk]
) -> GroundingVerdict

Return whether claim is supported by spans.

Pure.

Parameters:

  • claim (Claim) –

    The claim to check.

  • spans (Sequence[Chunk]) –

    The chunks the claim cites, already resolved by the caller.

Invariants
  • A claim with no spans is never supported.

Judge

Bases: Protocol

LLM-as-a-judge: a prompt, the reply it drew and a rubric to a graded verdict.

judge async

judge(
    *, prompt: str, output: Any, rubric: str
) -> JudgeVerdict

Grade output, the reply to prompt, against rubric.

Raises when the judging model could not answer; the caller decides what an unjudged sample means.

Rationale: ADR-0001.

Limiter

Bases: Protocol

In-process concurrency: at most N calls at once to one group of models.

A call takes a slot INSIDE its deadline and OUTSIDE its retry loop: waiting spends the caller's deadline, and a retry keeps the slot it holds. Cross-process limits stay with whatever is in front of the model (a gateway answers 429 and the wire policy waits).

Rationale: ADR-0021.

slot

slot(group: str) -> AbstractAsyncContextManager[float]

Hold one slot of group for the block; the value is the seconds waited for it.

Raises:

  • SlotWaitExceeded –

    The wait passed the limiter's bound; the adapter reports rate_limited / local_limiter.

Pricer

Bases: Protocol

A price table: a call's usage and who answered become what it cost.

price

price(usage: Usage, answered: Answered) -> Cost

Return the cost of usage on the model answered names.

Never raises: a model the table does not know is Cost(None, "unknown", reason="unknown_model"), and usage with nothing measured is reason="no_usage".

Rationale: ADR-0019.

PromptSource

Bases: Protocol

Prompt management: a name and a label to a source template and its provenance.

Rendering is the caller's; the port never compiles a template.

get

get(name: str, *, label: str = 'production') -> Prompt

Return the prompt name as served under label.

Synchronous, as prompt-management SDKs are; an async caller uses asyncio.to_thread. A managed store that is down is not an error: the result is the repo seed, with source == "repo". An adapter with no seed for name has nothing to serve and raises.

Invariants
  • text is never empty.

Reranker

Bases: Protocol

Cross-encoder reranking: a query and documents to one order-only score per document.

rerank async

rerank(query: str, documents: Sequence[str]) -> Reranked

Score each document against query, in input order.

No documents return an ok with no scores. Never raises for a model, wire or gate failure: the failure is the outcome, and the caller keeps the order it already had. A reply that does not score every document exactly once is invalid, never a partial ranking.

Rationale: ADR-0001.

Retriever

Bases: Protocol

RAG retrieval: a query to ranked chunks with provenance.

Hybrid search and reranking are the adapter's concern; the port promises order and provenance.

search async

search(
    q: str,
    *,
    k: int = 8,
    date_from: date | None = None,
    date_to: date | None = None
) -> list[Chunk]

Return up to k chunks for q, best first.

An empty or blank q returns []. Finding nothing is an empty list, never an exception.

Parameters:

  • q (str) –

    The query.

  • k (int, default: 8 ) –

    The most chunks to return.

  • date_from (date | None, default: None ) –

    Lower bound on Chunk.as_of; None leaves it open.

  • date_to (date | None, default: None ) –

    Upper bound on Chunk.as_of; None leaves it open.

TokenCounter

Bases: Protocol

Counts a request's input tokens before it is sent.

Rationale: ADR-0024.

count async

count(
    model: str,
    *,
    system: str,
    messages: Sequence[Message],
    tools: Sequence[Tool] = ()
) -> Counted

Count what model would read for this request.

Never raises: a model that cannot be counted is Counted(None, "unsupported"), a count request that failed is Counted(None, method, failure=...).

Parameters:

  • model (str) –

    The model id.

  • system (str) –

    The system prompt.

  • messages (Sequence[Message]) –

    The history that would be sent.

  • tools (Sequence[Tool], default: () ) –

    The tools that would be offered.

Tracer

Bases: Protocol

LLM observability: one generation observation per model call.

Synchronous, as tracing SDKs are; the async wrapper around a model call is observability.observe. Fail-open.

start_generation

start_generation(
    *,
    name: str,
    model: str,
    provider: str,
    input: Any,
    metadata: Mapping[str, Any],
    session_id: str | None,
    prompt_name: str | None
) -> Generation

Open one generation observation.

Never raises: a sink that is down or refusing returns a no-op Generation.

Parameters:

  • name (str) –

    The category a cost view groups by; the adapter stores it verbatim.

  • model (str) –

    The model id.

  • provider (str) –

    The provider name.

  • input (Any) –

    What the model was sent.

  • metadata (Mapping[str, Any]) –

    Extra fields for the sink.

  • session_id (str | None) –

    When set, groups the generation under that session.

  • prompt_name (str | None) –

    When set, links the managed prompt of that name.

Deadline dataclass

Deadline(
    total_s: float,
    attempt_s: float | None = None,
    first_event_s: float | None = None,
    started: float | None = None,
)

How long one call may take, on the event loop's monotonic clock.

Attributes:

  • total_s (float) –

    The whole call: slot wait, every wire request and re-ask, every backoff, every fallback tier.

  • attempt_s (float | None) –

    One wire request; None means total_s.

  • first_event_s (float | None) –

    Time to the first streamed event; None means not enforced.

  • started (float | None) –

    When the call started, on the loop's clock; None until start stamps it.

Raises:

  • ValueError –

    A bound is not a positive finite number.

Rationale: ADR-0021.

at property

at: float

When the call must be over, on the loop's clock.

Raises:

  • ValueError –

    The deadline was never started.

start

start(now: float) -> Deadline

This deadline, started at now; one already started is returned as is.

remaining

remaining(now: float) -> float

Seconds left at now, never below 0; total_s when not started.

attempt_bound

attempt_bound(now: float) -> float

The bound on one wire request started at now: attempt_s, never past the deadline.

RetryRule dataclass

RetryRule(
    attempts: int,
    wait_initial_s: float,
    wait_max_s: float,
    multiplier: float = 2.0,
    jitter: float = 0.1,
    honour_retry_after: bool = True,
    retry_after_max_s: float = 120.0,
)

How one failure kind is retried.

Attributes:

  • attempts (int) –

    Tries of one wire request for this kind, the first included; 1 means no retry.

  • wait_initial_s (float) –

    The wait before the second try, in seconds.

  • wait_max_s (float) –

    The cap on one wait, in seconds.

  • multiplier (float) –

    Each later wait is the previous one times this.

  • jitter (float) –

    A fraction of each wait added or removed at random; 0 disables it.

  • honour_retry_after (bool) –

    Wait what the server's retry-after-ms / Retry-After says instead of the computed wait.

  • retry_after_max_s (float) –

    A server hint longer than this ends the retry; the Failure keeps the hint so the caller can reschedule.

Raises:

  • ValueError –

    attempts is below 1, a wait is negative, the cap is below the initial wait, multiplier is below 1, or jitter is outside [0, 1].

wait_s

wait_s(n: int, *, spread: float = 0.0) -> float

The wait after the n-th failed try (from 1), capped, with spread in [-1, 1] of jitter. Pure.

SlotWaitExceeded

SlotWaitExceeded(group: str, waited_s: float)

Bases: Exception

Raised by a Limiter whose wait for a slot passed its max_wait_s.

An adapter turns it into rate_limited with provider_code="local_limiter", not retried.

Attributes:

  • group –

    The concurrency group.

  • waited_s –

    How long the call waited, in seconds.

WirePolicy dataclass

WirePolicy(
    rules: Mapping[FailureKind, RetryRule] = dict(),
    max_total_attempts: int = 60,
)

Which failure kinds one wire request is retried on, and how.

Attributes:

  • rules (Mapping[FailureKind, RetryRule]) –

    Failure kind to its RetryRule; a kind not listed is not retried.

  • max_total_attempts (int) –

    Wire requests one call may make across every kind and every request, a hard ceiling.

Raises:

  • ValueError –

    A rule's key is not in RETRYABLE, or max_total_attempts is below 1.

Rationale: ADR-0021.

default classmethod

default() -> WirePolicy

rate_limited 3 tries; overloaded, server, transport, timeout 2 tries; waits 0.5-8 s.

none classmethod

none() -> WirePolicy

No retry: every request is sent once (an interactive first-token path).

from_mapping classmethod

from_mapping(
    raw: Mapping[str, Any], *, path: str = "policy"
) -> WirePolicy

Read a policy written as data: {<kind>: {<RetryRule field>: value}, max_total_attempts: n}.

Parameters:

  • raw (Mapping[str, Any]) –

    The policy, e.g. a tier table's [policies.<name>] section.

  • path (str, default: 'policy' ) –

    Where the policy sits, named in an error.

Raises:

  • ValueError –

    An unknown or non-retryable kind, an unknown field, or a value a RetryRule refuses; the message starts with the offending path.

Chunk dataclass

Chunk(
    id: str,
    text: str,
    title: str = "",
    source: str = "",
    url: str = "",
    as_of: date | None = None,
    score: float | None = None,
    meta: Mapping[str, Any] = dict(),
)

One retrievable unit of text, with its provenance.

Attributes:

  • id (str) –

    The id a Claim cites.

  • title (str) –

    Display title; empty when the source has none.

  • source (str) –

    The corpus or feed the chunk came from (e.g. "wiki", "tickets", "web").

  • url (str) –

    The chunk's address; empty when it has none.

  • as_of (date | None) –

    The date the content speaks for; None when undated.

  • score (float | None) –

    Adapter-relative relevance (a cross-encoder's raw logit, a fused rank score).

  • meta (Mapping[str, Any]) –

    Adapter extras, read by name.

Invariants
  • score is order-only: it ranks one adapter's results and is never a threshold or compared across adapters.

CitedAnswer dataclass

CitedAnswer(
    question: str,
    claims: tuple[Claim, ...],
    chunks: Mapping[str, Chunk],
    model: str = "",
    extras: Mapping[str, Any] = dict(),
)

What a Citer returns: the cited claims and the chunks they may cite.

Carrying the chunks lets a Grounder read the cited text without a second retrieval.

Attributes:

  • question (str) –

    The question answered.

  • claims (tuple[Claim, ...]) –

    The claims, in the order written.

  • chunks (Mapping[str, Chunk]) –

    Chunk id to chunk; the universe the claims may cite.

  • model (str) –

    The model the adapter reports; empty when the adapter has none (a stub).

  • extras (Mapping[str, Any]) –

    The adapter's own per-answer report (retries, latency, ...). A consumer reads a key by name; the port defines none.

unknown_ids

unknown_ids() -> tuple[str, ...]

Return the cited ids that are not in chunks.

A non-empty result means the citer broke the Citer contract.

Claim dataclass

Claim(
    text: str,
    evidence_ids: tuple[str, ...] = (),
    meta: Mapping[str, Any] = dict(),
)

A sentence the writer asserts, with the ids of the chunks it rests on.

Attributes:

  • evidence_ids (tuple[str, ...]) –

    Ids of the cited chunks; empty means the claim is ungrounded.

  • meta (Mapping[str, Any]) –

    Adapter extras for this claim, read by name.

Embedded dataclass

Embedded(
    vectors: tuple[tuple[float, ...], ...],
    outcome: Outcome,
    stamp: str = "",
    reason: str | None = None,
    input_tokens: int | None = None,
    retries: int = 0,
    usage_v2: Usage | None = None,
    cost: Cost | None = None,
    answered: Answered | None = None,
    failure: Failure | None = None,
)

Bases: _Outcomed

What an Embedder returns: one vector per input, in input order, all in one space.

Attributes:

  • vectors (tuple[tuple[float, ...], ...]) –

    One vector per input; () exactly when outcome != "ok".

  • outcome (Outcome) –

    How the call ended.

  • stamp (str) –

    The vector space (model and dims). It belongs on every stored vector.

  • reason (str | None) –

    One line naming the failure; None on ok. Derived from failure when left None.

  • input_tokens (int | None) –

    Tokens the wire reported; None when it reported no usage (unknown, never 0).

  • retries (int) –

    Requests the adapter re-sent after a retryable failure (its RetryPolicy), counted whether the call then landed or not.

  • usage_v2 (Usage | None) –

    Token counts summed over every request; None when no request was made.

  • cost (Cost | None) –

    What the call cost; None only when no Pricer was configured.

  • answered (Answered | None) –

    Who produced the vectors.

  • failure (Failure | None) –

    The failure's kind and detail; None exactly when outcome == "ok".

Invariants
  • Two stamps never compare: a store holding two is a reindex in progress, not an index.
  • A batch that half-landed is a failed batch; no partial result is returned.

Rationale: ADR-0001, ADR-0019.

Generated dataclass

Generated(
    value: T | None,
    outcome: Outcome,
    model: str = "",
    reason: str | None = None,
    retries: int | None = 0,
    problems: tuple[Any, ...] = (),
    usage: Mapping[str, int | None] | None = None,
    sources: tuple[str, ...] | None = None,
    finish_reason: str | None = None,
    stamp: Mapping[str, Any] = dict(),
    usage_v2: Usage | None = None,
    cost: Cost | None = None,
    finish: Finish | None = None,
    answered: Answered | None = None,
    attempts: tuple[Attempt, ...] = (),
    warnings: tuple[CallWarning, ...] = (),
    failure: Failure | None = None,
    messages: tuple[Message, ...] = (),
    steps: int | None = None,
)

What a Generator returns.

Attributes:

  • value (T | None) –

    The parsed value. None when outcome != "ok", except truncated, which keeps a length-cut reply that still parsed so a caller that accepts cut text can read it.

  • outcome (Outcome) –

    How the call ended.

  • model (str) –

    The model id the request named. Deprecated: answered says who answered.

  • reason (str | None) –

    One line naming the failure; None on ok.

  • retries (int | None) –

    Re-asks used; None when the run died mid-flight and the count is unknown, never 0.

  • problems (tuple[Any, ...]) –

    The validator's findings on the kept value once the re-asks ran out. The caller prunes the value or drops it.

  • usage (Mapping[str, int | None] | None) –

    {"input", "output"} token counts; None, or a None count, is unmeasured. Deprecated: derived from usage_v2.

  • sources (tuple[str, ...] | None) –

    URLs the model server fetched while answering (a research-server extension); None when the reply carried none.

  • finish_reason (str | None) –

    The provider's finish reason, when it sent one. Deprecated: derived from finish.raw.

  • stamp (Mapping[str, Any]) –

    Extras the call path attaches by name (e.g. scores, generation).

  • usage_v2 (Usage | None) –

    Every bucket, summed over every wire request; None when no request was made.

  • cost (Cost | None) –

    What the call cost; None only when no Pricer was configured and no gateway reported a cost.

  • finish (Finish | None) –

    How the last reply ended; None when the provider said nothing.

  • answered (Answered | None) –

    Who produced the result.

  • attempts (tuple[Attempt, ...]) –

    Every wire request or fallback step that did not answer, in order.

  • warnings (tuple[CallWarning, ...]) –

    Settings dropped or altered on the way to the model.

  • failure (Failure | None) –

    The failure's kind and detail; None exactly when outcome == "ok".

  • messages (tuple[Message, ...]) –

    The new messages a Chat call produced (assistant turns and tool results, in order), ready to append to the history; () for a Generator call.

  • steps (int | None) –

    The model requests a Chat call made; None for a Generator call.

Invariants
  • When left None, usage is usage_v2.as_mapping(), finish_reason is finish.raw and reason is failure.reason.

Rationale: ADR-0001, ADR-0019, ADR-0020.

GenRequest dataclass

GenRequest(
    model: str,
    prompt: str,
    system: str = "",
    images: tuple[tuple[bytes, str], ...] = (),
    settings: Mapping[str, Any] = dict(),
    self_hosted: bool = True,
    timeout_s: float = 120.0,
    output_retries: int = 1,
    label: str = "chat",
    accept_truncated: bool = False,
    deadline: Deadline | None = None,
    concurrency_group: str | None = None,
    policy: WirePolicy | None = None,
    stream_internally: bool = False,
    cache_prefix: bool = False,
)

One model call, already resolved from a role: model id, rendered prompt and wire settings.

Attributes:

  • model (str) –

    The model id.

  • prompt (str) –

    The rendered user prompt.

  • system (str) –

    The rendered system prompt; empty for none.

  • images (tuple[tuple[bytes, str], ...]) –

    (bytes, MIME type) pairs for a vision model.

  • settings (Mapping[str, Any]) –

    Sent to the model verbatim (max_tokens, temperature, seed, top_p, extra_body).

  • self_hosted (bool) –

    True for a self-hosted tier. An adapter may take a lease on that model group, and a role sends it the thinking switch.

  • timeout_s (float) –

    The call's bound, in seconds: Deadline(total_s=timeout_s) when deadline is None.

  • output_retries (int) –

    Re-asks allowed when the validator finds problems.

  • label (str) –

    A name for the call in an adapter's logs; call.request_for sets the role name.

  • accept_truncated (bool) –

    Return a length-cut reply that parsed as ok, with a CallWarning, instead of truncated. Off by default.

  • deadline (Deadline | None) –

    The call's clocks: total, per wire request, first event. None means Deadline(total_s=timeout_s). A started deadline (call.generate starts one) carries what an earlier step already spent.

  • concurrency_group (str | None) –

    The Limiter group the call takes a slot in; None means the model id.

  • policy (WirePolicy | None) –

    How the adapter retries a wire request; None means the adapter's own default.

  • stream_internally (bool) –

    Request a stream and assemble it, so a cancelled or timed-out call closes a connection the server can act on. Off by default.

  • cache_prefix (bool) –

    Ask for a prompt-cache breakpoint after the system text, the call's stable prefix (ADR-0024). A request, not a guarantee: Usage.cache_read says if it was served.

Rationale: ADR-0021, ADR-0024.

clocks property

clocks: Deadline

deadline, or Deadline(total_s=timeout_s) when none was given.

GroundingVerdict dataclass

GroundingVerdict(
    supported: bool,
    reason: str = "",
    score: float | None = None,
    checker: str = "",
)

What a Grounder returns for one claim.

Attributes:

  • supported (bool) –

    The gate.

  • reason (str) –

    One line saying why.

  • score (float | None) –

    The adapter's own scale (an entailment model's probability); None for a rule-based check that has none. Reported, never compared across adapters.

  • checker (str) –

    The adapter that produced the verdict (e.g. "number_law", "hhem", "lettucedetect").

JsonOutput

Bases: dict[str, Any]

A hand-written JSON schema used as a Generator output type.

The value is a plain dict, and the schema reaches the model unchanged. Callers use json_output, which mints one subclass per schema; each Generator adapter translates that subclass into its framework's structured-output object (pydantic-ai: StructuredDict), so the facade (call.json_output) imports no adapter.

Attributes:

  • json_schema (Mapping[str, Any]) –

    The schema sent to the model.

  • schema_name (str) –

    The schema's title.

JudgeVerdict dataclass

JudgeVerdict(
    passed: bool,
    score: float,
    reason: str = "",
    judge: str = "",
)

What a Judge returns.

Attributes:

  • passed (bool) –

    The gate.

  • score (float) –

    The judge's own reading, from 0 to 1.

  • reason (str) –

    The judge's one- or two-sentence justification, recorded as the score comment.

  • judge (str) –

    The adapter that produced the verdict (e.g. "pydantic_evals", "stub").

Prompt dataclass

Prompt(
    name: str,
    text: str,
    version: int | None = None,
    source: str = "repo",
    config: Mapping[str, Any] = dict(),
    system: str = "",
)

What a PromptSource returns: a source template and its provenance.

Attributes:

  • name (str) –

    The prompt's name.

  • text (str) –

    The source template. The caller renders it; an adapter never compiles it.

  • version (int | None) –

    The managed copy's integer version; None for the repo seed.

  • source (str) –

    "repo" for the repo seed, otherwise the managed copy's origin (e.g. "langfuse:v7").

  • config (Mapping[str, Any]) –

    The adapter's per-version extras (content sha, pack version, model role, ...), read by name.

  • system (str) –

    The system template of a two-message (chat) prompt; empty for a text prompt.

Invariants
  • version is None exactly when source == "repo". The repo seed is served when the managed copy is absent, unreachable or disagrees with the seed.

Reranked dataclass

Reranked(
    scores: tuple[float, ...],
    outcome: Outcome,
    model: str = "",
    reason: str | None = None,
    answered: Answered | None = None,
    failure: Failure | None = None,
    attempts: tuple[Attempt, ...] = (),
)

Bases: _Outcomed

What a Reranker returns: one score per document, in input order.

Attributes:

  • scores (tuple[float, ...]) –

    One score per document; () exactly when outcome != "ok".

  • outcome (Outcome) –

    How the call ended.

  • model (str) –

    The model that scored.

  • reason (str | None) –

    One line naming the failure; None on ok. Derived from failure when left None.

  • answered (Answered | None) –

    Who scored.

  • failure (Failure | None) –

    The failure's kind and detail; None exactly when outcome == "ok".

  • attempts (tuple[Attempt, ...]) –

    Every failed request the reranker's wire policy re-sent, in order; () without a policy.

Invariants
  • Scores are model-relative and order-only (one model emits raw logits, another 0..1). Rank by them; never threshold them or compare them across models.

Seed dataclass

Seed(
    text: str,
    sha: str,
    config: Mapping[str, Any] = dict(),
    tags: Sequence[str] = (),
    system: str = "",
    kind: str = "",
)

What the repo says a prompt is.

A PromptSource serves a seed as a Prompt with source == "repo", and a managed store mirrors it.

Attributes:

  • text (str) –

    The source template.

  • sha (str) –

    The content sha that names text.

  • config (Mapping[str, Any]) –

    Carried beside the text (pack version, model role, ...).

  • tags (Sequence[str]) –

    Tags for the managed copy.

  • system (str) –

    The system template; non-empty makes the seed a chat prompt (system + user).

  • kind (str) –

    "chat" or "text" forces the managed prompt's type; empty derives it from system.

Answered dataclass

Answered(
    requested: str,
    model: str | None = None,
    provider: str | None = None,
    tier: str | None = None,
    deployment: str | None = None,
    response_id: str | None = None,
)

Who produced a result.

Attributes:

  • requested (str) –

    The model id the request carried (a route or an alias is fine).

  • model (str | None) –

    The model name the response reported; None when it reported none. Never an echo of requested.

  • provider (str | None) –

    The provider that served it, when the caller's provider table names it.

  • tier (str | None) –

    The tier that produced this result, when the caller's routing names it.

  • deployment (str | None) –

    The gateway deployment id, when a gateway reported one.

  • response_id (str | None) –

    The response's own id, when it had one.

Rationale: ADR-0019.

Attempt dataclass

Attempt(
    n: int,
    tier: str | None,
    model: str,
    failure: Failure,
    waited_s: float,
    usage: Usage | None = None,
)

One wire request or fallback step that did not answer, and the wait that followed it.

Attributes:

  • n (int) –

    The attempt's number, from 1.

  • tier (str | None) –

    The tier it ran on, when known.

  • model (str) –

    The model it asked.

  • failure (Failure) –

    How it failed.

  • waited_s (float) –

    The backoff slept before the next attempt, in seconds.

  • usage (Usage | None) –

    What it consumed, when it reported anything.

Rationale: ADR-0019.

CallWarning dataclass

CallWarning(kind: WarningKind, setting: str, detail: str)

A setting that was dropped or altered on its way to the model.

Attributes:

  • kind (WarningKind) –

    What happened to it.

  • setting (str) –

    The setting's name.

  • detail (str) –

    One line saying what happened.

Rationale: ADR-0019.

Cost dataclass

Cost(
    usd: Decimal | None,
    source: CostSource,
    priced_model: str | None = None,
    reason: str | None = None,
)

What a call cost, in US dollars, and where that number came from.

Attributes:

  • usd (Decimal | None) –

    The cost; None when unknown, never 0 for unknown.

  • source (CostSource) –

    Where usd came from.

  • priced_model (str | None) –

    The model reference the price was looked up under.

  • reason (str | None) –

    Why the cost is unknown: no_usage, unknown_model, no_pricer or pricer_error.

Rationale: ADR-0019.

Finish dataclass

Finish(unified: UnifiedFinish, raw: str | None)

How the last reply of a call ended.

Attributes:

  • unified (UnifiedFinish) –

    The closed-vocabulary reading of raw; other for a value no mapping knows.

  • raw (str | None) –

    The provider's own string, unmodified.

Rationale: ADR-0019.

Usage dataclass

Usage(
    input: int | None = None,
    output: int | None = None,
    cache_read: int | None = None,
    cache_write: int | None = None,
    reasoning: int | None = None,
    requests: int = 0,
    unmeasured: int = 0,
    details: Mapping[str, int] = dict(),
)

Token counts summed over every wire request of one call.

Attributes:

  • input (int | None) –

    All input tokens, cached included; None when no request reported it.

  • output (int | None) –

    All output tokens, reasoning included; None when no request reported it.

  • cache_read (int | None) –

    The part of input served from a provider cache.

  • cache_write (int | None) –

    The part of input written to a provider cache.

  • reasoning (int | None) –

    The part of output spent reasoning.

  • requests (int) –

    Wire requests folded in (output re-asks, tool steps, retries that returned a body).

  • unmeasured (int) –

    Of those, how many reported no usage at all.

  • details (Mapping[str, int]) –

    Provider extras under their own keys, summed key by key.

Invariants
  • cache_read <= input and reasoning <= output whenever both sides are known.

Rationale: ADR-0019.

complete property

complete: bool

True when at least one request was made, every one reported usage, and both totals are known.

as_mapping

as_mapping() -> Mapping[str, int | None]

Return the 1.x {"input", "output"} shape of Generated.usage.

failure

failure(
    kind: FailureKind,
    *,
    retry_after_s: float | None = None,
    status: int | None = None,
    provider_code: str | None = None,
    phase: Phase | None = None,
    in_stream: bool = False,
    reason: str = ""
) -> Failure

Return the Failure of kind with the given detail; the one constructor src/ uses.

Pure. The arguments are the Failure attributes of the same name; reason defaults to the kind itself, so a failure always names something.

Raises:

  • ValueError –

    kind is not a FailureKind.

json_output

json_output(
    schema: Mapping[str, Any], name: str
) -> type[JsonOutput]

Return a JsonOutput subclass that carries schema and name.

Pure.

Parameters:

  • schema (Mapping[str, Any]) –

    The JSON schema, sent to the model unchanged.

  • name (str) –

    The schema's title, and the name a test stub dispatches on.

llm_ports.ports.outcomes

Named failure reasons for a failed model call, and the HTTP-status rule every adapter shares.

outcome_for_status names the outcome an HTTP error status maps to: a status in REQUEST_FAULT blames the request, any other the server. failure_reason turns an exception into the one reason line a caller persists: the exception class, the status code when there is one, and the head of the message. A provider's error payload belongs in a log line, not in a persisted reason, and a proxy error can embed a keyed URL.

Invariants
  • Pure: no I/O.
  • A reason line never carries the response body and never exceeds REASON_MAX characters.

Rationale: ADR-0001.

GenerationFailed

GenerationFailed(outcome: str, reason: str | None)

Bases: RuntimeError

A non-ok, non-gated outcome raised for a caller whose contract is that a failure propagates.

The facade raises it for a caller that expects an exception (a Citer, or a caller that records a failed row from an except block).

Attributes:

  • outcome –

    The Outcome the call ended with.

  • reason –

    The one-line reason, or None.

outcome_for_status

outcome_for_status(status: int) -> Outcome

Return the outcome an HTTP error status maps to.

Pure. A status in REQUEST_FAULT is request_fault: retrying the same input fails the same way. Any other status is unavailable: the server, not the input, failed.

failure_reason

failure_reason(
    exc: BaseException, *, with_cause: bool = False
) -> str

Return the persisted reason line for exc.

The line is the exception class, the status code when the exception carries one, and the first line of its message cut before any body: payload.

Parameters:

  • exc (BaseException) –

    The exception to name.

  • with_cause (bool, default: False ) –

    Append <- <the cause's reason>, cut by the same rule. A wrapper's own line can name nothing ("Exceeded maximum output retries") while its cause ("Invalid JSON: EOF while parsing a string") names the fix.

Returns:

  • str –

    A line of at most REASON_MAX characters; the cause keeps up to half of it.

exception_chain

exception_chain(
    exc: BaseException, limit: int = CHAIN_LIMIT
) -> Iterator[BaseException]

Yield exc, then its __cause__ (else __context__), and so on down the chain.

SDK layers (pydantic-ai, openai, httpx, a transport) each wrap the one below, so the root cause is never on top.

Parameters:

  • exc (BaseException) –

    The outermost exception.

  • limit (int, default: CHAIN_LIMIT ) –

    The most exceptions to yield. A cycle ends the chain early.