Skip to content

llm_ports.adapters

llm_ports.adapters

Adapters: every module that imports a vendor library lives here, one extra each.

Import an adapter by its path (llm_ports.adapters.pydantic_ai, llm_ports.adapters.langfuse, ...); nothing here is imported by the rest of the package.

llm_ports.adapters.pydantic_ai

Adapters on pydantic-ai: generator, chat, embedder, output schemas, replay, the openai_compatible kind.

PydanticAIGenerator, PydanticAIChat, PydanticAIEmbedder, replay, and OpenAICompatibleFactory, the openai_compatible provider kind.

The failure-classification table every adapter shares (classify_core, WireFacts) and the pydantic-ai categories over it (classify_failure) are exported here too.

Needs the pydantic-ai extra (pip install 'llm-ports[pydantic-ai]').

IndexOrderedTransport

IndexOrderedTransport(
    inner: AsyncBaseTransport | None = None,
)

Bases: AsyncBaseTransport

An httpx transport that re-sorts a 200 reply's index-keyed list (INDEXED_LISTS) by index.

Parameters:

  • inner (AsyncBaseTransport | None, default: None ) –

    The transport that sends the request; None uses httpx's own.

Invariants
  • A non-200 reply, a path not in INDEXED_LISTS, an unreadable body, or a list already in index order is returned unchanged.

handle_async_request async

handle_async_request(request: Request) -> Response

Send request through the inner transport and return the reply with its list in index order.

aclose async

aclose() -> None

Close the inner transport.

Classifier

Bases: Protocol

An extra classifier: knowledge a gateway or provider adds ahead of the shared table.

WireFacts dataclass

WireFacts(
    status: int | None = None,
    headers: Mapping[str, str] = dict(),
    finish_raw: str | None = None,
    error_code: str | None = None,
    error_type: str | None = None,
    error_message: str | None = None,
    refusal: bool = False,
)

What the wire showed of a call's last response, with no body.

Attributes:

  • status (int | None) –

    The HTTP status, when a response arrived.

  • headers (Mapping[str, str]) –

    The response headers, names lower-cased.

  • finish_raw (str | None) –

    The reply's finish reason, verbatim.

  • error_code (str | None) –

    The provider's error code, verbatim.

  • error_type (str | None) –

    The provider's error type, verbatim.

  • error_message (str | None) –

    The provider's error text, at most ERROR_MESSAGE_MAX characters. Never the prompt or the completion.

  • refusal (bool) –

    True when the reply carried an explicit refusal field.

CaptureLike

Bases: Protocol

What PydanticAIGenerator reads from a capture after a call; the type of its capture=.

A capture may also carry replies (a sequence of WireReply) and last (the latest WireFacts); the generator reads them when present and falls back to pydantic-ai's own messages when not.

finish_reason property

finish_reason: str | None

The last reply's finish_reason, or None.

usage property

usage: Mapping[str, int | None]

input and output token counts; None for a count the server did not report.

sources property

sources: tuple[str, ...] | None

The reply's top-level sources strings, or None.

WireReply dataclass

WireReply(
    usage: Usage,
    finish_raw: str | None = None,
    model: str | None = None,
    response_id: str | None = None,
    refusal: bool = False,
    headers: Mapping[str, str] = dict(),
)

One 2xx reply of a call, as the wire showed it.

Attributes:

  • usage (Usage) –

    The reply's usage, one request; unmeasured when it carried no usage block.

  • finish_raw (str | None) –

    The first choice's finish_reason, verbatim.

  • model (str | None) –

    The model the body reported.

  • response_id (str | None) –

    The body's id.

  • refusal (bool) –

    True when the first choice's message carried a non-empty refusal.

  • headers (Mapping[str, str]) –

    The response headers, names lower-cased.

PydanticAIChat

PydanticAIChat(
    model: Model | None = None,
    *,
    model_for: (
        Callable[[R], AbstractAsyncContextManager[Model]]
        | None
    ) = None,
    capture: Callable[
        [], AbstractContextManager[CaptureLike]
    ] = capture,
    gated: tuple[type[BaseException], ...] = (),
    wrap: Callable[[Model], Model] | None = None,
    pricer: Pricer | None = None,
    gateway: GatewayReader | None = None,
    extra: Sequence[Classifier] = (),
    on_event: EventHook | None = None,
    policy: WirePolicy | None = None,
    limiter: Limiter | None = None
)

Bases: AdapterCore[ChatRequest], Chat

The Chat port on pydantic-ai; the module docstring states the contract.

Parameters:

  • model (Model | None, default: None ) –

    A fixed pydantic-ai Model.

  • model_for (Callable[[R], AbstractAsyncContextManager[Model]] | None, default: None ) –

    A per-request model factory (e.g. openai_compatible(...), the same object a PydanticAIGenerator takes, so both share one client).

  • capture (Callable[[], AbstractContextManager[CaptureLike]], default: capture ) –

    The capture factory; the default is filled by openai_compatible's response hook.

  • gated (tuple[type[BaseException], ...], default: () ) –

    Exception types that classify as gated.

  • wrap (Callable[[Model], Model] | None, default: None ) –

    Applied to each model model_for yields, e.g. a replay.Recorder.

  • pricer (Pricer | None, default: None ) –

    Prices a call the gateway did not price.

  • gateway (GatewayReader | None, default: None ) –

    Reads a gateway's response headers: its cost and deployment.

  • extra (Sequence[Classifier], default: () ) –

    Classifiers consulted before the shared table, in order.

  • on_event (EventHook | None, default: None ) –

    Receives the generator's events (llm_call_internal_error, llm_cost_unknown, llm_usage_unmeasured, llm_wire_retry, llm_deadline_expired, ...).

  • policy (WirePolicy | None, default: None ) –

    How a failed model request is retried when the request carries none.

  • limiter (Limiter | None, default: None ) –

    Caps concurrent calls per ChatRequest.concurrency_group (else per model id).

Raises:

  • ValueError –

    When neither or both of model and model_for are given.

stream async

stream(
    req: ChatRequest,
    output: type[T] = str,
    *,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None
) -> AsyncGenerator[StreamEvent]

Run one call, yielding its events as they happen and Finished last, exactly once.

The result's stamp carries first_event_s (seconds to the first event; None when there was none), limiter_wait_s and wire_requests.

Raises:

  • CancelledError –

    The consumer was cancelled; the call's task is cancelled first.

OpenAICompatible

OpenAICompatible(
    base_url: str,
    api_key: str | None,
    *,
    max_retries: int,
    transport: (
        AsyncBaseTransport | AsyncBaseTransport | None
    ),
    profile: ModelProfileSpec | None,
    settings: ModelSettings | None,
    client: AsyncClient | None
)

A ModelFor over one OpenAI-wire chat-completions endpoint, on one reused client per loop.

Built by openai_compatible, which documents the arguments. The provider (and the SDK client it wraps) is built once per event loop, with the SDK's own retries at max_retries; each request gets an OpenAIChatModel for its model id on it.

stats

stats() -> Mapping[str, int]

The client's clients_built, clients_open and requests_sent.

aclose async

aclose() -> None

Close the clients built for this endpoint. Idempotent.

PydanticAIEmbedder

PydanticAIEmbedder(
    model: EmbeddingModel | None = None,
    *,
    stamp: str,
    dims: int | None = None,
    model_for: EmbedModelFor | None = None,
    batch_size: int = 16,
    query_instruct: str = "",
    gated: tuple[type[BaseException], ...] = (),
    retry: RetryPolicy | None = None,
    pricer: Pricer | None = None,
    gateway: GatewayReader | None = None,
    on_event: EventHook | None = None,
    policy: WirePolicy | None = None
)

The Embedder port on pydantic-ai; the module docstring states the contract.

Parameters:

  • model (EmbeddingModel | None, default: None ) –

    A fixed pydantic-ai EmbeddingModel.

  • stamp (str) –

    The name of the space the vectors live in; an unnamed space cannot be stored.

  • dims (int | None, default: None ) –

    When given, every vector must be this long, or the call is invalid.

  • model_for (EmbedModelFor | None, default: None ) –

    A per-call model factory.

  • batch_size (int, default: 16 ) –

    Inputs per request.

  • query_instruct (str, default: '' ) –

    The task line prefixed to input_type="query" inputs; empty sends queries plain.

  • gated (tuple[type[BaseException], ...], default: () ) –

    Exception types that classify as gated.

  • retry (RetryPolicy | None, default: None ) –

    Deprecated: policy=retry.wire() is the same. Pass at most one of the two.

  • policy (WirePolicy | None, default: None ) –

    Which failed requests are re-sent, and how; None (and no retry) sends each request once.

  • pricer (Pricer | None, default: None ) –

    Prices a call the gateway did not price; None leaves cost None unless a gateway reported one.

  • gateway (GatewayReader | None, default: None ) –

    Reads a gateway's response headers (e.g. the module adapters.litellm_proxy).

  • on_event (EventHook | None, default: None ) –

    Receives llm_pricer_failed, llm_gateway_header_invalid and llm_wire_retry; without one they go to the stdlib logger llm_ports.adapters.pydantic_ai.

Raises:

  • ValueError –

    When neither or both of model and model_for are given, both retry and policy are given, stamp is empty, or batch_size is below 1.

stamp property

stamp: str

The name of the space this embedder's vectors live in.

embed async

embed(
    texts: Sequence[str],
    *,
    input_type: EmbedInput = "document"
) -> Embedded

Embed texts in order; a model, wire or gate failure returns a named outcome, never raises.

input_type="query" applies query_instruct. Empty texts returns ok with no vectors and zero input tokens. A reply that breaks the count or dimension check is invalid. A failure the generator would call invalid_output, truncated or refused is server (unavailable) here, since an embeddings reply has no parse step, token ceiling or refusal; its reason ends with the root cause of the exception chain (refused, timed out, DNS).

Returns:

  • Embedded –

    An Embedded stamped with this embedder's space. input_tokens is None when any

  • Embedded –

    batch did not report usage, never 0.

Rationale: ADR-0001, ADR-0007, ADR-0019.

OpenAICompatibleFactory

OpenAICompatibleFactory(
    *,
    gated: tuple[type[BaseException], ...] = (),
    probe_deadline_s: float = 5.0
)

The openai_compatible provider kind.

Parameters:

  • gated (tuple[type[BaseException], ...], default: () ) –

    Exception types the generator classifies as gated (a refusal raised by the caller's transport).

  • probe_deadline_s (float, default: 5.0 ) –

    The probe's deadline, in seconds.

Attributes:

  • kind (str) –

    openai_compatible.

build

build(
    spec: ProviderSpec,
    *,
    url: str | None,
    secret: str | None,
    models: Sequence[ModelInfo],
    deps: BuildDeps
) -> Built

Build the provider's Generator, Embedder source and Probe.

Raises:

  • TableInvalid –

    The provider names no base_url, or sets an option this kind does not take.

PydanticAIGenerator

PydanticAIGenerator(
    model: Model | None = None,
    *,
    model_for: (
        Callable[[R], AbstractAsyncContextManager[Model]]
        | None
    ) = None,
    capture: Callable[
        [], AbstractContextManager[CaptureLike]
    ] = capture,
    gated: tuple[type[BaseException], ...] = (),
    wrap: Callable[[Model], Model] | None = None,
    pricer: Pricer | None = None,
    gateway: GatewayReader | None = None,
    extra: Sequence[Classifier] = (),
    on_event: EventHook | None = None,
    policy: WirePolicy | None = None,
    limiter: Limiter | None = None
)

Bases: AdapterCore[GenRequest]

The Generator port on pydantic-ai; the module docstring states the contract.

Parameters:

  • model (Model | None, default: None ) –

    A fixed pydantic-ai Model.

  • model_for (Callable[[R], AbstractAsyncContextManager[Model]] | None, default: None ) –

    A per-request model factory.

  • capture (Callable[[], AbstractContextManager[CaptureLike]], default: capture ) –

    The capture factory; the default is filled by openai_compatible's response hook.

  • gated (tuple[type[BaseException], ...], default: () ) –

    Exception types that classify as gated.

  • wrap (Callable[[Model], Model] | None, default: None ) –

    Applied to each model that model_for yields (not to a fixed model), e.g. a replay.Recorder.

  • pricer (Pricer | None, default: None ) –

    Prices a call the gateway did not price (e.g. adapters.genai_prices.GenaiPricesPricer); None leaves cost None unless a gateway reported one.

  • gateway (GatewayReader | None, default: None ) –

    Reads a gateway's response headers (e.g. the module adapters.litellm_proxy): its cost, which takes precedence over pricer, and its deployment id.

  • extra (Sequence[Classifier], default: () ) –

    Classifiers consulted before the shared table, in order (e.g. adapters.litellm_proxy.classify_gateway).

  • on_event (EventHook | None, default: None ) –

    Receives llm_call_internal_error, llm_cost_unknown, llm_usage_unmeasured, llm_pricer_failed, llm_gateway_header_invalid, llm_wire_retry and llm_deadline_expired; without one they go to the stdlib logger llm_ports.adapters.pydantic_ai.

  • policy (WirePolicy | None, default: None ) –

    How a failed model request is retried when the request carries no policy of its own; None is WirePolicy.default(). WirePolicy.none() sends every request once.

  • limiter (Limiter | None, default: None ) –

    Caps concurrent calls per GenRequest.concurrency_group (else per model id); None runs every call at once.

Raises:

  • ValueError –

    When neither or both of model and model_for are given.

generate async

generate(
    req: GenRequest,
    output: type[T],
    *,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None
) -> Generated[T]

Run one generation; a model, wire or gate failure returns a named outcome, never raises.

Parameters:

  • req (GenRequest) –

    The call's prompt, model and limits.

  • output (type[T]) –

    str for prose, a pydantic model, or a JsonOutput subclass.

  • validate (Validator[T] | None, default: None ) –

    Returns the problems in a parsed value. A non-empty list triggers a re-ask while req.output_retries remain; after that the value is kept with its problems. An exception it raises is internal, never a model outage.

  • retry_message (RetryMessage | None, default: None ) –

    Renders the problems into the re-ask message; by default they are joined with "; ".

Returns:

  • Generated[T] –

    A Generated whose outcome is ok or a named failure. Its attempts hold every

  • Generated[T] –

    retried request; stamp["limiter_wait_s"] the seconds waited for a slot (None without

  • Generated[T] –

    a limiter); stamp["wire_requests"] the requests sent.

Raises:

  • CancelledError –

    The caller cancelled; the slot is released first.

run async

run(
    model: Model,
    req: GenRequest,
    output: type[T],
    validate: Validator[T] | None,
    retry_message: RetryMessage | None,
) -> Generated[T]

Run one agent on model and classify the result; the arguments are as for generate.

A subclass that supplies model on its own transport calls this to share the run, the validator and the classification.

classify_core

classify_core(
    exc: BaseException | None,
    facts: WireFacts,
    *,
    finish: UnifiedFinish | None = None,
    extra: Sequence[Classifier] = (),
    gated: tuple[type[BaseException], ...] = (),
    categorize: Categorize | None = None
) -> Failure | None

Return the failure a call ended with, or None for a success; the module docstring gives the order.

Parameters:

  • exc (BaseException | None) –

    The exception the call raised; None when it returned.

  • facts (WireFacts) –

    What the wire showed of the last response.

  • finish (UnifiedFinish | None, default: None ) –

    The unified reading of facts.finish_raw, from the adapter's own mapping.

  • extra (Sequence[Classifier], default: () ) –

    Classifiers consulted first, in order.

  • gated (tuple[type[BaseException], ...], default: () ) –

    Exception types that mean the caller's gate refused the call.

  • categorize (Categorize | None, default: None ) –

    The framework's own categories (step 4).

Raises:

  • CancelledError –

    Re-raised when exc is one; a cancellation is never an outcome.

Rationale: ADR-0020.

is_context_overflow

is_context_overflow(
    codes: Sequence[str | None],
    messages: Sequence[str | None],
) -> bool

True when a provider code or message names a context overflow (step 5). Pure.

kind_for_status

kind_for_status(status: int) -> FailureKind

Return the kind an HTTP error status maps to (step 6). Pure.

A status not in STATUS_KINDS is server (outcome unavailable), as 1.x mapped it: only the listed 4xx say the input itself was refused.

raised_in_provider

raised_in_provider(e: BaseException) -> bool

True when e was raised by code in PROVIDER_MODULES (step 9). Pure.

The owner is the innermost frame of e's traceback whose package is neither the standard library nor a neutral helper (pydantic, anyio): a KeyError from json inside a caller's validator is owned by the validator; an IndexError in pydantic_ai.models.openai by the provider client.

retry_after_s

retry_after_s(headers: Mapping[str, str]) -> float | None

Return the server's wait hint in seconds, from retry-after-ms or Retry-After.

Pure. Retry-After may be seconds or an HTTP date. A hint in the past, unreadable or not finite is None.

worst

worst(failures: Sequence[Failure]) -> Failure

Return the worst of failures by SEVERITY; the first of equals. Pure.

Raises:

  • ValueError –

    failures is empty.

parse_usage

parse_usage(block: Any) -> Usage

Return one reply's Usage from its OpenAI-wire usage block. Pure.

prompt_tokens and completion_tokens are the inclusive totals; prompt_tokens_details.cached_tokens (OpenAI, llama.cpp) or cache_read_input_tokens (a gateway relaying Anthropic) is the cached part; cache_creation_input_tokens the written part; completion_tokens_details.reasoning_tokens the reasoning part. Other integer keys go to details under their own names (nested ones as parent.key). No block at all is one unmeasured request.

classify

classify(
    exc: BaseException,
    finish_reason: str | None = None,
    *,
    gated: tuple[type[BaseException], ...] = ()
) -> tuple[Outcome, str]

Name the outcome a failed run maps to, with a one-line reason (the 1.x signature).

A thin wrapper over classify_failure, which also returns the kind and its detail. Pure.

Parameters:

  • exc (BaseException) –

    The exception the run raised.

  • finish_reason (str | None, default: None ) –

    The last reply's finish_reason, when the wire reported one.

  • gated (tuple[type[BaseException], ...], default: () ) –

    Exception types that mean the caller's gate refused the call.

Returns:

  • tuple[Outcome, str] –

    The outcome and its reason.

Rationale: ADR-0001, ADR-0020.

classify_failure

classify_failure(
    exc: BaseException | None,
    facts: WireFacts,
    *,
    extra: Sequence[Classifier] = (),
    gated: tuple[type[BaseException], ...] = ()
) -> Failure | None

Return the failure a pydantic-ai call ended with, or None when it succeeded.

Parameters:

  • exc (BaseException | None) –

    The exception the run raised; None when it returned.

  • facts (WireFacts) –

    What the wire showed of the last response (its finish reason, status, error code).

  • extra (Sequence[Classifier], default: () ) –

    Classifiers consulted first, in order (e.g. adapters.litellm_proxy.classify_gateway).

  • gated (tuple[type[BaseException], ...], default: () ) –

    Exception types that mean the caller's gate refused the call.

Raises:

  • CancelledError –

    Re-raised when exc is one.

Rationale: ADR-0020.

openai_finish

openai_finish(raw: str | None) -> UnifiedFinish | None

Return the unified reading of an OpenAI-wire finish_reason; None when there was none. Pure.

openai_compatible_embeddings

openai_compatible_embeddings(
    base_url: str,
    model_name: str,
    api_key: str | None = None,
    *,
    timeout_s: float = 120.0,
    max_retries: int = 0,
    transport: (
        AsyncBaseTransport | AsyncBaseTransport | None
    ) = None
) -> EmbedModelFor

Build a model_for over any OpenAI-wire /embeddings endpoint.

Each call opens one httpx2.AsyncClient (the client pydantic-ai's OpenAI provider takes).

Parameters:

  • base_url (str) –

    The URL up to where /embeddings starts, e.g. http://localhost:4000/v1.

  • model_name (str) –

    The embedding model the endpoint serves.

  • api_key (str | None, default: None ) –

    The bearer key; None sends a placeholder, for servers that need no key.

  • timeout_s (float, default: 120.0 ) –

    The client timeout per call.

  • max_retries (int, default: 0 ) –

    The OpenAI SDK's own retry count. Defaults to 0, so retries are not stacked on those of a gateway in front.

  • transport (AsyncBaseTransport | AsyncBaseTransport | None, default: None ) –

    Replaces httpx2's own transport (a caller's admitting or counting transport, or a MockTransport in a test). An httpx2 transport is used as is; a legacy httpx one is carried across a bridge. It always sits under the index-order re-sort.

Returns:

  • EmbedModelFor –

    An EmbedModelFor that yields an OpenAIEmbeddingModel.

Raises:

  • ValueError –

    api_key holds whitespace, a control or a non-ASCII character. The value is not shown.

Rationale: ADR-0009.

count_url

count_url(base_url: str) -> str

The gateway's count route for an OpenAI-wire base URL: /utils/token_counter at its root. Pure.

translate_settings

translate_settings(
    settings: Mapping[str, Any], info: ModelInfo | None
) -> tuple[dict[str, Any], tuple[str, ...]]

Spell a request's provider-neutral thinking switch the way its model needs. Pure.

Parameters:

  • settings (Mapping[str, Any]) –

    The request's settings; a schema 2 role puts thinking: bool in them.

  • info (ModelInfo | None) –

    The request's model, or None when the table does not know it (the settings pass through unchanged, so pydantic-ai's own unified thinking applies).

Returns:

  • dict[str, Any] –

    The settings to send, and the names of settings the model cannot take (dropped):

  • tuple[str, ...] –

    unified keeps thinking (pydantic-ai's ModelSettings["thinking"]); chat_template

  • tuple[dict[str, Any], tuple[str, ...]] –

    moves it to extra_body.chat_template_kwargs.enable_thinking; none drops it, and names

  • tuple[dict[str, Any], tuple[str, ...]] –

    it when the role asked to think.

openai_compatible

openai_compatible(
    base_url: str,
    api_key: str | None = None,
    *,
    max_retries: int = 0,
    transport: (
        AsyncBaseTransport | AsyncBaseTransport | None
    ) = None,
    profile: ModelProfileSpec | None = None,
    settings: ModelSettings | None = None,
    client: AsyncClient | None = None
) -> OpenAICompatible

Build a model_for over any OpenAI-wire chat-completions endpoint.

One httpx2.AsyncClient (the client pydantic-ai's OpenAI provider takes) is built per event loop, with explicit pool limits, a 5 s connect bound and the capture hook, and reused by every call; the per-request bound is passed per call. aclose() closes it.

Parameters:

  • base_url (str) –

    The URL up to where /chat/completions starts, e.g. http://localhost:4000/v1.

  • api_key (str | None, default: None ) –

    The bearer key; None sends a placeholder, for servers that need no key.

  • max_retries (int, default: 0 ) –

    The OpenAI SDK's own retry count. Defaults to 0, so retries are not stacked on those of a gateway in front.

  • transport (AsyncBaseTransport | AsyncBaseTransport | None, default: None ) –

    Replaces httpx2's own transport (a caller's admitting or counting transport, or a MockTransport in a test). An httpx2 transport is used as is; a legacy httpx one is carried across a bridge, so the provider still gets an httpx2 client.

  • profile (ModelProfileSpec | None, default: None ) –

    A pydantic-ai ModelProfileSpec: a partial ModelProfile merged over the one the provider resolves for the model, or a function from that resolved profile to the profile to use. E.g. it moves an agent whose output type the caller does not construct onto native structured output. None keeps the provider's.

  • settings (ModelSettings | None, default: None ) –

    Model-level ModelSettings, applied under each run's own settings (pydantic-ai merges run settings over model settings, key by key at the top level). None sets none.

  • client (AsyncClient | None, default: None ) –

    A caller's own httpx2.AsyncClient, used as given and never closed here (add record_response to its response hooks to keep the wire capture).

Returns:

  • OpenAICompatible –

    A ModelFor (OpenAICompatible) that yields an OpenAIChatModel for req.model, carrying

  • OpenAICompatible –

    profile and settings, with stats() and aclose().

Raises:

  • ValueError –

    api_key holds whitespace, a control or a non-ASCII character (the HTTP stack would quote it, value and all, when it refused the header). The value is not shown.

Rationale: ADR-0009.

framework_output

framework_output(output: Any) -> Any

Map the port's JsonOutput to pydantic-ai's StructuredDict; any other type passes unchanged.

strict_json_schema

strict_json_schema(
    model: type[BaseModel],
) -> dict[str, Any]

Return a response_format: json_schema, strict: true schema that OpenAI and llama.cpp accept.

OpenAI's strict mode refuses what pydantic's model_json_schema() emits: every object needs additionalProperties: false and every property listed in required (nullable via anyOf null). llama.cpp's grammar compiler accepts the strict form too, so one shape serves both. The transform is pydantic-ai's OpenAIJsonSchemaTransformer.

structured_dict

structured_dict(
    schema: Mapping[str, Any], name: str
) -> type[dict[str, Any]]

Wrap a hand-written JSON schema as an output type that returns a plain dict.

The schema is compiled into the grammar unchanged, so maxLength, enum, maxItems and required all survive. name is the schema's title and the name a FunctionModel stub dispatches on. No shape validation runs beyond what the grammar enforces.

llm_ports.adapters.pydantic_ai.replay

Record a live model's replies once and replay them in CI, for pydantic-ai models.

Example

rec = Recorder(real_model) # wraps any pydantic-ai Model; keeps every reply ... # run the pipeline once against the live model json.dump(rec.replies, f) # {key(messages): reply text} model = replay(json.load(f)) # a FunctionModel answering from that dict

key(messages) (version 1) is the sha of the first user prompt plus the attempt number (the count of model responses already in the conversation, so an output retry's second attempt has its own key). key_v2 (version 2, 1.24) covers everything that changes the answer: the model, the system prompt and instructions, the whole message list, the settings (sorted) and the output schema, plus the attempt; a system-prompt change no longer replays the old answer. Recorder(version=2) writes {"version": 2, "entries": {...}} (cassette()); replay reads both shapes. A changed request misses the lookup, the call raises NotRecorded naming the parts that differ from the nearest entry, and the golden set must be re-recorded. ReplayModel.unplayed() names the entries a run never asked for (the llm_replay pytest fixture fails a partly played cassette with it).

A version-2 Recorder given secrets refuses (SecretInCassette) to keep a reply containing any of them. Only the reply text is kept: no headers, no request bodies.

Recording keeps every raw reply parsed in attempts (an agent returns only the last attempt; a grader wants every one); an unparsable reply is kept as {"_unparsed": head}, never dropped. Either wrapper plugs into PydanticAIGenerator as its wrap= model wrapper; recording(make) and recorder(make) build a generator with the wrapper on and return it with the list or dict it fills.

The module depends on pydantic-ai, so it is imported by its path and needs the pydantic-ai extra.

NotRecorded

Bases: RuntimeError

The replay has no reply for this request: it changed since the recording.

SecretInCassette

Bases: ValueError

A reply about to be recorded contains a bound secret; nothing was written.

Recorder

Recorder(
    wrapped: Any,
    replies: dict[str, Any] | None = None,
    *,
    version: int = 1,
    secrets: Iterable[str] = ()
)

Bases: WrapperModel

A model wrapper that keeps every reply's text in replies.

Version 1 keys by key(messages) and keeps the text; version 2 keys by key_v2 and keeps {"text", "parts"} (the key's parts, for NotRecorded's diff). Pass replies to share one dict across several wrapped models.

Parameters:

  • wrapped (Any) –

    The live model.

  • replies (dict[str, Any] | None, default: None ) –

    The dict to fill; None starts an empty one.

  • version (int, default: 1 ) –

    1 (the 1.x key, the default) or 2.

  • secrets (Iterable[str], default: () ) –

    Values that must never be written; a reply containing one raises SecretInCassette (version 2).

cassette

cassette() -> dict[str, Any]

The recording as written to disk: version 2's {"version", "entries"}, version 1's dict.

request async

request(
    messages: list[ModelMessage],
    model_settings: ModelSettings | None,
    model_request_parameters: ModelRequestParameters,
) -> ModelResponse

Forward to the wrapped model and record the reply under its key.

ReplayModel

Bases: FunctionModel

The model replay returns; unplayed() names the entries no request asked for.

unplayed

unplayed() -> tuple[str, ...]

The recorded keys never replayed, sorted.

Recording

Recording(wrapped: Any, attempts: list[Any] | None = None)

Bases: WrapperModel

A model wrapper that keeps every raw reply, parsed, in attempts.

Pass attempts to share one list across several wrapped models (one list per run). An unparsable reply is kept as {"_unparsed": head}, its first UNPARSED_HEAD characters.

request async

request(
    messages: list[ModelMessage],
    model_settings: ModelSettings | None,
    model_request_parameters: ModelRequestParameters,
) -> ModelResponse

Forward to the wrapped model and append the reply, parsed, to attempts.

key

key(messages: list[ModelMessage]) -> str

Return the recording's lookup key, <sha of the first user prompt>:<attempt>.

key_parts

key_parts(
    messages: Sequence[ModelMessage],
    *,
    model: str,
    system: str,
    settings: Mapping[str, Any] | None,
    output_schema: Any,
    attempt: int
) -> dict[str, str]

The digest of each part key_v2 covers (KEY_PARTS). Pure.

key_v2

key_v2(
    messages: Sequence[ModelMessage],
    *,
    model: str,
    system: str,
    settings: Mapping[str, Any] | None,
    output_schema: Any,
    attempt: int
) -> str

Return the version-2 lookup key, <sha of every part but the attempt>:<attempt>. Pure.

replay

replay(recorded: Mapping[str, Any]) -> ReplayModel

Return a model that answers from a recording: a version-1 dict or a version-2 cassette.

The returned model raises NotRecorded for a request the recording never saw.

recording

recording(
    make: MakeGenerator[G],
) -> tuple[G, list[Any]]

Return a generator whose every raw reply lands, parsed, in the returned list (Recording).

recorder

recorder(
    make: MakeGenerator[G], *, version: int = 1
) -> tuple[G, dict[str, Any]]

Return a generator whose every reply lands in the returned dict under its key (Recorder).

llm_ports.adapters.genai_prices

A Pricer over genai-prices: what a call cost, from the model that answered and its usage.

genai-prices ships a dated price table for the hosted providers and computes a call's price as a Decimal from its token counts (input inclusive of cached tokens, as Usage is). This module adapts it to ports.Pricer and adds the two things the table cannot know:

  • a self-hosted model costs what the caller says: overrides maps a model id to a per-request price (often Decimal(0) for a local box, stated, never assumed);
  • a model the table does not know is Cost(None, "unknown", reason="unknown_model"), never free.
Invariants
  • price never raises: a failing lookup is Cost(None, "unknown", reason="pricer_error").
  • Unknown is None: a call with unmeasured usage is reason="no_usage", never priced from a partial count.
  • No I/O: the bundled table is read; genai-prices' own UpdatePrices refresh is the caller's choice.
Example

gen = PydanticAIGenerator(model_for=..., pricer=GenaiPricesPricer(overrides={"local-8b": Decimal(0)}))

Rationale: ADR-0019.

GenaiPricesPricer

GenaiPricesPricer(
    *,
    provider_for: (
        Callable[[Answered], str | None] | None
    ) = None,
    overrides: Mapping[str, Decimal] | None = None
)

Price a call from genai-prices' table, with caller overrides for models it cannot know.

Parameters:

  • provider_for (Callable[[Answered], str | None] | None, default: None ) –

    The genai-prices provider id for a call (e.g. "openai"), or None to let genai-prices match the model id across providers.

  • overrides (Mapping[str, Decimal] | None, default: None ) –

    A per-request price in US dollars by model id, looked up under the requested id first and then the answering model's; it wins over the table.

Rationale: ADR-0019.

price

price(usage: Usage, answered: Answered) -> Cost

Return what the call cost; usd is None when it cannot be known. Never raises.

llm_ports.adapters.litellm_proxy

What a LiteLLM proxy in front of the model says about a call, read from its response.

A LiteLLM proxy knows things the response body does not: which deployment served an alias (x-litellm-model-id, x-litellm-model-api-base), what the call cost at that deployment's price (x-litellm-response-cost), and how many retries and fallbacks it spent (x-litellm-attempted-retries, x-litellm-attempted-fallbacks). It also refuses a call in two ways that look alike on the wire: a 429 because its parallel-request limit is full (comes back in seconds), and a 429 because a budget or an upstream quota is spent (does not).

The module satisfies ports.GatewayReader (pass the module itself as gateway=), and classify_gateway is an extra classifier for the generator. It imports no LiteLLM and no HTTP client: it reads header mappings and WireFacts.

Invariants
  • Pure: no I/O.
  • A cost header that is present but not a plausible dollar amount (a finite number, 0 or between COST_MIN_USD and COST_MAX_USD) raises ValueError from cost_from_headers; the generator treats it as absent and reports it.
  • classify_gateway names only quota; a plain 429 is left to the shared table (rate_limited, retryable, with the server's Retry-After).
Example

gen = PydanticAIGenerator(model_for=openai_compatible(url, key), gateway=litellm_proxy, extra=(litellm_proxy.classify_gateway,))

Rationale: ADR-0019, ADR-0020.

cost_from_headers

cost_from_headers(
    headers: Mapping[str, str],
) -> Decimal | None

Return the cost the proxy reported (x-litellm-response-cost), or None when it reported none.

Pure.

A reported zero (0, -0, 0.000) is Decimal(0).

Raises:

  • ValueError –

    The header is present but not a finite number of dollars that is 0 or between COST_MIN_USD and COST_MAX_USD (exclusive of the maximum).

answered_from_headers

answered_from_headers(
    headers: Mapping[str, str],
) -> tuple[str | None, str | None]

Return (deployment id, upstream api base) as the proxy reported them; None for each it did not. Pure.

attempts_from_headers

attempts_from_headers(
    headers: Mapping[str, str],
) -> tuple[int | None, int | None]

Return (retries, fallbacks) the proxy spent on the call; None for each it did not report. Pure.

retry_hint_s

retry_hint_s(text: str) -> float | None

Return a provider's "Please retry in 8h26m23.4s" hint in seconds; None when the text has none. Pure.

classify_gateway

classify_gateway(
    exc: BaseException | None, facts: WireFacts
) -> Failure | None

Name a 429 that no wait will clear as quota; leave everything else to the shared table.

A 429 is quota when its error names a budget (the proxy's budget for a key, team or provider), a per-day limit, or asks for a retry longer than QUOTA_HINT_MAX_S (an upstream provider's spent quota). A 429 from the proxy's parallel-request limit is left alone: the shared table makes it rate_limited, retryable, with the server's Retry-After. Pure.

Rationale: ADR-0020.

llm_ports.adapters.gepa

Tune a prompt's text components with GEPA, reflective prompt evolution (extra [gepa]).

optimize(seed, train, val, evaluate=..., reflect=...) runs the gepa core engine over a candidate: a mapping of named text components (an instruction block, a system prompt) to their text. The caller owns everything domain-specific through two callables:

  • evaluate(candidate, items) runs the program with the candidate's texts on each item and returns one Rollout per item: the output, a score in [0, 1] (higher is better) and the feedback a reflection model should read (grader reasons, validation problems).
  • reflect(prompt) is the reflection model: prompt text in, completion text out.

GEPA evaluates candidates on minibatches of train, asks reflect to rewrite one component from the rollouts' feedback, keeps a rewrite only when it beats its parent on the same minibatch, and tracks every kept candidate's mean score on val. Optimized reports the best candidate by that mean next to the seed's own, so a caller promotes only when improved is true and val was held out from anything the caller tuned by hand.

Invariants
  • Never raises: too few items is Insufficient; an evaluate that raises, a rollout count that does not match its batch, a GEPA engine error, a reflect that failed on every call, or one that failed max_reflection_failures times in a row is Unavailable.
  • A reflection failure never passes silently: GEPA itself logs a failed reflection and moves on, so optimize counts every reflect call and failure and reports both on Optimized.
  • GEPA's own log lines are kept (Optimized.log_tail, Unavailable.log_tail), never printed.
  • A non-finite score is scored 0.0 and its feedback says so; a score outside [0, 1] is clamped.
  • The seed candidate is never mutated; Optimized.candidate is a fresh dict.
  • max_metric_calls bounds the item evaluations GEPA spends (its own budget accounting).
  • The search always ends without the budget: GEPA swallows a failed reflection and an evaluator that caches spends no budget, so a dead reflection model would otherwise spin. max_reflection_failures consecutive failures stop it as Unavailable; timeout_s, when set, stops it on wall-clock time and Optimized.stopped says so.

Rationale: ADR-0017.

Rollout dataclass

Rollout(output: O, score: float, feedback: str = '')

One item run with one candidate, and how it scored.

Attributes:

  • output (O) –

    What the program produced (opaque to GEPA; shown to the reflection model).

  • score (float) –

    In [0, 1], higher is better. GEPA sums it on minibatches and averages it on val.

  • feedback (str) –

    Why the item scored as it did, in words the reflection model can act on.

Optimized dataclass

Optimized(
    candidate: dict[str, str],
    seed_score: float,
    best_score: float,
    improved: bool,
    candidates: int,
    metric_calls: int,
    n_train: int,
    n_val: int,
    reflections: int,
    reflection_failures: int,
    log_tail: tuple[str, ...] = (),
    stopped: str = "budget",
)

The best candidate GEPA found, scored on the validation items.

Attributes:

  • candidate (dict[str, str]) –

    component -> text of the best candidate by mean validation score.

  • seed_score (float) –

    The seed candidate's mean validation score.

  • best_score (float) –

    The best candidate's mean validation score.

  • improved (bool) –

    best_score > seed_score; a caller promotes only when this is true.

  • candidates (int) –

    Candidates GEPA kept (the seed included).

  • metric_calls (int) –

    Item evaluations spent.

  • n_train (int) –

    Train items given.

  • n_val (int) –

    Validation items given.

  • reflections (int) –

    reflect calls made.

  • reflection_failures (int) –

    reflect calls that raised (GEPA skipped those steps).

  • log_tail (tuple[str, ...]) –

    GEPA's last log lines.

  • stopped (str) –

    Why the search ended: "budget" (max_metric_calls spent) or "timeout".

Unavailable dataclass

Unavailable(reason: str, log_tail: tuple[str, ...] = ())

The search could not run or finish (an evaluator, reflection or engine failure).

Attributes:

  • reason (str) –

    The exception's type and message, or why no reflection succeeded.

  • log_tail (tuple[str, ...]) –

    GEPA's last log lines.

RolloutAdapter

RolloutAdapter(
    evaluate: Callable[
        [Mapping[str, str], Sequence[I]],
        Sequence[Rollout[O]],
    ],
    describe: Callable[[I], str] = str,
)

A GEPA adapter over a caller's batch evaluator.

Implements GEPA's GEPAAdapter protocol: evaluate runs the caller's evaluator and returns an EvaluationBatch; make_reflective_dataset turns each rollout into the record GEPA's default proposer shows the reflection model (Inputs, Generated Outputs, Feedback).

Parameters:

  • evaluate (Callable[[Mapping[str, str], Sequence[I]], Sequence[Rollout[O]]]) –

    (candidate, items) -> one Rollout per item.

  • describe (Callable[[I], str], default: str ) –

    Renders an item for the reflection model; str by default.

Raises:

  • ValueError –

    From evaluate, when the evaluator returns a rollout count that does not match its batch (GEPA treats it as a systemic failure; optimize names it).

evaluate

evaluate(
    batch: list[I],
    candidate: dict[str, str],
    capture_traces: bool = False,
) -> EvaluationBatch[dict[str, Any], O]

Run candidate on batch through the caller's evaluator.

make_reflective_dataset

make_reflective_dataset(
    candidate: dict[str, str],
    eval_batch: EvaluationBatch[dict[str, Any], O],
    components_to_update: list[str],
) -> dict[str, list[dict[str, Any]]]

Return the per-component records the reflection model reads.

optimize

optimize(
    seed: Mapping[str, str],
    train: Sequence[I],
    val: Sequence[I],
    *,
    evaluate: Callable[
        [Mapping[str, str], Sequence[I]],
        Sequence[Rollout[O]],
    ],
    reflect: Callable[[str], str],
    describe: Callable[[I], str] = str,
    max_metric_calls: int = DEFAULT_METRIC_CALLS,
    minibatch_size: int = 3,
    seed_value: int = 0,
    run_dir: str | None = None,
    min_train: int = MIN_TRAIN,
    min_val: int = MIN_VAL,
    max_reflection_failures: int = MAX_REFLECTION_FAILURES,
    timeout_s: float | None = None
) -> Optimized | Insufficient | Unavailable

Evolve seed's text components with GEPA and return the best by validation score.

Parameters:

  • seed (Mapping[str, str]) –

    component -> text to start from; every component may be rewritten.

  • train (Sequence[I]) –

    Items GEPA's minibatches and reflections draw from.

  • val (Sequence[I]) –

    Held-out items every kept candidate is scored on; disjoint from train.

  • evaluate (Callable[[Mapping[str, str], Sequence[I]], Sequence[Rollout[O]]]) –

    (candidate, items) -> one Rollout per item; score per-item failures 0.0 with feedback rather than raising (a raise ends the search as Unavailable).

  • reflect (Callable[[str], str]) –

    The reflection model, prompt text in, completion text out.

  • describe (Callable[[I], str], default: str ) –

    Renders an item for the reflection model.

  • max_metric_calls (int, default: DEFAULT_METRIC_CALLS ) –

    The item-evaluation budget.

  • minibatch_size (int, default: 3 ) –

    Train items per reflection step.

  • seed_value (int, default: 0 ) –

    GEPA's random seed (minibatch sampling, component order).

  • run_dir (str | None, default: None ) –

    A directory GEPA checkpoints to, so a stopped run resumes; None keeps no state.

  • min_train (int, default: MIN_TRAIN ) –

    Train items below which nothing is searched.

  • min_val (int, default: MIN_VAL ) –

    Validation items below which nothing is searched.

  • max_reflection_failures (int, default: MAX_REFLECTION_FAILURES ) –

    Consecutive failed reflect calls that end the search as Unavailable (a reflection model that is down or out of quota).

  • timeout_s (float | None, default: None ) –

    Wall-clock seconds after which the search stops and reports what it has (Optimized.stopped == "timeout"); None sets no time bound.

Returns:

llm_ports.adapters.http

Adapters over HTTP (httpx): a reranker, a grounder client, a health probe, a token counter, and an index-ordered transport.

IndexOrderedTransport

IndexOrderedTransport(
    inner: AsyncBaseTransport | None = None,
)

Bases: AsyncBaseTransport

An httpx transport that re-sorts a 200 reply's index-keyed list (INDEXED_LISTS) by index.

Parameters:

  • inner (AsyncBaseTransport | None, default: None ) –

    The transport that sends the request; None uses httpx's own.

Invariants
  • A non-200 reply, a path not in INDEXED_LISTS, an unreadable body, or a list already in index order is returned unchanged.

handle_async_request async

handle_async_request(request: Request) -> Response

Send request through the inner transport and return the reply with its list in index order.

aclose async

aclose() -> None

Close the inner transport.

GrounderUnavailable

Bases: Exception

The sidecar did not answer with a verdict: a transport error, a non-200 or a malformed reply.

HttpGrounder

HttpGrounder(
    url: str,
    *,
    timeout_s: float = 20.0,
    client: Client | None = None
)

A Grounder whose model is behind {url}/verdict.

Parameters:

  • url (str) –

    The sidecar's base URL; a trailing slash is dropped.

  • timeout_s (float, default: 20.0 ) –

    The httpx timeout, in seconds, of the client this grounder builds.

  • client (Client | None, default: None ) –

    A ready httpx client, used instead of building one from timeout_s.

Attributes:

  • checker (str) –

    "http" until the first verdict, then the sidecar's own checker name.

Invariants
  • Every verdict is the sidecar's, the no-spans rule included (its own Grounder enforces it): one judge, one checker name on every verdict.
  • A dead sidecar raises for a no-spans claim rather than answering for it.

verdict

verdict(
    claim: Claim, spans: Sequence[Chunk]
) -> GroundingVerdict

Return the sidecar's verdict on claim against spans.

Raises:

HttpProbe

HttpProbe(
    url: str,
    *,
    secret: str | None = None,
    deadline_s: float = PROBE_DEADLINE_S,
    transport: AsyncBaseTransport | None = None,
    clock: Callable[[], float] = monotonic
)

A Probe that GETs one URL.

Parameters:

  • url (str) –

    The URL to ask, e.g. http://localhost:4000/v1/models.

  • secret (str | None, default: None ) –

    Sent as a bearer token; None sends no Authorization header.

  • deadline_s (float, default: PROBE_DEADLINE_S ) –

    The whole probe's deadline, connection included.

  • transport (AsyncBaseTransport | None, default: None ) –

    Replaces httpx's own transport (a test's MockTransport).

  • clock (Callable[[], float], default: monotonic ) –

    Monotonic seconds; injectable for tests.

probe async

probe() -> Health

Ask the URL once. Never raises.

Returns:

  • Health –

    up with the listed model ids (empty when the list is empty); down when the backend

  • Health –

    could not be reached, timed out or answered with an error status; unknown when the

  • Health –

    probe failed or could not send its request, the credential was refused (reason

  • Health –

    auth: ...), the backend has no endpoint to judge it by, or a 200 body is not a model

  • Health –

    list. The reason names an exception by its type only.

HttpReranker

HttpReranker(
    url: str,
    model: str,
    *,
    api_key: str | None = None,
    timeout_s: float = DEFAULT_TIMEOUT_S,
    transport: AsyncBaseTransport | None = None,
    policy: WirePolicy | None = None,
    on_event: EventHook | None = None
)

A Reranker over one /rerank endpoint.

Each call opens its own httpx client, and keeps it across the call's retries.

Parameters:

  • url (str) –

    The full /rerank endpoint URL.

  • model (str) –

    The model name sent on every request and reported on every result.

  • api_key (str | None, default: None ) –

    Sent as a bearer token when set.

  • timeout_s (float, default: DEFAULT_TIMEOUT_S ) –

    The per-call httpx timeout, in seconds.

  • transport (AsyncBaseTransport | None, default: None ) –

    Replaces httpx's own transport (a consumer's counting transport, or httpx.MockTransport in a test). It always sits under IndexOrderedTransport.

  • policy (WirePolicy | None, default: None ) –

    Which failed requests are re-sent, and how; None sends each request once.

  • on_event (EventHook | None, default: None ) –

    Receives llm_wire_retry; without one it goes to the stdlib logger llm_ports.adapters.http.

Raises:

  • ValueError –

    At construction, when url or model is empty.

rerank async

rerank(query: str, documents: Sequence[str]) -> Reranked

Score documents against query, in input order.

An empty documents returns ok with no scores and sends no request. A failure is a named outcome, never an exception (see the module invariants).

HttpTokenCounter

HttpTokenCounter(
    url: str,
    *,
    api_key: str | None = None,
    key: str = "total_tokens",
    models: Collection[str] | None = None,
    timeout_s: float = COUNT_TIMEOUT_S,
    transport: AsyncBaseTransport | None = None
)

A TokenCounter over one count endpoint.

Parameters:

  • url (str) –

    The full count URL, e.g. http://gateway:4000/utils/token_counter.

  • api_key (str | None, default: None ) –

    Sent as a bearer token when set.

  • key (str, default: 'total_tokens' ) –

    The reply's integer key holding the count.

  • models (Collection[str] | None, default: None ) –

    The model ids this counter may count; others are unsupported with no request. None counts every model.

  • timeout_s (float, default: COUNT_TIMEOUT_S ) –

    The per-count timeout, in seconds.

  • transport (AsyncBaseTransport | None, default: None ) –

    Replaces httpx's own transport (a test's MockTransport).

Raises:

  • ValueError –

    At construction, when url is empty.

count async

count(
    model: str,
    *,
    system: str,
    messages: Sequence[Message],
    tools: Sequence[Tool] = ()
) -> Counted

Count what model would read; the module invariants state the outcomes.

llm_ports.adapters.langfuse.tracer

The Tracer port over the Langfuse Python SDK 5 (extra [langfuse]).

Example

tracer = LangfuseTracer(Langfuse(...)) # the caller builds the client async with observe(tracer, name="summarize", model=m, provider="openai", input=prompt) as obs: ...

This adapter owns the generation lifecycle, the usage block, end, score and the managed-prompt link. The caller owns building the client (keys, host, its OpenTelemetry provider and export filter) and what goes into the calls (names, metadata).

Langfuse SDK 5 is OpenTelemetry underneath:

  • A generation is start_observation(as_type="generation"), opened as a root in the current OTel trace (trace_context={"trace_id": ...}), so the Langfuse trace id is the id every other span of that request carries. Outside a span a fresh trace id is minted.
  • Trace-level fields (session, trace name, tags) ride propagate_attributes around the open. name is stored verbatim as the generation name, the trace name and the one tag, so a cost dashboard can group by it.
  • Usage is usage_details with the measured sides of {"input", "output"}; the server adds total and prices by model.
  • Export is a background batch: end does not block on the network.

provider has no Langfuse field; it rides metadata["provider"] unless the caller set that key.

Invariants
  • An unreported token count is never sent as 0: only the measured sides go, and with neither measured no usage block is sent.
  • Fails open: a client that raises on open yields a NoopGeneration.
  • Every failure (open, end, score) emits at most one warning event per warn_hold_s through on_event (default: the stdlib logger llm_ports.adapters.langfuse.tracer), and never raises into the call it describes.
  • The prompt link is best effort and silent.

Rationale: ADR-0007.

LangfuseGeneration

LangfuseGeneration(
    gen: Any, warn: Callable[[str, Exception], None]
)

Generation over an SDK 5 LangfuseGeneration.

end is the one terminal call: its fields go in a single update, then end() closes the span. Failures go to warn, never raised.

end

end(**fields: Unpack[EndFields]) -> None

Close the generation with its output, usage, cost and metadata; never raises.

Usage with neither count reported sends no usage block (and an unreported side is omitted, never 0). cost_details is sent only when the call's cost is known in dollars; otherwise Langfuse's own model pricing may apply, and metadata.cost_source says which. level="ERROR" sends ERROR and a cancelled call ("CANCELLED") sends WARNING, each with status_message. finish, answered and failure already reach the sink as metadata on the library's own call path; they are accepted here for the port's signature.

score

score(
    name: str,
    value: float,
    comment: str | None = None,
    metadata: Mapping[str, Any] | None = None,
) -> None

Attach a numeric score to the generation, with its metadata when given; never raises.

LangfuseTracer

LangfuseTracer(
    client: Any,
    *,
    prompt_label: str = DEFAULT_PROMPT_LABEL,
    prompt_cache_ttl_s: int = DEFAULT_PROMPT_CACHE_TTL_S,
    warn_hold_s: float = DEFAULT_WARN_HOLD_S,
    clock: Callable[[], float] = monotonic,
    on_event: EventHook | None = None
)

Tracer whose sink is Langfuse.

Build one per client; the SDK client is per process.

Parameters:

  • client (Any) –

    A Langfuse SDK 5 client, built by the caller.

  • prompt_label (str, default: DEFAULT_PROMPT_LABEL ) –

    The managed-prompt label a generation links to.

  • prompt_cache_ttl_s (int, default: DEFAULT_PROMPT_CACHE_TTL_S ) –

    The SDK's cache TTL for that prompt fetch.

  • warn_hold_s (float, default: DEFAULT_WARN_HOLD_S ) –

    The minimum seconds between two warning events.

  • clock (Callable[[], float], default: monotonic ) –

    A monotonic clock, injectable for tests.

  • on_event (EventHook | None, default: None ) –

    The warning sink; defaults to the stdlib logger llm_ports.adapters.langfuse.tracer.

start_generation

start_generation(
    *,
    name: str,
    model: str,
    provider: str,
    input: Any,
    metadata: Mapping[str, Any],
    session_id: str | None,
    prompt_name: str | None
) -> Generation

Open a generation as a root span in the current OTel trace; never raises.

With prompt_name, the generation links to the prompt's labelled version (a cached fetch), so the prompt's generation count follows the live version across re-syncs. Any failure to open returns a NoopGeneration.

usage_details

usage_details(
    prompt: int | None, completion: int | None
) -> dict[str, int] | None

Return the Langfuse usage block, or None when the provider reported neither count.

When only one count is reported, only that side is sent: an unreported side is unknown, and booking it as 0 would make a half-measured call look cheaper than it was. Pure.

Rationale: ADR-0007, ADR-0019.

trace_id

trace_id(client: Any) -> str

Return the current OTel trace id as 32 hex digits, or a fresh one from the client outside a span.

llm_ports.adapters.langfuse.prompt_source

The PromptSource port over Langfuse prompt management.

Example

source = LangfusePromptSource(Langfuse(...), seed_for) prompt = source.get("summarize") # Prompt(text, version, source="langfuse:v7" | "repo")

The repository seed leads; the managed label mirrors it and never leads. Per get(name, label):

  1. get_prompt(name, label, fallback=<seed>, cache_ttl_seconds=ttl) goes through the SDK's own cache (an expired entry is served and refreshed on a background thread).
  2. If the reply is the fallback, or its config.content_sha is not the seed's sha, or the seed's config carries an identity the reply's config does not, the managed copy is absent or stale: the seed is pushed under the label with create_prompt(...) and refetched uncached.
  3. If the reply is still the fallback, or its text is not the seed's, the seed is served as Prompt(source="repo"). Otherwise the result is Prompt(source="langfuse:vN") with the managed version and its stored config.

seed_for(name) -> Seed is injected: the adapter knows nothing about where a repository prompt lives or how it is fingerprinted (prompts.Pack.seed builds one). With label_from_seed=True, a seed whose config names a label (a pack's manifest label) is served and mirrored under that label instead of the label argument. A seed with a non-empty system (or kind="chat") is a chat prompt: pushed as type="chat" with the message list, fetched as chat, and compared message by message on role and content. Langfuse refuses a type change on an existing name, so kind can force chat for a seed with no system message.

The client is duck-typed: anything with the SDK's get_prompt / create_prompt signatures. The caller builds it, so this module imports no SDK.

Invariants
  • Fails soft: an unreachable store, a refused push or a malformed reply each serves the seed as source == "repo" and emits a warning event through on_event (default: the stdlib logger llm_ports.adapters.langfuse.prompt_source); none of them raises.
  • The SDK never caches a fallback reply, so the down verdict is held per name for one TTL here: a store that is down costs one round-trip per TTL, not one per call.

Rationale: ADR-0008.

LangfusePromptSource

LangfusePromptSource(
    client: Any,
    seed_for: Callable[[str], Seed],
    *,
    cache_ttl_s: int = DEFAULT_CACHE_TTL_S,
    clock: Callable[[], float] = monotonic,
    on_event: EventHook | None = None,
    label_from_seed: bool = False
)

PromptSource whose managed store is Langfuse.

One instance per client; the SDK caches.

Parameters:

  • client (Any) –

    A Langfuse client, or anything with its get_prompt / create_prompt signatures.

  • seed_for (Callable[[str], Seed]) –

    Returns the repository seed for a prompt name.

  • cache_ttl_s (int, default: DEFAULT_CACHE_TTL_S ) –

    The SDK cache TTL, and how long a down verdict is held per name.

  • clock (Callable[[], float], default: monotonic ) –

    A monotonic clock, injectable for tests.

  • on_event (EventHook | None, default: None ) –

    The warning sink; defaults to the stdlib logger llm_ports.adapters.langfuse.prompt_source.

  • label_from_seed (bool, default: False ) –

    Opt in to the seed's own label: a seed whose config["label"] is set is served and mirrored under it, whatever label get is given. Off by default.

get

get(name: str, *, label: str = 'production') -> Prompt

Return prompt name, mirrored to the managed store under label.

Serves the managed version when it matches the seed, else the seed as source == "repo"; the module docstring gives the steps. A store failure never raises. With label_from_seed, a seed's config["label"] replaces label.

llm_ports.adapters.langfuse.experiments

Record a graded evaluation as a Langfuse dataset experiment (extra [langfuse]).

seed writes a golden set into a dataset, idempotently: each item's id is the caller's or a hash of its input, and Langfuse upserts an item by id, so a re-seed rewrites rather than duplicates. run drives the SDK's own experiment runner (Langfuse.run_experiment): it opens one trace per dataset item, links the trace to the item, and books each grade as a score on it. record_rows records rows already graded offline (a golden-set run that graded a recording) as one run on a dataset: the task hands back the row's public output plus its id, and the evaluator looks the row up by that id, because an SDK evaluator sees input, output, expected output and metadata, never the item.

The SDK logs a task or evaluator that raised and drops or leaves that item ungraded, so a run that only counted what came back would report a clean ratio over a shrunken denominator. Here a run that graded fewer items than it was handed is an invalid outcome naming the shortfall.

Example

run = record_rows(client, dataset="summary_golden_v1", run_name="pack-1a2b", rows=rows, output=lambda r: {"text": r["text"]}, grade=lambda r: [("echo", r["echo"], "NUMERIC")]) if not run.ok: ... # the run did not finish looking; run.reason says why

Invariants
  • Never raises for a server or SDK failure: the failure is the outcome.
  • max_concurrency defaults to 1: an offline grade drives one model or CLI at a time.

Rationale: ADR-0001.

ExperimentRun dataclass

ExperimentRun(
    outcome: Outcome,
    handed: int,
    graded: int = 0,
    reason: str | None = None,
    result: Any = None,
)

What run and record_rows return.

Attributes:

  • outcome (Outcome) –

    ok when every item was graded; invalid when the SDK dropped or left an item ungraded; unavailable when the run itself failed.

  • handed (int) –

    Items the run was handed.

  • graded (int) –

    Items that came back with at least one grade.

  • reason (str | None) –

    One line naming the failure; None on ok.

  • result (Any) –

    The SDK's ExperimentResult (its run URL, item results); None when the run failed.

ok property

ok: bool

True when outcome == "ok".

Seeded dataclass

Seeded(
    outcome: Outcome,
    handed: int,
    written: int = 0,
    ids: tuple[str, ...] = (),
    reason: str | None = None,
)

What seed returns.

Attributes:

  • outcome (Outcome) –

    ok when every item was written; invalid when some writes failed; unavailable when the dataset could be neither created nor read.

  • handed (int) –

    Items seed was handed.

  • written (int) –

    Items the store accepted.

  • ids (tuple[str, ...]) –

    The id of every item written, in input order.

  • reason (str | None) –

    One line naming the failure; None on ok.

ok property

ok: bool

True when outcome == "ok".

run

run(
    client: Any,
    *,
    run_name: str,
    items: Sequence[Any],
    task: Callable[..., Any],
    grade: Callable[..., Iterable[Grade]],
    metadata: Mapping[str, str] | None = None,
    max_concurrency: int = 1
) -> ExperimentRun

Run task over dataset items and book grade's scores on each item's trace.

Parameters:

  • client (Any) –

    A Langfuse SDK 5 client, built by the caller.

  • run_name (str) –

    The experiment and run name.

  • items (Sequence[Any]) –

    Dataset items (client.get_dataset(name).items), so each trace links to its item.

  • task (Callable[..., Any]) –

    The SDK task: called with item= (and other keywords), returns the item's output.

  • grade (Callable[..., Iterable[Grade]]) –

    Called with the SDK evaluator's keywords (output=, expected_output=, ...); returns the item's Grades.

  • metadata (Mapping[str, str] | None, default: None ) –

    Run metadata.

  • max_concurrency (int, default: 1 ) –

    Items in flight at once.

Returns:

  • ExperimentRun –

    The run, whose outcome is ok only when every handed item was graded. Never raises.

record_rows

record_rows(
    client: Any,
    *,
    dataset: str,
    run_name: str,
    rows: Sequence[Mapping[str, Any]],
    output: Callable[
        [Mapping[str, Any]], Mapping[str, Any]
    ],
    grade: Callable[[Mapping[str, Any]], Iterable[Grade]],
    metadata: Mapping[str, str] | None = None,
    id_field: str = "id"
) -> ExperimentRun

Record rows graded offline as one run on dataset.

Only items whose id matches a row's id_field are run; an item with no row is skipped, not counted.

Parameters:

  • client (Any) –

    A Langfuse SDK 5 client.

  • dataset (str) –

    The dataset's name.

  • run_name (str) –

    The run name.

  • rows (Sequence[Mapping[str, Any]]) –

    The graded rows, each carrying id_field.

  • output (Callable[[Mapping[str, Any]], Mapping[str, Any]]) –

    A row to the public output recorded on its trace.

  • grade (Callable[[Mapping[str, Any]], Iterable[Grade]]) –

    A row to its Grades.

  • metadata (Mapping[str, str] | None, default: None ) –

    Run metadata.

  • id_field (str, default: 'id' ) –

    The row key that matches a dataset item's id.

Returns:

  • ExperimentRun –

    The run (see run); unavailable when the dataset could not be read. Never raises.

input_id

input_id(value: Any) -> str

A stable id for a dataset item: the first 32 hex digits of the SHA-256 of its canonical JSON.

Parameters:

  • value (Any) –

    The item's input; any JSON-serialisable value (a non-JSON leaf is stringified).

Returns:

  • str –

    The id; equal inputs (key order aside) give equal ids. Never raises.

seed

seed(
    client: Any,
    *,
    dataset: str,
    items: Sequence[Mapping[str, Any]],
    input: Callable[[Mapping[str, Any]], Any],
    expected: (
        Callable[[Mapping[str, Any]], Any] | None
    ) = None,
    metadata: (
        Callable[[Mapping[str, Any]], Any] | None
    ) = None,
    item_id: (
        Callable[[Mapping[str, Any]], str] | None
    ) = None,
    description: str | None = None
) -> Seeded

Write items into dataset, creating the dataset when missing; a re-seed upserts.

Parameters:

  • client (Any) –

    A Langfuse SDK 5 client.

  • dataset (str) –

    The dataset's name.

  • items (Sequence[Mapping[str, Any]]) –

    The golden rows, in any shape the projections read.

  • input (Callable[[Mapping[str, Any]], Any]) –

    A row to the item's input.

  • expected (Callable[[Mapping[str, Any]], Any] | None, default: None ) –

    A row to the item's expected output; omitted, none is written.

  • metadata (Callable[[Mapping[str, Any]], Any] | None, default: None ) –

    A row to the item's metadata; omitted, none is written.

  • item_id (Callable[[Mapping[str, Any]], str] | None, default: None ) –

    A row to the item's id; omitted, input_id of the item's input, so the same input is the same item across seeds.

  • description (str | None, default: None ) –

    The dataset's description, written on creation.

Returns:

  • Seeded –

    ok with every id when every item was written; invalid naming the shortfall when some

  • Seeded –

    writes failed; unavailable when the dataset could be neither created nor read. Never

  • Seeded –

    raises.

Invariants
  • Idempotent: seeding the same rows twice leaves one item per id.
  • A failed write never stops the rest; the outcome counts it.

llm_ports.adapters.langfuse.labels

Labelled model replies as Langfuse scores, with a Langfuse annotation queue per role (extra [langfuse]).

LangfuseExampleStore is a learn.WritableExampleStore and an evals.labels.LabelQueue for topics llm:<role> (evals.labels.topic). An example is a score on the labelled generation, named by source and role:

  • label-<role>: a HUMAN label against a categorical score config of the same name (good / bad), given in Langfuse's annotation-queue UI or relayed by put from another surface that names the person (Example.by). A relayed label completes that reply's queue item, as the annotation UI does.
  • label.model-<role>: a stronger model's label, written by put.
  • label.judge-<role>: the production judge's verdict as a label, written by put.

The separator is - because Langfuse refuses a score-config name outside CONFIG_NAME (no :); reads also accept the earlier label:<role> spelling, so scores written under it stay readable. Every role gets one score config and one annotation queue, both named label-<role>, created on first enqueue. enqueue skips a reply the queue already holds (pending or done), and queued reads what it holds, so a nightly draw neither duplicates items nor grows the backlog unbounded. put writes each score under a deterministic id (source, role, subject, and the person for a human label), so a relabel replaces the earlier score instead of adding a second one. The SDK is synchronous; each call runs in a worker thread.

Example

store = LangfuseExampleStore(Langfuse(...)) # the caller builds the client await store.enqueue(topic("summarize"), [(observation_id, trace_id)]) read = await store.examples([topic("summarize")])

Invariants
  • Never raises for a server or SDK failure: the failure is the outcome.
  • A score this adapter cannot read as an example (another name, an unknown value, no observation subject) is skipped, never guessed.
  • A topic that is not llm:<role> holds no examples here and is refused by put/enqueue.
  • A role whose label name Langfuse would refuse (CONFIG_NAME) is refused here as a request_fault before any write, never sent to fail remotely.
  • A human label is written only with the person who gave it (by); an outcome label never.
  • A failure reason carries the server's own validation message when it sends one.

Rationale: ADR-0010, ADR-0011, ADR-0014.

LangfuseExampleStore

LangfuseExampleStore(
    client: Any,
    *,
    description: str = "Is this reply good? (labelled data)"
)

Labelled model replies stored as Langfuse scores, queued for humans in annotation queues.

Parameters:

  • client (Any) –

    A Langfuse SDK 5 client, built by the caller.

  • description (str, default: 'Is this reply good? (labelled data)' ) –

    The description given to each created score config and queue.

enqueue async

enqueue(
    topic: str, subjects: Sequence[tuple[str, str]]
) -> Written

Add each (observation id, trace id) the role's annotation queue does not hold yet.

Returns:

  • Written –

    How many were newly queued, or a failure outcome. Never raises.

queued async

queued(topic: str) -> Queued

Read the observation ids the role's annotation queue holds, pending and done. Never raises.

A role with no queue yet reads ok and empty; nothing is created by reading.

put async

put(examples: Sequence[Example]) -> Written

Write labels as categorical scores. Never raises.

A human label is one a person gave on another surface and code relays: it is written under label-<role> against the role's score config (created when missing), keyed by the person as well, and the role's queue item for that reply is then marked completed.

Returns:

  • Written –

    How many scores landed; request_fault (none written) when any example is an

  • Written –

    outcome label, a human label with no by, not on an llm:<role> topic, not

  • Written –

    good/bad, or has no ref["trace"]. unavailable when the server failed; a

  • Written –

    failure while completing a queue item comes after every score landed, so written

  • Written –

    then counts them all.

examples async

examples(
    topics: Sequence[str],
    *,
    since: datetime | None = None,
    source: Source | None = None
) -> ExamplesRead

Read every source's labels on the replies of topics. Never raises.

A topic that is not llm:<role> contributes nothing; an outcome source reads empty.

score_name

score_name(source: Labeller, role: str) -> str

Return the score name a source's label on a role reply is stored under.

parse_name

parse_name(name: str) -> tuple[Labeller, str] | None

Return (source, role) for a label score name in either spelling, or None for any other score.

example_of_score

example_of_score(d: Any) -> Example | None

Return a v3 score object as an Example, or None when it is not a label score this adapter reads.

llm_ports.adapters.optuna

Tune a combiner's hyperparameters with Optuna, walk-forward (extra [optuna]).

tune(build, events, space) splits the resolved events in time (forecast.walk.holdout_start): the tuner only ever sees the TRAIN events, those forecast before the holdout starts. Each trial builds a fresh combiner from its parameters (build(params)) and replays it over the train events with walk_forward; its objective is the mean Brier score over the resolved train events, with an abstention scored as 0.5 (Brier 0.25), so a configuration cannot win by answering less. The best parameters are then replayed over EVERY event, learning through the holdout as they would live, and scored on the holdout only: Tuned.holdout is out of sample.

The study goes to Optuna's storage (an RDB URL, an Optuna storage object, or None for memory) under study_name; a reused name adds trials to that study. The TPE sampler is seeded.

SqlJournal(engine, table) is an Optuna journal backend on one SQL table the caller owns (journal_storage(...) wraps it in Optuna's JournalStorage). Optuna's own RDB storage creates its tables wherever the connection's search path points and checks its schema through Alembic, which reads the default schema; behind a transaction-mode pooler (PgBouncer) a session-level search path does not hold, so the RDB storage cannot be pinned to one schema there. A journal is one append-only table: the caller creates it (a migration; create=True for tests and SQLite), names it schema-qualified, and every statement carries the name.

dashboard_app(storage) / serve_dashboard(storage) put optuna-dashboard over any of those storages, a SqlJournal included, on demand (extra [optuna-dashboard]): the stock dashboard reads only an RDB URL or a journal file, so a journal table needs this launcher.

Invariants
  • Never raises: too few events is Insufficient; a storage or Optuna failure is Unavailable; a trial whose build or replay raises is pruned as failed, not fatal.
  • The objective never reads a holdout event.

Rationale: ADR-0015.

Range dataclass

Range(low: float, high: float, log: bool = False)

One hyperparameter's search range.

Attributes:

  • low (float) –

    The smallest value tried.

  • high (float) –

    The largest value tried.

  • log (bool) –

    Sample on a log scale (a learning rate); low must then be > 0.

Tuned dataclass

Tuned(
    params: dict[str, float],
    study_name: str,
    trials: int,
    failed_trials: int,
    holdout_start: datetime,
    n_train: int,
    train_brier: float,
    holdout: Score,
)

The best parameters a study found, and how they scored out of sample.

Attributes:

  • params (dict[str, float]) –

    name -> value of the best trial.

  • study_name (str) –

    The Optuna study the trials went to.

  • trials (int) –

    Trials completed in this call.

  • failed_trials (int) –

    Trials whose build or replay raised in this call.

  • holdout_start (datetime) –

    The first holdout event's forecast time.

  • n_train (int) –

    Resolved train events the objective scored.

  • train_brier (float) –

    The best trial's objective (mean Brier over the train events).

  • holdout (Score) –

    The best parameters' Score on the resolved holdout events they answered.

Unavailable dataclass

Unavailable(reason: str)

The study could not run (storage unreachable, every trial failed, an Optuna error).

Attributes:

  • reason (str) –

    The exception's type and message, or why no trial completed.

SqlJournal

SqlJournal(
    engine: Engine, table: str, *, create: bool = False
)

Bases: BaseJournalBackend

An Optuna journal backend on one SQL table (n bigint primary key, log text).

Log n is the journal's 0-based log number. An append reads the next number and inserts the whole batch in one transaction; the primary key makes a concurrent append that read the same number fail, and it retries (APPEND_RETRIES) from the new end, so the journal stays gapless and in order across processes. Any SQLAlchemy dialect (PostgreSQL behind a pooler, SQLite).

Parameters:

  • engine (Engine) –

    A synchronous SQLAlchemy engine.

  • table (str) –

    The table, name or schema.name.

  • create (bool, default: False ) –

    Create the table if it is missing (tests, SQLite); a production table comes from the caller's migration.

Raises:

  • ValueError –

    table is not one or two plain identifiers.

read_logs

read_logs(log_number_from: int) -> Iterable[dict[str, Any]]

Every log numbered log_number_from or later, in order.

append_logs

append_logs(logs: list[dict[str, Any]]) -> None

Append logs after the current end, atomically.

objective_brier

objective_brier(
    preds: Sequence[float | None], events: Sequence[Event]
) -> float | None

Return the mean Brier score over the resolved events, an abstention scored as ABSTAIN_P; None when none resolved.

tune

tune(
    build: Callable[[dict[str, float]], Combiner],
    events: Sequence[Event],
    space: Mapping[str, Range],
    *,
    study_name: str,
    storage: Any = None,
    n_trials: int = 40,
    holdout: float = 0.2,
    seed: int = 0,
    min_train: int = MIN_TRAIN
) -> Tuned | Insufficient | Unavailable

Search space for the combiner parameters with the lowest walk-forward Brier score.

Parameters:

  • build (Callable[[dict[str, float]], Combiner]) –

    Makes a fresh combiner from one trial's name -> value.

  • events (Sequence[Event]) –

    Every event, resolved and open, in any order.

  • space (Mapping[str, Range]) –

    name -> Range to search.

  • study_name (str) –

    The Optuna study name; trials join an existing study of that name.

  • storage (Any, default: None ) –

    An Optuna storage (URL or object); None keeps the study in memory.

  • n_trials (int, default: 40 ) –

    Trials to run in this call.

  • holdout (float, default: 0.2 ) –

    The share of the resolved events held out, in (0, 1).

  • seed (int, default: 0 ) –

    The TPE sampler's seed.

  • min_train (int, default: MIN_TRAIN ) –

    Resolved train events below which nothing is searched.

Returns:

journal_storage

journal_storage(
    engine: Engine, table: str, *, create: bool = False
) -> Any

Optuna's JournalStorage over a SqlJournal; pass it to tune(storage=...).

dashboard_app

dashboard_app(storage: Any) -> Any

Return optuna-dashboard's WSGI app over storage (extra [optuna-dashboard]).

storage is anything Optuna opens: a journal_storage(...), an RDB URL, an Optuna storage object. The app reads the studies live; nothing is copied. Mount it under any WSGI server.

Raises:

  • ModuleNotFoundError –

    The optuna-dashboard extra is not installed (the message names it).

serve_dashboard

serve_dashboard(
    storage: Any,
    *,
    host: str = "127.0.0.1",
    port: int = 8080
) -> None

Serve optuna-dashboard over storage until interrupted (extra [optuna-dashboard]).

An on-demand study UI: run it while a person looks, stop it after. It binds host (loopback by default; reach it through a tunnel) and blocks.

Raises:

  • ModuleNotFoundError –

    The optuna-dashboard extra is not installed (the message names it).

llm_ports.adapters.otel

OpenTelemetry GenAI spans for model calls (extra [otel]).

OtelTracer is the Tracer port over OpenTelemetry: compose it like any other sink (TeeTracer(LangfuseTracer(client), OtelTracer())) and every call the library makes is a span. The two helpers below are for a call the library does not make.

Example

with gen_ai_span(tracer, operation="chat", model=m, provider="openai") as rec: reply = await client.call(...) rec.set_usage(reply.in_tok, reply.out_tok)

record_gen_ai(tracer, operation="embeddings", model=m, provider="openai", latency_ms=840)

A span records the shape of a model call (who, which model, how long, how many tokens) in the trace the rest of the request already lives in, named and attributed by the OpenTelemetry GenAI semantic conventions. Attribute keys come from opentelemetry-semantic-conventions' constants, so an upstream rename arrives with a version bump.

tracer is any OTel Tracer, or None when tracing is off; every function then does nothing, so a call site needs no conditional. attributes carries the caller's own dimensions under its own prefix, outside the reserved gen_ai.* namespace.

  • gen_ai_span opens a child of the ambient context (start_as_current_span), so the model call lands inside the existing job or request span. Only span creation is guarded: an exception from the caller's block propagates, and the SDK records it on the span.
  • record_gen_ai records a call after it is over, for a call that learns its latency and usage only then. The span is back-dated by the measured latency_ms, so its duration is the call's.
  • gen_ai.system is no longer emitted (1.24): the conventions removed it in v1.37.0 for gen_ai.provider.name. SYSTEM stays exported for 1.x imports; nothing sets it.
Invariants
  • Prompt and completion bodies are never carried, and no switch turns them on.
  • None is not zero: a provider that reported neither count gets the unmeasured_attr flag and no token attributes; a reported zero stays zero.

Rationale: ADR-0007, ADR-0025.

OtelGeneration

OtelGeneration(
    span: Any,
    meters: _Meters | None,
    base: Mapping[str, Any],
    clock: Callable[[], float],
    *,
    tracer: Any | None = None
)

The Generation an OtelTracer opens; ends its span once and never raises.

end

end(
    *,
    output: Any = None,
    usage: Mapping[str, int | None] | None = None,
    level: str | None = None,
    status_message: str | None = None,
    metadata: Mapping[str, Any] | None = None,
    usage_v2: Usage | None = None,
    cost: Cost | None = None,
    finish: Finish | None = None,
    answered: Answered | None = None,
    failure: Failure | None = None
) -> None

Set the result's attributes, record the metrics, end the span; output is never carried.

tool

tool(
    *,
    name: str,
    call_id: str,
    duration_s: float,
    is_error: bool
) -> None

Write one tool execution as an INTERNAL child span execute_tool {name}; never raises.

score

score(
    name: str,
    value: float,
    comment: str | None = None,
    metadata: Mapping[str, Any] | None = None,
) -> None

Do nothing: scores belong to the trace store (an ended span takes no new data).

OtelTracer

OtelTracer(
    tracer_provider: Any | None = None,
    meter_provider: Any | None = None,
    *,
    semconv: Literal["pinned"] = "pinned",
    clock: Callable[[], float] = monotonic
)

A Tracer that writes each generation as an OpenTelemetry span; the module docstring states the contract.

Parameters:

  • tracer_provider (Any | None, default: None ) –

    The provider spans come from; None takes the global one.

  • meter_provider (Any | None, default: None ) –

    When given, the duration histogram and token counters are recorded on it.

  • semconv (Literal['pinned'], default: 'pinned' ) –

    The attribute-name set; only "pinned" (observability.semconv) exists.

  • clock (Callable[[], float], default: monotonic ) –

    A monotonic clock for the duration metric, injectable for tests.

start_generation

start_generation(
    *,
    name: str,
    model: str,
    provider: str,
    input: Any,
    metadata: Mapping[str, Any],
    session_id: str | None,
    prompt_name: str | None
) -> Generation

Open a CLIENT span for one call; input is never carried. A failure to open yields a no-op.

GenAiRecorder

GenAiRecorder(
    span: Any, *, unmeasured_attr: str = UNMEASURED
)

Attaches a call's results to its span once the call returns.

A no-op over span=None. Never raises.

set_usage

set_usage(
    input_tokens: int | None, output_tokens: int | None
) -> None

Set the token counts as reported; when neither is reported, set the unmeasured flag, never zeros.

span_name

span_name(operation: str, model: str) -> str

Return the conventions' span name, {operation} {model}; the call's identity lives in attributes.

attributes_for

attributes_for(
    *,
    operation: str,
    model: str,
    provider: str,
    attributes: Mapping[str, Any] | None = None
) -> dict[str, Any]

Return the convention attributes plus the caller's own, with None values dropped. Pure.

gen_ai_span

gen_ai_span(
    tracer: Any | None,
    *,
    operation: str,
    model: str,
    provider: str,
    attributes: Mapping[str, Any] | None = None,
    unmeasured_attr: str = UNMEASURED
) -> Iterator[GenAiRecorder]

Open a child span around one live model call; the module docstring states the contract.

Yields:

  • GenAiRecorder –

    A GenAiRecorder over the span, or a no-op recorder when tracer is None or span

  • GenAiRecorder –

    creation failed.

record_gen_ai

record_gen_ai(
    tracer: Any | None,
    *,
    operation: str,
    model: str,
    provider: str,
    latency_ms: int,
    input_tokens: int | None = None,
    output_tokens: int | None = None,
    error: str | None = None,
    attributes: Mapping[str, Any] | None = None,
    unmeasured_attr: str = UNMEASURED
) -> None

Record a span for a call that already finished, back-dated by latency_ms; never raises.

A non-empty error sets the span status to ERROR, with the message cut to ERROR_MAX characters; without one, the status is left unset.

llm_ports.adapters.pluggy

Plugins found by entry point and loaded with pluggy (extra [market]): action packs and provider kinds.

Two entry-point groups share one loader:

  • llm_ports.market (market.ENTRY_POINT_GROUP): action packs. A module declares its actions on its own ActionRegistry and returns it from a market_actions hook; load_packs adds each pack to the scheduler's registry.
  • llm_ports.providers (call.providers.ENTRY_POINT_GROUP): provider kinds. A module calls register(factory) from a register_providers hook; load_providers returns the kinds.

    PACK = ActionRegistry("reports")

    @PACK.action(feature="report", action="weekly", cadence="7d", lane=3) async def weekly(session, work_key): ...

    @hookimpl def market_actions() -> ActionRegistry: return PACK

    pyproject.toml

    [project.entry-points."llm_ports.market"] reports = "reports_pkg.market"

The process calls each loader once, at its composition root, passing the value of its pin setting (PACKS_PIN_ENV, PROVIDERS_PIN_ENV by convention; the library reads no environment).

Invariants
  • The hook specs are added before any plugin registers, so pluggy refuses a hook implementation whose arguments the spec does not declare.
  • Built-ins register first, by explicit name, needing no distribution metadata; entry points follow, sorted by name. No order derives from a set or from install order.
  • The pin: None loads the built-ins and every installed plugin; "" the built-ins only; "a,b" the built-ins and exactly those, and a pinned name that is not installed raises.
  • Strict loading (the default) raises PluginLoadError (PackLoadError for packs) on any failure, naming the group, the plugin and the stage: import, register (pluggy refused it, or it contributed a wrong object), hook (its hook raised), duplicate (a key or a name another plugin holds; both are named), pin, pending (an unknown hook name, or no hook). load_packs(strict=False) records each failure as a PackLoad instead, for a diagnostic.
  • A registry is frozen after load: a strict load_packs that finishes freezes the ActionRegistry it loaded into, and load_providers returns its ProviderRegistry frozen, so a registration after the composition root raises.
  • Every load emits plugins_loaded (group, names, origins) once it finishes.
  • market_pack_failed carries stage in the PackLoad.stage vocabulary (PackStage) and plugin_stage in the PluginLoadError.stage vocabulary (PluginStage).

Rationale: ADR-0018 (packs), ADR-0022 (providers).

Loaded dataclass

Loaded(
    group: str,
    plugins: tuple[str, ...],
    origin: Mapping[str, str] = dict(),
)

Which plugins a process loaded into one registry, and from where.

Attributes:

  • group (str) –

    The entry-point group.

  • plugins (tuple[str, ...]) –

    Plugin names in load order: built-ins as given, then entry points sorted by name.

  • origin (Mapping[str, str]) –

    Plugin name to builtin, caller, or <distribution>==<version>.

PluginLoadError

PluginLoadError(
    message: str,
    *,
    group: str,
    plugin: str,
    stage: PluginStage
)

Bases: ImportError

A plugin that did not load; raised at the composition root, so the process does not start.

Attributes:

  • group (str) –

    The entry-point group.

  • plugin (str) –

    The plugin's name (comma-joined for several pinned-but-missing names).

  • stage (PluginStage) –

    Where it failed.

PackLoadError

PackLoadError(
    message: str,
    *,
    group: str,
    plugin: str,
    stage: PluginStage
)

Bases: PluginLoadError

An action pack that did not load (load_packs, strict).

PackSpec

The hooks a pack module may implement.

market_actions

market_actions() -> ActionRegistry

Return the pack's actions.

ProviderHooks

The hooks a provider plugin may implement.

register_providers

register_providers(
    register: Callable[[ProviderFactory], None],
) -> None

Call register(factory) once per provider kind the plugin contributes.

PackLoad dataclass

PackLoad(
    name: str,
    dist: str | None,
    outcome: PackOutcome,
    actions: tuple[str, ...] = (),
    stage: PackStage | None = None,
    reason: str | None = None,
    origin: str | None = None,
)

One pack, loaded or not.

Attributes:

  • name (str) –

    The entry point's name (the pack's name, and its pluggy plugin name).

  • dist (str | None) –

    The distribution that declares it, or None when unknown (and for a built-in).

  • outcome (PackOutcome) –

    ok, failed or blocked.

  • actions (tuple[str, ...]) –

    The feature.action names the pack added; empty unless ok.

  • stage (PackStage | None) –

    Where a failed pack stopped; None unless failed.

  • reason (str | None) –

    Why it failed; None unless failed.

  • origin (str | None) –

    builtin, <dist>==<version>, or None when unknown.

load_packs

load_packs(
    registry: ActionRegistry,
    *,
    group: str = ENTRY_POINT_GROUP,
    block: Iterable[str] = (),
    entry_points: EntryPoints | None = None,
    on_event: EventHook | None = None,
    builtins: Sequence[tuple[str, object]] = (),
    pin: str | None = None,
    strict: bool = True
) -> tuple[PackLoad, ...]

Load the built-in packs, then every pack listed under group, into registry.

Parameters:

  • registry (ActionRegistry) –

    The scheduler's registry; each pack joins it through include. Under strict, a load that finishes freezes it (ActionRegistry.freeze), so a declaration after the composition root raises; declare the caller's own actions first, or pass them as builtins. A diagnostic load (strict=False) leaves it open.

  • group (str, default: ENTRY_POINT_GROUP ) –

    The entry-point group to read.

  • block (Iterable[str], default: () ) –

    Pack names to skip (pluggy's blocked set); for tests. Production pins instead.

  • entry_points (EntryPoints | None, default: None ) –

    The entry points to load instead of the installed ones (or a callable that returns them).

  • on_event (EventHook | None, default: None ) –

    Receives market_pack_loaded, market_pack_failed (stage as PackLoad.stage, plugin_stage as PluginLoadError.stage), market_pack_blocked and plugins_loaded; without one, the stdlib logger llm_ports.market.

  • builtins (Sequence[tuple[str, object]], default: () ) –

    (name, module) packs that ship with the caller, registered first, in order.

  • pin (str | None, default: None ) –

    The pin setting's value; None when it is unset (see the module docstring).

  • strict (bool, default: True ) –

    Raise on the first failure. False records each failure and loads the rest.

Returns:

  • tuple[PackLoad, ...] –

    One PackLoad per pack, in load order.

Raises:

  • PackLoadError –

    Under strict, the first pack that failed, naming its stage.

  • RuntimeError –

    registry is already frozen; nothing is loaded.

packs_loaded

packs_loaded(
    loads: Iterable[PackLoad],
    *,
    group: str = ENTRY_POINT_GROUP
) -> Loaded

The Loaded record of the packs that joined: names in load order, with their origins.

failed

failed(loads: Iterable[PackLoad]) -> tuple[PackLoad, ...]

The loads that failed (a blocked pack is a decision, not a failure).

builtin

builtin(
    name: str, *factories: ProviderFactory
) -> tuple[str, object]

Return a built-in provider plugin that registers factories, for load_providers(builtins=...).

Parameters:

  • name (str) –

    The plugin's name.

  • factories (ProviderFactory, default: () ) –

    The kinds it contributes.

load_providers

load_providers(
    *,
    builtins: Sequence[tuple[str, object]] = (),
    pin: str | None = None,
    entry_points: EntryPoints | None = None,
    on_event: EventHook | None = None
) -> ProviderRegistry

Load the built-in provider plugins, then every one listed under llm_ports.providers.

Parameters:

  • builtins (Sequence[tuple[str, object]], default: () ) –

    (name, plugin) pairs registered first, in order (builtin(...) makes one).

  • pin (str | None, default: None ) –

    The pin setting's value; None when it is unset (see the module docstring).

  • entry_points (EntryPoints | None, default: None ) –

    The entry points to load instead of the installed ones (or a callable that returns them).

  • on_event (EventHook | None, default: None ) –

    Receives plugins_loaded; without one, the stdlib logger llm_ports.call.

Returns:

  • ProviderRegistry –

    The kinds, frozen, with loaded naming each plugin and its origin.

Raises:

  • PluginLoadError –

    The first plugin that failed, naming its stage; a kind registered by two plugins is duplicate and names both.

llm_ports.adapters.pydantic_evals

The Judge port implemented on pydantic-evals' LLMJudge.

pydantic-evals' LLMJudge renders the Input/Output/Rubric prompt and binds the verdict to its GradingOutput grammar (reason, pass, score); this adapter picks the model and translates the verdict into a JudgeVerdict. Install with the evals extra (llm-ports[evals]).

The model source is the caller's, as for the generator and the embedder:

  • model=: a fixed pydantic-ai Model (tests pass a FunctionModel).
  • model_for=: () -> async context manager yielding a Model, called per request, so the judge's own call goes through the caller's transport (admission control, leases, a work queue) like every other call. Build it with profile=JUDGE_PROFILE.

The grade uses native structured output. pydantic-evals' agent asks for GradingOutput as a tool call by default; JUDGE_PROFILE puts the judge on NativeOutput, a strict json_schema response_format, which is the mode a grammar-compiling server enforces. verdict_of refuses a reason that still carries tool-call markup, or a leaked verdict that contradicts the parsed one, so the caller records that the judge could not run, and why.

A call that sent images is judged with them (images=, beside the port's text-only signature), as pydantic-evals multimodal Input.

Invariants
  • PydanticEvalsJudge.judge raises when the judging model could not answer, as the port requires; the caller decides what an unjudged sample means.
  • No default verdict exists anywhere in this module.

Rationale: ADR-0001.

PydanticEvalsJudge

PydanticEvalsJudge(
    role: Role,
    *,
    model: Model | None = None,
    model_for: JudgeModelFor | None = None
)

The Judge port on pydantic-evals; the module docstring states the contract.

role carries the judge's model name and sampling (Role.model_settings()). Exactly one of model and model_for is required.

Raises:

  • ValueError –

    When neither or both of model and model_for are given.

model_name property

model_name: str

The judge's resolved model name, from its role.

judge async

judge(
    *,
    prompt: str,
    output: Any,
    rubric: str,
    images: Sequence[tuple[bytes, str]] = ()
) -> JudgeVerdict

Grade output against rubric, with prompt and any images as the judged input.

A failure of the judging model's call propagates: the caller decides what an unjudged sample means.

Raises:

  • ValueError –

    When the reply lacks a score or a verdict, or its reason is unreadable (verdict_of).

Rationale: ADR-0001.

clean_reason

clean_reason(reason: str, passed: bool) -> str

Return the reason cut where the grade's own JSON leaked into it.

Pure.

Parameters:

  • reason (str) –

    The judge's reason text.

  • passed (bool) –

    The parsed verdict, checked against any verdict the leaked tail states.

Raises:

  • ValueError –

    When the reason carries tool-call markup (its verdict is unreadable), or its leaked tail states a verdict that contradicts passed.

verdict_of

verdict_of(out: dict[str, Any]) -> JudgeVerdict

Translate an LLMJudge.evaluate result into the port's JudgeVerdict.

out holds score and pass, each with the model's reason. Pure.

Raises:

  • ValueError –

    When either key is missing (the judge did not answer the question), or when clean_reason refuses the reason.

judge_inputs

judge_inputs(
    prompt: str, images: Sequence[tuple[bytes, str]] = ()
) -> str | list[Any]

Build the LLMJudge Input: the text alone, or the text then each image as BinaryContent.

pydantic-evals renders a sequence Input part by part, an image as multimodal content.

llm_ports.adapters.sqlglot

A read-only AST guard for SQL an agent wrote (extra [sql], sqlglot).

The guard is one layer of a defense stack, not the stack: run its result as a role that cannot write, in a READ ONLY transaction with a statement timeout. It parses with sqlglot, validates the tree, and regenerates the SQL from the tree. The returned string is the only thing to execute, so comments and formatting tricks do not survive the parser.

A frozen SqlGuardPolicy holds the rules' data; the consumer owns its schema allowlist.

Invariants
  • An admitted statement is exactly one statement, and it is a SELECT (UNION, CTE and subquery are fine).
  • No DML, DDL, transaction or utility node appears anywhere in the tree (FORBIDDEN_NODES), and there is no SELECT ... INTO.
  • Every real table is schema-qualified and inside allowed_schemas. CTE names are exempt, so the system catalogs are denied unless a policy names them.
  • No function in deny_functions is called (POSTGRES_DENY_FUNCTIONS: sleep, file, backend-signal, advisory-lock, dblink and xml-dump functions).
  • LIMIT is at most max_rows: injected when absent, lowered when larger. A non-integer LIMIT is refused.
  • A policy may tighten the structural rules but never loosen them: forbidden_nodes keeps every type in FORBIDDEN_NODES, and single_statement stays True. Both are refused at construction, so no instance widens what the guard admits.
  • The guard fails closed: anything unparseable raises.
Example

POLICY = SqlGuardPolicy(allowed_schemas=("reporting",), deny_functions=POSTGRES_DENY_FUNCTIONS, max_rows=200) safe_sql = guard(agent_sql, POLICY) # raises SqlGuardError with a reason the agent can act on

SqlGuardError

Bases: ValueError

The statement was refused; the message is a reason the agent can act on.

SqlGuardPolicy dataclass

SqlGuardPolicy(
    allowed_schemas: tuple[str, ...],
    deny_functions: frozenset[str],
    max_rows: int,
    forbidden_nodes: tuple[
        type[Expression], ...
    ] = FORBIDDEN_NODES,
    single_statement: bool = True,
    dialect: str = "postgres",
)

What the guard admits.

Attributes:

  • allowed_schemas (tuple[str, ...]) –

    The queryable schemas, lower-case. An ordered tuple, so the error text is stable.

  • deny_functions (frozenset[str]) –

    Function names refused anywhere in the tree, lower-case.

  • max_rows (int) –

    The largest LIMIT an admitted statement carries; at least 1.

  • forbidden_nodes (tuple[type[Expression], ...]) –

    Node types refused anywhere in the tree; a superset of FORBIDDEN_NODES.

  • single_statement (bool) –

    Always True.

  • dialect (str) –

    The sqlglot dialect that parses and regenerates the statement.

Raises:

  • ValueError –

    At construction, for a policy that loosens the structural rules, a max_rows below 1, or a schema or function name that is not lower-case.

schemas_text property

schemas_text: str

The allowed schemas as one comma-separated string, for error text.

guard

guard(sql: str, policy: SqlGuardPolicy) -> str

Validate sql under policy and return the regenerated statement to execute.

Parameters:

  • sql (str) –

    The statement the agent wrote.

  • policy (SqlGuardPolicy) –

    The rules to enforce.

Returns:

  • str –

    The statement regenerated from its parse tree in policy.dialect, comments stripped, with

  • str –

    a LIMIT of at most policy.max_rows.

Raises:

  • SqlGuardError –

    The statement is empty, unparseable, or breaks a rule; the message names the rule.

llm_ports.adapters.lettucedetect

LettuceDetect, a token-span hallucination detector, behind the Grounder port.

The detector returns the offending tokens of a claim with a confidence each, and reads a context of up to 4,096 tokens. It costs what a validator may, not what a hot path may.

The lettucedetect extra (llm-ports[lettucedetect]: torch, transformers and the model, hundreds of MB) is imported lazily, and the detector is injectable, so the port's shape is tested anywhere with a fake (llm_ports.testing.StubDetector) and the real model only where it is installed.

score is 1 minus the worst flagged span's confidence. It is this adapter's own scale, never compared with another checker's.

Invariants
  • score is None when any flagged span carries no confidence: unknown is never zero, and 1.0 would read as fully grounded.
  • A context longer than MAX_CONTEXT_CHARS (a character proxy for the model's 4,096-token window) is still judged, and the verdict's reason starts with WINDOW_REASON, so a reader can see the judgement rested on a truncated premise.

Detector

Bases: Protocol

The slice of lettucedetect.models.inference.HallucinationDetector this adapter uses.

predict

predict(
    *, context: list[str], answer: str, output_format: str
) -> list[dict[str, Any]]

Return the spans of answer that context does not support.

LettuceDetectGrounder

LettuceDetectGrounder(
    detector: Detector | None = None, *, model: str = MODEL
)

A Grounder over LettuceDetect token spans.

A claim is supported when the model flags no span of it as unsupported by the cited chunks' text.

Parameters:

  • detector (Detector | None, default: None ) –

    The detector to use; None loads the real model named by model.

  • model (str, default: MODEL ) –

    The checkpoint load_detector loads when detector is None.

Invariants
  • A claim with no spans is unsupported.

verdict

verdict(
    claim: Claim, spans: Sequence[Chunk]
) -> GroundingVerdict

Return whether the text of spans supports claim.

load_detector

load_detector(model: str = MODEL) -> Detector

Load the real LettuceDetect model.

Raises:

  • RuntimeError –

    lettucedetect is not installed. The message names the extra to install, so a caller never sees a bare ImportError.

llm_ports.adapters.scoringrules

Proper scores beyond the binary case, on scoringrules (extra [scoringrules]).

forecast.score scores a binary forecast with the standard library. A range or a level is not binary: a stated interval is scored by the interval score (width plus a penalty for each miss, scaled by 2 / alpha), and a stated distribution by the CRPS. Each function scores many forecasts at once and returns one score per forecast; lower is better. Names and argument order are scoringrules' own (obs first), so a reader of its documentation reads these unchanged.

Invariants
  • Never raises: inputs of unequal length, empty, or that scoringrules rejects give None.
  • A returned list is aligned with the inputs, and every score in it is finite.

Rationale: ADR-0013.

brier_score

brier_score(
    obs: Sequence[bool], fct: Sequence[float]
) -> list[float] | None

Return the Brier score of each binary forecast (a cross-check of forecast.score.brier_score).

Parameters:

  • obs (Sequence[bool]) –

    Whether each event happened.

  • fct (Sequence[float]) –

    The forecast probabilities, aligned with obs.

Returns:

  • list[float] | None –

    One score per forecast, or None for unusable input. Never raises.

interval_score

interval_score(
    obs: Sequence[float],
    lower: Sequence[float],
    upper: Sequence[float],
    alpha: float,
) -> list[float] | None

Return the interval score of each stated [lower, upper] range at level 1 - alpha.

Parameters:

  • obs (Sequence[float]) –

    What the world did.

  • lower (Sequence[float]) –

    Each range's lower end.

  • upper (Sequence[float]) –

    Each range's upper end.

  • alpha (float) –

    The range's stated miss rate, in (0, 1) (0.32 for a one-sigma range).

Returns:

  • list[float] | None –

    One score per range, or None for unusable input or an alpha outside (0, 1). Never raises.

crps_normal

crps_normal(
    obs: Sequence[float],
    mu: Sequence[float],
    sigma: Sequence[float],
) -> list[float] | None

Return the CRPS of each stated normal distribution N(mu, sigma) at what was observed.

Returns:

  • list[float] | None –

    One score per forecast, or None for unusable input or a sigma <= 0. Never raises.

llm_ports.adapters.sklearn

An isotonic Calibrator on scikit-learn (extra [sklearn]).

IsotonicCalibrator is a scikit-learn estimator: fit(X, y, sample_weight=None) runs sklearn.isotonic.IsotonicRegression and keeps its knots as a forecast.PiecewiseLinear in map_, so the fitted map is applied and stored without scikit-learn; predict_proba(X) reads it.

Invariants
  • Never raises across the port: too few pairs, one outcome only, unaligned inputs or a failed fit leave map_ None and the reason in insufficient_, and predict_proba returns None.
  • The fitted map is monotone non-decreasing and inside [0, 1].

Rationale: ADR-0013.

IsotonicCalibrator

IsotonicCalibrator(min_pairs: int = MIN_PAIRS)

Bases: Params

Fits a monotone calibration map by isotonic regression.

Parameters:

  • min_pairs (int, default: MIN_PAIRS ) –

    The floor of usable pairs; MIN_PAIRS by default.

Attributes:

  • min_pairs –

    As given.

  • map_ (PiecewiseLinear | None) –

    The fitted map, or None when the last fit produced none. Set by fit.

  • insufficient_ (Insufficient | None) –

    Why the last fit produced no map, or None when it did. Set by fit.

fit

fit(
    X: Sequence[float],
    y: Sequence[bool],
    sample_weight: Sequence[float] | None = None,
) -> Self

Fit the map.

Parameters:

  • X (Sequence[float]) –

    Stated probabilities.

  • y (Sequence[bool]) –

    Whether each event happened, aligned with X.

  • sample_weight (Sequence[float] | None, default: None ) –

    Optional weights aligned with X, to calibrate toward a reweighted population. A pair is dropped when its probability is invalid or its weight is not a finite real > 0. Inputs of different lengths are not aligned (a derived array that lost a row would misalign every pair after it), so nothing is fitted.

Returns:

  • Self –

    The estimator, with map_ or insufficient_ set. Never raises.

predict_proba

predict_proba(
    X: Sequence[float],
) -> list[tuple[float, float]] | None

Return (P(no), P(yes)) per stated probability; None with no fitted map. Never raises.

llm_ports.adapters.venn_abers

A Venn-ABERS Calibrator (extra [venn-abers]), for voices with few resolved forecasts.

Venn-ABERS (Vovk, Petej and Fedorova) fits two isotonic regressions per test point, one assuming each outcome, and returns the pair (p0, p1) that brackets the calibrated probability; its validity guarantee holds at any sample size, so it is the calibrator for a small history. The width of (p0, p1) says how much the history can say. VennAbersCalibrator is a scikit-learn estimator: fit(X, y), predict_proba(X) (the merged probability) and predict_interval(X).

Invariants
  • Never raises across the port: too few pairs, one outcome only, or a failed fit leave model_ None and the reason in insufficient_, and both predictions return None; a failed prediction for one point returns its stated probability unchanged.
  • Every returned probability is in [0, 1].

Rationale: ADR-0013.

VennAbersCalibrator

VennAbersCalibrator(min_pairs: int = MIN_PAIRS)

Bases: Params

Fits a Venn-ABERS map.

Parameters:

  • min_pairs (int, default: MIN_PAIRS ) –

    The floor of usable pairs; MIN_PAIRS by default.

Attributes:

  • min_pairs –

    As given.

  • model_ (Any) –

    The fitted venn_abers.VennAbers, or None when the last fit produced none.

  • n_ (int) –

    The pairs the model was fitted on (0 with no model).

  • insufficient_ (Insufficient | None) –

    Why the last fit produced no model, or None when it did.

fit

fit(X: Sequence[float], y: Sequence[bool]) -> Self

Fit the map.

Parameters:

  • X (Sequence[float]) –

    Stated probabilities.

  • y (Sequence[bool]) –

    Whether each event happened, aligned with X.

Returns:

  • Self –

    The estimator, with model_ or insufficient_ set. Never raises.

predict_interval

predict_interval(
    X: Sequence[float],
) -> list[tuple[float, float]] | None

Return (p0, p1) per stated probability, the bracket around its calibrated value.

Returns:

  • list[tuple[float, float]] | None –

    The brackets, None with no fitted model. A point with an invalid probability gets

  • list[tuple[float, float]] | None –

    (0.5, 0.5), a failed prediction (p, p). Never raises.

predict_proba

predict_proba(
    X: Sequence[float],
) -> list[tuple[float, float]] | None

Return (P(no), P(yes)) per stated probability, P(yes) = p1 / (1 - p0 + p1).

Returns:

  • list[tuple[float, float]] | None –

    The rows, None with no fitted model. Never raises.

llm_ports.adapters.river

A learned Combiner on River's online logistic regression (extra [river]).

RiverLogitPool regresses the outcome on each voice's log-odds, one feature per voice, plus a silent flag per voice it has heard before and that did not speak this time: a voice's silence can carry information. It learns one event at a time, so its weights follow drift without a nightly refit. Until it has heard warmup outcomes it does not combine ({}), so a caller falls back to its baseline rather than read an untrained model. It speaks the Combiner protocol, River's own: predict_proba_one(x) and learn_one(x, y), with x as voice -> probability.

Invariants
  • Never raises across the port: a River failure leaves the state unchanged (learn_one) or returns {} (predict_proba_one).
  • A combined probability is in [0, 1].

Rationale: ADR-0013.

RiverLogitPool

RiverLogitPool(
    lr: float = 0.05, l2: float = 0.0, warmup: int = WARMUP
)

An online logistic logit pool.

Parameters:

  • lr (float, default: 0.05 ) –

    The SGD learning rate.

  • l2 (float, default: 0.0 ) –

    The L2 penalty on the weights.

  • warmup (int, default: WARMUP ) –

    Outcomes heard before predict_proba_one answers; WARMUP by default.

Attributes:

  • seen (int) –

    Outcomes heard so far.

  • failures (int) –

    Updates River refused (state left unchanged); a rising count means the pool is not learning.

  • voices (set[str]) –

    Every voice heard so far.

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return {False: P(no), True: P(yes)}; {} before warmup or with no usable voice. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Update the weights on one resolved event; an event with no usable voice is ignored. Never raises.

weights

weights() -> dict[str, float]

Return the current feature weights (voice:<v>, silent:<v>) for a readout.

llm_ports.adapters.mlflow

The learning loop's run ledger and model registry on MLflow (extra [mlflow]).

MlflowRunLedger(client) records each RunLedger.append as one MLflow run, so loops land in MLflow's tracking UI and API next to every other experiment:

  • experiment: prefix + kind (created on first append; artifact_location sets where its runs' artifacts go).
  • tags: llm_ports.kind, llm_ports.subject, llm_ports.decision (the recorded verdict's decision, absent on an ungated run) and mlflow.runName.
  • params / metrics: every scalar param and every finite numeric metric whose name MLflow accepts. A verdict, a model or a string metric is not a number MLflow can chart, so the whole row is also JSON in the run's tags (ROW_TAG/000, ROW_TAG/001, ... of at most 8000 characters each, then ROW_PARTS_TAG = the count, written last), which latest reads back from the search result itself: no artifact store, no download. A row over MAX_ROW_JSON characters, or a subject over 8000 (MLflow would cut its tag short), is refused. latest matches the row's own kind, so two prefixes whose experiment names collide (a + bc, ab + c) never cross-read.
  • Written.ref / LedgerRow.ref: the MLflow run id.

A run ends FINISHED once every part is written, FAILED when a write fails part-way; latest reads FINISHED runs whose row is complete, so a half-written or foreign run never reads as a row. Runs end through update_run (MLflow's set_terminated prints a URL to stdout on a tracking server), so their end_time is unset. Runs are ordered by start time in milliseconds, stamped strictly increasing per ledger object; two ledger objects writing one kind in the same millisecond have no defined order.

MlflowModelRegistry(client) maps ModelRegistry onto MLflow's model registry: a registered model per name, a model version per register, aliases as MLflow aliases. A version's source is the artifact_uri when one is given (a tracking server refuses a local path as a source, so use a remote URI such as s3://... there, or runs:/<id>/..., whose run id is passed along as MLflow's server requires); with none it is a backing run in the experiment registry_experiment (runs:/<id>/llm_ports), since MLflow requires a source. The fitted model is JSON in the version tag llm_ports.model (at most MAX_MODEL_JSON characters; a larger model belongs at an artifact_uri), run_ref in llm_ports.run_ref. register does not wait for the version to turn READY (MLflow's default waits and logs while it does); a backend that copies the artifact first, Databricks for one, reports it PENDING_REGISTRATION until then.

The client is the caller's (mlflow.MlflowClient(tracking_uri, registry_uri)): a tracking server, Databricks, or a local sqlite:/// file. MLflow's client is synchronous; every call runs in a worker thread, and one ledger or registry object is safe to share between threads. Against an unreachable server MLflow retries each call before failing (MLFLOW_HTTP_REQUEST_MAX_RETRIES, MLFLOW_HTTP_REQUEST_BACKOFF_FACTOR, MLFLOW_HTTP_REQUEST_TIMEOUT; minutes by default), so a loop that must report unavailable promptly sets those lower in its environment.

Invariants
  • Never raises: MLflow refusing a value (INVALID_PARAMETER_VALUE) is a request_fault; any other failure, a server down included, is unavailable. A param or metric MLflow would refuse (its name, an unconvertible number) is left out of MLflow's columns, never a failed write; it stays in the row.
  • Its calls write nothing to stdout or stderr.
  • Passes llm_ports.testing.conformance (check_run_ledger, check_model_registry and their broken-storage checks) against a tracking server and a local SQLite store.

Rationale: ADR-0016.

llm_ports.adapters.mlflow.ledger

MlflowRunLedger: the learning loop's run ledger on MLflow tracking (see the package docstring).

MlflowRunLedger

MlflowRunLedger(
    client: MlflowClient,
    *,
    prefix: str = "",
    artifact_location: str | None = None,
    clock: Callable[[], float] = time
)

A RunLedger over MLflow tracking: one run per row (see the module docstring).

Record runs through client.

Parameters:

  • client (MlflowClient) –

    The caller's MLflow client.

  • prefix (str, default: '' ) –

    Prepended to a kind to name its experiment.

  • artifact_location (str | None, default: None ) –

    Where a new experiment's artifacts go; None for the server's default.

  • clock (Callable[[], float], default: time ) –

    Wall-clock seconds, for each run's start time; injectable for tests.

append async

append(
    kind: str,
    *,
    subject: str,
    params: Mapping[str, Any],
    metrics: Mapping[str, Any]
) -> Written

Record one run as an MLflow run. Never raises.

latest async

latest(
    kind: str,
    *,
    subject: str | None = None,
    decisions: Sequence[Decision] | None = None
) -> RowRead

Read the newest matching FINISHED run. Never raises; an unknown kind has no rows.

llm_ports.adapters.mlflow.registry

MlflowModelRegistry: the model registry port on MLflow's registry (see the package docstring).

MlflowModelRegistry

MlflowModelRegistry(
    client: MlflowClient,
    *,
    registry_experiment: str = "llm_ports.registry"
)

A ModelRegistry over MLflow's model registry (see the module docstring).

Register models through client (its registry URI).

Parameters:

  • client (MlflowClient) –

    The caller's MLflow client.

  • registry_experiment (str, default: 'llm_ports.registry' ) –

    The experiment holding the backing run of a version registered with no artifact_uri.

register async

register(
    name: str,
    *,
    model: Mapping[str, Any],
    artifact_uri: str | None = None,
    run_ref: str | None = None,
    tags: Mapping[str, str] | None = None
) -> Written

Create an MLflow model version. Never raises.

set_alias async

set_alias(name: str, alias: str, version: int) -> Written

Set an MLflow alias. Never raises; MLflow refuses latest and v<N> as aliases.

by_alias async

by_alias(name: str, alias: str) -> VersionRead

Read the version an MLflow alias points at. Never raises.

get async

get(name: str, version: int) -> VersionRead

Read one MLflow model version. Never raises.