Skip to content

llm_ports.observability

llm_ports.observability

Observation of port calls.

observe, record, the fan-out TeeTracer, the null sink, event hooks, the caller's context variables, and the pinned OpenTelemetry names (observability.semconv).

NoopGeneration

A Generation that records nothing.

It is what a sink-less Tracer returns, and so what call.generate observes into when no trace hook is given.

end

end(**_kw: Any) -> None

Record nothing.

score

score(*_a: Any, **_kw: Any) -> None

Record nothing.

NoopTracer

A Tracer for a process with no observability sink.

Every start_generation returns a NoopGeneration, so a producer never branches on whether a sink is configured.

start_generation

start_generation(**_kw: Any) -> NoopGeneration

Return a NoopGeneration.

Observation

Bases: Protocol

What a trace hook yields: one open observation of one call.

Invariants
  • Neither method raises.

update

update(
    *,
    output: Any = None,
    usage_details: Mapping[str, Any] | None = None,
    metadata: Mapping[str, Any] | None = None
) -> None

Record what the call produced so far; a None argument leaves that field as it was.

score

score(
    *,
    name: str,
    value: float,
    comment: str | None = None,
    metadata: Mapping[str, Any] | None = None
) -> None

Attach a numeric score to the observation, with metadata stored on the score.

PortObservation

PortObservation(gen: Generation)

An Observation over a Generation.

Holds the last update (and the call's result, once attached) and ends the generation once, through flush, fail or cancel. Never raises.

metadata property

metadata: Mapping[str, Any] | None

The last update's metadata; None when no update carried any.

update

update(
    *,
    output: Any = None,
    usage_details: Mapping[str, Any] | None = None,
    metadata: Mapping[str, Any] | None = None
) -> None

Hold the call's output, usage and metadata; a None argument keeps the held value.

Parameters:

  • output (Any, default: None ) –

    What the call produced.

  • usage_details (Mapping[str, Any] | None, default: None ) –

    The provider's {"input": ..., "output": ...} counts; each side is read through opt_int.

  • metadata (Mapping[str, Any] | None, default: None ) –

    Replaces the held metadata.

attach

attach(result: Generated[Any]) -> None

Hold the call's result, so the end carries its usage, cost, finish, who answered and failure.

score

score(
    *,
    name: str,
    value: float,
    comment: str | None = None,
    metadata: Mapping[str, Any] | None = None
) -> None

Attach a numeric score to the underlying generation.

usage_counts

usage_counts() -> tuple[int | None, int | None]

Return (input, output) exactly as reported; None where the provider said nothing.

tool

tool(
    *,
    name: str,
    call_id: str,
    duration_s: float,
    is_error: bool
) -> None

Record one tool execution under this generation (record_tool). Never raises.

flush

flush() -> None

End the generation with what was held (the clean exit).

fail

fail(
    exc: BaseException,
    metadata: Mapping[str, Any] | None = None,
) -> None

End the generation at level ERROR with the error line, cut to STATUS_MAX characters.

Parameters:

  • exc (BaseException) –

    The exception that ended the call.

  • metadata (Mapping[str, Any] | None, default: None ) –

    The metadata to end with; None uses the last update's.

cancel

cancel(
    exc: BaseException,
    metadata: Mapping[str, Any] | None = None,
) -> None

End the generation at level CANCELLED: the caller cancelled or interrupted the call.

TeeGeneration

TeeGeneration(
    tee: TeeTracer, opened: list[tuple[_Sink, Generation]]
)

The Generation a TeeTracer opens: every end and score reaches every sink's generation.

end

end(**kw: Any) -> None

End every sink's generation with the same fields; never raises.

tool

tool(
    *,
    name: str,
    call_id: str,
    duration_s: float,
    is_error: bool
) -> None

Hand one tool execution to every sink's generation that records tools; never raises.

score

score(
    name: str,
    value: float,
    comment: str | None = None,
    metadata: Mapping[str, Any] | None = None,
) -> None

Score every sink's generation; never raises.

TeeTracer

TeeTracer(
    *tracers: Tracer,
    on_event: EventHook | None = None,
    hold_s: float = 300.0,
    clock: Callable[[], float] = monotonic
)

A Tracer over several sinks; the module docstring states the contract.

Parameters:

  • tracers (Tracer, default: () ) –

    The sinks, in the order each call reaches them.

  • on_event (EventHook | None, default: None ) –

    Receives tracer_sink_failed; without one, the stdlib logger llm_ports.observability.

  • hold_s (float, default: 300.0 ) –

    The fewest seconds between two tracer_sink_failed events from one sink.

  • clock (Callable[[], float], default: monotonic ) –

    A monotonic clock, injectable for tests.

stats

stats() -> Mapping[str, Mapping[str, int]]

Per sink ("<index>:<class>"): generations started, ended, and calls that failed.

start_generation

start_generation(
    *,
    name: str,
    model: str,
    provider: str,
    input: Any,
    metadata: Mapping[str, Any],
    session_id: str | None,
    prompt_name: str | None
) -> Generation

Open the generation on every sink; a sink that raises gets a NoopGeneration.

current_feature

current_feature() -> str | None

Return the active category, or None outside any llm_feature block.

current_source

current_source() -> str

Return the active traffic source: LIVE_SOURCE unless an llm_source block says otherwise.

Never None, so a metric label is always set.

llm_feature

llm_feature(name: str) -> Iterator[None]

Tag every LLM generation emitted within this block with the category name.

Propagates through async/await and nests (the inner block wins). Costs nothing when tracing is disabled.

Safe around async generators. ContextVar.reset(token) raises ValueError when the token was created in a different Context, which happens when the block wraps a body that yields and the generator is closed or garbage-collected outside the task that entered it (a streaming response that consumes an async generator, for example). On that error the variable is cleared instead, so a stale category never leaks into whatever runs next.

llm_source

llm_source(name: str) -> Iterator[None]

Tag every generation inside this block as traffic of name.

An eval harness sets e.g. "prompt_eval" around a golden replay, so its failed graders write no live gauge and a rate alert on the live series never reads a replay as a live failure. Teardown follows llm_feature: a token from another Context is cleared, not reset.

log_events

log_events(logger: str) -> EventHook

Return an EventHook that writes each event to the stdlib logger named logger.

The record is "<event> <fields>" at the event's level.

observe async

observe(
    tracer: Tracer,
    *,
    name: str,
    model: str,
    provider: str,
    input: Any,
    metadata: Mapping[str, Any] | None = None,
    session_id: str | None = None,
    prompt_name: str | None = None,
    clock: Callable[[], float] = monotonic
) -> AsyncIterator[PortObservation]

Open one generation around the block and end it once; see the module docstring.

On an exception, the end carries the opening metadata, overlaid by the last update's, plus latency_ms; the exception then propagates. A cancellation or interrupt ends it the same way at level CANCELLED.

Parameters:

  • tracer (Tracer) –

    The tracer that opens the generation.

  • name (str) –

    The category a cost view groups by.

  • model (str) –

    The model id.

  • provider (str) –

    The provider name.

  • input (Any) –

    What the model is sent.

  • metadata (Mapping[str, Any] | None, default: None ) –

    Opening metadata.

  • session_id (str | None, default: None ) –

    When set, groups the generation under that session.

  • prompt_name (str | None, default: None ) –

    When set, links the managed prompt of that name.

  • clock (Callable[[], float], default: monotonic ) –

    The monotonic clock the latency is measured on.

Yields:

opt_int

opt_int(v: Any) -> int | None

Return int(v), or None when the provider reported nothing readable.

Replaces int(v or 0), which reads "no usage block arrived" as "this call used zero tokens".

record

record(
    tracer: Tracer,
    *,
    name: str,
    model: str,
    provider: str,
    input: Any,
    output: Any,
    started_at: float,
    ended_at: float,
    usage: Usage | None = None,
    cost: Cost | None = None,
    finish: Finish | None = None,
    answered: Answered | None = None,
    failure: Failure | None = None,
    metadata: Mapping[str, Any] | None = None,
    session_id: str | None = None,
    prompt_name: str | None = None
) -> None

Record one finished generation in one call, for a model call the library did not make.

A subprocess, a vendor's agent loop: whatever made the call reports it here, and every sink receives the same fields generate sends. Never raises.

Parameters:

  • tracer (Tracer) –

    The tracer that records it.

  • name (str) –

    The category a cost view groups by.

  • model (str) –

    The model id requested.

  • provider (str) –

    The provider name.

  • input (Any) –

    What the model was sent.

  • output (Any) –

    What it produced.

  • started_at (float) –

    When the call started, in seconds on any clock ended_at shares.

  • ended_at (float) –

    When it ended. Before started_at, the latency is omitted.

  • usage (Usage | None, default: None ) –

    Its usage; None records it as unmeasured.

  • cost (Cost | None, default: None ) –

    What it cost; None when not priced.

  • finish (Finish | None, default: None ) –

    How its last reply ended.

  • answered (Answered | None, default: None ) –

    Who answered.

  • failure (Failure | None, default: None ) –

    Its failure; None when it succeeded (the level is then ERROR).

  • metadata (Mapping[str, Any] | None, default: None ) –

    Extra fields for the sink.

  • session_id (str | None, default: None ) –

    When set, groups the generation under that session.

  • prompt_name (str | None, default: None ) –

    When set, links the managed prompt of that name.

record_tool

record_tool(
    gen: Generation,
    *,
    name: str,
    call_id: str,
    duration_s: float,
    is_error: bool
) -> None

Hand one finished tool execution to gen, when its sink records tools. Never raises.

The seam is optional and duck-typed, so a 1.x Generation keeps working: a generation with a tool(name=, call_id=, duration_s=, is_error=) method receives it (OtelTracer's opens an INTERNAL child span execute_tool {name}; RecordingTracer's records it); any other ignores it, and the call's observation still carries the tool names, durations and error count in its metadata.

Rationale: ADR-0023, ADR-0025.

llm_ports.observability.semconv

Every OpenTelemetry attribute and metric name the library emits, spelled once.

The GenAI semantic convention is at stability development: it has no release, and it renames (cache_creation became cache_write, gen_ai.system became gen_ai.provider.name, the token usage histogram became per-kind counters). So no call site spells a name: each is a constant here, checked against the source commit SEMCONV_SOURCE. A rename upstream is a one-file diff plus a changelog line. A scheduled CI job (.github/workflows/semconv-drift.yml) diffs these constants against the convention's registry at HEAD and opens an issue on drift.

Names under llm_ports. are the library's own (the convention has no key for them).

Invariants
  • CONVENTION_ATTRIBUTES and LIBRARY_ATTRIBUTES are every attribute key an OtelTracer span may carry; a unit test pins the emitted key set to them.
  • No body attribute (gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions) is named here, so none can be emitted.

Rationale: ADR-0025.