llm_ports.observability¶
llm_ports.observability
¶
Observation of port calls.
observe, record, the fan-out TeeTracer, the null sink, event hooks, the caller's context
variables, and the pinned OpenTelemetry names (observability.semconv).
NoopGeneration
¶
NoopTracer
¶
A Tracer for a process with no observability sink.
Every start_generation returns a NoopGeneration, so a producer never branches on whether a
sink is configured.
Observation
¶
Bases: Protocol
What a trace hook yields: one open observation of one call.
Invariants
- Neither method raises.
update
¶
update(
*,
output: Any = None,
usage_details: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None
) -> None
Record what the call produced so far; a None argument leaves that field as it was.
score
¶
score(
*,
name: str,
value: float,
comment: str | None = None,
metadata: Mapping[str, Any] | None = None
) -> None
Attach a numeric score to the observation, with metadata stored on the score.
PortObservation
¶
PortObservation(gen: Generation)
An Observation over a Generation.
Holds the last update (and the call's result, once attached) and ends the generation once,
through flush, fail or cancel. Never raises.
metadata
property
¶
metadata: Mapping[str, Any] | None
The last update's metadata; None when no update carried any.
update
¶
update(
*,
output: Any = None,
usage_details: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None
) -> None
Hold the call's output, usage and metadata; a None argument keeps the held value.
Parameters:
-
output(Any, default:None) –What the call produced.
-
usage_details(Mapping[str, Any] | None, default:None) –The provider's
{"input": ..., "output": ...}counts; each side is read throughopt_int. -
metadata(Mapping[str, Any] | None, default:None) –Replaces the held metadata.
attach
¶
attach(result: Generated[Any]) -> None
Hold the call's result, so the end carries its usage, cost, finish, who answered and failure.
score
¶
score(
*,
name: str,
value: float,
comment: str | None = None,
metadata: Mapping[str, Any] | None = None
) -> None
Attach a numeric score to the underlying generation.
usage_counts
¶
usage_counts() -> tuple[int | None, int | None]
Return (input, output) exactly as reported; None where the provider said nothing.
tool
¶
tool(
*,
name: str,
call_id: str,
duration_s: float,
is_error: bool
) -> None
Record one tool execution under this generation (record_tool). Never raises.
fail
¶
fail(
exc: BaseException,
metadata: Mapping[str, Any] | None = None,
) -> None
End the generation at level ERROR with the error line, cut to STATUS_MAX characters.
Parameters:
-
exc(BaseException) –The exception that ended the call.
-
metadata(Mapping[str, Any] | None, default:None) –The metadata to end with; None uses the last update's.
cancel
¶
cancel(
exc: BaseException,
metadata: Mapping[str, Any] | None = None,
) -> None
End the generation at level CANCELLED: the caller cancelled or interrupted the call.
TeeGeneration
¶
TeeGeneration(
tee: TeeTracer, opened: list[tuple[_Sink, Generation]]
)
The Generation a TeeTracer opens: every end and score reaches every sink's generation.
tool
¶
tool(
*,
name: str,
call_id: str,
duration_s: float,
is_error: bool
) -> None
Hand one tool execution to every sink's generation that records tools; never raises.
score
¶
score(
name: str,
value: float,
comment: str | None = None,
metadata: Mapping[str, Any] | None = None,
) -> None
Score every sink's generation; never raises.
TeeTracer
¶
TeeTracer(
*tracers: Tracer,
on_event: EventHook | None = None,
hold_s: float = 300.0,
clock: Callable[[], float] = monotonic
)
A Tracer over several sinks; the module docstring states the contract.
Parameters:
-
tracers(Tracer, default:()) –The sinks, in the order each call reaches them.
-
on_event(EventHook | None, default:None) –Receives
tracer_sink_failed; without one, the stdlib loggerllm_ports.observability. -
hold_s(float, default:300.0) –The fewest seconds between two
tracer_sink_failedevents from one sink. -
clock(Callable[[], float], default:monotonic) –A monotonic clock, injectable for tests.
stats
¶
stats() -> Mapping[str, Mapping[str, int]]
Per sink ("<index>:<class>"): generations started, ended, and calls that failed.
start_generation
¶
start_generation(
*,
name: str,
model: str,
provider: str,
input: Any,
metadata: Mapping[str, Any],
session_id: str | None,
prompt_name: str | None
) -> Generation
Open the generation on every sink; a sink that raises gets a NoopGeneration.
current_feature
¶
current_feature() -> str | None
Return the active category, or None outside any llm_feature block.
current_source
¶
current_source() -> str
Return the active traffic source: LIVE_SOURCE unless an llm_source block says otherwise.
Never None, so a metric label is always set.
llm_feature
¶
llm_feature(name: str) -> Iterator[None]
Tag every LLM generation emitted within this block with the category name.
Propagates through async/await and nests (the inner block wins). Costs nothing when tracing is disabled.
Safe around async generators. ContextVar.reset(token) raises ValueError when the token was
created in a different Context, which happens when the block wraps a body that yields and
the generator is closed or garbage-collected outside the task that entered it (a streaming
response that consumes an async generator, for example). On that error the variable is cleared
instead, so a stale category never leaks into whatever runs next.
llm_source
¶
llm_source(name: str) -> Iterator[None]
Tag every generation inside this block as traffic of name.
An eval harness sets e.g. "prompt_eval" around a golden replay, so its failed graders write
no live gauge and a rate alert on the live series never reads a replay as a live failure.
Teardown follows llm_feature: a token from another Context is cleared, not reset.
log_events
¶
log_events(logger: str) -> EventHook
Return an EventHook that writes each event to the stdlib logger named logger.
The record is "<event> <fields>" at the event's level.
observe
async
¶
observe(
tracer: Tracer,
*,
name: str,
model: str,
provider: str,
input: Any,
metadata: Mapping[str, Any] | None = None,
session_id: str | None = None,
prompt_name: str | None = None,
clock: Callable[[], float] = monotonic
) -> AsyncIterator[PortObservation]
Open one generation around the block and end it once; see the module docstring.
On an exception, the end carries the opening metadata, overlaid by the last update's, plus
latency_ms; the exception then propagates. A cancellation or interrupt ends it the same way
at level CANCELLED.
Parameters:
-
tracer(Tracer) –The tracer that opens the generation.
-
name(str) –The category a cost view groups by.
-
model(str) –The model id.
-
provider(str) –The provider name.
-
input(Any) –What the model is sent.
-
metadata(Mapping[str, Any] | None, default:None) –Opening metadata.
-
session_id(str | None, default:None) –When set, groups the generation under that session.
-
prompt_name(str | None, default:None) –When set, links the managed prompt of that name.
-
clock(Callable[[], float], default:monotonic) –The monotonic clock the latency is measured on.
Yields:
-
AsyncIterator[PortObservation]–The open
PortObservation.
opt_int
¶
opt_int(v: Any) -> int | None
Return int(v), or None when the provider reported nothing readable.
Replaces int(v or 0), which reads "no usage block arrived" as "this call used zero tokens".
record
¶
record(
tracer: Tracer,
*,
name: str,
model: str,
provider: str,
input: Any,
output: Any,
started_at: float,
ended_at: float,
usage: Usage | None = None,
cost: Cost | None = None,
finish: Finish | None = None,
answered: Answered | None = None,
failure: Failure | None = None,
metadata: Mapping[str, Any] | None = None,
session_id: str | None = None,
prompt_name: str | None = None
) -> None
Record one finished generation in one call, for a model call the library did not make.
A subprocess, a vendor's agent loop: whatever made the call reports it here, and every sink
receives the same fields generate sends. Never raises.
Parameters:
-
tracer(Tracer) –The tracer that records it.
-
name(str) –The category a cost view groups by.
-
model(str) –The model id requested.
-
provider(str) –The provider name.
-
input(Any) –What the model was sent.
-
output(Any) –What it produced.
-
started_at(float) –When the call started, in seconds on any clock
ended_atshares. -
ended_at(float) –When it ended. Before
started_at, the latency is omitted. -
usage(Usage | None, default:None) –Its usage; None records it as unmeasured.
-
cost(Cost | None, default:None) –What it cost; None when not priced.
-
finish(Finish | None, default:None) –How its last reply ended.
-
answered(Answered | None, default:None) –Who answered.
-
failure(Failure | None, default:None) –Its failure; None when it succeeded (the level is then ERROR).
-
metadata(Mapping[str, Any] | None, default:None) –Extra fields for the sink.
-
session_id(str | None, default:None) –When set, groups the generation under that session.
-
prompt_name(str | None, default:None) –When set, links the managed prompt of that name.
record_tool
¶
record_tool(
gen: Generation,
*,
name: str,
call_id: str,
duration_s: float,
is_error: bool
) -> None
Hand one finished tool execution to gen, when its sink records tools. Never raises.
The seam is optional and duck-typed, so a 1.x Generation keeps working: a generation with a
tool(name=, call_id=, duration_s=, is_error=) method receives it (OtelTracer's opens an
INTERNAL child span execute_tool {name}; RecordingTracer's records it); any other ignores
it, and the call's observation still carries the tool names, durations and error count in its
metadata.
Rationale: ADR-0023, ADR-0025.
llm_ports.observability.semconv¶
Every OpenTelemetry attribute and metric name the library emits, spelled once.
The GenAI semantic convention is at stability development: it has no release, and it renames
(cache_creation became cache_write, gen_ai.system became gen_ai.provider.name, the token
usage histogram became per-kind counters). So no call site spells a name: each is a constant here,
checked against the source commit SEMCONV_SOURCE. A rename upstream is a one-file diff plus a
changelog line. A scheduled CI job (.github/workflows/semconv-drift.yml) diffs these constants
against the convention's registry at HEAD and opens an issue on drift.
Names under llm_ports. are the library's own (the convention has no key for them).
Invariants
CONVENTION_ATTRIBUTESandLIBRARY_ATTRIBUTESare every attribute key anOtelTracerspan may carry; a unit test pins the emitted key set to them.- No body attribute (
gen_ai.input.messages,gen_ai.output.messages,gen_ai.system_instructions) is named here, so none can be emitted.
Rationale: ADR-0025.