Skip to content

llm_ports.testing

llm_ports.testing

Test doubles for every port, a fake clock, and conformance suites.

testing.conformance (no extra) checks any implementation of a port against its contract; testing.pytest_plugin is the pytest plugin (loaded by pytest's entry point, never by import llm_ports); testing.langfuse (the langfuse extra) is imported by path.

StubChat

StubChat(
    script: Sequence[Turn], *, model: str = STUB_CHAT_MODEL
)

Bases: Chat

A Chat with no wire; the module docstring states the contract.

Parameters:

  • script (Sequence[Turn]) –

    The model turns, one per model request.

  • model (str, default: STUB_CHAT_MODEL ) –

    The model id the stub reports in answered.model.

Attributes:

  • requests (list[ChatRequest]) –

    Every ChatRequest received, in call order.

unused property

unused: tuple[Turn, ...]

The scripted turns no request has played yet.

assert_spent

assert_spent() -> None

Raise ContractViolation("script_unused") when scripted turns were never played.

stream async

stream(
    req: ChatRequest,
    output: type[T] = str,
    *,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None
) -> AsyncGenerator[StreamEvent]

Play the script for req, running its tools; retry_message is unused (no re-ask).

ToolCall dataclass

ToolCall(
    name: str,
    args: Mapping[str, Any] = dict(),
    call_id: str | None = None,
)

One scripted tool call.

Attributes:

  • name (str) –

    The tool's name.

  • args (Mapping[str, Any]) –

    The arguments the model sends.

  • call_id (str | None) –

    The call id; None numbers it.

Turn dataclass

Turn(
    text: str = "",
    calls: tuple[ToolCall, ...] = (),
    reasoning: str = "",
    usage: Usage | None = None,
    failed: Failure | None = None,
)

One scripted model turn: final text, tool calls, or a failure. Build one with the class methods.

Attributes:

  • text (str) –

    The reply text (final when there are no calls).

  • calls (tuple[ToolCall, ...]) –

    The tool calls asked for.

  • reasoning (str) –

    Reasoning text streamed before the reply.

  • usage (Usage | None) –

    The request's usage; None is unmeasured.

  • failed (Failure | None) –

    The failure the request ends with.

reply classmethod

reply(
    text: str,
    *,
    reasoning: str = "",
    usage: Usage | None = None
) -> Turn

A final reply. Pure.

tool classmethod

tool(
    name: str,
    args: Mapping[str, Any] | None = None,
    *,
    call_id: str | None = None
) -> Turn

One tool call. Pure.

tools classmethod

tools(*calls: ToolCall, usage: Usage | None = None) -> Turn

Parallel tool calls in one turn. Pure.

failure classmethod

failure(kind: FailureKind, **detail: Any) -> Turn

A request that fails with kind; detail is ports.failure's keywords. Pure.

FakeClock

FakeClock(start: float = 0.0)

A clock that starts at start and moves only on advance or sleep.

Parameters:

  • start (float, default: 0.0 ) –

    The first reading.

Attributes:

  • sleeps (list[float]) –

    Every sleep duration asked for, in order.

advance

advance(s: float) -> None

Move the clock forward by s seconds.

Raises:

  • ValueError –

    s is negative; a monotonic clock never goes back.

sleep async

sleep(s: float) -> None

Record the sleep and advance by it (a negative sleep is zero), without waiting.

Reply

Bases: Protocol

A StubGenerator reply function: (request, output type) to a reply.

A reply is a plain value, a ScriptedReply (Reply.ok, Reply.problems), a Failure (Reply.failed(kind, **detail)) or a whole Generated. Any function of that shape is a Reply.

Example

gen = StubGenerator([Reply.problems("draft", ["too short"]), Reply.ok("final")])

ScriptedReply dataclass

ScriptedReply(
    value: Any,
    problems: tuple[Any, ...] = (),
    usage: Usage | None = None,
    finish: UnifiedFinish | None = None,
    answered_model: str | None = None,
)

One scripted model reply: a value, with what the provider would have reported about it.

Build one with Reply.ok or Reply.problems.

Attributes:

  • value (Any) –

    The parsed value.

  • problems (tuple[Any, ...]) –

    Problems the reply carries regardless of validate (a validator not satisfied).

  • usage (Usage | None) –

    The request's usage; None is unmeasured.

  • finish (UnifiedFinish | None) –

    How the reply ended; None (a bare 1.x reply value) reports no finish.

  • answered_model (str | None) –

    The model the reply says answered; None uses the stub's.

StubGenerator

StubGenerator(
    reply: Reply | Sequence[Any] | Any,
    *,
    model: str = STUB_MODEL
)

A Generator with no wire; the module docstring states the contract.

Parameters:

  • reply (Reply | Sequence[Any] | Any) –

    A function of (request, output type), a script (a sequence of reply objects), or any other value, returned for every request.

  • model (str, default: STUB_MODEL ) –

    The model id the stub reports in answered.model.

Attributes:

  • requests (list[GenRequest]) –

    Every GenRequest received, in call order (a re-ask does not add one).

unused property

unused: tuple[Any, ...]

The scripted replies no request has played yet; () for a reply function or a constant.

assert_spent

assert_spent() -> None

Raise ContractViolation("script_unused") when scripted replies were never played.

generate async

generate(
    req: GenRequest,
    output: type[T],
    *,
    validate: Validator[T] | None = None,
    retry_message: RetryMessage | None = None
) -> Generated[T]

Answer req, re-asking while problems remain and re-asks are left; retry_message is unused.

LiveRequestBlocked

LiveRequestBlocked(host: str)

Bases: RuntimeError

A test sent a request to a live host without opting in.

Attributes:

  • host –

    The host the request was for.

Name the host the request was for.

RecordedGeneration dataclass

RecordedGeneration(
    opened: dict[str, Any],
    ended: dict[str, Any] | None = None,
    ends: int = 0,
    scores: list[tuple[str, float, str | None]] = list(),
    score_metadata: list[Mapping[str, Any] | None] = list(),
    tools: list[dict[str, Any]] = list(),
)

One start_generation call: what it was opened with, how it ended and how it was scored.

Attributes:

  • opened (dict[str, Any]) –

    The keyword arguments start_generation received.

  • ended (dict[str, Any] | None) –

    The keyword arguments of end, or None while the generation is open.

  • scores (list[tuple[str, float, str | None]]) –

    Every (name, value, comment) passed to score, in call order.

  • score_metadata (list[Mapping[str, Any] | None]) –

    The metadata of each of those scores, in the same order (None where none).

  • ends (int) –

    How many times end was called (a generation is ended exactly once).

  • tools (list[dict[str, Any]]) –

    Every tool execution handed to tool, as its keyword arguments, in order.

tool

tool(
    *,
    name: str,
    call_id: str,
    duration_s: float,
    is_error: bool
) -> None

Record one tool execution of a chat call.

end

end(**fields: Unpack[EndFields]) -> None

Record how the generation ended, replacing any earlier end, and count the ends.

ended holds the 1.x five keys always and each 1.24 keyword only when it carried a value.

score

score(
    name: str,
    value: float,
    comment: str | None = None,
    metadata: Mapping[str, Any] | None = None,
) -> None

Record one score and its metadata.

RecordingTracer

RecordingTracer()

A Tracer that records every generation.

Attributes:

start_generation

start_generation(
    *,
    name: str,
    model: str,
    provider: str,
    input: Any,
    metadata: Mapping[str, Any],
    session_id: str | None,
    prompt_name: str | None
) -> RecordedGeneration

Open, record and return one generation.

StubCiter

A Citer that asserts nothing.

Every chunk id is known to the answer, and the answer has no claims. model is the empty string, since a stub has no model.

answer async

answer(q: str, chunks: Sequence[Chunk]) -> CitedAnswer

Return an answer to q with no claims, keyed over chunks.

StubDetector

StubDetector()

An adapters.lettucedetect.Detector with no model.

It flags every digit run in the answer that no context string contains, in LettuceDetect's span shape with confidence 0.9: a stand-in for the number law.

Attributes:

  • calls (list[dict[str, Any]]) –

    The keyword arguments of every predict call, in call order.

predict

predict(
    *, context: list[str], answer: str, output_format: str
) -> list[dict[str, Any]]

Return one span for each digit run in answer that no context string contains.

StubEmbedder

StubEmbedder(
    *,
    dims: int = 8,
    stamp: str = "stub@8",
    fail: Outcome | None = None
)

An Embedder with no wire: each text hashes to a deterministic unit vector.

Parameters:

  • dims (int, default: 8 ) –

    The length of every vector.

  • stamp (str, default: 'stub@8' ) –

    The embedding stamp reported on every result.

  • fail (Outcome | None, default: None ) –

    An outcome other than ok; when set, every call fails with it.

Attributes:

  • calls (list[tuple[tuple[str, ...], EmbedInput]]) –

    (texts, input_type) for every call, in call order.

Invariants
  • The same text always maps to the same vector.

stamp property

stamp: str

The embedding stamp reported on every result.

vector

vector(text: str) -> tuple[float, ...]

Return the unit vector for text, derived from its sha256 digest.

embed async

embed(
    texts: Sequence[str],
    *,
    input_type: EmbedInput = "document"
) -> Embedded

Return one vector per text, or fail with the configured outcome.

StubPromptSource

StubPromptSource(texts: Mapping[str, str])

A PromptSource with no managed store, serving fixed in-memory texts.

Each name serves its text as the repo seed (version None, source "repo"). The requested label is recorded in config["label"], so a test can see it was passed through.

Parameters:

  • texts (Mapping[str, str]) –

    The text served for each prompt name.

get

get(name: str, *, label: str = 'production') -> Prompt

Return the fixed text for name as a repo-seed Prompt.

Raises:

  • KeyError –

    name was not given. The stub never invents text: with no seed, the port has nothing to serve.

StubReranker

StubReranker(
    *,
    model: str = "stub-reranker",
    fail: Outcome | None = None
)

A Reranker with no model: a document scores the count of its words found in the query.

Words are case-folded, so a test can predict the order.

Parameters:

  • model (str, default: 'stub-reranker' ) –

    The model name reported on every result.

  • fail (Outcome | None, default: None ) –

    An outcome other than ok; when set, every call fails with it.

Attributes:

  • calls (list[tuple[str, tuple[str, ...]]]) –

    (query, documents) for every call, in call order.

rerank async

rerank(query: str, documents: Sequence[str]) -> Reranked

Score each document by word overlap with query, or fail with the configured outcome.

No documents is ok with no scores even when the stub is set to fail, as the port says.

ContractViolation

ContractViolation(invariant: str, detail: str)

Bases: AssertionError

An implementation broke one invariant of its port's contract.

Attributes:

  • invariant (str) –

    The invariant's name (round_trip, never_raises, ...).

Name the broken invariant and what was seen.

run_tool async

run_tool(
    tool: Tool, args: Mapping[str, Any], budget: int | None
) -> tuple[ToolOutput, bool, float]

Run one handler under its timeout and cut its output to budget; never raises but cancellation.

Returns:

  • ToolOutput –

    The output (an error output for a timeout, a raise, or a non-ToolOutput return), whether

  • bool –

    it was cut, and the seconds the handler ran.

block_live_requests

block_live_requests() -> Callable[[], None]

Block every live model request until the returned function is called; see the module docstring.

is_loopback

is_loopback(host: str) -> bool

Whether host is this machine: localhost, or a loopback address. Pure.

llm_ports.testing.conformance

Conformance suites: the contract every implementation of a port must keep.

A consumer that writes its own adapter (a wrapper, a second provider, a database-backed ledger) runs the port's check_* in its own CI; the library runs the same checks against its reference adapters and its stubs, so the stub cannot silently differ from the adapter. Each check is a coroutine that returns the case names it checked and raises ContractViolation naming the first case broken. CHECKS maps every runtime-checkable port to its check; a test proves it total.

Rationale: ADR-0016.

ContractViolation

ContractViolation(invariant: str, detail: str)

Bases: AssertionError

An implementation broke one invariant of its port's contract.

Attributes:

  • invariant (str) –

    The invariant's name (round_trip, never_raises, ...).

Name the broken invariant and what was seen.

check_chat async

check_chat(
    make: Callable[[Sequence[Turn]], Chat],
) -> tuple[str, ...]

Check a Chat against the port's contract; the module docstring lists the cases.

Parameters:

  • make (Callable[[Sequence[Turn]], Chat]) –

    Builds a fresh chat that answers its model requests from the script, in order.

Returns:

  • tuple[str, ...] –

    The invariants checked, in order.

Raises:

check_token_counter async

check_token_counter(
    counter: TokenCounter,
    *,
    model: str,
    uncountable: str | None = None
) -> tuple[str, ...]

Check a TokenCounter against the port's contract; the module docstring lists the cases.

Parameters:

  • counter (TokenCounter) –

    The counter.

  • model (str) –

    A model id it counts.

  • uncountable (str | None, default: None ) –

    A model id it cannot count, when it has one.

Returns:

  • tuple[str, ...] –

    The invariants checked, in order.

Raises:

check_generator async

check_generator(
    make: Callable[[Sequence[Any]], Generator],
) -> tuple[str, ...]

Check a Generator against the port's contract; the module docstring lists the cases.

Parameters:

  • make (Callable[[Sequence[Any]], Generator]) –

    Builds a fresh generator that answers its requests from the script, in order.

Returns:

  • tuple[str, ...] –

    The invariants checked, in order.

Raises:

check_example_store async

check_example_store(
    store: WritableExampleStore,
) -> tuple[str, ...]

Check a working, writable ExampleStore against the port's contract.

Returns:

  • tuple[str, ...] –

    The invariants checked, in order.

Raises:

check_model_registry async

check_model_registry(
    registry: ModelRegistry,
) -> tuple[str, ...]

Check a working ModelRegistry against the port's contract.

Returns:

  • tuple[str, ...] –

    The invariants checked, in order.

Raises:

check_run_ledger async

check_run_ledger(ledger: RunLedger) -> tuple[str, ...]

Check a working RunLedger against the port's contract.

Returns:

  • tuple[str, ...] –

    The invariants checked, in order.

Raises:

check_unavailable_model_registry async

check_unavailable_model_registry(
    registry: ModelRegistry,
) -> tuple[str, ...]

Check that a ModelRegistry whose storage is broken reports it.

Returns:

  • tuple[str, ...] –

    The invariants checked.

Raises:

  • ContractViolation –

    A write that claims success, or an unreadable registry read as empty.

check_unavailable_run_ledger async

check_unavailable_run_ledger(
    ledger: RunLedger,
) -> tuple[str, ...]

Check that a RunLedger whose storage is broken reports it.

Returns:

  • tuple[str, ...] –

    The invariants checked.

Raises:

  • ContractViolation –

    A write that claims success, or an unreadable ledger read as empty.

check_unreadable_example_store async

check_unreadable_example_store(
    store: ExampleStore,
) -> tuple[str, ...]

Check that an ExampleStore whose storage is broken never reads as empty.

Returns:

  • tuple[str, ...] –

    The invariants checked.

Raises:

check_citer async

check_citer(make: Callable[[], Citer]) -> tuple[str, ...]

No chunks yield no claims; every citation is a chunk id it was given.

check_embedder async

check_embedder(
    make: Callable[[], Embedder],
) -> tuple[str, ...]

One vector per text in one space; no texts is not an error; the stamp is constant.

check_evaluator async

check_evaluator(
    make: Callable[[], Evaluator],
) -> tuple[str, ...]

The board has every key, always.

check_gateway_reader async

check_gateway_reader(
    make: Callable[[], GatewayReader],
) -> tuple[str, ...]

No headers report no cost and no deployment.

check_generation async

check_generation(
    make: Callable[[], Generation],
) -> tuple[str, ...]

end takes the full 1.24 payload (or the 1.x five) and score takes metadata; neither raises.

check_grounder async

check_grounder(
    make: Callable[[], Grounder],
) -> tuple[str, ...]

A claim with no spans is never supported.

check_judge async

check_judge(make: Callable[[], Judge]) -> tuple[str, ...]

A judgement is a JudgeVerdict with a numeric score (a judge may raise when it cannot answer).

check_limiter async

check_limiter(
    make: Callable[[], Limiter],
) -> tuple[str, ...]

A free slot is held at once and released: the wait is a non-negative number of seconds.

check_pricer async

check_pricer(make: Callable[[], Pricer]) -> tuple[str, ...]

An unknown model and unmeasured usage both price to usd=None; neither raises.

check_probe async

check_probe(make: Callable[[], Probe]) -> tuple[str, ...]

probe returns a Health in a known state and never raises.

check_prompt_source async

check_prompt_source(
    make: Callable[[], PromptSource], name: str
) -> tuple[str, ...]

get(name) serves non-empty text under that name.

Parameters:

  • make (Callable[[], PromptSource]) –

    Builds the source.

  • name (str) –

    A prompt the source has (at least as a repo seed).

check_reranker async

check_reranker(
    make: Callable[[], Reranker],
) -> tuple[str, ...]

One score per document in input order; no documents is an ok with no scores.

check_retriever async

check_retriever(
    make: Callable[[], Retriever],
) -> tuple[str, ...]

A blank query is []; a search returns at most k chunks.

check_tracer async

check_tracer(make: Callable[[], Tracer]) -> tuple[str, ...]

start_generation returns a Generation that passes check_generation; nothing raises.

llm_ports.testing.pytest_plugin

The llm_ports pytest plugin, registered through the pytest11 entry point.

Installing llm-ports in a test environment turns it on; -p no:llm_ports turns it off.

  • Live model requests are blocked in every test (testing._live.block_live_requests): a request that reaches a real network transport for a non-loopback host raises LiveRequestBlocked naming the host. A test opts in with @pytest.mark.llm_live; it then runs only when --llm-live is passed, and is skipped otherwise.
  • Fixtures: stub_generator (a factory; every script it built must be spent by teardown), recording_tracer, fake_clock, and llm_replay (a cassette by path: replayed, or recorded with --llm-record; a partly played cassette fails at teardown).

The plugin is test-only: it is loaded by pytest, never by import llm_ports.

pytest_addoption

pytest_addoption(parser: Parser) -> None

Add --llm-live and --llm-record.

pytest_configure

pytest_configure(config: Config) -> None

Register the llm_live marker.

stub_generator

stub_generator() -> Iterator[Callable[..., StubGenerator]]

Build StubGenerators; a scripted one with replies left over fails the test at teardown.

recording_tracer

recording_tracer() -> RecordingTracer

A fresh RecordingTracer.

fake_clock

fake_clock() -> FakeClock

A fresh FakeClock at 0.0.

llm_replay

llm_replay(
    request: FixtureRequest,
) -> Iterator[Callable[..., Any]]

Return open(path, live=None, *, secrets=()) for a pydantic-ai model over a cassette.

Replaying (the default), it returns replay(cassette) and fails at teardown when an entry was never played. With --llm-record and a live model, it returns a version-2 Recorder over live and writes the cassette at teardown. Needs the pydantic-ai extra.

llm_ports.testing.langfuse

Test doubles for the Langfuse adapters (extra [langfuse]).

LangfuseRig is a real Langfuse SDK 5 client whose exporter is in memory. SDK 5 is OpenTelemetry underneath, so the exported span is the wire, and observed() reads each span back as the observation the server stores. Each rig builds under a fresh public key: the SDK's resource manager is a singleton per key, and a reused key hands a later test an earlier test's exporter. Scores are an ingestion-API call rather than a span, so the rig records create_score calls.

RefusingLangfuse is a client whose server is down: every call raises ConnectionError, so a test can prove an adapter over it never raises into the caller.

FakeLangfusePrompts fakes the two prompt-management calls LangfusePromptSource makes: get_prompt returns what create_prompt last stored (versions numbered in order), else the fallback.

LangfuseRig

LangfuseRig()

A real SDK 5 client exporting to memory.

Attributes:

  • exporter (InMemorySpanExporter) –

    The in-memory span exporter.

  • client (Langfuse) –

    The Langfuse client under test.

  • scores (list[dict[str, Any]]) –

    The keyword arguments of every create_score call, in call order.

spans

spans() -> list[Any]

Flush the client and return every finished span.

attrs

attrs() -> dict[str, Any]

Return the attributes of the one finished span.

Raises:

  • ValueError –

    The rig has not exported exactly one span.

observed

observed() -> list[dict[str, Any]]

Return every finished span as an observation (see the module-level observed).

RefusingLangfuse

A Langfuse client whose server is down.

Every SDK call the adapters make raises ConnectionError with error as its message.

FakeLangfusePrompts

FakeLangfusePrompts(stored: SimpleNamespace | None = None)

The prompt-management half of a Langfuse client, in memory.

Attributes:

  • stored –

    The prompt get_prompt returns, or None to return the fallback.

  • created (list[dict[str, Any]]) –

    The keyword arguments of every create_prompt call.

  • fetches (list[tuple[str, str, str, int]]) –

    (name, label, type, cache_ttl_seconds) for every get_prompt call.

get_prompt

get_prompt(
    name: str,
    *,
    label: str,
    type: str,
    fallback: Any,
    cache_ttl_seconds: int
) -> SimpleNamespace

Record the fetch and return the stored prompt, or the fallback when nothing is stored.

create_prompt

create_prompt(**kw: Any) -> None

Record the call and store its prompt as the next version.

observed

observed(span: Any) -> dict[str, Any]

Return one exported span as the observation the server stores.

The langfuse.* attributes are unpacked: metadata is one JSON attribute per key, and usage is one JSON blob.

Invariants
  • Only keys the span carries appear; an absent field is absent, never None.