llm_ports.testing¶
llm_ports.testing
¶
Test doubles for every port, a fake clock, and conformance suites.
testing.conformance (no extra) checks any implementation of a port against its contract;
testing.pytest_plugin is the pytest plugin (loaded by pytest's entry point, never by
import llm_ports); testing.langfuse (the langfuse extra) is imported by path.
StubChat
¶
StubChat(
script: Sequence[Turn], *, model: str = STUB_CHAT_MODEL
)
Bases: Chat
A Chat with no wire; the module docstring states the contract.
Parameters:
-
script(Sequence[Turn]) –The model turns, one per model request.
-
model(str, default:STUB_CHAT_MODEL) –The model id the stub reports in
answered.model.
Attributes:
-
requests(list[ChatRequest]) –Every
ChatRequestreceived, in call order.
assert_spent
¶
assert_spent() -> None
Raise ContractViolation("script_unused") when scripted turns were never played.
stream
async
¶
stream(
req: ChatRequest,
output: type[T] = str,
*,
validate: Validator[T] | None = None,
retry_message: RetryMessage | None = None
) -> AsyncGenerator[StreamEvent]
Play the script for req, running its tools; retry_message is unused (no re-ask).
ToolCall
dataclass
¶
ToolCall(
name: str,
args: Mapping[str, Any] = dict(),
call_id: str | None = None,
)
One scripted tool call.
Attributes:
-
name(str) –The tool's name.
-
args(Mapping[str, Any]) –The arguments the model sends.
-
call_id(str | None) –The call id; None numbers it.
Turn
dataclass
¶
Turn(
text: str = "",
calls: tuple[ToolCall, ...] = (),
reasoning: str = "",
usage: Usage | None = None,
failed: Failure | None = None,
)
One scripted model turn: final text, tool calls, or a failure. Build one with the class methods.
Attributes:
-
text(str) –The reply text (final when there are no calls).
-
calls(tuple[ToolCall, ...]) –The tool calls asked for.
-
reasoning(str) –Reasoning text streamed before the reply.
-
usage(Usage | None) –The request's usage; None is unmeasured.
-
failed(Failure | None) –The failure the request ends with.
FakeClock
¶
FakeClock(start: float = 0.0)
A clock that starts at start and moves only on advance or sleep.
Parameters:
-
start(float, default:0.0) –The first reading.
Attributes:
-
sleeps(list[float]) –Every
sleepduration asked for, in order.
Reply
¶
Bases: Protocol
A StubGenerator reply function: (request, output type) to a reply.
A reply is a plain value, a ScriptedReply (Reply.ok, Reply.problems), a Failure
(Reply.failed(kind, **detail)) or a whole Generated. Any function of that shape is a Reply.
Example
gen = StubGenerator([Reply.problems("draft", ["too short"]), Reply.ok("final")])
ScriptedReply
dataclass
¶
ScriptedReply(
value: Any,
problems: tuple[Any, ...] = (),
usage: Usage | None = None,
finish: UnifiedFinish | None = None,
answered_model: str | None = None,
)
One scripted model reply: a value, with what the provider would have reported about it.
Build one with Reply.ok or Reply.problems.
Attributes:
-
value(Any) –The parsed value.
-
problems(tuple[Any, ...]) –Problems the reply carries regardless of
validate(a validator not satisfied). -
usage(Usage | None) –The request's usage; None is unmeasured.
-
finish(UnifiedFinish | None) –How the reply ended; None (a bare 1.x reply value) reports no finish.
-
answered_model(str | None) –The model the reply says answered; None uses the stub's.
StubGenerator
¶
StubGenerator(
reply: Reply | Sequence[Any] | Any,
*,
model: str = STUB_MODEL
)
A Generator with no wire; the module docstring states the contract.
Parameters:
-
reply(Reply | Sequence[Any] | Any) –A function of
(request, output type), a script (a sequence of reply objects), or any other value, returned for every request. -
model(str, default:STUB_MODEL) –The model id the stub reports in
answered.model.
Attributes:
-
requests(list[GenRequest]) –Every
GenRequestreceived, in call order (a re-ask does not add one).
unused
property
¶
unused: tuple[Any, ...]
The scripted replies no request has played yet; () for a reply function or a constant.
assert_spent
¶
assert_spent() -> None
Raise ContractViolation("script_unused") when scripted replies were never played.
generate
async
¶
generate(
req: GenRequest,
output: type[T],
*,
validate: Validator[T] | None = None,
retry_message: RetryMessage | None = None
) -> Generated[T]
Answer req, re-asking while problems remain and re-asks are left; retry_message is unused.
LiveRequestBlocked
¶
LiveRequestBlocked(host: str)
Bases: RuntimeError
A test sent a request to a live host without opting in.
Attributes:
-
host–The host the request was for.
Name the host the request was for.
RecordedGeneration
dataclass
¶
RecordedGeneration(
opened: dict[str, Any],
ended: dict[str, Any] | None = None,
ends: int = 0,
scores: list[tuple[str, float, str | None]] = list(),
score_metadata: list[Mapping[str, Any] | None] = list(),
tools: list[dict[str, Any]] = list(),
)
One start_generation call: what it was opened with, how it ended and how it was scored.
Attributes:
-
opened(dict[str, Any]) –The keyword arguments
start_generationreceived. -
ended(dict[str, Any] | None) –The keyword arguments of
end, or None while the generation is open. -
scores(list[tuple[str, float, str | None]]) –Every
(name, value, comment)passed toscore, in call order. -
score_metadata(list[Mapping[str, Any] | None]) –The
metadataof each of those scores, in the same order (None where none). -
ends(int) –How many times
endwas called (a generation is ended exactly once). -
tools(list[dict[str, Any]]) –Every tool execution handed to
tool, as its keyword arguments, in order.
tool
¶
tool(
*,
name: str,
call_id: str,
duration_s: float,
is_error: bool
) -> None
Record one tool execution of a chat call.
end
¶
end(**fields: Unpack[EndFields]) -> None
Record how the generation ended, replacing any earlier end, and count the ends.
ended holds the 1.x five keys always and each 1.24 keyword only when it carried a value.
score
¶
score(
name: str,
value: float,
comment: str | None = None,
metadata: Mapping[str, Any] | None = None,
) -> None
Record one score and its metadata.
RecordingTracer
¶
RecordingTracer()
A Tracer that records every generation.
Attributes:
-
generations(list[RecordedGeneration]) –Every
RecordedGeneration, in open order.
start_generation
¶
start_generation(
*,
name: str,
model: str,
provider: str,
input: Any,
metadata: Mapping[str, Any],
session_id: str | None,
prompt_name: str | None
) -> RecordedGeneration
Open, record and return one generation.
StubCiter
¶
A Citer that asserts nothing.
Every chunk id is known to the answer, and the answer has no claims. model is the empty
string, since a stub has no model.
answer
async
¶
answer(q: str, chunks: Sequence[Chunk]) -> CitedAnswer
Return an answer to q with no claims, keyed over chunks.
StubDetector
¶
StubDetector()
An adapters.lettucedetect.Detector with no model.
It flags every digit run in the answer that no context string contains, in LettuceDetect's span shape with confidence 0.9: a stand-in for the number law.
Attributes:
-
calls(list[dict[str, Any]]) –The keyword arguments of every
predictcall, in call order.
predict
¶
predict(
*, context: list[str], answer: str, output_format: str
) -> list[dict[str, Any]]
Return one span for each digit run in answer that no context string contains.
StubEmbedder
¶
StubEmbedder(
*,
dims: int = 8,
stamp: str = "stub@8",
fail: Outcome | None = None
)
An Embedder with no wire: each text hashes to a deterministic unit vector.
Parameters:
-
dims(int, default:8) –The length of every vector.
-
stamp(str, default:'stub@8') –The embedding stamp reported on every result.
-
fail(Outcome | None, default:None) –An outcome other than
ok; when set, every call fails with it.
Attributes:
-
calls(list[tuple[tuple[str, ...], EmbedInput]]) –(texts, input_type)for every call, in call order.
Invariants
- The same text always maps to the same vector.
vector
¶
vector(text: str) -> tuple[float, ...]
Return the unit vector for text, derived from its sha256 digest.
StubPromptSource
¶
StubPromptSource(texts: Mapping[str, str])
A PromptSource with no managed store, serving fixed in-memory texts.
Each name serves its text as the repo seed (version None, source "repo"). The requested
label is recorded in config["label"], so a test can see it was passed through.
Parameters:
-
texts(Mapping[str, str]) –The text served for each prompt name.
StubReranker
¶
StubReranker(
*,
model: str = "stub-reranker",
fail: Outcome | None = None
)
A Reranker with no model: a document scores the count of its words found in the query.
Words are case-folded, so a test can predict the order.
Parameters:
-
model(str, default:'stub-reranker') –The model name reported on every result.
-
fail(Outcome | None, default:None) –An outcome other than
ok; when set, every call fails with it.
Attributes:
-
calls(list[tuple[str, tuple[str, ...]]]) –(query, documents)for every call, in call order.
ContractViolation
¶
ContractViolation(invariant: str, detail: str)
Bases: AssertionError
An implementation broke one invariant of its port's contract.
Attributes:
-
invariant(str) –The invariant's name (
round_trip,never_raises, ...).
Name the broken invariant and what was seen.
run_tool
async
¶
run_tool(
tool: Tool, args: Mapping[str, Any], budget: int | None
) -> tuple[ToolOutput, bool, float]
Run one handler under its timeout and cut its output to budget; never raises but cancellation.
Returns:
-
ToolOutput–The output (an error output for a timeout, a raise, or a non-
ToolOutputreturn), whether -
bool–it was cut, and the seconds the handler ran.
block_live_requests
¶
block_live_requests() -> Callable[[], None]
Block every live model request until the returned function is called; see the module docstring.
is_loopback
¶
is_loopback(host: str) -> bool
Whether host is this machine: localhost, or a loopback address. Pure.
llm_ports.testing.conformance¶
Conformance suites: the contract every implementation of a port must keep.
A consumer that writes its own adapter (a wrapper, a second provider, a database-backed ledger)
runs the port's check_* in its own CI; the library runs the same checks against its reference
adapters and its stubs, so the stub cannot silently differ from the adapter. Each check is a
coroutine that returns the case names it checked and raises ContractViolation naming the first
case broken. CHECKS maps every runtime-checkable port to its check; a test proves it total.
Rationale: ADR-0016.
ContractViolation
¶
ContractViolation(invariant: str, detail: str)
Bases: AssertionError
An implementation broke one invariant of its port's contract.
Attributes:
-
invariant(str) –The invariant's name (
round_trip,never_raises, ...).
Name the broken invariant and what was seen.
check_chat
async
¶
Check a Chat against the port's contract; the module docstring lists the cases.
Parameters:
-
make(Callable[[Sequence[Turn]], Chat]) –Builds a fresh chat that answers its model requests from the script, in order.
Returns:
-
tuple[str, ...]–The invariants checked, in order.
Raises:
-
ContractViolation–The first invariant the chat breaks.
check_token_counter
async
¶
check_token_counter(
counter: TokenCounter,
*,
model: str,
uncountable: str | None = None
) -> tuple[str, ...]
Check a TokenCounter against the port's contract; the module docstring lists the cases.
Parameters:
-
counter(TokenCounter) –The counter.
-
model(str) –A model id it counts.
-
uncountable(str | None, default:None) –A model id it cannot count, when it has one.
Returns:
-
tuple[str, ...]–The invariants checked, in order.
Raises:
-
ContractViolation–The first invariant the counter breaks.
check_generator
async
¶
check_generator(
make: Callable[[Sequence[Any]], Generator],
) -> tuple[str, ...]
Check a Generator against the port's contract; the module docstring lists the cases.
Parameters:
-
make(Callable[[Sequence[Any]], Generator]) –Builds a fresh generator that answers its requests from the script, in order.
Returns:
-
tuple[str, ...]–The invariants checked, in order.
Raises:
-
ContractViolation–The first invariant the generator breaks.
check_example_store
async
¶
check_example_store(
store: WritableExampleStore,
) -> tuple[str, ...]
Check a working, writable ExampleStore against the port's contract.
Returns:
-
tuple[str, ...]–The invariants checked, in order.
Raises:
-
ContractViolation–The first invariant the store breaks.
check_model_registry
async
¶
check_model_registry(
registry: ModelRegistry,
) -> tuple[str, ...]
Check a working ModelRegistry against the port's contract.
Returns:
-
tuple[str, ...]–The invariants checked, in order.
Raises:
-
ContractViolation–The first invariant the registry breaks.
check_run_ledger
async
¶
check_run_ledger(ledger: RunLedger) -> tuple[str, ...]
Check a working RunLedger against the port's contract.
Returns:
-
tuple[str, ...]–The invariants checked, in order.
Raises:
-
ContractViolation–The first invariant the ledger breaks.
check_unavailable_model_registry
async
¶
check_unavailable_model_registry(
registry: ModelRegistry,
) -> tuple[str, ...]
Check that a ModelRegistry whose storage is broken reports it.
Returns:
-
tuple[str, ...]–The invariants checked.
Raises:
-
ContractViolation–A write that claims success, or an unreadable registry read as empty.
check_unavailable_run_ledger
async
¶
check_unavailable_run_ledger(
ledger: RunLedger,
) -> tuple[str, ...]
Check that a RunLedger whose storage is broken reports it.
Returns:
-
tuple[str, ...]–The invariants checked.
Raises:
-
ContractViolation–A write that claims success, or an unreadable ledger read as empty.
check_unreadable_example_store
async
¶
check_unreadable_example_store(
store: ExampleStore,
) -> tuple[str, ...]
Check that an ExampleStore whose storage is broken never reads as empty.
Returns:
-
tuple[str, ...]–The invariants checked.
Raises:
-
ContractViolation–The read raised or claimed success.
check_citer
async
¶
check_citer(make: Callable[[], Citer]) -> tuple[str, ...]
No chunks yield no claims; every citation is a chunk id it was given.
check_embedder
async
¶
check_embedder(
make: Callable[[], Embedder],
) -> tuple[str, ...]
One vector per text in one space; no texts is not an error; the stamp is constant.
check_evaluator
async
¶
check_evaluator(
make: Callable[[], Evaluator],
) -> tuple[str, ...]
The board has every key, always.
check_gateway_reader
async
¶
check_gateway_reader(
make: Callable[[], GatewayReader],
) -> tuple[str, ...]
No headers report no cost and no deployment.
check_generation
async
¶
check_generation(
make: Callable[[], Generation],
) -> tuple[str, ...]
end takes the full 1.24 payload (or the 1.x five) and score takes metadata; neither raises.
check_grounder
async
¶
check_grounder(
make: Callable[[], Grounder],
) -> tuple[str, ...]
A claim with no spans is never supported.
check_judge
async
¶
check_judge(make: Callable[[], Judge]) -> tuple[str, ...]
A judgement is a JudgeVerdict with a numeric score (a judge may raise when it cannot answer).
check_limiter
async
¶
check_limiter(
make: Callable[[], Limiter],
) -> tuple[str, ...]
A free slot is held at once and released: the wait is a non-negative number of seconds.
check_pricer
async
¶
check_pricer(make: Callable[[], Pricer]) -> tuple[str, ...]
An unknown model and unmeasured usage both price to usd=None; neither raises.
check_probe
async
¶
check_probe(make: Callable[[], Probe]) -> tuple[str, ...]
probe returns a Health in a known state and never raises.
check_prompt_source
async
¶
check_prompt_source(
make: Callable[[], PromptSource], name: str
) -> tuple[str, ...]
get(name) serves non-empty text under that name.
Parameters:
-
make(Callable[[], PromptSource]) –Builds the source.
-
name(str) –A prompt the source has (at least as a repo seed).
check_reranker
async
¶
check_reranker(
make: Callable[[], Reranker],
) -> tuple[str, ...]
One score per document in input order; no documents is an ok with no scores.
llm_ports.testing.pytest_plugin¶
The llm_ports pytest plugin, registered through the pytest11 entry point.
Installing llm-ports in a test environment turns it on; -p no:llm_ports turns it off.
- Live model requests are blocked in every test (
testing._live.block_live_requests): a request that reaches a real network transport for a non-loopback host raisesLiveRequestBlockednaming the host. A test opts in with@pytest.mark.llm_live; it then runs only when--llm-liveis passed, and is skipped otherwise. - Fixtures:
stub_generator(a factory; every script it built must be spent by teardown),recording_tracer,fake_clock, andllm_replay(a cassette by path: replayed, or recorded with--llm-record; a partly played cassette fails at teardown).
The plugin is test-only: it is loaded by pytest, never by import llm_ports.
stub_generator
¶
stub_generator() -> Iterator[Callable[..., StubGenerator]]
Build StubGenerators; a scripted one with replies left over fails the test at teardown.
llm_replay
¶
llm_replay(
request: FixtureRequest,
) -> Iterator[Callable[..., Any]]
Return open(path, live=None, *, secrets=()) for a pydantic-ai model over a cassette.
Replaying (the default), it returns replay(cassette) and fails at teardown when an entry was
never played. With --llm-record and a live model, it returns a version-2 Recorder over
live and writes the cassette at teardown. Needs the pydantic-ai extra.
llm_ports.testing.langfuse¶
Test doubles for the Langfuse adapters (extra [langfuse]).
LangfuseRig is a real Langfuse SDK 5 client whose exporter is in memory. SDK 5 is
OpenTelemetry underneath, so the exported span is the wire, and observed() reads each span
back as the observation the server stores. Each rig builds under a fresh public key: the SDK's
resource manager is a singleton per key, and a reused key hands a later test an earlier test's
exporter. Scores are an ingestion-API call rather than a span, so the rig records create_score
calls.
RefusingLangfuse is a client whose server is down: every call raises ConnectionError, so a
test can prove an adapter over it never raises into the caller.
FakeLangfusePrompts fakes the two prompt-management calls LangfusePromptSource makes:
get_prompt returns what create_prompt last stored (versions numbered in order), else the
fallback.
LangfuseRig
¶
LangfuseRig()
A real SDK 5 client exporting to memory.
Attributes:
-
exporter(InMemorySpanExporter) –The in-memory span exporter.
-
client(Langfuse) –The
Langfuseclient under test. -
scores(list[dict[str, Any]]) –The keyword arguments of every
create_scorecall, in call order.
RefusingLangfuse
¶
A Langfuse client whose server is down.
Every SDK call the adapters make raises ConnectionError with error as its message.
FakeLangfusePrompts
¶
FakeLangfusePrompts(stored: SimpleNamespace | None = None)
The prompt-management half of a Langfuse client, in memory.
Attributes:
-
stored–The prompt
get_promptreturns, or None to return the fallback. -
created(list[dict[str, Any]]) –The keyword arguments of every
create_promptcall. -
fetches(list[tuple[str, str, str, int]]) –(name, label, type, cache_ttl_seconds)for everyget_promptcall.
get_prompt
¶
get_prompt(
name: str,
*,
label: str,
type: str,
fallback: Any,
cache_ttl_seconds: int
) -> SimpleNamespace
Record the fetch and return the stored prompt, or the fallback when nothing is stored.
create_prompt
¶
create_prompt(**kw: Any) -> None
Record the call and store its prompt as the next version.
observed
¶
observed(span: Any) -> dict[str, Any]
Return one exported span as the observation the server stores.
The langfuse.* attributes are unpacked: metadata is one JSON attribute per key, and usage is
one JSON blob.
Invariants
- Only keys the span carries appear; an absent field is absent, never None.