Changelog¶
All notable changes to this project are documented here. The format is based on
Keep a Changelog, and the project adheres to
Semantic Versioning. Why a design is the way it is lives in
docs/adr/.
Unreleased¶
[1.26.1] - 2026-11-20¶
Fixed¶
PydanticEvalsJudgesendsthinkingspelled for its model. It passedRole.model_settings()straight toLLMJudge, so a schema 2 role's provider-neutralthinking: Falsereached achat_templatemodel (llama.cpp, vLLM) as a key it ignores; the template default (thinking ON) spent the whole token budget reasoning and every verdict failed "token limit exceeded before any response was generated". The judge now runs the settings throughtranslate_settings, as the generator's factory does, with the tier'sModelInfo. A role whose model override is not its tier's model passes through unchanged.translate_settingsnow lives besideModelInfo(llm_ports.call.models), so theevalsextra no longer needs thepydantic-aiadapter's dependencies to reach it;adapters.pydantic_aistill exports it.
[1.26.0] - 2026-11-12¶
Added¶
call.graders.schema_complete(..., model=, nullable=): declared answer paths for a dict reply. A strict grammar replies a dict, which carries no annotations, so every None in it read as unanswered, including a null the prompt orders ("else null"). Pass the declared output model (model=Out, itsnullable_paths), name the paths (nullable={"items[].stop"}, for a field typed nullable whose prompt still forbids null), or both (joined). A declaration replaces a pydantic value's own annotations. Bind it into a table with{**GRADERS, "schema_complete": partial(schema_complete, model=Out)}. With neither keyword, grading is unchanged.
[1.25.0] - 2026-11-12¶
Behaviour changes (read first)¶
- The wire capture no longer reads a streamed (
text/event-stream) 2xx body. Reading it buffered the whole stream before the first event reached the caller. A streamed call's usage and finish come from pydantic-ai's own parse of the stream. A JSON body is captured as before. Backends.aclose()now closes theopenai_compatibleprovider's client. The factory's settings-translating wrapper had noaclose, so the close never reached the client it wraps.generate(counter=...)andRole.context. A role withcontext="check"or"trim"now counts its prompt before the wire and can returnrequest_fault/context_overflowwithprovider_code="preflight"and zero generation requests. The defaultcontext="off"changes nothing.
Added¶
- The
Chatport (ADR-0023).ports.Chatwithchatandstream;ChatRequest,Message,Part,Tool,ToolOutput, and the stream eventsTextDelta,ReasoningDelta,ToolCallStarted,ToolCallFinished,StepFinished,Finished.Generatedgainsmessagesandsteps.Chat.chathas a default body (streamdrained), so an implementation that subclassesChatwrites onlystream. call.chat/call.stream/call.chat_request_for: capability check (tools,json_schema,vision), admit gate, cache plan, pre-flight, one trace observation with a child span per tool, graders. Eventsllm_tool_failed,llm_tool_truncated,llm_max_steps.call.TurnOptionsdeclares the optional keywords both take once (tools,hooks,system,settings,metadata,max_steps,counter).ports.Validator[T]andports.RetryMessage: named aliases for thevalidateandretry_messagecallables, now used by everyGeneratorandChatsignature. The types are unchanged.adapters.pydantic_ai.PydanticAIChaton pydantic-ai's agent loop, sharing the generator's wire policy, deadline, limiter, classification and result contract (AdapterCore, which now also takesmodel=/model_for=for both). Theopenai_compatiblefactory builds one per provider (Backends.chat_for).- Context is measured (ADR-0024).
ports.TokenCounter/Counted;adapters.http.HttpTokenCounter(LiteLLM's/utils/token_countershape), built by the factory for a model statingcount_tokens(Backends.counter_for).call.context:Budget,fit,TrimPolicy,trim,cut_middle,cache_plan,volatile_system.Rolegainscontext,trim,max_steps,tool_output_chars,cache_prefix;GenRequestgainscache_prefix. Eventsllm_count_failed,llm_trimmed,llm_context_overflow,llm_count_optimistic;stamp["count_requests"]andstamp["trimmed"]. - Tool spans.
observability.record_toolandPortObservation.tool;OtelTraceropens an INTERNALexecute_tool {name}child span per tool;TeeTracerfans it out;RecordingTracerrecords it. - Testing.
testing.StubChat(playsTurns, really runs the tools),testing.run_tool;testing.conformance.check_chat(eleven cases) andcheck_token_counter, both inCHECKS.
[1.24.0] - 2026-11-12¶
Behaviour changes (read first)¶
PydanticAIGeneratornow retries a failed wire request by default. Its default policy isWirePolicy.default():rate_limited3 tries (0.5 s doubling to 8 s, with jitter, honouring the server'sretry-after-ms/Retry-Afterup to 120 s), andoverloaded,server,timeoutandtransport2 tries.quota,authand a request-side kind are never retried. 1.x sent one request;PydanticAIGenerator(policy=WirePolicy.none())keeps that. Every retried request is anAttempton the result (ADR-0021).- A call's timeout is now a wall-clock bound on the WHOLE call.
generatestarts aDeadlinefrom the role's timeout and ends the call at it, across the slot wait, every wire request, every backoff and every fallback tier; an expiry isunavailablewith kindtimeoutandphase="total". 1.x's timeout bounded each HTTP phase, which a slow trickle never tripped. -
A tier table's
[policies.<name>]is now validated at load. A value aRetryRulecannot hold (an unknown kind such asquota, a missingwait_max_s, a negative wait) isTableInvalidnaming its path. 1.23.0 stored any table. -
gen_ai.systemis no longer emitted bygen_ai_span/record_gen_ai(the GenAI conventions removed it at v1.37.0 forgen_ai.provider.name, which is still set).SYSTEMstays exported; nothing sets it (ADR-0025). - A cancelled or interrupted call now ends its generation, once, at level
CANCELLED(Langfuse:WARNING), with its latency; theCancelledError/KeyboardInterruptstill propagates. 1.x left the generation open. tracer_trace(provider=)no longer defaults to"openai": None takes the tier's provider from a schema 2 table, else"unknown".StubGeneratorruns the re-ask loop: problems fromvalidate(orReply.problems) earnreq.output_retriesre-asks andretriescounts them (1.x always reported 0); a scriptedFailurereports the attempt number asretries, not None. AStubRerankerset to fail still answers no documents withok, as the port says.- The pytest plugin blocks live requests in any test session where
llm-portsis installed (entry pointpytest11llm_ports): a request to a non-loopback host over a realhttpx2/httpxtransport raisesLiveRequestBlocked. Opt in per test with@pytest.mark.llm_liveplus--llm-live; turn the plugin off with-p no:llm_ports. - The observation ContextVar's name is
"llm_feature"(was"langfuse_feature"); the accessor functions are unchanged.
Added¶
market.prune_mirror(withPrunableMirror,MirrorRow,PruneReport): aftersync_all, retires each enabled mirror row whose action the registry no longer declares, via the consumer'sretire_action. A registered row is never retired; a pack that failed to load keeps its rows (market_prune_held); each retired row is amarket_action_prunedevent andPruneReport.countis the gauge. Never raises.- Wire policy (ADR-0021):
WirePolicy(kind toRetryRule, plusmax_total_attempts) andRetryRule, exported fromllm_ports,llm_ports.portsandllm_ports.call.resilience. One retry loop (adapters._retry, tenacity underneath) is shared by the generator, the embedder and the reranker; the provider SDK's retries stay 0. ARole.policy, else the tier's declaredpolicy(Tier.wire_policy), else the generator's default applies.x-should-retry: falseends the retry; a server hint aboveretry_after_max_sstops it with the hint kept. - Deadline:
Deadline(total_s, attempt_s=None, first_event_s=None)onGenRequest.deadline;GenRequest.clocksreads it (orDeadline(total_s=timeout_s)).attempt_sbounds one request (a retriedtimeout),first_event_sthe wait for a stream's first event.call.DEADLINE_GRACE_Sis how long after its deadline the facade cuts off a generator that ignores it. - Fallback:
generatewalks a role's tier and itsfallbackchain; a failure whose kind is in the tier'sfallback_onhands the call to the next tier with what is left of the deadline.answered.tiernames who answered,attemptscarries every earlier tier's failure (eachAttempt.tierset),stamp["fallback_depth"]how far it went.Hooks.generator_forserves a fallback tier from another provider's generator. Eventsllm_fallback,llm_deadline_expired. - Limiter: the
Limiterport (slot(group), yields the seconds waited) and the in-processcall.resilience.LocalLimiter(slots, default=None, max_wait_s=None)withLocalLimiter.for_models. The slot is taken inside the deadline and kept across retries;stamp["limiter_wait_s"]reports the wait. A wait pastmax_wait_sisrate_limitedwithprovider_code="local_limiter".GenRequest.concurrency_group(the model'sgroup, else its key) names the slot. PydanticAIGenerator(policy=, limiter=),.stats()(clients_built,clients_open,requests_sent,calls_in_flight) and.aclose().openai_compatible(..., client=)takes a caller'shttpx2.AsyncClient; without one it builds ONE per event loop and reuses it (1.x opened a client per call). It now returns anOpenAICompatible(still amodel_for).Role.stream_internally/GenRequest.stream_internally: the adapter requests a stream and assembles it, so a cancelled or timed-out call closes a connection the server can stop on.PydanticAIEmbedder(policy=);HttpReranker(policy=, on_event=);Reranked.attempts.BuildDeps.policy,.limiterand.pricerare typed (WirePolicy,Limiter,Pricer) andOpenAICompatibleFactorypasses them, withon_event, to every generator and embedder it builds.-
RetryPolicy.wire(): the 1.x policy as aWirePolicy. -
OtelTracer(adapters.otel, extraotel, ADR-0025): theTracerport over OpenTelemetry. A CLIENT span"{operation} {model}"per generation with the result's usage, cost, finish, who answered anderror.type = failure.kind; given ameter_provider, thegen_ai.client.inference.durationhistogram and one counter per measured token kind. Never a body, never an env read. observability.semconv: every GenAI attribute and metric name, pinned tosemantic-conventions-genai@e6658f1;tools/semconv_drift.pyand the weeklysemconv-drift.ymljob diff them against HEAD (an issue on drift; exit 2 when it cannot look).TeeTracer: oneTracerover several sinks, each isolated;stats()per sink (started, ended, failed) and a throttledtracer_sink_failedevent.observability.record: one call records a finished outside call with the same end payloadgeneratesends.Generation.endgains keyword-onlyusage_v2,cost,finish,answered,failure, passed only to anendthat accepts them, so a 1.xGenerationkeeps working. The Langfuse adapter sendscost_detailsonly when the cost is known (andmetadata.cost_source).- Testing kit:
Reply.ok/Reply.problems(aScriptedReply), aStubGeneratorSCRIPT (one reply per request; running out or leaving replies is aContractViolation),FakeClock, and the pytest plugin's fixturesstub_generator,recording_tracer,fake_clock,llm_replay. - Conformance for every port (
testing.conformance, now a package):check_generator(driven by a script; the stub andPydanticAIGeneratorboth pass),check_embedder,check_reranker,check_tracer,check_generation,check_pricer,check_probe,check_retriever,check_citer,check_grounder,check_evaluator,check_prompt_source,check_judge,check_gateway_reader,check_limiter;CHECKSmaps every runtime-checkable port to its check (a test proves it total). - Replay key v2 (
adapters.pydantic_ai.replay):key_v2covers the model, system prompt and instructions, the whole message list, the settings and the output schema;Recorder(version=2)writes{"version": 2, "entries": ...}and refuses (SecretInCassette) a reply containing a bound secret;replayreads v1 and v2, a miss names the parts that differ, andReplayModel.unplayed()names the entries a run never asked for. A v1 cassette still replays.
Changed¶
PydanticAIEmbedder(retry=RetryPolicy(...))runs through the shared loop (RetryPolicy.wire()); a server'sRetry-Afteris now honoured up towait_max_s.RetryPolicyis deprecated in favour ofWirePolicy; passing bothretry=andpolicy=is aValueError.- The
pydantic-aifloor is 2.64 (first_event_timeout); thehttpextra now carriestenacity>=9(the reranker's retry loop).
[1.23.0] - 2026-11-12¶
Behaviour fixes (read first)¶
- A reply the token ceiling cut is no longer
ok. Afinish_reasonoflengthis nowtruncated(kindtruncated) on the success path too, even when the cut text parsed; the cut value is kept on the result.Role.accept_truncated=True(orGenRequest.accept_truncated) restoresok, with aCallWarningon the result (ADR-0020). - A caller's bug is no longer a model outage. An exception raised by the caller's own code (a
KeyErrorin avalidatefunction, a bug in amodel_for) is nowrequest_faultwith kindinternal, neverunavailable: it is not retried, failed over or counted against a backend. It is reported as the eventllm_call_internal_errorwith its traceback (ADR-0020). Who raised decides, so a provider's fault staysunavailableas in 1.x: an exception raised inside a provider client while it read a response that arrived (a 200 withchoices: []) is kindmalformed_response(retryable); pydantic-ai'sUsageLimitExceeded, an application'sRunCancelledandConcurrencyLimitExceeded(the agent's own queue) arecaller_limit(not retryable); aFallbackExceptionGroupis its worst member. A caller's configuration isinternaltoo, with no request sent: pydantic-ai'sUserError(an output mode the model lacks), and an HTTP client'sInvalidURLorUnsupportedProtocol(a base URL such ashttp://[::1/v1,ftp://...or one with no scheme), which 1.x reported as a retryableunavailable.
Changed¶
adapters.pluggy.load_packsraisesPackLoadError(anImportError) on the first pack that fails, naming the group, the pack and the stage (import,register,hook,duplicate,pin,pending). 1.22 recorded the failure and carried on, so a scheduler booted without a producer it was meant to run.load_packs(..., strict=False)keeps the old record-and-continue behaviour for a diagnostic. Entry points now load sorted by name, not in install order, and a misspelled hook name fails atpendinginstead of loading a plugin that does nothing (ADR-0022).- A strict
load_packsfreezes theActionRegistryit loaded into when it finishes, so a declaration after the composition root raisesRuntimeError(declare the process's own actions first, or pass them asbuiltins); callingload_packson an already frozen registry raises before loading anything.strict=Falseleaves the registry open. market_pack_failedkeeps 1.22'sstagevocabulary (PackLoad.stage:import,validate,pack,join, pluspin) and addsplugin_stageinPluginLoadError.stage's vocabulary.ActionRegistry: re-declaring a heldfeature.actionwith any spec object other than the identical one raisesDuplicateAction(aValueError) naming both origins, even when the metadata is equal. 1.22 replaced the held spec, so an equal declaration with a different handler swapped the handler in silently. Re-adding the identical object stays a no-op.- A
desired_ttlneeds bothcache_tableandage_col(the output row its age is measured against); without them nothing can judge the TTL, so the declaration is refused at import. GateCachefails closed until its first successful read:check(andadmit) returngate_unread: ...for every category, so a boot that forgotrefresh()refuses instead of admitting ungated.GateCache(..., fail_open=True)admits instead. Either way, each judgement from never-read gates emitsmarket_gate_unread(WARNING) withserving: "closed"or"open". A failed boot read keeps the gate closed until a laterrefresh()succeeds, and the cache never reads on its own: callawait gates.refresh()at boot and on a schedule.- The
marketextra needspluggy>=1.7(was>=1.5). call.providers.buildrefuses a secret whose value holds whitespace, a control or a non-ASCII character (TableInvalidatproviders.<name>.secret, naming the secret, never its value): no header can carry it, and httpx quotes a refused header, value and all, in its error and its DEBUG trace. A trailing newline from a secret file is the usual cause.adapters.pydantic_ai.openai_compatibleandopenai_compatible_embeddingsraiseValueErrorat construction for anapi_keythat holds whitespace, a control or a non-ASCII character (the message never shows the value). 1.22 accepted it and every call then failed with an error, and a DEBUG log line, quotingBearer <key>.build, theopenai_compatiblekind, these two andHttpProbeshare one validator.- Every
call.generateresult carriesstamp["tiers"], the fingerprint of the tier table that routed it (absent only for a hand-builtTier). - A prompt pack now fails at load where it used to load (ADR-0026):
- a template that
{% include %}s,{% import %}s or{% extends %}a file not listed intemplatesor the newincludes; - a dynamic include whose target cannot be read statically;
- a pack file holding a carriage return (CRLF or a lone CR);
- a
graderslist naming the same grader twice; - an
includesentry that is not a relative POSIX path inside the pack (.., a leading/, a drive, a\, an empty or hidden segment, orpack.yaml), or that resolves outside it.
Each raises PromptPackInvalid naming the file or field. Templates render from exactly the
manifest's files; nothing else in the pack directory is read. A partial in a subdirectory or
with any extension (partials/h.j2, rubric.md) loads once it is listed in includes.
- PackManifest.golden is resolved against the pack directory (it was documented as
repo-relative in 1.22.0, which nothing read) and must stay inside it.
- call.graders.GRADERS is a read-only mapping (types.MappingProxyType): assigning into it raises
TypeError. Consumers must stop writing into it (monkeypatch.setitem(GRADERS, ...) now
fails) and pass a table instead: {**GRADERS, "mine": fn} to score_all / score_one /
board_graders / call.Hooks(graders=...) / prompts.resolve(registry=) /
evals.golden.resolve_pack(registry=). A test that needs the default table changed replaces the
module global: monkeypatch.setattr(llm_ports.call.graders, "GRADERS", {**GRADERS, "mine": fn})
(score_one reads it at call time).
- Pack.stamp() gains the key identity. The sha key, stamp_str() and content_sha are
unchanged for every pack, byte for byte.
- PackManifest.graders accepts bare grader names (resolved through a registry) as well as dotted
paths.
- LangfusePromptSource re-pushes a mirror whose config.identity differs from the seed's even
when its content sha matches. get's default label is unchanged (production).
- The pydantic-ai and evals extras need pydantic-ai-slim / pydantic-evals 2.62 or newer
(was 2.57): 2.62 is the first release with every ModelAPIError category class the failure
table reads (ModelAuthenticationError, ModelNotFoundError and ModelRequestRejectedError
arrived in 2.62). Verified against 2.62.0 and 2.66.0.
- PydanticAIEmbedder's retry predicate is Failure.retryable: auth, not_found, quota and
internal are no longer retried.
- Generated.model is documented as the requested id (deprecated in favour of answered);
usage and finish_reason are derived from usage_v2 and finish. Their 1.22.0 values are
pinned by a golden test.
Added¶
- Tier table
schema_version = 2(ADR-0022):[providers.<name>](a registeredkind, and by name only abase_urlkey and asecretname),[models.<key>](call.models.ModelInfo: wire id, provider,supportsfrom the closedCapabilityvocabulary,thinkingstyle, limits that are None when unknown,slots,group,self_hosted,usd_per_requestas aDecimal),[policies.<name>](read-only), and per tierpolicy,fallbackandfallback_on. Every refusal isTableInvalid(ValueError)naming the dotted path. A table withoutschema_versionloads to the same values as 1.22.0, a[providers]or[policies]section in it included (ignored, as 1.22 ignored it, and named once per load by allm_table_ignoredWARNING).Tier.policyis read-only and hashable (nested tables read as mappings, arrays as tuples), so aTierhashes as it did in 1.22. NewTierfields (info,policy,fallback,fallback_on,table_fingerprint) andTierTablefields (model_info,providers,policies,schema_version) all default, so 1.x construction is unchanged. TierTable.fingerprint()andTierTable.require(role, *caps), which proves at import that a role's tier and every fallback tier state the capabilities it needs.- The capability pre-check in
call.generate: a request whose model does not statevision,webor structured output (json_schema, orjson_objectin its place) endsrequest_faultwith anunsupported: ...reason and no wire request. A schema 1 table is not checked, and onellm_call_warningper process says so; a role that overrides its tier's model id is named once. - On a schema 2 tier,
Role.model_settings()sends a provider-neutralthinking: bool. call.providers:ProviderFactory,ProviderRegistry(a duplicate kind raises naming both plugins; frozen after load),build(table, urls=, secrets=, deps=, factories=), which raises at boot for an unknown kind, an unresolved URL key or a secret that resolves to nothing, and returnsBackends(generator_for,chat_for,embedder_for,reranker_for,probe_for,aclose).buildemitsllm_backends_builtwith the fingerprint and the plugin origins; no secret value is logged.BuildDepsandBuilthold typed placeholders (policy,limiter,pricer,chat) for the resilience, result-contract and chat rows.adapters.pydantic_ai.OpenAICompatibleFactory, theopenai_compatiblekind (generator, embedder per model that statesembed, probe), andtranslate_settings, which spellsthinkingperThinkingStyle(unified,chat_template,none;nonewarns once per model asllm_call_warningwhen a role asked to think).ports.Probe,ports.Health,ports.HealthState;adapters.http.HttpProbe(GETone URL under a 5 s deadline:upwith the listed model ids,downwhen unreachable or an error status,unknownwhen the probe itself failed or could not send its request, the credential was refused (401/403, reasonauth: HTTP <status>), the endpoint is missing (404/405), or a 200 body is not a model list; never raises). AHealth.reasonnames an exception by its type only, never its message, and a secret no header can carry is refused before any request.adapters.pluggy:load_providersfor the second entry-point groupllm_ports.providers(hookregister_providers(register), markerprovider_hookimpl,builtin(name, *factories)), on the same loader as packs;load_packs(builtins=, pin=, strict=);PluginLoadError,PackLoadError,PluginStage;PackLoad.origin;packs_loaded(loads)returning acall.providers.Loaded; the eventplugins_loaded(group, names, origins); the conventional pin setting namesPACKS_PIN_ENVandPROVIDERS_PIN_ENV(the library reads no environment).market:DuplicateAction,ActionRegistry(sources=...)(asourceid outside it is refused),ActionRegistry.freeze(),.frozen,.origin(feature, action),add(..., origin=)andinclude(..., origin=);KillSwitch(..., registry=)checks the paused set against the registry on every parse;GATE_UNREAD;LastKnown.known.tests/contract/api_surface.json: the public__all__of every module, snapshotted; a name that disappears fails the suite.prompts.byte_sha(parts): sha256 over named byte strings, length-prefixed so a re-split, rename or reorder never collides. Pinned vectors freeze it.Pack.identity:byte_shaover the manifest's bytes and every file of the closed template set, by sorted path.Pack.files()lists that set;Pack.seed()builds the repoSeeda managed mirror copies, carryingcontent_sha,identityand the manifest's label in its config.PackManifest.includes,PackManifest.schema_version(onlyprompts.SCHEMA_VERSION= 1 loads), andlabel, the neutral alias oflangfuse_label(YAML key and read-only property).LangfusePromptSource(..., label_from_seed=True)opts in to serving and mirroring under a seed'sconfig["label"](the manifest labelPack.seed()carries) instead ofget'slabel.prompts.resolve(pack, registry=, import_roots=)andPack.resolved(...), returningResolvedPack: theoutput_schematype imported only from the caller's module roots, each grader looked up in aGraderRegistry(or a plain mapping), the golden path kept inside the pack directory, andoutput_schema_sha. Every miss isPromptPackInvalidnaming the field.prompts.schema_shahashes an output type's canonical JSON schema.prompts.scan(root, registry=, import_roots=)reports resolution failures beside load failures, andprompts.registry(root, registry=, import_roots=)raises on the first one, so a mistyped grader fails when the registry is built. Without the arguments both behave as in 1.22.0.prompts.served_shas(root, key="identity")serves identities forunevaluated.evals.golden.fire_pack(resolved, role=, generator_for=, ...): reads the pack's golden set (evals.golden.read_golden: a JSON list, an object withitems, or JSON lines), fires it, grades it on the pack's graders and returns aGoldenRunstamped withidentity,output_schema_sha, the role's tier and model, and the table fingerprint the caller passes. It never raises for a golden-set defect: no golden set, no graders, a missing file (the reason names the manifest value and the resolved path), an unreadable or empty file, or an item whose slots do not render each returncould_not_runwith the reason before anything fires, and emit agolden_runWARNING.evals.golden.resolve_packresolves withcall.graders.GRADERSbehind the caller's registry. Thegolden_itemevent gainsidentity.Outcomegains a seventh member,refused: the model or its provider declined (acontent_filterfinish, an explicitrefusal, pydantic-ai'sContentFilterError).ports.Failureandports.failure(...): the failure'skind(18 kinds,FailureKind), withoutcomeandretryablederived from it (OUTCOME_FOR,RETRYABLE), andretry_after_s,status,provider_code,phase,in_stream,reason. OnGenerated,EmbeddedandRerankedasfailure; the 1.xreasonis derived from it (ADR-0020).- The result contract (ADR-0019):
Usage(inclusive buckets:input,output,cache_read,cache_write,reasoning,requests,unmeasured,details; unknown is None),Cost(usdas aDecimalor None,source,priced_model,reason),Answered(who answered: the body'smodel, the gateway'sdeployment,response_id),Finish(unified,raw),CallWarning,Attempt.Generatedgainsusage_v2(summed over every request of the call),cost,finish,answered,attempts,warnings,failure;Embeddedgainsusage_v2,cost,answered,failure;Rerankedgainsanswered,failure. - The
PricerandGatewayReaderports, andPydanticAIGenerator(pricer=, gateway=, extra=, on_event=),PydanticAIEmbedder(pricer=, gateway=, on_event=). Cost precedence: the gateway's cost header, then the pricer, then unknown. A cost never fails a call: a refused cost header, a customGatewayReaderthat raises, or a sum that overflows leaves the gateway cost unknown and emitsllm_gateway_header_invalid. adapters.genai_prices.GenaiPricesPricer(new extra[prices],genai-prices>=0.2.3, also in[all]): prices a call from genai-prices' dated table, with per-requestoverridesfor a self-hosted model; an unknown model isCost(None, "unknown", reason="unknown_model").adapters.litellm_proxy(no extra, no vendor import): reads a LiteLLM proxy's cost, deployment and attempt headers (aGatewayReader), andclassify_gatewaynames a spent budget or quota 429quota(never retryable) while a parallel-limit 429 staysrate_limitedwith its wait. A cost header outside 0 or [COST_MIN_USD,COST_MAX_USD) = [1e-12, 1e7) dollars is implausible and treated as absent; a signed zero is 0.- The shared classification table
adapters._classify.classify_core(exported fromadapters.pydantic_aiwithWireFacts,Classifier,kind_for_status,retry_after_s,is_context_overflow,raised_in_provider,PROVIDER_MODULES,CALLER_CONFIG_ERRORS,worst,SEVERITY), used by the generator, the embedder and the HTTP reranker; andadapters.pydantic_ai.classify_failure,openai_finish,OPENAI_FINISH,parse_usage,WireReply. - Events:
llm_cost_unknown(once per model per process),llm_usage_unmeasured,llm_pricer_failed,llm_gateway_header_invalid,llm_call_internal_error. Thellm_call_failedevent carrieskind; the call's observation metadata carries the kind, the cost and its source, who answered, the unified finish, the attempt count and every usage bucket. testing.Reply.failed(kind, **detail): aStubGeneratorreply function returns it (or aports.failure(...)) to fail a call with that kind.testing.Replyis now aProtocolwith the same call shape, so any function that was aReplystill is. The stubs fillfailureandanswered;StubGenerator(model=)(defaulttesting.STUB_MODEL) is the model it reports as having answered, never the request's id.
Fixed¶
adapters.langfuse.tracer.usage_detailsno longer books an unreported side as 0: only the measured sides are sent.
[1.22.0] - 2026-11-11¶
Added¶
llm_ports.market, the declaration side of a capacity-driven work market (ADR-0018):ActionRegistrywith theaction(...)decorator andActionSpec, which refuse at import an action with none ofdesired_ttl,cadenceormax_silence_h(and a malformed interval, an unknown silence clock, a negative lane, a non-positive weight or cost);KillSwitch, the operator's pausedfeature.actionset read from a setting the caller names;GateCacheandLastKnown, cached control reads that serve the last known value on a failed read, withGateCache.admitusable ascall.Hooks.admit;ActionMirrorandsync_all, the store copy an operator tunes. Every component reports through an injectedon_eventhook.adapters.pluggy.load_packs(extra[market]): action packs from other distributions, listed under thellm_ports.marketentry-point group, validated by pluggy and joined all or nothing; each entry point endsok,failed(with its stage) orblocked.
[1.21.0] - 2026-11-09¶
Fixed¶
adapters.gepa.optimizeno longer spins on a dead reflection model. GEPA logs a failed reflection and starts the next iteration, soraise_on_exceptionnever sees it, and a caller whose evaluator caches spends no budget: a reflection model out of daily quota was called every few seconds for hours.max_reflection_failures(default 5) consecutive failedreflectcalls now stop the search asUnavailablenaming the last error, through GEPA's ownstop_callbacks.
Added¶
optimize(timeout_s=...): a wall-clock bound (GEPA'sTimeoutStopCondition);Optimized.stoppedis"timeout"when it ended the search, else"budget".
Changed¶
- Releasing is the version-bump PR alone:
tag-version.yml(ci-kit'sci-version-tag) tagsvX.Y.Zat the bump commit when it merges and dispatchesrelease.ymlon the tag; a daily run re-dispatches a tag whose release never ran and fails when a release left its GitHub release or PyPI files missing.release.ymlandci.ymlreadgithub.ref_typeinstead of the event, so a dispatched tag run is a release, not a dev upload, and the docs site deploys only from the newest release tag.
[1.20.0] - 2026-11-09¶
Added¶
adapters.gepa(extra[gepa], inall):optimize(seed, train, val, evaluate=, reflect=)runs GEPA reflective prompt evolution over a caller's batch evaluator (Rollout(output, score, feedback)per item) and reflection callable, and returnsOptimized | Insufficient | Unavailable.Optimized.improvedcompares the best candidate to the seed on the validation items. GEPA swallows a failed reflection, sooptimizecounts them: every call failing isUnavailable, some failing isOptimized.reflection_failures. GEPA's log lines are kept on the outcome, never printed.RolloutAdapteris theGEPAAdapterunderneath.adapters.optuna.dashboard_app(storage)/serve_dashboard(storage, host=, port=)(extra[optuna-dashboard]): optuna-dashboard over any Optuna storage, aSqlJournaltable included, on demand and bound to loopback by default.
[1.19.0] - 2026-11-08¶
Added¶
forecast.skill:brier_skill(obs, fct, ref, clusters=None), a voice's Brier skill over a per-event REFERENCE forecast (a symbol's base rate, a prior) on the pairs both scored, with a seeded cluster-bootstrap interval (events sharing a day's tape resample together).Score'sbrier_skill_scorestays skill over climatology. Returns aSkill(exported fromllm_ports.forecast). Standard library only, like the rest offorecast.
Fixed¶
release.yml: the PyPI upload usesskip-existing, and a follow-up step fails the job unless PyPI lists every built file. v1.18.0 shipped its wheel but not its sdist (PyPI refused the sdist's attestation with "Failed to refresh TUF"), and a rerun could not recover because it failed first on the wheel that had already uploaded. Now a rerun uploads only the missing files.
[1.18.0] - 2026-11-08¶
Added¶
forecast.score:mce(curve), the largest bin gap of a reliability curve, anddecompose(curve), Murphy's decomposition of the Brier score intoreliability,resolutionanduncertainty(aDecomposition, exported fromllm_ports.forecast). It is exact on a curve over cells that each forecast one rate. Consumers that hand-rolled the decomposition besidereliability()andece()can now take all three from the core.
[1.17.0] - 2026-11-08¶
Added¶
forecast.platt:PlattCalibrator, aCalibratorthat fitssigmoid(a * logit(p) + b)by ridge-stabilised Newton steps with the standard library, andPlatt, the fitted map (applied and stored as JSON without a fitting library). A fitted slopea <= 0inverts the voice's ranking, so the fit is refused (insufficient_.reasonstartsinverted slope) and the values are kept infitted_.
[1.16.0] - 2026-11-08¶
Added¶
evals.golden: a golden item may carryuser, and then fires as a system and user pair (the rendered pack as the system prompt,useras the prompt), the shape of a producer whose fixed instructions ride as the system for the prefix cache.slotsis optional for a pack with none.fire_one's result names thesystemit sent (empty for a slotted item, which still sends none, so a consumer facade with nosystemkeyword keeps working).
[1.15.0] - 2026-11-08¶
Seams for bigger ML engines (ADR-0016): every addition is optional, so no consumer changes.
Added¶
learn.Written.refandlearn.LedgerRow.ref: the id a backend gave what it wrote (not part of equality).MemoryRunLedgerandJsonlRunLedgermint a UUID per row and read it back, soRun.recorded.reflinks a run to whatever it produced.learn.Fit.artifact_uri: where a model too big formodellives;loop.runrecords it in the row aslearn_artifact_uri(reserved), only when set.learn.ModelRegistryport (register,set_alias,by_alias,get; versions from 1 per name, aliases move) withModelVersionandVersionRead, andlearn.registry:MemoryModelRegistryandJsonlModelRegistry(an append-only event log). Anartifact_uriorrun_refof the wrong type, or aversionthat is not anint(Written.ref's string passed back, abool), is arequest_fault, never a raise.testing.conformance:check_run_ledger,check_example_store,check_model_registryand their broken-storage checks (check_unavailable_run_ledger,check_unreadable_example_store,check_unavailable_model_registry), raisingContractViolationnaming the broken invariant. Every in-repo backend passes them.adapters.mlflow(new extra[mlflow]:mlflow-skinny, SQLAlchemy, Alembic):MlflowRunLedger(one MLflow run per row; scalar params and numeric metrics logged, the whole row as JSON split over run tags, read back from the search result with no artifact store) andMlflowModelRegistry(MLflow model versions and aliases; a version with noartifact_urigets a backing run as its source; aruns:/<id>/...URI passes its run id, as a server requires). The ledger matches a row's own kind, so prefixes whose experiment names collide never cross-read, and refuses a subject over 8000 characters (MLflow would cut its tag). Both pass the conformance suites against a realmlflow serverand a local SQLite store, write nothing to stdout or stderr, and are safe to share between threads. Fullmlflowis a dev dependency only, for the server tests.learn.Run.as_dict()(a run as plain fields, for an admin endpoint's response) andlearn.failed(run): red when nothing was measured (could_not_run), the ledger row did not land, orapplyraised;failandinsufficientstay green.- Guide
docs/guides/ml-engines.mdand exampleexamples/ml_engine_seams.py.
[1.14.0] - 2026-11-08¶
Added¶
adapters.optuna.SqlJournalandjournal_storage(engine, table, create=): an Optuna journal backend on one SQL table(n bigint primary key, log text)the caller owns, schema-qualified in every statement. Appends are gapless across processes (the primary key fails a racing append, which retries from the new end). For PostgreSQL behind a transaction-mode pooler, where Optuna's RDB storage cannot be pinned to a schema (its tables follow the session search path and its Alembic check reads the default schema).
[1.13.0] - 2026-11-08¶
Added¶
forecast.walk:Event(x,forecast_time,resolve_time,resolution) andwalk_forward, a combiner replayed in forecast-time order that hears each outcome only at its resolve time and returns one probability (None = abstained) per event;holdout_start(the time split) andcompare(several combiners scored on the same resolved, answered events).forecast.PriorPool: wraps anyCombinerwith a prior voice; shrinks the combined log-odds toward the prior's (shrink), extremizes (alpha), and answers with the prior while the inner combiner abstains.forecast.BestVoice: the best-single-voice baseline, online (lowest running Brier score among the voices that spoke, once each hasmin_noutcomes).adapters.optuna.tune(extra[optuna], also inforecast): a seeded TPE search over a combiner's parameters whose objective is the walk-forward Brier score on the train split (abstention scored as 0.5); the best parameters are replayed over every event and scored on the holdout only. ReturnsTuned,InsufficientorUnavailable, never raises; the study goes to any Optuna storage. Rationale: ADR-0015.
[1.12.0] - 2026-11-08¶
Added¶
LangfuseExampleStore.putrelays a person's label: ahumanexample withbynaming the person is written aslabel-<role>against the role's categorical score config (created when missing), keyed by the person, and the reply's annotation-queue item is markedCOMPLETED(labels.COMPLETED). A consumer's own label surface feeds the same store,coverageandagreementas the annotation UI. Rationale: ADR-0014.
Changed¶
putrefuses ahumanlabel with an emptyby(was: everyhumanlabel) as arequest_fault; anoutcomelabel is still refused.
[1.11.0] - 2026-11-07¶
Added¶
llm_ports.forecast, standard library only:ForecastandResolved(Metaculus / Good Judgment, schema.orgClaim/ClaimReviewand W3C PROV field names;resolutionisyes/no/annulled);score(obs, fct)(brier_score,log_score,brier_skill_score,overconfidence; scoringrules' names and argument order),reliabilityandece; theCalibratorport (the scikit-learn estimator protocol:fit(X, y),predict_proba(X),get_params/set_params) withInsufficientandPiecewiseLinear(a fitted monotone map, JSON round-trip);HedgeMap, a scikit-learn estimator for the probability each source's hedge word has earned (beta-binomial shrinkage toward the word's pooled rate); theCombinerport (River'slearn_one(x, y)/predict_proba_one(x)) andLogitPool(equal-weight log-odds pool, optional extremizing). Aforecast-standaloneimport contract keeps it liftable. Rationale: ADR-0013.- Adapters, one extra each and grouped as
forecast(not inall):adapters.scoringrules(brier_score,interval_score,crps_normal),adapters.sklearn.IsotonicCalibrator(optionalsample_weight),adapters.venn_abers.VennAbersCalibrator(small-n, with a(p0, p1)bracket frompredict_interval) andadapters.river.RiverLogitPool(online logistic logit pool with silent-voice flags). - The forecasts guide and
examples/forecasts.py.
Changed¶
- The coverage floor rises to 97.9%.
[1.10.0] - 2026-11-07¶
Added¶
llm_ports.evals.calibration.trustandTrust: whether a judge's verdicts may act, from its agreement with human labels (at leastMIN_LABELLEDpairs, kappa or its interval's lower end atMIN_KAPPA) and its seeded-bad replay. Never raises; a judge with no labels, or a replay row it cannot read, is untrusted. Rationale: ADR-0012.llm_ports.learn.stats.kappa_ci(Fleiss-Cohen-Everitt 95% interval,Z95) andAgreement.kappa_low/kappa_high, filled byexamples.agreement.llm_ports.learn.examples.pairs: per topic, the(measured, reference)pairs behindagreement, optionally for one labeller (by, one judge model), so a caller pools them.- The trace port's
scoretakes an optionalmetadata, stored on the score (Langfuse adapter,PortObservation,RecordingTracer.score_metadata), so a judge label records its model. - The learning-loops guide maps predict / resolve / score / consume onto the loop stages and
shows a judge earning trust;
examples/learning_loop.pyruns the consume step.
[1.9.1] - 2026-11-07¶
Fixed¶
llm_ports.adapters.langfuse.labels: label score names arelabel-<role>,label.model-<role>andlabel.judge-<role>. The:spelling is not a valid Langfuse score-config name, so the firstenqueuefor any role failed with a 400 and created no queue. Reads accept both spellings, so scores already written underlabel.judge:<role>stay readable.- A role whose label name Langfuse would refuse is now a
request_faultfromenqueueandputbefore any write (newCONFIG_NAME), and anunavailablereason carries the server's own validation message instead of only the status code. examplessplits its score-name filter intoNAMES_PER_READnames per request, so the query string stays bounded as roles grow. Rationale: ADR-0011.
[1.9.0] - 2026-11-07¶
Added¶
llm_ports.adapters.langfuse.experiments.seedand its resultSeeded: write a golden set into a Langfuse dataset, creating it when missing. An item's id is the caller's orinput_id(input)(a SHA-256 of the canonical JSON), and Langfuse upserts by id, so a re-seed rewrites rather than duplicates. A failed write is aninvalidoutcome naming the shortfall; an unreadable dataset isunavailable. Never raises.
1.8.0 - 2026-11-07¶
Added¶
- Two built-in call graders in
llm_ports.call.graders, for research (web-fetching) tiers:cites_urlcounts the distinct http(s) URLs in the reply's string fields and passes at one or more;no_refusalcounts stock ungrounded-refusal phrases ("I don't have real-time access", "as of my training", "please consult ...") and passes at zero. Both are inGRADERS, so a role names them by string. WEB_ONLY_GRADERS({"cites_url"}):graders_for_inputsdrops them for a call withoutweb=True, the mirror ofTEXT_ONLY_GRADERS, so a text-only call is never failed for citing no pages.
1.7.0 - 2026-11-07¶
Added¶
llm_ports.evals.labels.LabelQueue.queued(topic)and its result typeQueued(pending,done,seen, a named outcome): what a topic's review queue already holds, so a scheduled draw can skip it and bound the backlog.LangfuseExampleStore.queuedreads the role's annotation queue items (paged); a role with no queue reads empty and creates nothing.
Changed¶
LangfuseExampleStore.enqueueskips a reply the queue already holds (pending or done) and returns how many were newly queued, so re-running the same deterministic draw adds no duplicate items. A customLabelQueueimplementation now also needsqueued.
1.6.0 - 2026-11-07¶
Added¶
llm_ports.evals.labels: labelled model replies on the learning-loop core. A label is alearn.Exampleon topicllm:<role>(topic,role_of);drawpicks which unlabelled production replies to queue for a human, closing each role'slearn.examples.coveragegap and stratified by the judge's verdict;label_ofturns anyJudge's verdict into amodelorjudgeexample (neverhuman);LabelQueueis the port a review surface implements.llm_ports.adapters.langfuse.labels.LangfuseExampleStore: alearn.WritableExampleStoreand aLabelQueueover Langfuse scores (label:<role>human, given in the annotation UI;label.model:<role>andlabel.judge:<role>written byputunder a stable id) and one annotation queue per role, created on first use.llm_ports.adapters.langfuse.experiments:runandrecord_rowsrecord a graded evaluation as a Langfuse dataset experiment; a run that graded fewer items than it was handed isinvalid.
1.5.0 - 2026-11-07¶
Added¶
llm_ports.learn: the learning-loop core, standard library only (ADR-0010).Clause(with arole:bar,floor,veto),Verdict(pass | fail | insufficient | could_not_run, every clause kept,blocking),Example(a label from ahuman,model,judgeoroutcome) andLedgerRow.gate:at_least,at_most,equals,check,decide(gate, clauses, unevaluated=),could_not_run.loop.run(loop, ledger)drives aLoop'sevidence,fit,clausesandapplyand records exactly one row per run underlearn_verdict, never raising;loop.recordrecords a gate run outside a loop.examples:coverage,agreement,few_shot.stats:cohen_kappa,gwet_ac1,bh_fdr(Benjamini-Yekutieli by default).stores:MemoryRunLedger,JsonlRunLedger,MemoryExampleStore,JsonlExampleStore,union. Guidedocs/guides/learning-loops.md, exampleexamples/learning_loop.py.
Changed¶
evals.calibration.cohen_kappaandgwet_ac1now live inlearn.statsand are re-exported unchanged.
1.4.0 - 2026-11-06¶
Added¶
llm_ports.evals.judge.JudgeLaneandSample: run a sampled judge as background tasks off the caller's await path.start(role, prompt, result, mod=, run=, eligible=, seen=)draws the reply (should_sample), reads what the call saw, and schedules the consumer'srun(sample)in a freshcontextvars.Context, holding a strong reference until it finishes;pending()anddrain()serve shutdown and tests. Events<name>_sampled|_seen_failed|_sampler_failedgo toon_event(the stdlib logger by default). Merged intoevals.judge: there is still one judge module.llm_ports.evals.golden: golden-set evaluation of a prompt pack.fire(items, role=, pack=, source=, output=, generator_for=, generate=)renders each item's slots through the pack and fires it through the call path (never raises for a model or wire failure);rows_of,monitors,grade(rows, role=, extra=)(the role's call graders plus extra row graders on oneEvaluatorboard) andcitations_resolve(cited, known).llm_ports.evals.calibration: calibrate a judge in both directions.cohen_kappaandgwet_ac1over binary(judge, human)pairs (None where undefined);seed_placeholderandseed_figurebuild seeded-bad replies a judge must fail;tallyandverdictread a replay of good samples beside their seeded copies (exit codesCALIBRATED,FINDING,BLIND).llm_ports.adapters.pydantic_ai.replay.recording(make)andrecorder(make): a generator plus the list (or dict) its raw attempts land in, so a golden run records every attempt.
1.3.0 - 2026-11-06¶
Added¶
llm_ports.grounding.numbers.shorten(text, width, placeholder="…"): cut a text for display on a word boundary, never inside a digit run. When the first word alone is too long the hard cut steps back to the start of any number it would split, so a shortened "25bp" can never read "2…". Every digit run in the result is a whole digit run of the input. Pure, no I/O.
1.2.0 - 2026-11-06¶
Added¶
llm_ports.call.resilience:RetryPolicy(tries per request, backoff start and cap) andFailureBudget(a long run's per-item outcomes;exhaustedaftermax_consecutivefailures in a row, or a failure rate abovemax_rateoncemin_attemptsitems were tried). Pure, no I/O.PydanticAIEmbedder(retry=RetryPolicy(...))re-sends ONE request that failedunavailable(timeout, refused, 5xx) with exponential backoff (tenacity, now in thepydantic-aiextra); the batches that landed are not redone, andrequest_fault/gatedare never retried. Default None: each request is sent once, as in 1.1.0.Embedded.retries: the requests re-sent during the call, on success and failure alike.
1.1.0 - 2026-11-04¶
Added¶
openai_compatibletakesprofile=(a pydantic-aiModelProfileSpec) andsettings=(model-levelModelSettings) and passes both to theOpenAIChatModelit yields, so a caller can move an agent whose output type it does not construct onto native structured output without rebuilding the model on the builder's client. Both default to None; with neither given the model is built exactly as in 1.0.1.
1.0.1 - 2026-11-03¶
Fixed¶
openai_compatibleandopenai_compatible_embeddingshand pydantic-ai's OpenAI provider anhttpx2.AsyncClient, so pydantic-ai 2.65'sPydanticAIDeprecationWarning(a legacyhttpx.AsyncClienton an OpenAI-compatible provider, removed in pydantic-ai 3) no longer fires on every call.transport=still takes a legacy httpx transport, bridged onto the httpx2 client; it now also takes an httpx2 one.classifymaps anhttpx2.HTTPStatusError. (ADR-0009)
Changed¶
- The
pydantic-aiextra nameshttpx2>=2.12(already installed by the OpenAI SDK). The default transport under both builders is httpx2's, which verifies TLS against the OS trust store instead ofcertifi. - pytest fails on a
DeprecationWarningorUserWarningraised at a call this package makes.
1.0.0 - 2026-11-03¶
The first stable release: the API of 1.0.0rc1, unchanged. From here the public surface (every
name without a leading underscore, __all__ per package) is covered by semantic versioning.
Changed¶
- Development status is Production/Stable; the README installs without
--pre. - No change was needed here to adopt rc1 in its first consumer (mjdevaccount/trade-data#2755).
1.0.0rc1 - 2026-11-03¶
Added¶
- A documentation site (mkdocs-material + mkdocstrings,
mkdocs.yml): concepts, a guide per example, an API reference page per subpackage generated from the docstrings, the ADRs and this changelog. The newdocsCI job builds it with--strict; a release tag deploys it to GitHub Pages. examples/: named outcomes as a fallbackmatch, grounded prose with slots, a prompt pack, the print gate, a judge and an adversary, chunk/embed/score, OpenTelemetry spans (all network-free), and an OpenAI-compatible endpoint (run when its env vars are set). Each asserts its claims;tests/examples/runs them, and every Python block in the README and docs, on everypytestleg.- Strict typing:
mypy --strictover the package andpyright --verifytypesat 100% (the public surface fully typed), run by the newtypesCI job.CaptureLikemembers are read-only properties, so any capture with those attributes satisfies it;StubEmbedderandStubRerankertakefail: Outcome;PackManifest.model_configis aClassVar. - CI:
vulture(dead code) inlint; thecore-onlyjob (the core with no extra, then withhttponly). - Releases publish to PyPI through trusted publishing (no stored token), with PEP 740
attestations; a tag still attaches the wheel and sdist to a GitHub release, independently of the
upload. A manual run of
release.ymluploads a.devNpre-release as the dry run. - Package metadata: project URLs, classifiers and keywords.
- CI:
actionlintover the workflows;ci.ymlis callable (workflow_call), and a release runs all of it before building.
Changed¶
- The README is rewritten for a first reader: the one idea, a 30-line quickstart that needs no network or key (run in CI), the concepts with their ADRs, and the layer diagram.
- Breaking: the package is layered (
src/llm_ports/, PyPA src layout). The 33 flat modules move into subpackages, bottom-up:ports(withports.outcomes, wasfailure) <observability(context,observation,NoopTracer) |grounding(numbers,slots, the belt,NumberLawGrounder) |prompts(packs,manifest,RepoPromptSource) <call(generate,call.roles,call.graders,call.schema) <evals(GauntletEvaluator,evals.judge,evals.refute) |rag(rag.chunking,rag.metrics) |provenance(wascontent) <adapters(pydantic_ai,pydantic_evals,langfuse,otel,http,lettucedetect,sqlglot) |testing(testing.langfuse).referenceis dissolved into the packages above. The root keeps its names. The map isdocs/REDESIGN.md§1. - Every module declares
__all__, sorted (ruffRUF022); a test asserts each name resolves. - import-linter contracts (
pyproject.toml) hold the layers, keep adapters independent and keep vendor imports insideadaptersandtesting;tests/contract/runs them on every leg. - Tests mirror the package; fixtures live in
tests/fixtures/. - Breaking: every framework is an extra.
pip install llm-portsinstalls pydantic, jinja2 and pyyaml; pydantic-ai and httpx move to thepydantic-aiandhttpextras (all= every extra butlettucedetect). An adapter imported without its extra fails with the install line.tests/contract/test_core_imports.pyand thecore-onlyCI job hold the core vendor-free. - Breaking: the tier key
boxis renamedself_hosted(Tier.self_hosted,GenRequest.self_hosted), default true.GenRequest.labeldefaults to"chat". - Breaking: names moved to the layer that owns them.
outcome_for_statusandREQUEST_FAULT->ports.outcomes;strict_json_schema->adapters.pydantic_ai;EventHook(and the newlog_events) ->observability, socallno longer exportsEventHookorObservation;IndexOrderedTransport/INDEXED_LISTSare imported fromadapters.http(oradapters.pydantic_ai), the moduleindex_orderis private; the Protocols live inports._protocols, still exported fromports. - Breaking: helpers that were never the contract are private:
capture,record_response,Capture(CaptureLikestays public),call.trace_output, the prompt source'skind/wire/served. - Breaking:
evals.judge.rubrictakes no domain vocabulary. The number law's examples are the consumer's, passed asexamples=numbers.LawExamples(paraphrase=..., fact_names=...); with the 0.12 examples the text is byte-identical to 0.12 (the golden test). load_tierstakes a mapping as well as a path; checks and errors name it<mapping>.__version__is read from the installed metadata.- Logger names follow the modules (
llm_ports.adapters.langfuse.tracer,.prompt_source,llm_ports.call). - Long functions split on merit (
call.generate, the pydantic-ai generator); magic numbers named (CHAIN_LIMIT,DEFAULT_TIMEOUT_S,SYSTEM_CAP,SCHEMA_CAP). uv.lockrefreshed to the latest releases (pydantic-ai-slim and pydantic-evals 2.65.0).
Deprecated¶
- The tier key
box: a table that still sets it loads, with aDeprecationWarningpointing at theload_tierscaller; a tier setting bothboxandself_hostedis refused.
Fixed¶
- The
pydantic-aiextra names httpx: the adapter builds an httpx client, and from pydantic-ai 2.6x / openai 3.4x neither depends on httpx (both moved to httpx2). The run's usage is read as the property pydantic-ai made it, so 2.65 no longer warns onusage().
0.12.0 - 2026-11-03¶
No behaviour change and no public name change; the release carries documentation, tooling and dependency floors.
Added¶
numbers.law_text(mode, *, web): the number law's one English statement, beside the check it states (numbers.loose).judge_sampling.rubriccomposes it; its output is byte-identical to 0.11.0 for every grounding mode and web flag, pinned bytests/test_rubric_golden.py.- Architecture decision records
docs/adr/0001–0008(MADR), holding the rationale and the measured evidence that used to live in docstrings. CONTRIBUTING.md: the docstring rules, where history goes, how to release.- A
pre-commithook that rejects deployment vocabulary inllm_ports/, run in CI. - CI: pytest on Python 3.12, 3.13, 3.14 and 3.15 (Ubuntu) and 3.13 (Windows); the suite at every
dependency's declared lower bound (
--resolution lowest-direct); a branch-coverage floor with Codecov upload over OIDC;ruff format --check;twine check --stricton the build;zizmorover the workflows; every workflow defaults topermissions: {}.
Changed¶
- Every docstring states the contract in Google style, with invariants in an
Invariants:section; ruff'sDrules (Google convention) enforce the layout. - The tree is formatted with
ruff format. - Dependency floors match what is tested:
pydantic>=2.12(already required bypydantic-ai-slim>=2.57) and, for thesqlextra,sqlglot>=30.22. Older sqlglot dropsLIMIT ALLat parse time, sosql_guard.guardcapped the query instead of refusing it.
Fixed¶
content.print_gateandDataClass.allowsdocument theValueErrorthey raise for an audience outsideAUDIENCES; the old text saidprint_gatenever raised.
0.11.0 - 2026-11-03¶
Added¶
ports.Reranker, the tenth port, andReranked(scores in input order, order-only; never raises for a model, wire or gate failure). Also exported from the package root.reranker_http:HttpRerankerover the Cohere/Jina-shaped/rerankwire,payload,scores_from. Every document is scored exactly once (indices exactly0..n-1) and every score is finite, or the result isinvalid.observation:observe(tracer, ...)opens one generation around a block and ends it once (ERRORandlatency_mson an exception);PortObservation(wascall._PortObservation) withusage_counts()andfail(exc, metadata);opt_int. A call that reported neither usage count stampstokens_unmeasured: true.call.tracer_traceis built onobserve.langfuse_tracer(extralangfuse):LangfuseTracer(client, ..., on_event=),LangfuseGeneration,usage_details,trace_id.langfuse_prompt_source:LangfusePromptSource(client, seed_for, ..., on_event=); the repo leads and the label mirrors. Duck-typed client, no SDK import.otel_gen_ai(extraotel):gen_ai_span,record_gen_ai(back-dated),GenAiRecorder; attribute names fromopentelemetry-semantic-conventions, emittinggen_ai.provider.namebeside the deprecatedgen_ai.system.sql_guard(extrasql):guard(sql, policy),SqlGuardPolicy(withdialect, default postgres),SqlGuardError,FORBIDDEN_NODES,POSTGRES_DENY_FUNCTIONS.testing.StubReranker;testing_langfuse(extralangfuse):LangfuseRig,observed,FakeLangfusePrompts,RefusingLangfuse.
Changed¶
index_order.IndexOrderedTransportmoved out ofpydantic_ai_embedder(still importable there, the same object) and also orders/rerankreplies'results.
0.10.0 - 2026-11-03¶
Added¶
pydantic_evals_judge(extraevals):PydanticEvalsJudge(role, model=|model_for=),JUDGE_PROFILE,clean_reason,verdict_of,judge_inputs.judge_sampling:Seen,seen_for,schema_of,judge_input,reply_text,sample_key,selected,rubric(role, seen, *, platform, mode, today, tz_label),verdict_label.lettucedetect_grounder(extralettucedetect):LettuceDetectGrounder,load_detector,Detector,MODEL,MAX_CONTEXT_CHARS,WINDOW_REASON.grounder_http:HttpGrounder,GrounderUnavailable,payload.claim_refute:refute_slots,refute_prompt(pack, ...),refute(generate, role, *, pack, ...),RefuteClaim,GenerateFn.json_schema:research_schema,RESEARCH_ANALYSIS,ANALYSIS_KEY,without_analysis,research(generate, role, ...).testing.StubDetector.
0.9.0 - 2026-11-03¶
Added¶
chunking:chunk_text,break_floor,max_chars_for_chunks,content_hash,should_ingest,CHUNK_CHARS,CHUNK_OVERLAP.retrieval_metrics:first_rank,hit_at_k,reciprocal_rank,agree_at_k,mean,clears.reference:GauntletEvaluator,NumberLawGrounder,RepoPromptSource,NoopTracer,NoopGeneration.pydantic_ai_replay:Recorder,replay,Recording,key,NotRecorded.testing.StubCiter,testing.StubPromptSource;ports.Seed(also exported from the root).call_graders.TEXT_ONLY_GRADERS,graders_for_inputs.
Changed¶
call.generatedropsTEXT_ONLY_GRADERSfor a call withimages=or the newweb=True, so an image or a web tier no longer failsnumbers_from_inputsby construction. With notracehook the call observes intotracer_trace(NoopTracer()), the package's one null sink.
0.8.1 - 2026-11-03¶
Fixed¶
IndexOrderedTransportre-sorts an/embeddingsreply'sdatabyindex. pydantic-ai dropsindex, so a server answering out of order put every vector on its neighbour's row, which the count check cannot see.openai_compatible_embeddingsalways wraps its transport with it.
0.8.0 - 2026-11-03¶
Added¶
ports.Embedder, the ninth port, andEmbedded(vectors, outcome, stamp, reason, input_tokens): one vector per text in one named space;input_tokens=Nonewhen the wire reported no usage.pydantic_ai_embedder.PydanticAIEmbedderon pydantic-ai'sOpenAIEmbeddingModel: count and dimension checks (a short batch or a wrong dimension isinvalid, never a partial result), batching, an asymmetric query prefix, and the generator's never-raise outcome classification.openai_compatible_embeddings(base_url, model_name, key, transport=).testing.StubEmbedder: deterministic hashed unit vectors.
0.7.0 - 2026-11-03¶
Added¶
content:ContentItem(prov:Entity),Activity(prov:Activity),DataClass(rightspublic < owned < licensed < scraped,private,link_required;jointakes the most restrictive),CEILINGper audience, andprint_gate(item, audience) -> GateVerdict: ok, or one named refusal (rights,unclaimed,uncited,unsourced,link).
0.6.0 - 2026-10-31¶
Added¶
slots.to_prose(text): context a digit-free grammar can read with no figure to reach for. Every digit run is removed whole (numbers.DIGITS), every character outsidePROSE_CHARSbecomes a space.
Fixed¶
__version__matches the package version (it said 0.3.0).
0.5.0 - 2026-10-31¶
Added¶
call_graders.nullable_paths(model): the nullable leaf paths of a pydantic model.
Changed¶
call_graders.schema_completereads a None on a nullable pydantic field as an answer; a None on a required field still fails, and a dict value is graded as before.
0.4.0 - 2026-10-29¶
Added¶
numbers.loose(text, allowed, words_ok): the number law in one function. Digit runs by value not inallowed, then spelled numbers and number-word tails not inwords_ok, in order and named.grounding.loose,slots.loose;grounding.beltandslots.beltcountloose.
Changed¶
numbers.uncitedisloosesorted into the cited-spans shape (same results on every pinned case).call_graders.numbers_from_inputsscores the distinct figures ofloose, so spelled numbers count; the prompt's own words and a%license them.
0.3.0 - 2026-10-28¶
Added¶
slots: grounded prose by construction.PROSE_CHARS,prose_pattern,slot_pattern(the digit-free grammar),SLOT,ID,cited,fill,problems(unknown slot, stray brace, cited twice),beltover filled text.numbers.canonandnumbers.digits: digit runs compare by value ("2.20" == "2.2").
Changed¶
grounding.beltcompares digit runs by value.
0.2.0 - 2026-10-23¶
Added¶
grounding:Problem(path, reason, detail=""),describe,retry_message,belt(text, allowed, words_ok, *, strip_links),allowed_numbers(texts).failure_reason(exc, *, with_cause=False): an optional<- <cause reason>tail, cut by the same body-safe rule and withinREASON_MAX.
0.1.1 - 2026-10-22¶
Added¶
py.typed(PEP 561).- A release workflow: a
v*tag builds the wheel and sdist from the tagged commit, refuses a tag that does not match the package version, and attaches both to a GitHub release.
0.1.0 - 2026-10-22¶
Added¶
- The ports (Protocols and frozen value types), roles over a consumer's tier table, prompt packs
named by content sha, per-call graders,
generate()with named outcomes andHooks, the pydantic-aiGeneratoradapter, and test doubles.