llm_ports.learn¶
llm_ports.learn
¶
The learning-loop core: labelled examples, one gate, one run ledger, one loop runner.
Every learning loop is the same moves: read evidence, fit or measure, hold the result to a gate, and record the run. This package holds the parts that do not depend on what is learned:
- values:
Example,Clause,Verdict,LedgerRow(and the store results). gate: clause helpers anddecide, the one way a verdict is reached.stats: agreement (cohen_kappa,gwet_ac1) and false-discovery control (bh_fdr).examples:coverage,agreement,few_shotover labelled examples.- ports:
RunLedger,ExampleStore,WritableExampleStore,ModelRegistry,Loop;storesandregistryimplement them with the standard library;loop.rundrives aLoop(evidence, fit, clauses, apply) and records exactly one row;failedsays whether a run reads red.
Invariants
- Standard library only: imports nothing else from
llm_portsand no third-party package, so it can be lifted into any codebase (enforced by an import-linter contract).
Rationale: ADR-0010, ADR-0016 (the seams to bigger engines).
ExampleStore
¶
Bases: Protocol
Labelled examples, read by topic.
examples
async
¶
examples(
topics: Sequence[str],
*,
since: datetime | None = None,
source: Source | None = None
) -> ExamplesRead
Read the examples of topics, labelled at or after since, from source or any source.
Returns:
-
ExamplesRead–The examples, or an
unavailableoutcome; an unreadable store never reads as empty.
Fit
dataclass
¶
Fit(
model: Mapping[str, Any] = dict[str, Any](),
metrics: Mapping[str, Any] = dict[str, Any](),
artifact_uri: str | None = None,
)
What a loop's fit stage produced.
Attributes:
-
model(Mapping[str, Any]) –The fitted parameters a consumer applies (a weight, a threshold, coefficients); JSON-able.
{}for a loop that measures without fitting. -
metrics(Mapping[str, Any]) –What the fit measured (sample size, error, lift); JSON-able.
-
artifact_uri(str | None) –Where a model too big for
modelwas written (a file, a bucket, a tracker's artifact store); None whenmodelis the whole model.loop.runrecords it in the ledger row.
Invariants
- Plain data, so a fit made in a worker process pickles back whole.
Loop
¶
Bases: Protocol
One learning loop: read evidence, fit, say which clauses the fit must clear, act on it.
loop.run drives the stages, decides the verdict with gate.decide, calls apply and
records one ledger row, so a loop holds only what is specific to it. A loop whose verdict
needs a policy other than the default (unevaluated) sets the unevaluated attribute.
subject
property
¶
subject: str
What it learns about (a symbol, a role, a model); "" for one subject.
evidence
async
¶
evidence() -> E
Read the evidence (examples, outcomes, series). May raise; run records the failure.
fit
¶
fit(evidence: E) -> Fit
Fit or measure. May raise; run records the failure.
Pure (no I/O, no shared state), so a heavier engine can ship it to a remote worker.
clauses
¶
Return the gate's evaluated clauses for this fit. May raise; run records the failure.
apply
async
¶
Act on the verdict: store the fit on pass, keep the incumbent otherwise, or nothing.
Called once per run that reached a decision (never on could_not_run), before the run
is recorded, so the row says what was done.
Returns:
-
str–One word for the record (
stored,held,noop, ...). May raise;runrecords -
str–apply_failedand keeps the verdict.
ModelRegistry
¶
Bases: Protocol
Numbered versions of fitted models, with named aliases (champion) pointing at one each.
Mirrors a tracker's model registry (versions from 1 per name; an alias moves, a version never changes), so a backend is a thin mapping.
register
async
¶
register(
name: str,
*,
model: Mapping[str, Any],
artifact_uri: str | None = None,
run_ref: str | None = None,
tags: Mapping[str, str] | None = None
) -> Written
Register a new version of name (created on first use).
Parameters:
-
name(str) –The registered model's name.
-
model(Mapping[str, Any]) –The fitted parameters; JSON-able.
-
artifact_uri(str | None, default:None) –Where a model too big for
modellives; None whenmodelis whole. -
run_ref(str | None, default:None) –The ledger row that produced it (
Written.ref). -
tags(Mapping[str, str] | None, default:None) –Free string labels.
Returns:
by_alias
async
¶
by_alias(name: str, alias: str) -> VersionRead
Read the version alias of name points at.
Returns:
-
VersionRead–The version,
okwith None when the model or alias does not exist, orunavailable.
get
async
¶
get(name: str, version: int) -> VersionRead
Read one version of name.
Returns:
-
VersionRead–The version,
okwith None when it does not exist,request_faultwhenversionis -
VersionRead–not an
int(Written.refis its decimal string), orunavailable.
RunLedger
¶
Bases: Protocol
The record of every run: what ran, on what, with which knobs, and what it decided.
append
async
¶
append(
kind: str,
*,
subject: str,
params: Mapping[str, Any],
metrics: Mapping[str, Any]
) -> Written
Record one run.
Parameters:
-
kind(str) –The loop's run kind.
-
subject(str) –What the run was about; "" for a loop with one subject.
-
params(Mapping[str, Any]) –The knobs the run used; JSON-able.
-
metrics(Mapping[str, Any]) –What it measured; JSON-able. A gated run carries its verdict under
VERDICT_KEY.
Returns:
latest
async
¶
latest(
kind: str,
*,
subject: str | None = None,
decisions: Sequence[Decision] | None = None
) -> RowRead
Read the newest row of one kind.
Parameters:
-
kind(str) –The run kind.
-
subject(str | None, default:None) –Only rows about this subject; None for any.
-
decisions(Sequence[Decision] | None, default:None) –Only rows whose recorded verdict's decision is one of these (a row with no verdict never matches); None for any row. A gate's decisive read passes
("pass", "fail"), so a latercould_not_rundoes not close what a decisive run opened.
Returns:
-
RowRead–The row (None when there is none), or an
unavailableoutcome.
WritableExampleStore
¶
Bases: ExampleStore, Protocol
A store that also accepts examples written by code.
Clause
dataclass
¶
Clause(
name: str,
passed: bool | None,
value: Value = None,
threshold: Value = None,
op: str = "",
role: Role = "bar",
)
One condition of a gate, evaluated.
Attributes:
-
name(str) –The clause's name, unique within its gate (
n_at_least_200,kappa, ...). -
passed(bool | None) –True or False when evaluated; None when it could not be (missing evidence).
-
value(Value) –The measured value; None when it could not be computed.
-
threshold(Value) –The bar it was held to; None for a boolean check.
-
op(str) –How
valuewas compared tothreshold(>=,<=,==); "" for a boolean check. -
role(Role) –What failing it means:
bar(default),floororveto(seeRole). The gate sets it when it evaluates the clause, so one condition may be avetoin one branch and afloorin another.
Invariants
- A None
passedis never read as a pass (gate.decidefails it closed).
Example
dataclass
¶
Example(
topic: str,
subject: str,
value: str | float,
source: Source,
by: str = "",
at: datetime | None = None,
reason: str = "",
ref: Mapping[str, str] = dict[str, str](),
)
One labelled example: something a loop learns from, with where it came from.
Attributes:
-
topic(str) –What is learned (
llm:summary,trend:1d,fit:calibration). -
subject(str) –The thing labelled, unique within the topic (an observation id,
item@day). -
value(str | float) –The label: a class (
good,bad) or a number (a forward return). -
source(Source) –human,model,judgeoroutcome. -
by(str) –Who labelled: a person's id, a model's name, a grader's name; "" when unknown.
-
at(datetime | None) –When the label was given; None when the store does not say.
-
reason(str) –The labeller's reason; "" when none was given.
-
ref(Mapping[str, str]) –Provenance: trace id, table and key, code version. Not part of equality.
Invariants
- A
modelorjudgelabel is measured againsthumanlabels, never counted as one.
ExamplesRead
dataclass
¶
ExamplesRead(
examples: tuple[Example, ...],
outcome: StoreOutcome,
reason: str | None = None,
)
What ExampleStore.examples returns.
Attributes:
-
examples(tuple[Example, ...]) –Every example read;
()exactly whenoutcome != "ok". -
outcome(StoreOutcome) –ok, orunavailablewhen the store could not be read. -
reason(str | None) –One line naming the failure; None on
ok.
Invariants
- A store that could not be read never reads as a store with no examples.
LedgerRow
dataclass
¶
LedgerRow(
kind: str,
subject: str,
params: Mapping[str, Any],
metrics: Mapping[str, Any],
at: datetime,
ref: str | None = None,
)
One recorded run of a loop.
Attributes:
-
kind(str) –The loop's run kind.
-
subject(str) –What the run was about; "" when the loop has one subject.
-
params(Mapping[str, Any]) –The knobs the run used.
-
metrics(Mapping[str, Any]) –What it measured; a gated run carries
Verdict.as_dictunderVERDICT_KEY. -
at(datetime) –When it was recorded.
-
ref(str | None) –The id the ledger gave the row (
Written.refof its append); None when the ledger mints none. Not part of equality.
ModelVersion
dataclass
¶
ModelVersion(
name: str,
version: int,
model: Mapping[str, Any],
artifact_uri: str | None,
run_ref: str | None,
at: datetime,
tags: Mapping[str, str] = dict[str, str](),
)
One registered version of a model: what a loop fitted, kept so it can be promoted.
Attributes:
-
name(str) –The registered model's name.
-
version(int) –Its number within
name, from 1, increasing by one per registration. -
model(Mapping[str, Any]) –The fitted parameters (
Fit.model); JSON-able. -
artifact_uri(str | None) –Where a model too big for
modellives (a file, a bucket, a tracker's artifact store); None whenmodelis the whole model. -
run_ref(str | None) –The ledger row that produced it (
Written.ref); None when unknown. -
at(datetime) –When it was registered.
-
tags(Mapping[str, str]) –Free labels; string to string.
RowRead
dataclass
¶
RowRead(
row: LedgerRow | None,
outcome: StoreOutcome,
reason: str | None = None,
)
What RunLedger.latest returns.
Attributes:
-
row(LedgerRow | None) –The newest matching row; None when there is none or the ledger could not be read.
-
outcome(StoreOutcome) –ok, orunavailablewhen the ledger could not be read. -
reason(str | None) –One line naming the failure; None on
ok.
Invariants
- "No row yet" (
ok, None) is distinct from "could not read" (unavailable).
Verdict
dataclass
¶
Verdict(
gate: str,
subject: str,
decision: Decision,
clauses: tuple[Clause, ...] = (),
blocker: str | None = None,
reason: str = "",
)
What a gate decided about one subject, with every clause it read.
Attributes:
-
gate(str) –The gate's name.
-
subject(str) –What was judged (a model, a symbol, a role); "" when the gate has one subject.
-
decision(Decision) –pass,fail,insufficientorcould_not_run. -
clauses(tuple[Clause, ...]) –Every clause, in the order the gate evaluated them.
-
blocker(str | None) –The first clause, in order, that set the decision; None on
passand oncould_not_run. Every other clause that did not pass is inblocking. -
reason(str) –One line for a person: why it passed, what blocked it, or why nothing ran.
Invariants
decision == "pass"exactly when there is at least one clause and every clause passed.
VersionRead
dataclass
¶
VersionRead(
version: ModelVersion | None,
outcome: StoreOutcome,
reason: str | None = None,
)
What a ModelRegistry read returns.
Attributes:
-
version(ModelVersion | None) –The version; None when there is none or the registry could not be read.
-
outcome(StoreOutcome) –ok,request_faultfor a version that is not anint, orunavailablewhen the registry could not be read. -
reason(str | None) –One line naming the failure; None on
ok.
Invariants
- "No such version or alias" (
ok, None) is distinct from "could not read" (unavailable).
Written
dataclass
¶
Written(
written: int,
outcome: StoreOutcome,
reason: str | None = None,
ref: str | None = None,
)
What a store or ledger write returns.
Attributes:
-
written(int) –Items the store accepted.
-
outcome(StoreOutcome) –okwhen every item landed;unavailablewhen the store failed (writtencounts what landed first);request_faultwhen the store refused the input. -
reason(str | None) –One line naming the failure; None on
ok. -
ref(str | None) –The id the backend gave what it wrote (a run id, a model version), so a caller can link it to an external tracker; None when the backend mints none or nothing landed. Not part of equality.
Run
dataclass
¶
Run(
kind: str,
subject: str,
verdict: Verdict,
fit: Fit | None,
applied: str,
recorded: Written,
seconds: float,
)
One run of a loop.
Attributes:
-
kind(str) –The loop's run kind.
-
subject(str) –The loop's subject.
-
verdict(Verdict) –The gate's verdict (
could_not_runwhen a stage failed). -
fit(Fit | None) –The fit, or None when the run stopped before it finished.
-
applied(str) –What
applyreturned;apply_failedwhen it raised; "" when not called. -
recorded(Written) –The ledger's answer to the row this run wrote; its
refis the row's id when the ledger mints one (what a model registry'srun_reflinks to). -
seconds(float) –Wall time of the stages, excluding the ledger write.
as_dict
¶
as_dict() -> dict[str, Any]
Return the run as plain fields, for an HTTP response or a scheduler's log.
Returns:
-
dict[str, Any]–kind,subject,decision,reason,blocker(from the verdict),applied, -
dict[str, Any]–recorded(the ledger's outcome),recorded_reason,ref(the row's id, or None), -
dict[str, Any]–metrics(the fit's;{}without a fit) andseconds.failedreads it.
failed
¶
failed(result: Run | Mapping[str, Any]) -> bool
Return whether a run should read red to whatever scheduled it.
Red is a run that measured nothing (could_not_run), whose ledger row did not land, or
whose apply raised (apply_failed). A fail or insufficient verdict is a measurement,
not a failure, and reads green.
Parameters:
-
result(Run | Mapping[str, Any]) –A
Run, or itsRun.as_dict(as a scheduler holds it after an HTTP hop).
Returns:
-
bool–True when the run is red. Never raises; a mapping missing a key reads red.
record
async
¶
record(
ledger: RunLedger,
kind: str,
verdict: Verdict,
*,
params: Mapping[str, Any] | None = None,
metrics: Mapping[str, Any] | None = None
) -> Written
Record one gate verdict as a ledger row, for a gate run outside a Loop.
Parameters:
-
ledger(RunLedger) –Where the run is recorded.
-
kind(str) –The run kind.
-
verdict(Verdict) –The verdict; its
subjectis the row's subject. -
params(Mapping[str, Any] | None, default:None) –The knobs the gate used.
-
metrics(Mapping[str, Any] | None, default:None) –What it measured;
VERDICT_KEYis set fromverdict.
Returns:
-
Written–The ledger's answer. Never raises.
run
async
¶
Run loop once and record it in ledger.
Parameters:
-
loop(Loop[E]) –The loop. An
unevaluatedattribute, when present, is passed togate.decide; a value other than "fail" or "insufficient" makes the runcould_not_run. -
ledger(RunLedger) –Where the run is recorded.
-
clock(Callable[[], float], default:monotonic) –A monotonic clock in seconds, for
seconds; injectable for tests.
Returns:
-
Run–The run. Never raises for a stage's exception (recorded as
could_not_run, or as -
Run–apply_failedforapply), a loop attribute that raises or a fit whosemetricsor -
Run–modelis not a mapping (could_not_run, stagelooporfit), or a ledger failure -
Run–(
Run.recordedcarries it). A loop whosekindcannot be read is recorded under its -
Run–class name.
llm_ports.learn.gate¶
One gate for every loop: evaluated clauses in, a verdict out.
A gate is a function from evidence to clauses; decide turns the clauses into a Verdict the
same way for every loop, so a verdict means the same thing on every page that shows one. The
first rule that applies wins:
- no clauses:
could_not_run. - a
vetoclause evaluated False:fail(it disqualifies, whatever the sample). - a
floorclause not True:insufficient(not enough evidence yet; abarmeasured on too small a sample is noise, so it cannotfail). - with
unevaluated="fail"(default), any clause not True:fail. Withunevaluated="insufficient", a clause evaluated False:fail, else a clause that is None:insufficient. - otherwise
pass.
So a promote / hold / reject gate maps onto it exactly: disqualifiers are veto, unproven
conditions are floor, unevaluated="insufficient"; then pass is promote, insufficient is
hold and fail is reject, with reject winning over hold.
The clause helpers (at_least, at_most, equals, check) build a Clause from a measured
value. A value that is None, NaN or infinite gives passed=None, which never passes.
Invariants
- Pure: no clock, no I/O.
decidenever passes a verdict with an unevaluated clause, and never passes zero clauses.- The blocker is the FIRST clause, in the order given, that the deciding rule names.
Rationale: ADR-0010.
at_least
¶
at_least(
name: str,
value: Value,
threshold: float,
*,
role: Role = "bar"
) -> Clause
Return a clause that passes when value >= threshold.
Parameters:
-
name(str) –The clause's name.
-
value(Value) –The measured value; None, NaN or infinite leaves the clause unevaluated.
-
threshold(float) –The bar.
-
role(Role, default:'bar') –What failing it means (
bar,floorfor a sample size,veto).
Returns:
-
Clause–The evaluated clause. Never raises.
at_most
¶
at_most(
name: str,
value: Value,
threshold: float,
*,
role: Role = "bar"
) -> Clause
Return a clause that passes when value <= threshold.
Parameters:
-
name(str) –The clause's name.
-
value(Value) –The measured value; None, NaN or infinite leaves the clause unevaluated.
-
threshold(float) –The bar.
-
role(Role, default:'bar') –What failing it means (
bar,floor,veto).
Returns:
-
Clause–The evaluated clause. Never raises.
equals
¶
equals(
name: str,
value: Value,
expected: Value,
*,
role: Role = "bar"
) -> Clause
Return a clause that passes when value == expected.
Parameters:
-
name(str) –The clause's name.
-
value(Value) –The measured value; None leaves the clause unevaluated.
-
expected(Value) –What it must equal.
-
role(Role, default:'bar') –What failing it means (
bar,floor,veto).
Returns:
-
Clause–The evaluated clause. Never raises.
check
¶
check(
name: str,
passed: bool | None,
value: Value = None,
*,
role: Role = "bar"
) -> Clause
Return a clause from a condition the caller already evaluated.
Parameters:
-
name(str) –The clause's name.
-
passed(bool | None) –True, False, or None when it could not be evaluated.
-
value(Value, default:None) –What was measured, for the record; optional.
-
role(Role, default:'bar') –What failing it means (
bar,floor,veto).
Returns:
-
Clause–The clause. Never raises.
decide
¶
decide(
gate: str,
clauses: Iterable[Clause],
*,
subject: str = "",
unevaluated: Unevaluated = "fail"
) -> Verdict
Return the verdict of one gate over its evaluated clauses (rules in the module docstring).
Parameters:
-
gate(str) –The gate's name.
-
clauses(Iterable[Clause]) –Every clause, in the order to report them; the blocker is the first one the deciding rule names.
-
subject(str, default:'') –What was judged; "" when the gate has one subject.
-
unevaluated(Unevaluated, default:'fail') –How a
barorvetoclause that is None reads:fail(default, fail closed as a failure) orinsufficient(fail closed as unproven). Afloorthat is None is alwaysinsufficient.
Returns:
-
Verdict–The verdict. Never raises.
Invariants
- An unevaluated clause never passes, under either policy.
could_not_run
¶
could_not_run(
gate: str, reason: str, *, subject: str = ""
) -> Verdict
Return the verdict of a gate that measured nothing.
Parameters:
-
gate(str) –The gate's name.
-
reason(str) –Why nothing was measured (the store was down, the data was empty, ...).
-
subject(str, default:'') –What would have been judged.
Returns:
-
Verdict–A
could_not_runverdict with no clauses. Never raises.
llm_ports.learn.loop¶
Run a learning loop once: evidence, fit, gate, apply, and exactly one ledger row, whatever happens.
run is the part every loop used to rewrite: it drives a Loop's stages, decides the verdict
with gate.decide, lets the loop act on it, and records the run. A stage that raises does not
escape: the run is recorded as could_not_run with the stage and the exception's type and
message, because a run that left no row is indistinguishable from a run that never happened.
An apply that raises keeps the verdict and is recorded as apply_failed.
Ledger row written for every run:
params: the loop'sparams.metrics: the fit'smetrics, then the reserved keysRESERVED:learn_verdict(Verdict.as_dict),learn_model(the fit'smodel),learn_apply(whatapplyreturned,apply_failed, or "" when it was not called),learn_apply_reasonandlearn_seconds, andlearn_artifact_uri(the fit'sartifact_uri) only when the fit set one. A fit metric of a reserved name is replaced.
record writes the same verdict row for a gate run outside a Loop. Run.as_dict is a run as
plain fields (an admin endpoint's response) and failed says whether it should read red.
Invariants
- Exactly one
RunLedger.appendper call ofrunorrecord. - Never raises for a loop's failure (a stage, an attribute, a malformed fit) or a ledger's; both are in the returned value.
Rationale: ADR-0010.
Run
dataclass
¶
Run(
kind: str,
subject: str,
verdict: Verdict,
fit: Fit | None,
applied: str,
recorded: Written,
seconds: float,
)
One run of a loop.
Attributes:
-
kind(str) –The loop's run kind.
-
subject(str) –The loop's subject.
-
verdict(Verdict) –The gate's verdict (
could_not_runwhen a stage failed). -
fit(Fit | None) –The fit, or None when the run stopped before it finished.
-
applied(str) –What
applyreturned;apply_failedwhen it raised; "" when not called. -
recorded(Written) –The ledger's answer to the row this run wrote; its
refis the row's id when the ledger mints one (what a model registry'srun_reflinks to). -
seconds(float) –Wall time of the stages, excluding the ledger write.
as_dict
¶
as_dict() -> dict[str, Any]
Return the run as plain fields, for an HTTP response or a scheduler's log.
Returns:
-
dict[str, Any]–kind,subject,decision,reason,blocker(from the verdict),applied, -
dict[str, Any]–recorded(the ledger's outcome),recorded_reason,ref(the row's id, or None), -
dict[str, Any]–metrics(the fit's;{}without a fit) andseconds.failedreads it.
failed
¶
failed(result: Run | Mapping[str, Any]) -> bool
Return whether a run should read red to whatever scheduled it.
Red is a run that measured nothing (could_not_run), whose ledger row did not land, or
whose apply raised (apply_failed). A fail or insufficient verdict is a measurement,
not a failure, and reads green.
Parameters:
-
result(Run | Mapping[str, Any]) –A
Run, or itsRun.as_dict(as a scheduler holds it after an HTTP hop).
Returns:
-
bool–True when the run is red. Never raises; a mapping missing a key reads red.
record
async
¶
record(
ledger: RunLedger,
kind: str,
verdict: Verdict,
*,
params: Mapping[str, Any] | None = None,
metrics: Mapping[str, Any] | None = None
) -> Written
Record one gate verdict as a ledger row, for a gate run outside a Loop.
Parameters:
-
ledger(RunLedger) –Where the run is recorded.
-
kind(str) –The run kind.
-
verdict(Verdict) –The verdict; its
subjectis the row's subject. -
params(Mapping[str, Any] | None, default:None) –The knobs the gate used.
-
metrics(Mapping[str, Any] | None, default:None) –What it measured;
VERDICT_KEYis set fromverdict.
Returns:
-
Written–The ledger's answer. Never raises.
run
async
¶
Run loop once and record it in ledger.
Parameters:
-
loop(Loop[E]) –The loop. An
unevaluatedattribute, when present, is passed togate.decide; a value other than "fail" or "insufficient" makes the runcould_not_run. -
ledger(RunLedger) –Where the run is recorded.
-
clock(Callable[[], float], default:monotonic) –A monotonic clock in seconds, for
seconds; injectable for tests.
Returns:
-
Run–The run. Never raises for a stage's exception (recorded as
could_not_run, or as -
Run–apply_failedforapply), a loop attribute that raises or a fit whosemetricsor -
Run–modelis not a mapping (could_not_run, stagelooporfit), or a ledger failure -
Run–(
Run.recordedcarries it). A loop whosekindcannot be read is recorded under its -
Run–class name.
llm_ports.learn.examples¶
Read labelled examples: how many per topic, who agrees with the human, which to show a judge.
coverage: per topic, how many distinct subjects each source labelled, against a target. Every topic asked about appears, an unlabelled one included: the gap is the point.pairs: per topic, the(measured, reference)binary label pairs on the subjects both sources labelled: what every agreement statistic reads, and what a caller pools across topics.agreement: per topic, how well each measured source (a judge, a model labeller) agrees with the reference source (a human), as Cohen's kappa with its 95% interval and Gwet's AC1.few_shot: a deterministic, class-balanced pick of examples, to put graded examples of a topic in front of a judge.
Invariants
- Pure: no store, no clock.
- One source's labels on one subject count once: the latest given wins (a relabel replaces).
- A statistic the examples cannot define is None, never a number.
Rationale: ADR-0010.
Coverage
dataclass
¶
Coverage(
topic: str,
counts: Mapping[str, int],
target: int,
reference: Source = "human",
positive: int = 0,
)
One topic's labelled examples.
Attributes:
-
topic(str) –The topic.
-
counts(Mapping[str, int]) –Distinct subjects labelled, per source (every source present, zeros included).
-
target(int) –The reference-source labels the topic should have.
-
reference(Source) –The source the target counts (
humanunless told otherwise). -
positive(int) –Of the reference-labelled subjects, how many carry the positive class.
Agreement
dataclass
¶
Agreement(
n: int,
agree: float | None,
kappa: float | None,
ac1: float | None,
kappa_low: float | None = None,
kappa_high: float | None = None,
)
How one source agrees with the reference source on one topic.
Attributes:
-
n(int) –Subjects both labelled.
-
agree(float | None) –The share of those on which they agree; None when
nis 0. -
kappa(float | None) –Cohen's kappa; None where it is undefined (
stats.cohen_kappa). -
ac1(float | None) –Gwet's AC1; None under two subjects.
-
kappa_low(float | None) –The 95% interval's lower end (
stats.kappa_ci); None where kappa is. -
kappa_high(float | None) –Its upper end; None where kappa is.
latest
¶
Return one example per (topic, subject, source): the latest given (by at).
An example with no at sorts before every dated one; among equals the later in examples wins.
coverage
¶
coverage(
examples: Iterable[Example],
topics: Iterable[str],
*,
target: int | Mapping[str, int],
reference: Source = "human",
positive: str = "good"
) -> dict[str, Coverage]
Return each topic's Coverage.
Parameters:
-
examples(Iterable[Example]) –Examples from any source.
-
topics(Iterable[str]) –The topics to report; a topic with no example reports zeros. A labelled topic missing from
topicsis reported too, so an example is never silently dropped. -
target(int | Mapping[str, int]) –The reference labels each topic should have, one number for all or a per-topic mapping (a topic missing from the mapping gets 0).
-
reference(Source, default:'human') –The source the target counts.
-
positive(str, default:'good') –The class counted in
Coverage.positive.
Returns:
-
dict[str, Coverage]–{topic: Coverage}, in the order oftopics, then any extra labelled topics sorted.
pairs
¶
pairs(
examples: Iterable[Example],
*,
reference: Source = "human",
measured: Source = "judge",
by: str | None = None,
positive: str = "good"
) -> dict[str, list[tuple[str, str]]]
Return, per topic, the (measured, reference) label pairs on the subjects both labelled.
Each label reads as binary: "yes" (stats.YES) when it is positive, else "no". Pairs are
ordered by subject, so the result is deterministic.
Parameters:
-
examples(Iterable[Example]) –Examples from any source.
-
reference(Source, default:'human') –The source taken as ground truth.
-
measured(Source, default:'judge') –The source compared with it.
-
by(str | None, default:None) –When given, only
measuredexamples whosebyis this labeller (one judge model) count; a label with anotherby, or none, is left out. -
positive(str, default:'good') –The positive class.
Returns:
-
dict[str, list[tuple[str, str]]]–{topic: [(measured, reference), ...]}for every topic with a reference label, in topic -
dict[str, list[tuple[str, str]]]–order; a topic where no subject was labelled by both maps to
[]. Never raises.
agreement
¶
agreement(
examples: Iterable[Example],
*,
reference: Source = "human",
measured: Sequence[Source] = ("judge", "model"),
positive: str = "good"
) -> dict[str, dict[str, Agreement]]
Return, per topic, each measured source's Agreement with the reference source.
Labels are read as binary: positive or not. Only subjects both sources labelled count. A
topic with no reference label is absent; a source with no shared subject reports n == 0.
Returns:
-
dict[str, dict[str, Agreement]]–{topic: {source: Agreement}}.
few_shot
¶
few_shot(
examples: Iterable[Example],
k: int,
*,
topic: str,
source: Source = "human"
) -> list[Example]
Return up to k examples of one topic from one source, balanced across classes.
The pick is deterministic (a hash of the subject orders each class) and round-robin across the label values, rarest class first, so a judge shown the examples sees every class the examples hold. Numeric labels are one class each, so a numeric topic is simply a stable pick.
Parameters:
-
examples(Iterable[Example]) –Examples from any topic and source.
-
k(int) –The most to return.
-
topic(str) –The topic to pick from.
-
source(Source, default:'human') –The source to pick from (
humanby default: ground truth).
Returns:
-
list[Example]–The picked examples, at most
k. Never raises.
llm_ports.learn.stats¶
The statistics a gate reads: agreement between two labellers, and a multiple-testing mask.
cohen_kappa/gwet_ac1/kappa_ci: chance-corrected agreement of two labellers on the same binary items. Read kappa; read AC1 beside it where one answer dominates (kappa's chance correction degenerates there).bh_fdr: which of a family of p-values survive false-discovery-rate control, Benjamini-Yekutieli by default (valid under arbitrary dependence), plain Benjamini-Hochberg on request.
Invariants
- Pure, standard library only.
- A statistic the data cannot define is None, never a number.
Rationale: ADR-0010.
cohen_kappa
¶
cohen_kappa(pairs: Pairs) -> float | None
Return Cohen's kappa over binary (a, b) label pairs.
Both labels are "yes" or anything else (read as no); abstentions belong outside pairs.
Returns:
-
float | None–Kappa, or None when it is undefined: under two pairs, or both labellers one-sided on the
-
float | None–same answer (chance agreement 1.0).
kappa_ci
¶
kappa_ci(
pairs: Pairs, z: float = Z95
) -> tuple[float, float] | None
Return the confidence interval of Cohen's kappa over binary (a, b) label pairs.
The interval is kappa +- z * SE, with SE the large-sample standard error of Fleiss, Cohen and
Everitt (1969), the one statsmodels' cohens_kappa reports; each end is clipped to [-1, 1].
Parameters:
-
pairs(Pairs) –As
cohen_kappa. -
z(float, default:Z95) –The normal quantile;
Z95(default) for a 95% interval.
Returns:
-
tuple[float, float] | None–(low, high), or None wherevercohen_kappais None.
gwet_ac1
¶
gwet_ac1(pairs: Pairs) -> float | None
Return Gwet's AC1 over binary (a, b) label pairs.
Chance agreement is 2 * pi * (1 - pi), with pi the pooled share of "yes", so the statistic
stays defined and bounded where one answer dominates.
Returns:
-
float | None–AC1, or None under two pairs.
bh_fdr
¶
bh_fdr(
pvals: Sequence[float | None],
alpha: float = 0.1,
*,
dependent: bool = True
) -> list[bool]
Return which p-values survive false-discovery-rate control at alpha, in input order.
The step-up compares each p to rank / (m * c) * alpha (never p * m * c / rank to alpha:
the two float orders disagree at the threshold on grid p-values such as k / (B + 1)).
Parameters:
-
pvals(Sequence[float | None]) –The family's p-values; a None is an untested hypothesis and never passes.
-
alpha(float, default:0.1) –The FDR level.
-
dependent(bool, default:True) –True (default) for Benjamini-Yekutieli, which multiplies the threshold's denominator by
c(m) = sum(1/i)and controls FDR under any dependence between the tests; False for plain Benjamini-Hochberg (valid under independence or PRDS).
Returns:
-
list[bool]–One bool per p-value. Never raises.
Invariants
- The family size
mcounts tested hypotheses only: a None neither passes nor loosens the bar for the others.
llm_ports.learn.stores¶
Ledgers and example stores that need nothing but the standard library.
MemoryRunLedger,MemoryExampleStore: in-process, for tests and one-shot scripts.JsonlRunLedger,JsonlExampleStore: one JSON object per line in a file, so a loop runs and keeps its history anywhere with zero setup.union: read several example stores as one.
A database-backed ledger is an adapter in the consuming application: it implements the two
RunLedger methods over its own table. Both ledgers here id each row with a fresh UUID
(Written.ref, LedgerRow.ref). The model registries are in learn.registry.
Invariants
- Never raises for a backend failure: an unreadable or unwritable file is an
unavailableoutcome; a value that is not JSON-able is arequest_fault. - An unreadable store never reads as an empty one.
Rationale: ADR-0010.
MemoryRunLedger
¶
MemoryRunLedger(*, now: Callable[[], datetime] = _utcnow)
A RunLedger held in a list (rows), oldest first.
Create an empty ledger.
Parameters:
-
now(Callable[[], datetime], default:_utcnow) –The clock that stamps each row; injectable for tests.
JsonlRunLedger
¶
JsonlRunLedger(
path: str | Path,
*,
now: Callable[[], datetime] = _utcnow
)
A RunLedger in a JSON-lines file: one row per line, appended, never rewritten.
Use the ledger at path (created on first append).
Parameters:
-
path(str | Path) –The file.
-
now(Callable[[], datetime], default:_utcnow) –The clock that stamps each row; injectable for tests.
MemoryExampleStore
¶
MemoryExampleStore(examples: Iterable[Example] = ())
A writable ExampleStore held in a list (items).
Create a store holding examples.
examples
async
¶
examples(
topics: Sequence[str],
*,
since: datetime | None = None,
source: Source | None = None
) -> ExamplesRead
Read matching examples. Never raises.
JsonlExampleStore
¶
JsonlExampleStore(path: str | Path)
A writable ExampleStore in a JSON-lines file: one example per line.
Use the store at path (created on first put).
examples
async
¶
examples(
topics: Sequence[str],
*,
since: datetime | None = None,
source: Source | None = None
) -> ExamplesRead
Read matching examples. Never raises; a missing file is an empty store.
union
¶
union(*stores: ExampleStore) -> ExampleStore
Return one read-only store over stores.
Returns:
-
ExampleStore–A store whose read is every store's read concatenated, or
unavailable(naming the store) -
ExampleStore–when any one cannot be read: a partial read would undercount a topic's labels silently.
llm_ports.learn.registry¶
Model registries that need nothing but the standard library.
MemoryModelRegistry: in-process, for tests and one-shot scripts.JsonlModelRegistry: an append-only JSON-lines event log (registerandaliasevents), folded on read, so every alias move stays on the record.
A tracker's registry (MLflow's, for one) is an adapter that implements the same four methods.
Invariants
- Never raises for a backend failure: an unreadable or unwritable file is an
unavailableoutcome; a model that is not JSON-able, an empty name or alias, anartifact_uriorrun_refof the wrong type, or an alias set to a version that does not exist is arequest_fault. - Versions of one name are numbered from 1 and never change once registered; only aliases move.
- One writer per
JsonlModelRegistryfile: the next version number is read from the file.
Rationale: ADR-0016.
MemoryModelRegistry
¶
MemoryModelRegistry(
*, now: Callable[[], datetime] = _utcnow
)
A ModelRegistry held in memory.
Create an empty registry.
Parameters:
-
now(Callable[[], datetime], default:_utcnow) –The clock that stamps each version; injectable for tests.
register
async
¶
register(
name: str,
*,
model: Mapping[str, Any],
artifact_uri: str | None = None,
run_ref: str | None = None,
tags: Mapping[str, str] | None = None
) -> Written
Register a new version. Never raises.
set_alias
async
¶
set_alias(name: str, alias: str, version: int) -> Written
Point alias at version. Never raises.
by_alias
async
¶
by_alias(name: str, alias: str) -> VersionRead
Read the version alias points at. Never raises.
JsonlModelRegistry
¶
JsonlModelRegistry(
path: str | Path,
*,
now: Callable[[], datetime] = _utcnow
)
A ModelRegistry in a JSON-lines event log: appended, never rewritten.
Use the registry at path (created on first write).
Parameters:
-
path(str | Path) –The file.
-
now(Callable[[], datetime], default:_utcnow) –The clock that stamps each event; injectable for tests.
register
async
¶
register(
name: str,
*,
model: Mapping[str, Any],
artifact_uri: str | None = None,
run_ref: str | None = None,
tags: Mapping[str, str] | None = None
) -> Written
Append a register event. Never raises.
set_alias
async
¶
set_alias(name: str, alias: str, version: int) -> Written
Append an alias event. Never raises.
by_alias
async
¶
by_alias(name: str, alias: str) -> VersionRead
Read the version alias points at. Never raises; a missing file is an empty registry.
get
async
¶
get(name: str, version: int) -> VersionRead
Read one version. Never raises; a missing file is an empty registry.