Skip to content

llm_ports.learn

llm_ports.learn

The learning-loop core: labelled examples, one gate, one run ledger, one loop runner.

Every learning loop is the same moves: read evidence, fit or measure, hold the result to a gate, and record the run. This package holds the parts that do not depend on what is learned:

  • values: Example, Clause, Verdict, LedgerRow (and the store results).
  • gate: clause helpers and decide, the one way a verdict is reached.
  • stats: agreement (cohen_kappa, gwet_ac1) and false-discovery control (bh_fdr).
  • examples: coverage, agreement, few_shot over labelled examples.
  • ports: RunLedger, ExampleStore, WritableExampleStore, ModelRegistry, Loop; stores and registry implement them with the standard library; loop.run drives a Loop (evidence, fit, clauses, apply) and records exactly one row; failed says whether a run reads red.
Invariants
  • Standard library only: imports nothing else from llm_ports and no third-party package, so it can be lifted into any codebase (enforced by an import-linter contract).

Rationale: ADR-0010, ADR-0016 (the seams to bigger engines).

ExampleStore

Bases: Protocol

Labelled examples, read by topic.

examples async

examples(
    topics: Sequence[str],
    *,
    since: datetime | None = None,
    source: Source | None = None
) -> ExamplesRead

Read the examples of topics, labelled at or after since, from source or any source.

Returns:

  • ExamplesRead –

    The examples, or an unavailable outcome; an unreadable store never reads as empty.

Fit dataclass

Fit(
    model: Mapping[str, Any] = dict[str, Any](),
    metrics: Mapping[str, Any] = dict[str, Any](),
    artifact_uri: str | None = None,
)

What a loop's fit stage produced.

Attributes:

  • model (Mapping[str, Any]) –

    The fitted parameters a consumer applies (a weight, a threshold, coefficients); JSON-able. {} for a loop that measures without fitting.

  • metrics (Mapping[str, Any]) –

    What the fit measured (sample size, error, lift); JSON-able.

  • artifact_uri (str | None) –

    Where a model too big for model was written (a file, a bucket, a tracker's artifact store); None when model is the whole model. loop.run records it in the ledger row.

Invariants
  • Plain data, so a fit made in a worker process pickles back whole.

Loop

Bases: Protocol

One learning loop: read evidence, fit, say which clauses the fit must clear, act on it.

loop.run drives the stages, decides the verdict with gate.decide, calls apply and records one ledger row, so a loop holds only what is specific to it. A loop whose verdict needs a policy other than the default (unevaluated) sets the unevaluated attribute.

kind property

kind: str

The run kind its ledger rows carry.

subject property

subject: str

What it learns about (a symbol, a role, a model); "" for one subject.

params property

params: Mapping[str, Any]

The knobs of this run (window, thresholds); JSON-able.

evidence async

evidence() -> E

Read the evidence (examples, outcomes, series). May raise; run records the failure.

fit

fit(evidence: E) -> Fit

Fit or measure. May raise; run records the failure.

Pure (no I/O, no shared state), so a heavier engine can ship it to a remote worker.

clauses

clauses(fit: Fit, evidence: E) -> Sequence[Clause]

Return the gate's evaluated clauses for this fit. May raise; run records the failure.

apply async

apply(verdict: Verdict, fit: Fit) -> str

Act on the verdict: store the fit on pass, keep the incumbent otherwise, or nothing.

Called once per run that reached a decision (never on could_not_run), before the run is recorded, so the row says what was done.

Returns:

  • str –

    One word for the record (stored, held, noop, ...). May raise; run records

  • str –

    apply_failed and keeps the verdict.

ModelRegistry

Bases: Protocol

Numbered versions of fitted models, with named aliases (champion) pointing at one each.

Mirrors a tracker's model registry (versions from 1 per name; an alias moves, a version never changes), so a backend is a thin mapping.

register async

register(
    name: str,
    *,
    model: Mapping[str, Any],
    artifact_uri: str | None = None,
    run_ref: str | None = None,
    tags: Mapping[str, str] | None = None
) -> Written

Register a new version of name (created on first use).

Parameters:

  • name (str) –

    The registered model's name.

  • model (Mapping[str, Any]) –

    The fitted parameters; JSON-able.

  • artifact_uri (str | None, default: None ) –

    Where a model too big for model lives; None when model is whole.

  • run_ref (str | None, default: None ) –

    The ledger row that produced it (Written.ref).

  • tags (Mapping[str, str] | None, default: None ) –

    Free string labels.

Returns:

  • Written –

    written == 1 and the new version number as a decimal string in ref, else a

  • Written –

    request_fault (a value the registry refuses) or unavailable outcome.

set_alias async

set_alias(name: str, alias: str, version: int) -> Written

Point alias of name at version, moving it from any earlier version.

Returns:

  • Written –

    written == 1 on success; request_fault when the version does not exist or is not an

  • Written –

    int; else

  • Written –

    unavailable.

by_alias async

by_alias(name: str, alias: str) -> VersionRead

Read the version alias of name points at.

Returns:

  • VersionRead –

    The version, ok with None when the model or alias does not exist, or unavailable.

get async

get(name: str, version: int) -> VersionRead

Read one version of name.

Returns:

  • VersionRead –

    The version, ok with None when it does not exist, request_fault when version is

  • VersionRead –

    not an int (Written.ref is its decimal string), or unavailable.

RunLedger

Bases: Protocol

The record of every run: what ran, on what, with which knobs, and what it decided.

append async

append(
    kind: str,
    *,
    subject: str,
    params: Mapping[str, Any],
    metrics: Mapping[str, Any]
) -> Written

Record one run.

Parameters:

  • kind (str) –

    The loop's run kind.

  • subject (str) –

    What the run was about; "" for a loop with one subject.

  • params (Mapping[str, Any]) –

    The knobs the run used; JSON-able.

  • metrics (Mapping[str, Any]) –

    What it measured; JSON-able. A gated run carries its verdict under VERDICT_KEY.

Returns:

  • Written –

    written == 1 on success, else an unavailable or request_fault outcome. A ledger

  • Written –

    that ids its rows puts the id in ref; the row read back carries the same ref.

latest async

latest(
    kind: str,
    *,
    subject: str | None = None,
    decisions: Sequence[Decision] | None = None
) -> RowRead

Read the newest row of one kind.

Parameters:

  • kind (str) –

    The run kind.

  • subject (str | None, default: None ) –

    Only rows about this subject; None for any.

  • decisions (Sequence[Decision] | None, default: None ) –

    Only rows whose recorded verdict's decision is one of these (a row with no verdict never matches); None for any row. A gate's decisive read passes ("pass", "fail"), so a later could_not_run does not close what a decisive run opened.

Returns:

  • RowRead –

    The row (None when there is none), or an unavailable outcome.

WritableExampleStore

Bases: ExampleStore, Protocol

A store that also accepts examples written by code.

put async

put(examples: Sequence[Example]) -> Written

Store examples.

Returns:

  • Written –

    How many landed, or a request_fault naming an example the store refuses (a store

  • Written –

    may refuse a human label written by code: a person did not give it).

Clause dataclass

Clause(
    name: str,
    passed: bool | None,
    value: Value = None,
    threshold: Value = None,
    op: str = "",
    role: Role = "bar",
)

One condition of a gate, evaluated.

Attributes:

  • name (str) –

    The clause's name, unique within its gate (n_at_least_200, kappa, ...).

  • passed (bool | None) –

    True or False when evaluated; None when it could not be (missing evidence).

  • value (Value) –

    The measured value; None when it could not be computed.

  • threshold (Value) –

    The bar it was held to; None for a boolean check.

  • op (str) –

    How value was compared to threshold (>=, <=, ==); "" for a boolean check.

  • role (Role) –

    What failing it means: bar (default), floor or veto (see Role). The gate sets it when it evaluates the clause, so one condition may be a veto in one branch and a floor in another.

Invariants
  • A None passed is never read as a pass (gate.decide fails it closed).

as_dict

as_dict() -> dict[str, Any]

Return the clause as a JSON-able mapping.

from_dict classmethod

from_dict(raw: Mapping[str, Any]) -> Clause

Rebuild a clause from as_dict output.

Raises:

  • KeyError –

    raw has no name.

Example dataclass

Example(
    topic: str,
    subject: str,
    value: str | float,
    source: Source,
    by: str = "",
    at: datetime | None = None,
    reason: str = "",
    ref: Mapping[str, str] = dict[str, str](),
)

One labelled example: something a loop learns from, with where it came from.

Attributes:

  • topic (str) –

    What is learned (llm:summary, trend:1d, fit:calibration).

  • subject (str) –

    The thing labelled, unique within the topic (an observation id, item@day).

  • value (str | float) –

    The label: a class (good, bad) or a number (a forward return).

  • source (Source) –

    human, model, judge or outcome.

  • by (str) –

    Who labelled: a person's id, a model's name, a grader's name; "" when unknown.

  • at (datetime | None) –

    When the label was given; None when the store does not say.

  • reason (str) –

    The labeller's reason; "" when none was given.

  • ref (Mapping[str, str]) –

    Provenance: trace id, table and key, code version. Not part of equality.

Invariants
  • A model or judge label is measured against human labels, never counted as one.

ExamplesRead dataclass

ExamplesRead(
    examples: tuple[Example, ...],
    outcome: StoreOutcome,
    reason: str | None = None,
)

What ExampleStore.examples returns.

Attributes:

  • examples (tuple[Example, ...]) –

    Every example read; () exactly when outcome != "ok".

  • outcome (StoreOutcome) –

    ok, or unavailable when the store could not be read.

  • reason (str | None) –

    One line naming the failure; None on ok.

Invariants
  • A store that could not be read never reads as a store with no examples.

ok property

ok: bool

True when outcome == "ok".

LedgerRow dataclass

LedgerRow(
    kind: str,
    subject: str,
    params: Mapping[str, Any],
    metrics: Mapping[str, Any],
    at: datetime,
    ref: str | None = None,
)

One recorded run of a loop.

Attributes:

  • kind (str) –

    The loop's run kind.

  • subject (str) –

    What the run was about; "" when the loop has one subject.

  • params (Mapping[str, Any]) –

    The knobs the run used.

  • metrics (Mapping[str, Any]) –

    What it measured; a gated run carries Verdict.as_dict under VERDICT_KEY.

  • at (datetime) –

    When it was recorded.

  • ref (str | None) –

    The id the ledger gave the row (Written.ref of its append); None when the ledger mints none. Not part of equality.

verdict property

verdict: Verdict | None

The recorded verdict, or None for an ungated (informational) run.

decision property

decision: Decision | None

The recorded verdict's decision, or None for an ungated run.

ModelVersion dataclass

ModelVersion(
    name: str,
    version: int,
    model: Mapping[str, Any],
    artifact_uri: str | None,
    run_ref: str | None,
    at: datetime,
    tags: Mapping[str, str] = dict[str, str](),
)

One registered version of a model: what a loop fitted, kept so it can be promoted.

Attributes:

  • name (str) –

    The registered model's name.

  • version (int) –

    Its number within name, from 1, increasing by one per registration.

  • model (Mapping[str, Any]) –

    The fitted parameters (Fit.model); JSON-able.

  • artifact_uri (str | None) –

    Where a model too big for model lives (a file, a bucket, a tracker's artifact store); None when model is the whole model.

  • run_ref (str | None) –

    The ledger row that produced it (Written.ref); None when unknown.

  • at (datetime) –

    When it was registered.

  • tags (Mapping[str, str]) –

    Free labels; string to string.

RowRead dataclass

RowRead(
    row: LedgerRow | None,
    outcome: StoreOutcome,
    reason: str | None = None,
)

What RunLedger.latest returns.

Attributes:

  • row (LedgerRow | None) –

    The newest matching row; None when there is none or the ledger could not be read.

  • outcome (StoreOutcome) –

    ok, or unavailable when the ledger could not be read.

  • reason (str | None) –

    One line naming the failure; None on ok.

Invariants
  • "No row yet" (ok, None) is distinct from "could not read" (unavailable).

ok property

ok: bool

True when outcome == "ok".

Verdict dataclass

Verdict(
    gate: str,
    subject: str,
    decision: Decision,
    clauses: tuple[Clause, ...] = (),
    blocker: str | None = None,
    reason: str = "",
)

What a gate decided about one subject, with every clause it read.

Attributes:

  • gate (str) –

    The gate's name.

  • subject (str) –

    What was judged (a model, a symbol, a role); "" when the gate has one subject.

  • decision (Decision) –

    pass, fail, insufficient or could_not_run.

  • clauses (tuple[Clause, ...]) –

    Every clause, in the order the gate evaluated them.

  • blocker (str | None) –

    The first clause, in order, that set the decision; None on pass and on could_not_run. Every other clause that did not pass is in blocking.

  • reason (str) –

    One line for a person: why it passed, what blocked it, or why nothing ran.

Invariants
  • decision == "pass" exactly when there is at least one clause and every clause passed.

passed property

passed: bool

True when decision == "pass".

blocking property

blocking: tuple[Clause, ...]

Every clause that did not pass (failed or not evaluated), in order.

as_dict

as_dict() -> dict[str, Any]

Return the verdict as a JSON-able mapping (what a ledger row stores under VERDICT_KEY).

from_dict classmethod

from_dict(raw: Mapping[str, Any]) -> Verdict | None

Rebuild a verdict from as_dict output; None when raw is not one.

Never raises: a mapping with an unknown decision or a malformed clause is not a verdict.

VersionRead dataclass

VersionRead(
    version: ModelVersion | None,
    outcome: StoreOutcome,
    reason: str | None = None,
)

What a ModelRegistry read returns.

Attributes:

  • version (ModelVersion | None) –

    The version; None when there is none or the registry could not be read.

  • outcome (StoreOutcome) –

    ok, request_fault for a version that is not an int, or unavailable when the registry could not be read.

  • reason (str | None) –

    One line naming the failure; None on ok.

Invariants
  • "No such version or alias" (ok, None) is distinct from "could not read" (unavailable).

ok property

ok: bool

True when outcome == "ok".

Written dataclass

Written(
    written: int,
    outcome: StoreOutcome,
    reason: str | None = None,
    ref: str | None = None,
)

What a store or ledger write returns.

Attributes:

  • written (int) –

    Items the store accepted.

  • outcome (StoreOutcome) –

    ok when every item landed; unavailable when the store failed (written counts what landed first); request_fault when the store refused the input.

  • reason (str | None) –

    One line naming the failure; None on ok.

  • ref (str | None) –

    The id the backend gave what it wrote (a run id, a model version), so a caller can link it to an external tracker; None when the backend mints none or nothing landed. Not part of equality.

ok property

ok: bool

True when outcome == "ok".

Run dataclass

Run(
    kind: str,
    subject: str,
    verdict: Verdict,
    fit: Fit | None,
    applied: str,
    recorded: Written,
    seconds: float,
)

One run of a loop.

Attributes:

  • kind (str) –

    The loop's run kind.

  • subject (str) –

    The loop's subject.

  • verdict (Verdict) –

    The gate's verdict (could_not_run when a stage failed).

  • fit (Fit | None) –

    The fit, or None when the run stopped before it finished.

  • applied (str) –

    What apply returned; apply_failed when it raised; "" when not called.

  • recorded (Written) –

    The ledger's answer to the row this run wrote; its ref is the row's id when the ledger mints one (what a model registry's run_ref links to).

  • seconds (float) –

    Wall time of the stages, excluding the ledger write.

as_dict

as_dict() -> dict[str, Any]

Return the run as plain fields, for an HTTP response or a scheduler's log.

Returns:

  • dict[str, Any] –

    kind, subject, decision, reason, blocker (from the verdict), applied,

  • dict[str, Any] –

    recorded (the ledger's outcome), recorded_reason, ref (the row's id, or None),

  • dict[str, Any] –

    metrics (the fit's; {} without a fit) and seconds. failed reads it.

failed

failed(result: Run | Mapping[str, Any]) -> bool

Return whether a run should read red to whatever scheduled it.

Red is a run that measured nothing (could_not_run), whose ledger row did not land, or whose apply raised (apply_failed). A fail or insufficient verdict is a measurement, not a failure, and reads green.

Parameters:

  • result (Run | Mapping[str, Any]) –

    A Run, or its Run.as_dict (as a scheduler holds it after an HTTP hop).

Returns:

  • bool –

    True when the run is red. Never raises; a mapping missing a key reads red.

record async

record(
    ledger: RunLedger,
    kind: str,
    verdict: Verdict,
    *,
    params: Mapping[str, Any] | None = None,
    metrics: Mapping[str, Any] | None = None
) -> Written

Record one gate verdict as a ledger row, for a gate run outside a Loop.

Parameters:

  • ledger (RunLedger) –

    Where the run is recorded.

  • kind (str) –

    The run kind.

  • verdict (Verdict) –

    The verdict; its subject is the row's subject.

  • params (Mapping[str, Any] | None, default: None ) –

    The knobs the gate used.

  • metrics (Mapping[str, Any] | None, default: None ) –

    What it measured; VERDICT_KEY is set from verdict.

Returns:

  • Written –

    The ledger's answer. Never raises.

run async

run(
    loop: Loop[E],
    ledger: RunLedger,
    *,
    clock: Callable[[], float] = monotonic
) -> Run

Run loop once and record it in ledger.

Parameters:

  • loop (Loop[E]) –

    The loop. An unevaluated attribute, when present, is passed to gate.decide; a value other than "fail" or "insufficient" makes the run could_not_run.

  • ledger (RunLedger) –

    Where the run is recorded.

  • clock (Callable[[], float], default: monotonic ) –

    A monotonic clock in seconds, for seconds; injectable for tests.

Returns:

  • Run –

    The run. Never raises for a stage's exception (recorded as could_not_run, or as

  • Run –

    apply_failed for apply), a loop attribute that raises or a fit whose metrics or

  • Run –

    model is not a mapping (could_not_run, stage loop or fit), or a ledger failure

  • Run –

    (Run.recorded carries it). A loop whose kind cannot be read is recorded under its

  • Run –

    class name.

llm_ports.learn.gate

One gate for every loop: evaluated clauses in, a verdict out.

A gate is a function from evidence to clauses; decide turns the clauses into a Verdict the same way for every loop, so a verdict means the same thing on every page that shows one. The first rule that applies wins:

  1. no clauses: could_not_run.
  2. a veto clause evaluated False: fail (it disqualifies, whatever the sample).
  3. a floor clause not True: insufficient (not enough evidence yet; a bar measured on too small a sample is noise, so it cannot fail).
  4. with unevaluated="fail" (default), any clause not True: fail. With unevaluated="insufficient", a clause evaluated False: fail, else a clause that is None: insufficient.
  5. otherwise pass.

So a promote / hold / reject gate maps onto it exactly: disqualifiers are veto, unproven conditions are floor, unevaluated="insufficient"; then pass is promote, insufficient is hold and fail is reject, with reject winning over hold.

The clause helpers (at_least, at_most, equals, check) build a Clause from a measured value. A value that is None, NaN or infinite gives passed=None, which never passes.

Invariants
  • Pure: no clock, no I/O.
  • decide never passes a verdict with an unevaluated clause, and never passes zero clauses.
  • The blocker is the FIRST clause, in the order given, that the deciding rule names.

Rationale: ADR-0010.

at_least

at_least(
    name: str,
    value: Value,
    threshold: float,
    *,
    role: Role = "bar"
) -> Clause

Return a clause that passes when value >= threshold.

Parameters:

  • name (str) –

    The clause's name.

  • value (Value) –

    The measured value; None, NaN or infinite leaves the clause unevaluated.

  • threshold (float) –

    The bar.

  • role (Role, default: 'bar' ) –

    What failing it means (bar, floor for a sample size, veto).

Returns:

  • Clause –

    The evaluated clause. Never raises.

at_most

at_most(
    name: str,
    value: Value,
    threshold: float,
    *,
    role: Role = "bar"
) -> Clause

Return a clause that passes when value <= threshold.

Parameters:

  • name (str) –

    The clause's name.

  • value (Value) –

    The measured value; None, NaN or infinite leaves the clause unevaluated.

  • threshold (float) –

    The bar.

  • role (Role, default: 'bar' ) –

    What failing it means (bar, floor, veto).

Returns:

  • Clause –

    The evaluated clause. Never raises.

equals

equals(
    name: str,
    value: Value,
    expected: Value,
    *,
    role: Role = "bar"
) -> Clause

Return a clause that passes when value == expected.

Parameters:

  • name (str) –

    The clause's name.

  • value (Value) –

    The measured value; None leaves the clause unevaluated.

  • expected (Value) –

    What it must equal.

  • role (Role, default: 'bar' ) –

    What failing it means (bar, floor, veto).

Returns:

  • Clause –

    The evaluated clause. Never raises.

check

check(
    name: str,
    passed: bool | None,
    value: Value = None,
    *,
    role: Role = "bar"
) -> Clause

Return a clause from a condition the caller already evaluated.

Parameters:

  • name (str) –

    The clause's name.

  • passed (bool | None) –

    True, False, or None when it could not be evaluated.

  • value (Value, default: None ) –

    What was measured, for the record; optional.

  • role (Role, default: 'bar' ) –

    What failing it means (bar, floor, veto).

Returns:

  • Clause –

    The clause. Never raises.

decide

decide(
    gate: str,
    clauses: Iterable[Clause],
    *,
    subject: str = "",
    unevaluated: Unevaluated = "fail"
) -> Verdict

Return the verdict of one gate over its evaluated clauses (rules in the module docstring).

Parameters:

  • gate (str) –

    The gate's name.

  • clauses (Iterable[Clause]) –

    Every clause, in the order to report them; the blocker is the first one the deciding rule names.

  • subject (str, default: '' ) –

    What was judged; "" when the gate has one subject.

  • unevaluated (Unevaluated, default: 'fail' ) –

    How a bar or veto clause that is None reads: fail (default, fail closed as a failure) or insufficient (fail closed as unproven). A floor that is None is always insufficient.

Returns:

  • Verdict –

    The verdict. Never raises.

Invariants
  • An unevaluated clause never passes, under either policy.

could_not_run

could_not_run(
    gate: str, reason: str, *, subject: str = ""
) -> Verdict

Return the verdict of a gate that measured nothing.

Parameters:

  • gate (str) –

    The gate's name.

  • reason (str) –

    Why nothing was measured (the store was down, the data was empty, ...).

  • subject (str, default: '' ) –

    What would have been judged.

Returns:

  • Verdict –

    A could_not_run verdict with no clauses. Never raises.

llm_ports.learn.loop

Run a learning loop once: evidence, fit, gate, apply, and exactly one ledger row, whatever happens.

run is the part every loop used to rewrite: it drives a Loop's stages, decides the verdict with gate.decide, lets the loop act on it, and records the run. A stage that raises does not escape: the run is recorded as could_not_run with the stage and the exception's type and message, because a run that left no row is indistinguishable from a run that never happened. An apply that raises keeps the verdict and is recorded as apply_failed.

Ledger row written for every run:

  • params: the loop's params.
  • metrics: the fit's metrics, then the reserved keys RESERVED: learn_verdict (Verdict.as_dict), learn_model (the fit's model), learn_apply (what apply returned, apply_failed, or "" when it was not called), learn_apply_reason and learn_seconds, and learn_artifact_uri (the fit's artifact_uri) only when the fit set one. A fit metric of a reserved name is replaced.

record writes the same verdict row for a gate run outside a Loop. Run.as_dict is a run as plain fields (an admin endpoint's response) and failed says whether it should read red.

Invariants
  • Exactly one RunLedger.append per call of run or record.
  • Never raises for a loop's failure (a stage, an attribute, a malformed fit) or a ledger's; both are in the returned value.

Rationale: ADR-0010.

Run dataclass

Run(
    kind: str,
    subject: str,
    verdict: Verdict,
    fit: Fit | None,
    applied: str,
    recorded: Written,
    seconds: float,
)

One run of a loop.

Attributes:

  • kind (str) –

    The loop's run kind.

  • subject (str) –

    The loop's subject.

  • verdict (Verdict) –

    The gate's verdict (could_not_run when a stage failed).

  • fit (Fit | None) –

    The fit, or None when the run stopped before it finished.

  • applied (str) –

    What apply returned; apply_failed when it raised; "" when not called.

  • recorded (Written) –

    The ledger's answer to the row this run wrote; its ref is the row's id when the ledger mints one (what a model registry's run_ref links to).

  • seconds (float) –

    Wall time of the stages, excluding the ledger write.

as_dict

as_dict() -> dict[str, Any]

Return the run as plain fields, for an HTTP response or a scheduler's log.

Returns:

  • dict[str, Any] –

    kind, subject, decision, reason, blocker (from the verdict), applied,

  • dict[str, Any] –

    recorded (the ledger's outcome), recorded_reason, ref (the row's id, or None),

  • dict[str, Any] –

    metrics (the fit's; {} without a fit) and seconds. failed reads it.

failed

failed(result: Run | Mapping[str, Any]) -> bool

Return whether a run should read red to whatever scheduled it.

Red is a run that measured nothing (could_not_run), whose ledger row did not land, or whose apply raised (apply_failed). A fail or insufficient verdict is a measurement, not a failure, and reads green.

Parameters:

  • result (Run | Mapping[str, Any]) –

    A Run, or its Run.as_dict (as a scheduler holds it after an HTTP hop).

Returns:

  • bool –

    True when the run is red. Never raises; a mapping missing a key reads red.

record async

record(
    ledger: RunLedger,
    kind: str,
    verdict: Verdict,
    *,
    params: Mapping[str, Any] | None = None,
    metrics: Mapping[str, Any] | None = None
) -> Written

Record one gate verdict as a ledger row, for a gate run outside a Loop.

Parameters:

  • ledger (RunLedger) –

    Where the run is recorded.

  • kind (str) –

    The run kind.

  • verdict (Verdict) –

    The verdict; its subject is the row's subject.

  • params (Mapping[str, Any] | None, default: None ) –

    The knobs the gate used.

  • metrics (Mapping[str, Any] | None, default: None ) –

    What it measured; VERDICT_KEY is set from verdict.

Returns:

  • Written –

    The ledger's answer. Never raises.

run async

run(
    loop: Loop[E],
    ledger: RunLedger,
    *,
    clock: Callable[[], float] = monotonic
) -> Run

Run loop once and record it in ledger.

Parameters:

  • loop (Loop[E]) –

    The loop. An unevaluated attribute, when present, is passed to gate.decide; a value other than "fail" or "insufficient" makes the run could_not_run.

  • ledger (RunLedger) –

    Where the run is recorded.

  • clock (Callable[[], float], default: monotonic ) –

    A monotonic clock in seconds, for seconds; injectable for tests.

Returns:

  • Run –

    The run. Never raises for a stage's exception (recorded as could_not_run, or as

  • Run –

    apply_failed for apply), a loop attribute that raises or a fit whose metrics or

  • Run –

    model is not a mapping (could_not_run, stage loop or fit), or a ledger failure

  • Run –

    (Run.recorded carries it). A loop whose kind cannot be read is recorded under its

  • Run –

    class name.

llm_ports.learn.examples

Read labelled examples: how many per topic, who agrees with the human, which to show a judge.

  • coverage: per topic, how many distinct subjects each source labelled, against a target. Every topic asked about appears, an unlabelled one included: the gap is the point.
  • pairs: per topic, the (measured, reference) binary label pairs on the subjects both sources labelled: what every agreement statistic reads, and what a caller pools across topics.
  • agreement: per topic, how well each measured source (a judge, a model labeller) agrees with the reference source (a human), as Cohen's kappa with its 95% interval and Gwet's AC1.
  • few_shot: a deterministic, class-balanced pick of examples, to put graded examples of a topic in front of a judge.
Invariants
  • Pure: no store, no clock.
  • One source's labels on one subject count once: the latest given wins (a relabel replaces).
  • A statistic the examples cannot define is None, never a number.

Rationale: ADR-0010.

Coverage dataclass

Coverage(
    topic: str,
    counts: Mapping[str, int],
    target: int,
    reference: Source = "human",
    positive: int = 0,
)

One topic's labelled examples.

Attributes:

  • topic (str) –

    The topic.

  • counts (Mapping[str, int]) –

    Distinct subjects labelled, per source (every source present, zeros included).

  • target (int) –

    The reference-source labels the topic should have.

  • reference (Source) –

    The source the target counts (human unless told otherwise).

  • positive (int) –

    Of the reference-labelled subjects, how many carry the positive class.

have property

have: int

Subjects the reference source labelled.

gap property

gap: int

Reference labels still missing to reach target; never negative.

covered property

covered: bool

True when the topic has its target of reference labels.

Agreement dataclass

Agreement(
    n: int,
    agree: float | None,
    kappa: float | None,
    ac1: float | None,
    kappa_low: float | None = None,
    kappa_high: float | None = None,
)

How one source agrees with the reference source on one topic.

Attributes:

  • n (int) –

    Subjects both labelled.

  • agree (float | None) –

    The share of those on which they agree; None when n is 0.

  • kappa (float | None) –

    Cohen's kappa; None where it is undefined (stats.cohen_kappa).

  • ac1 (float | None) –

    Gwet's AC1; None under two subjects.

  • kappa_low (float | None) –

    The 95% interval's lower end (stats.kappa_ci); None where kappa is.

  • kappa_high (float | None) –

    Its upper end; None where kappa is.

of classmethod

of(pairs: Pairs) -> Agreement

Return the agreement of (measured, reference) pairs; the empty sequence gives n == 0.

latest

latest(
    examples: Iterable[Example],
) -> dict[tuple[str, str, str], Example]

Return one example per (topic, subject, source): the latest given (by at).

An example with no at sorts before every dated one; among equals the later in examples wins.

coverage

coverage(
    examples: Iterable[Example],
    topics: Iterable[str],
    *,
    target: int | Mapping[str, int],
    reference: Source = "human",
    positive: str = "good"
) -> dict[str, Coverage]

Return each topic's Coverage.

Parameters:

  • examples (Iterable[Example]) –

    Examples from any source.

  • topics (Iterable[str]) –

    The topics to report; a topic with no example reports zeros. A labelled topic missing from topics is reported too, so an example is never silently dropped.

  • target (int | Mapping[str, int]) –

    The reference labels each topic should have, one number for all or a per-topic mapping (a topic missing from the mapping gets 0).

  • reference (Source, default: 'human' ) –

    The source the target counts.

  • positive (str, default: 'good' ) –

    The class counted in Coverage.positive.

Returns:

  • dict[str, Coverage] –

    {topic: Coverage}, in the order of topics, then any extra labelled topics sorted.

pairs

pairs(
    examples: Iterable[Example],
    *,
    reference: Source = "human",
    measured: Source = "judge",
    by: str | None = None,
    positive: str = "good"
) -> dict[str, list[tuple[str, str]]]

Return, per topic, the (measured, reference) label pairs on the subjects both labelled.

Each label reads as binary: "yes" (stats.YES) when it is positive, else "no". Pairs are ordered by subject, so the result is deterministic.

Parameters:

  • examples (Iterable[Example]) –

    Examples from any source.

  • reference (Source, default: 'human' ) –

    The source taken as ground truth.

  • measured (Source, default: 'judge' ) –

    The source compared with it.

  • by (str | None, default: None ) –

    When given, only measured examples whose by is this labeller (one judge model) count; a label with another by, or none, is left out.

  • positive (str, default: 'good' ) –

    The positive class.

Returns:

  • dict[str, list[tuple[str, str]]] –

    {topic: [(measured, reference), ...]} for every topic with a reference label, in topic

  • dict[str, list[tuple[str, str]]] –

    order; a topic where no subject was labelled by both maps to []. Never raises.

agreement

agreement(
    examples: Iterable[Example],
    *,
    reference: Source = "human",
    measured: Sequence[Source] = ("judge", "model"),
    positive: str = "good"
) -> dict[str, dict[str, Agreement]]

Return, per topic, each measured source's Agreement with the reference source.

Labels are read as binary: positive or not. Only subjects both sources labelled count. A topic with no reference label is absent; a source with no shared subject reports n == 0.

Returns:

  • dict[str, dict[str, Agreement]] –

    {topic: {source: Agreement}}.

few_shot

few_shot(
    examples: Iterable[Example],
    k: int,
    *,
    topic: str,
    source: Source = "human"
) -> list[Example]

Return up to k examples of one topic from one source, balanced across classes.

The pick is deterministic (a hash of the subject orders each class) and round-robin across the label values, rarest class first, so a judge shown the examples sees every class the examples hold. Numeric labels are one class each, so a numeric topic is simply a stable pick.

Parameters:

  • examples (Iterable[Example]) –

    Examples from any topic and source.

  • k (int) –

    The most to return.

  • topic (str) –

    The topic to pick from.

  • source (Source, default: 'human' ) –

    The source to pick from (human by default: ground truth).

Returns:

  • list[Example] –

    The picked examples, at most k. Never raises.

llm_ports.learn.stats

The statistics a gate reads: agreement between two labellers, and a multiple-testing mask.

  • cohen_kappa / gwet_ac1 / kappa_ci: chance-corrected agreement of two labellers on the same binary items. Read kappa; read AC1 beside it where one answer dominates (kappa's chance correction degenerates there).
  • bh_fdr: which of a family of p-values survive false-discovery-rate control, Benjamini-Yekutieli by default (valid under arbitrary dependence), plain Benjamini-Hochberg on request.
Invariants
  • Pure, standard library only.
  • A statistic the data cannot define is None, never a number.

Rationale: ADR-0010.

cohen_kappa

cohen_kappa(pairs: Pairs) -> float | None

Return Cohen's kappa over binary (a, b) label pairs.

Both labels are "yes" or anything else (read as no); abstentions belong outside pairs.

Returns:

  • float | None –

    Kappa, or None when it is undefined: under two pairs, or both labellers one-sided on the

  • float | None –

    same answer (chance agreement 1.0).

kappa_ci

kappa_ci(
    pairs: Pairs, z: float = Z95
) -> tuple[float, float] | None

Return the confidence interval of Cohen's kappa over binary (a, b) label pairs.

The interval is kappa +- z * SE, with SE the large-sample standard error of Fleiss, Cohen and Everitt (1969), the one statsmodels' cohens_kappa reports; each end is clipped to [-1, 1].

Parameters:

  • pairs (Pairs) –

    As cohen_kappa.

  • z (float, default: Z95 ) –

    The normal quantile; Z95 (default) for a 95% interval.

Returns:

  • tuple[float, float] | None –

    (low, high), or None wherever cohen_kappa is None.

gwet_ac1

gwet_ac1(pairs: Pairs) -> float | None

Return Gwet's AC1 over binary (a, b) label pairs.

Chance agreement is 2 * pi * (1 - pi), with pi the pooled share of "yes", so the statistic stays defined and bounded where one answer dominates.

Returns:

  • float | None –

    AC1, or None under two pairs.

bh_fdr

bh_fdr(
    pvals: Sequence[float | None],
    alpha: float = 0.1,
    *,
    dependent: bool = True
) -> list[bool]

Return which p-values survive false-discovery-rate control at alpha, in input order.

The step-up compares each p to rank / (m * c) * alpha (never p * m * c / rank to alpha: the two float orders disagree at the threshold on grid p-values such as k / (B + 1)).

Parameters:

  • pvals (Sequence[float | None]) –

    The family's p-values; a None is an untested hypothesis and never passes.

  • alpha (float, default: 0.1 ) –

    The FDR level.

  • dependent (bool, default: True ) –

    True (default) for Benjamini-Yekutieli, which multiplies the threshold's denominator by c(m) = sum(1/i) and controls FDR under any dependence between the tests; False for plain Benjamini-Hochberg (valid under independence or PRDS).

Returns:

  • list[bool] –

    One bool per p-value. Never raises.

Invariants
  • The family size m counts tested hypotheses only: a None neither passes nor loosens the bar for the others.

llm_ports.learn.stores

Ledgers and example stores that need nothing but the standard library.

  • MemoryRunLedger, MemoryExampleStore: in-process, for tests and one-shot scripts.
  • JsonlRunLedger, JsonlExampleStore: one JSON object per line in a file, so a loop runs and keeps its history anywhere with zero setup.
  • union: read several example stores as one.

A database-backed ledger is an adapter in the consuming application: it implements the two RunLedger methods over its own table. Both ledgers here id each row with a fresh UUID (Written.ref, LedgerRow.ref). The model registries are in learn.registry.

Invariants
  • Never raises for a backend failure: an unreadable or unwritable file is an unavailable outcome; a value that is not JSON-able is a request_fault.
  • An unreadable store never reads as an empty one.

Rationale: ADR-0010.

MemoryRunLedger

MemoryRunLedger(*, now: Callable[[], datetime] = _utcnow)

A RunLedger held in a list (rows), oldest first.

Create an empty ledger.

Parameters:

  • now (Callable[[], datetime], default: _utcnow ) –

    The clock that stamps each row; injectable for tests.

append async

append(
    kind: str,
    *,
    subject: str,
    params: Mapping[str, Any],
    metrics: Mapping[str, Any]
) -> Written

Record one run. Never raises.

latest async

latest(
    kind: str,
    *,
    subject: str | None = None,
    decisions: Sequence[Decision] | None = None
) -> RowRead

Read the newest matching row. Never raises.

JsonlRunLedger

JsonlRunLedger(
    path: str | Path,
    *,
    now: Callable[[], datetime] = _utcnow
)

A RunLedger in a JSON-lines file: one row per line, appended, never rewritten.

Use the ledger at path (created on first append).

Parameters:

  • path (str | Path) –

    The file.

  • now (Callable[[], datetime], default: _utcnow ) –

    The clock that stamps each row; injectable for tests.

append async

append(
    kind: str,
    *,
    subject: str,
    params: Mapping[str, Any],
    metrics: Mapping[str, Any]
) -> Written

Append one row. Never raises.

latest async

latest(
    kind: str,
    *,
    subject: str | None = None,
    decisions: Sequence[Decision] | None = None
) -> RowRead

Read the newest matching row. Never raises; a missing file is an empty ledger.

MemoryExampleStore

MemoryExampleStore(examples: Iterable[Example] = ())

A writable ExampleStore held in a list (items).

Create a store holding examples.

examples async

examples(
    topics: Sequence[str],
    *,
    since: datetime | None = None,
    source: Source | None = None
) -> ExamplesRead

Read matching examples. Never raises.

put async

put(examples: Sequence[Example]) -> Written

Store examples. Never raises.

JsonlExampleStore

JsonlExampleStore(path: str | Path)

A writable ExampleStore in a JSON-lines file: one example per line.

Use the store at path (created on first put).

examples async

examples(
    topics: Sequence[str],
    *,
    since: datetime | None = None,
    source: Source | None = None
) -> ExamplesRead

Read matching examples. Never raises; a missing file is an empty store.

put async

put(examples: Sequence[Example]) -> Written

Append examples. Never raises.

union

union(*stores: ExampleStore) -> ExampleStore

Return one read-only store over stores.

Returns:

  • ExampleStore –

    A store whose read is every store's read concatenated, or unavailable (naming the store)

  • ExampleStore –

    when any one cannot be read: a partial read would undercount a topic's labels silently.

llm_ports.learn.registry

Model registries that need nothing but the standard library.

  • MemoryModelRegistry: in-process, for tests and one-shot scripts.
  • JsonlModelRegistry: an append-only JSON-lines event log (register and alias events), folded on read, so every alias move stays on the record.

A tracker's registry (MLflow's, for one) is an adapter that implements the same four methods.

Invariants
  • Never raises for a backend failure: an unreadable or unwritable file is an unavailable outcome; a model that is not JSON-able, an empty name or alias, an artifact_uri or run_ref of the wrong type, or an alias set to a version that does not exist is a request_fault.
  • Versions of one name are numbered from 1 and never change once registered; only aliases move.
  • One writer per JsonlModelRegistry file: the next version number is read from the file.

Rationale: ADR-0016.

MemoryModelRegistry

MemoryModelRegistry(
    *, now: Callable[[], datetime] = _utcnow
)

A ModelRegistry held in memory.

Create an empty registry.

Parameters:

  • now (Callable[[], datetime], default: _utcnow ) –

    The clock that stamps each version; injectable for tests.

register async

register(
    name: str,
    *,
    model: Mapping[str, Any],
    artifact_uri: str | None = None,
    run_ref: str | None = None,
    tags: Mapping[str, str] | None = None
) -> Written

Register a new version. Never raises.

set_alias async

set_alias(name: str, alias: str, version: int) -> Written

Point alias at version. Never raises.

by_alias async

by_alias(name: str, alias: str) -> VersionRead

Read the version alias points at. Never raises.

get async

get(name: str, version: int) -> VersionRead

Read one version. Never raises.

JsonlModelRegistry

JsonlModelRegistry(
    path: str | Path,
    *,
    now: Callable[[], datetime] = _utcnow
)

A ModelRegistry in a JSON-lines event log: appended, never rewritten.

Use the registry at path (created on first write).

Parameters:

  • path (str | Path) –

    The file.

  • now (Callable[[], datetime], default: _utcnow ) –

    The clock that stamps each event; injectable for tests.

register async

register(
    name: str,
    *,
    model: Mapping[str, Any],
    artifact_uri: str | None = None,
    run_ref: str | None = None,
    tags: Mapping[str, str] | None = None
) -> Written

Append a register event. Never raises.

set_alias async

set_alias(name: str, alias: str, version: int) -> Written

Append an alias event. Never raises.

by_alias async

by_alias(name: str, alias: str) -> VersionRead

Read the version alias points at. Never raises; a missing file is an empty registry.

get async

get(name: str, version: int) -> VersionRead

Read one version. Never raises; a missing file is an empty registry.