Skip to content

Learning loops

Every learning loop is the same moves: read evidence, fit or measure, hold the result to a gate, act on the verdict, and record the run. llm_ports.learn holds the parts that do not depend on what is learned, on the standard library alone, so it lifts into any codebase.

A loop implements four stages and nothing else:

stage what it does may raise?
evidence() read what it learns from (a repository call, a file) yes: recorded as could_not_run
fit(evidence) fit or measure; returns Fit(model, metrics) yes: same
clauses(fit, evidence) the gate's evaluated clauses yes: same
apply(verdict, fit) act: store the fit on pass, keep the incumbent otherwise yes: recorded as apply_failed, the verdict stands

In the vocabulary of a learning-loop registry, the four stages map onto predict / resolve / score / consume:

registry move where it lives
predict your producer writes predictions (or an Example with source="judge" / "model")
resolve evidence() reads them back joined to what happened (source="outcome" or "human")
score fit() measures, clauses() holds the measurement to a gate
consume apply() acts on the verdict, and something downstream READS what it stored

A loop that never consumes is a report: name the reader of apply()'s result before you build it.

run(loop, ledger) drives them, decides the verdict with gate.decide, and appends exactly one ledger row per run, every exit included. It never raises.

The verdict

A clause is a condition, evaluated: passed is True, False, or None when it could not be evaluated. Its role says what failing it means:

  • bar (default): measured and did not clear: fail.
  • floor: a sample floor, or a condition not yet proven: insufficient. A failing floor also turns a failing bar into insufficient, because a bar measured on too small a sample is noise.
  • veto: a disqualifier: fail, whatever else holds.

A clause that is None never passes. By default it reads as a failure; a loop with unevaluated = "insufficient" reads it as unproven instead, which is how a promote / hold / reject gate maps on exactly: pass = promote, insufficient = hold, fail = reject.

The verdict keeps every clause (Verdict.blocking lists each one that did not pass) and names the first that decided it (blocker), so a page can show all the reasons, not only one.

The ledger

RunLedger has two methods, append and latest. MemoryRunLedger and JsonlRunLedger ship here; a database-backed ledger is an adapter in your application over its own table. A row's verdict lives under metrics["learn_verdict"] (VERDICT_KEY), a name chosen so it never collides with keys an existing ledger already uses. latest(kind, decisions=("pass", "fail")) reads the last decisive run, so a later could_not_run does not hide what the last real measurement said.

Labelled examples

Example(topic, subject, value, source, by, at, reason, ref) is one label from a human, a model, a production judge, or an outcome resolved from the world. An ExampleStore reads them by topic; stores are adapters over wherever labels already live, and stores.union reads several as one. Three pure readers serve a coverage report and a judge:

  • examples.coverage(examples, topics, target=...): per topic, how many reference labels exist and the gap to the target; a topic with no label is reported with zeros.
  • examples.agreement(examples): per topic, how a judge or model agrees with the human labels (share, Cohen's kappa with its 95% interval, Gwet's AC1).
  • examples.pairs(examples, by=...): the (measured, reference) pairs behind those numbers, per topic, optionally for one judge model only, so a caller can pool them across topics.
  • examples.few_shot(examples, k, topic=...): a deterministic pick balanced across classes, for a judge's prompt.

Consume: a judge earns trust from the labels

evals.calibration.trust(pairs, replay=...) is the consumer every judge shares: a judge's verdicts may act (page, gate, promote) only when it agrees with the human on at least min_labelled replies (kappa, or with bound="low" its interval's lower end, at min_kappa) AND its seeded-bad replay is calibrated. It returns a Trust carrying the numbers, never raises, and a judge with no labels is untrusted, so swapping the judge model resets trust instead of inheriting it. Record which model gave each judge label (by; on a trace handle, score(..., metadata={"by": model})) or pairs(by=...) cannot tell them apart.

The whole loop

examples/learning_loop.py
"""A learning loop from evidence to a recorded verdict, on the standard library alone.

The loop learns a decision threshold from labelled outcomes, holds the fit to a gate, applies it
only when the gate passes, and leaves exactly one ledger row per run, whatever happens. A loop
holds only what is specific to it (`evidence`, `fit`, `clauses`, `apply`); `run` does the rest.

The second half reads labelled examples the way a judge or a coverage report does: how many
human labels each topic has, how far a judge agrees with them, and a balanced few-shot pick. The
last step CONSUMES them: a judge model's verdicts are trusted only when its agreement with the
human labels, and its seeded-bad replay, earn it (`evals.calibration.trust`).
"""

import asyncio
import tempfile
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any

from llm_ports.evals import calibration
from llm_ports.learn import Clause, Example, Fit, Verdict, examples, gate, run
from llm_ports.learn.stores import JsonlRunLedger, MemoryExampleStore


@dataclass
class Threshold:
    """Learn the score above which outcomes are mostly wins, and adopt it only when earned."""

    scores: Sequence[tuple[float, bool]]  # (score, won)
    kind: str = "threshold"
    subject: str = "demo"
    params: Mapping[str, Any] = field(default_factory=lambda: {"min_n": 20, "min_hit": 0.6})
    adopted: float | None = None

    async def evidence(self) -> Sequence[tuple[float, bool]]:
        return self.scores  # a real loop reads a repository here

    def fit(self, evidence: Sequence[tuple[float, bool]]) -> Fit:
        best = max(
            sorted({s for s, _ in evidence}), key=lambda t: _hit([w for s, w in evidence if s >= t]), default=None
        )
        above = [w for s, w in evidence if best is not None and s >= best]
        return Fit(model={"threshold": best}, metrics={"n": len(evidence), "n_above": len(above), "hit": _hit(above)})

    def clauses(self, fit: Fit, evidence: Sequence[tuple[float, bool]]) -> Sequence[Clause]:
        return [
            gate.at_least("n", fit.metrics["n"], self.params["min_n"], role="floor"),
            gate.at_least("hit_rate", fit.metrics["hit"], self.params["min_hit"]),
        ]

    async def apply(self, verdict: Verdict, fit: Fit) -> str:
        if not verdict.passed:
            return "held"  # the incumbent stays
        self.adopted = fit.model["threshold"]
        return "stored"


class Unreachable(Threshold):
    """The same loop, with its store down."""

    async def evidence(self) -> Sequence[tuple[float, bool]]:
        raise ConnectionError("the outcomes table is unreachable")


def _hit(wins: Sequence[bool]) -> float | None:
    return sum(wins) / len(wins) if wins else None


async def main() -> None:
    with tempfile.TemporaryDirectory() as tmp:
        ledger = JsonlRunLedger(Path(tmp) / "runs.jsonl")

        # Too little evidence: `insufficient`, nothing adopted, and still one row.
        early = Threshold([(0.9, True), (0.2, False)])
        first = await run(early, ledger)
        assert (first.verdict.decision, first.verdict.blocker, first.applied) == ("insufficient", "n", "held")
        assert early.adopted is None

        # Enough evidence: the gate passes and the loop adopts the fitted threshold.
        data = [(s / 10, s >= 6) for s in range(10)] * 3
        later = Threshold(data)
        second = await run(later, ledger)
        assert second.verdict.passed and second.applied == "stored" and later.adopted == 0.6

        # A stage that raises is a recorded `could_not_run`, never an exception.
        broken = await run(Unreachable([]), ledger)
        assert broken.verdict.decision == "could_not_run" and broken.applied == ""
        assert broken.verdict.reason == "evidence: ConnectionError: the outcomes table is unreachable"

        # The ledger answers "what is the last decisive verdict?" without a later failure hiding it.
        decisive = await ledger.latest("threshold", decisions=("pass", "fail"))
        assert decisive.row is not None and decisive.row.verdict == second.verdict
        assert decisive.row.metrics["learn_model"] == {"threshold": 0.6}
        assert len(Path(ledger.path).read_text(encoding="utf-8").splitlines()) == 3, "one row per run"
        for got in (first, second, broken):
            print(f"{got.kind}: {got.verdict.decision:<13} applied={got.applied or '-':<7} {got.verdict.reason}")

    # Labelled examples: one contract over every store.
    store = MemoryExampleStore(
        [Example("llm:summary", f"r{i}", "good" if i % 3 else "bad", "human", by="operator") for i in range(9)]
        + [Example("llm:summary", f"r{i}", "good", "judge", by="9b-judge") for i in range(9)]
    )
    read = await store.examples(["llm:summary", "llm:triage"])
    assert read.ok
    cov = examples.coverage(read.examples, ["llm:summary", "llm:triage"], target=20)
    assert (cov["llm:summary"].have, cov["llm:summary"].gap) == (9, 11)
    assert cov["llm:triage"].have == 0, "a topic with no labels is reported, not dropped"

    judge = examples.agreement(read.examples)["llm:summary"]["judge"]
    assert judge.n == 9 and judge.agree is not None and round(judge.agree, 2) == 0.67  # the judge passes everything

    shots = examples.few_shot(read.examples, 4, topic="llm:summary")
    assert {e.value for e in shots} == {"good", "bad"}, "a judge shown examples sees every class"
    for topic, c in cov.items():
        print(f"{topic}: {c.have} human labels, gap {c.gap}")
    print(f"judge vs human on llm:summary: agree {judge.agree:.2f}, kappa {judge.kappa}")

    # Consume: may this judge model's verdicts act? Pool its pairs over every topic (`by` keeps one
    # model's labels, so a new model never inherits the old one's record) and hold them to both
    # directions: agreement with the human, and a seeded-bad replay it must fail.
    pooled = [p for ps in examples.pairs(read.examples, by="9b-judge").values() for p in ps]
    replay = [
        {"role": "summary", "kind": "good", "after": "pass"},
        {"role": "summary", "kind": "tbd", "after": "fail"},
    ]
    verdict = calibration.trust(pooled, replay=replay, min_labelled=9)
    assert not verdict.trusted and verdict.agreement.kappa == 0.0 and "under 0.6" in verdict.why  # passes everything
    assert (
        calibration.trust(examples.pairs(read.examples, by="new-judge")["llm:summary"], replay=replay).agreement.n == 0
    )
    print(f"trust 9b-judge: {verdict.trusted} ({verdict.why})")


asyncio.run(main())

Source: examples/learning_loop.py