Learning loops¶
Every learning loop is the same moves: read evidence, fit or measure, hold the result to a gate,
act on the verdict, and record the run. llm_ports.learn holds the parts that do not depend on
what is learned, on the standard library alone, so it lifts into any codebase.
A loop implements four stages and nothing else:
| stage | what it does | may raise? |
|---|---|---|
evidence() |
read what it learns from (a repository call, a file) | yes: recorded as could_not_run |
fit(evidence) |
fit or measure; returns Fit(model, metrics) |
yes: same |
clauses(fit, evidence) |
the gate's evaluated clauses | yes: same |
apply(verdict, fit) |
act: store the fit on pass, keep the incumbent otherwise |
yes: recorded as apply_failed, the verdict stands |
In the vocabulary of a learning-loop registry, the four stages map onto predict / resolve / score / consume:
| registry move | where it lives |
|---|---|
| predict | your producer writes predictions (or an Example with source="judge" / "model") |
| resolve | evidence() reads them back joined to what happened (source="outcome" or "human") |
| score | fit() measures, clauses() holds the measurement to a gate |
| consume | apply() acts on the verdict, and something downstream READS what it stored |
A loop that never consumes is a report: name the reader of apply()'s result before you build it.
run(loop, ledger) drives them, decides the verdict with gate.decide, and appends exactly one
ledger row per run, every exit included. It never raises.
The verdict¶
A clause is a condition, evaluated: passed is True, False, or None when it could not be
evaluated. Its role says what failing it means:
bar(default): measured and did not clear:fail.floor: a sample floor, or a condition not yet proven:insufficient. A failingflooralso turns a failingbarintoinsufficient, because a bar measured on too small a sample is noise.veto: a disqualifier:fail, whatever else holds.
A clause that is None never passes. By default it reads as a failure; a loop with
unevaluated = "insufficient" reads it as unproven instead, which is how a promote / hold /
reject gate maps on exactly: pass = promote, insufficient = hold, fail = reject.
The verdict keeps every clause (Verdict.blocking lists each one that did not pass) and names
the first that decided it (blocker), so a page can show all the reasons, not only one.
The ledger¶
RunLedger has two methods, append and latest. MemoryRunLedger and JsonlRunLedger ship
here; a database-backed ledger is an adapter in your application over its own table. A row's
verdict lives under metrics["learn_verdict"] (VERDICT_KEY), a name chosen so it never collides
with keys an existing ledger already uses. latest(kind, decisions=("pass", "fail")) reads the
last decisive run, so a later could_not_run does not hide what the last real measurement said.
Labelled examples¶
Example(topic, subject, value, source, by, at, reason, ref) is one label from a human, a
model, a production judge, or an outcome resolved from the world. An ExampleStore reads
them by topic; stores are adapters over wherever labels already live, and stores.union reads
several as one. Three pure readers serve a coverage report and a judge:
examples.coverage(examples, topics, target=...): per topic, how many reference labels exist and the gap to the target; a topic with no label is reported with zeros.examples.agreement(examples): per topic, how a judge or model agrees with the human labels (share, Cohen's kappa with its 95% interval, Gwet's AC1).examples.pairs(examples, by=...): the(measured, reference)pairs behind those numbers, per topic, optionally for one judge model only, so a caller can pool them across topics.examples.few_shot(examples, k, topic=...): a deterministic pick balanced across classes, for a judge's prompt.
Consume: a judge earns trust from the labels¶
evals.calibration.trust(pairs, replay=...) is the consumer every judge shares: a judge's
verdicts may act (page, gate, promote) only when it agrees with the human on at least
min_labelled replies (kappa, or with bound="low" its interval's lower end, at min_kappa) AND
its seeded-bad replay is calibrated. It returns a Trust carrying the numbers, never raises, and
a judge with no labels is untrusted, so swapping the judge model resets trust instead of
inheriting it. Record which model gave each judge label (by; on a trace handle,
score(..., metadata={"by": model})) or pairs(by=...) cannot tell them apart.
The whole loop¶
"""A learning loop from evidence to a recorded verdict, on the standard library alone.
The loop learns a decision threshold from labelled outcomes, holds the fit to a gate, applies it
only when the gate passes, and leaves exactly one ledger row per run, whatever happens. A loop
holds only what is specific to it (`evidence`, `fit`, `clauses`, `apply`); `run` does the rest.
The second half reads labelled examples the way a judge or a coverage report does: how many
human labels each topic has, how far a judge agrees with them, and a balanced few-shot pick. The
last step CONSUMES them: a judge model's verdicts are trusted only when its agreement with the
human labels, and its seeded-bad replay, earn it (`evals.calibration.trust`).
"""
import asyncio
import tempfile
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from llm_ports.evals import calibration
from llm_ports.learn import Clause, Example, Fit, Verdict, examples, gate, run
from llm_ports.learn.stores import JsonlRunLedger, MemoryExampleStore
@dataclass
class Threshold:
"""Learn the score above which outcomes are mostly wins, and adopt it only when earned."""
scores: Sequence[tuple[float, bool]] # (score, won)
kind: str = "threshold"
subject: str = "demo"
params: Mapping[str, Any] = field(default_factory=lambda: {"min_n": 20, "min_hit": 0.6})
adopted: float | None = None
async def evidence(self) -> Sequence[tuple[float, bool]]:
return self.scores # a real loop reads a repository here
def fit(self, evidence: Sequence[tuple[float, bool]]) -> Fit:
best = max(
sorted({s for s, _ in evidence}), key=lambda t: _hit([w for s, w in evidence if s >= t]), default=None
)
above = [w for s, w in evidence if best is not None and s >= best]
return Fit(model={"threshold": best}, metrics={"n": len(evidence), "n_above": len(above), "hit": _hit(above)})
def clauses(self, fit: Fit, evidence: Sequence[tuple[float, bool]]) -> Sequence[Clause]:
return [
gate.at_least("n", fit.metrics["n"], self.params["min_n"], role="floor"),
gate.at_least("hit_rate", fit.metrics["hit"], self.params["min_hit"]),
]
async def apply(self, verdict: Verdict, fit: Fit) -> str:
if not verdict.passed:
return "held" # the incumbent stays
self.adopted = fit.model["threshold"]
return "stored"
class Unreachable(Threshold):
"""The same loop, with its store down."""
async def evidence(self) -> Sequence[tuple[float, bool]]:
raise ConnectionError("the outcomes table is unreachable")
def _hit(wins: Sequence[bool]) -> float | None:
return sum(wins) / len(wins) if wins else None
async def main() -> None:
with tempfile.TemporaryDirectory() as tmp:
ledger = JsonlRunLedger(Path(tmp) / "runs.jsonl")
# Too little evidence: `insufficient`, nothing adopted, and still one row.
early = Threshold([(0.9, True), (0.2, False)])
first = await run(early, ledger)
assert (first.verdict.decision, first.verdict.blocker, first.applied) == ("insufficient", "n", "held")
assert early.adopted is None
# Enough evidence: the gate passes and the loop adopts the fitted threshold.
data = [(s / 10, s >= 6) for s in range(10)] * 3
later = Threshold(data)
second = await run(later, ledger)
assert second.verdict.passed and second.applied == "stored" and later.adopted == 0.6
# A stage that raises is a recorded `could_not_run`, never an exception.
broken = await run(Unreachable([]), ledger)
assert broken.verdict.decision == "could_not_run" and broken.applied == ""
assert broken.verdict.reason == "evidence: ConnectionError: the outcomes table is unreachable"
# The ledger answers "what is the last decisive verdict?" without a later failure hiding it.
decisive = await ledger.latest("threshold", decisions=("pass", "fail"))
assert decisive.row is not None and decisive.row.verdict == second.verdict
assert decisive.row.metrics["learn_model"] == {"threshold": 0.6}
assert len(Path(ledger.path).read_text(encoding="utf-8").splitlines()) == 3, "one row per run"
for got in (first, second, broken):
print(f"{got.kind}: {got.verdict.decision:<13} applied={got.applied or '-':<7} {got.verdict.reason}")
# Labelled examples: one contract over every store.
store = MemoryExampleStore(
[Example("llm:summary", f"r{i}", "good" if i % 3 else "bad", "human", by="operator") for i in range(9)]
+ [Example("llm:summary", f"r{i}", "good", "judge", by="9b-judge") for i in range(9)]
)
read = await store.examples(["llm:summary", "llm:triage"])
assert read.ok
cov = examples.coverage(read.examples, ["llm:summary", "llm:triage"], target=20)
assert (cov["llm:summary"].have, cov["llm:summary"].gap) == (9, 11)
assert cov["llm:triage"].have == 0, "a topic with no labels is reported, not dropped"
judge = examples.agreement(read.examples)["llm:summary"]["judge"]
assert judge.n == 9 and judge.agree is not None and round(judge.agree, 2) == 0.67 # the judge passes everything
shots = examples.few_shot(read.examples, 4, topic="llm:summary")
assert {e.value for e in shots} == {"good", "bad"}, "a judge shown examples sees every class"
for topic, c in cov.items():
print(f"{topic}: {c.have} human labels, gap {c.gap}")
print(f"judge vs human on llm:summary: agree {judge.agree:.2f}, kappa {judge.kappa}")
# Consume: may this judge model's verdicts act? Pool its pairs over every topic (`by` keeps one
# model's labels, so a new model never inherits the old one's record) and hold them to both
# directions: agreement with the human, and a seeded-bad replay it must fail.
pooled = [p for ps in examples.pairs(read.examples, by="9b-judge").values() for p in ps]
replay = [
{"role": "summary", "kind": "good", "after": "pass"},
{"role": "summary", "kind": "tbd", "after": "fail"},
]
verdict = calibration.trust(pooled, replay=replay, min_labelled=9)
assert not verdict.trusted and verdict.agreement.kappa == 0.0 and "under 0.6" in verdict.why # passes everything
assert (
calibration.trust(examples.pairs(read.examples, by="new-judge")["llm:summary"], replay=replay).agreement.n == 0
)
print(f"trust 9b-judge: {verdict.trusted} ({verdict.why})")
asyncio.run(main())
Source: examples/learning_loop.py