A standalone learning-loop core inside llm-ports¶
Decided in 1.5.0.
Context and problem statement¶
A consuming platform runs dozens of learning loops (signal weights, calibrations, judges, grids)
and each re-wrote the same parts: a gate with its own verdict shape, its own way of recording a
run, its own idea of "not enough evidence". Three gate shapes and an earned() boolean per fit
meant no page could read a verdict generically, a clause that could not be evaluated was handled
differently in each, and a loop that stopped deciding left no trace. LLM judges need the same
parts: labelled examples, agreement statistics, a trust gate. Where should one shared core live,
and what must it guarantee?
Decision drivers¶
- Usable in an unrelated codebase: no vendor, no dependency on the rest of this library.
- One verdict vocabulary (
pass | fail | insufficient | could_not_run) that can express every existing gate exactly, including a promote / hold / reject gate, without changing a threshold. - Missing evidence never passes, and a run that could not decide is recorded, not silent.
- Cheap to adopt: a new loop is four methods.
Considered options¶
- A new repository and distribution for the core.
- A subpackage of llm-ports, enforced standalone.
- MLflow tracking and registry as the ledger and gate store.
Decision outcome¶
Option 2: llm_ports.learn.
- llm-ports already has the release, typing, import-layer and executed-docs machinery a published core needs; a new repository would rebuild it and add a second pin to every consumer.
- An import-linter
forbiddencontract stopslearnimporting any otherllm_portsmodule or a third-party package, andtests/contract/test_learn_standalone.pycopies the package alone into an empty tree and runs a loop there, so "lift it out" is executed, not claimed. Splitting it into its own distribution later is a file move. - Its Protocols (
RunLedger,ExampleStore,Loop) live inlearn/, notports/: a deliberate exception to "a port lives inports/", becauseports/carries LLM types and the standalone contract forbids reaching it. learnsits in the bottom layer besideports;evals.calibrationre-exportscohen_kappaandgwet_ac1fromlearn.stats, so no consumer changes.- The gate gives each clause a role (
bar,floor,veto) and takes anunevaluatedpolicy (failorinsufficient); with those, a strict AND, a sample floor and a promote / hold / reject gate are all the samedecide. - A recorded verdict lives under its own metric key,
learn_verdict, because an existing ledger already usesgateandverdictfor other shapes. bh_fdrdefaults to Benjamini-Yekutieli (valid under any dependence between tests);dependent=Falsegives Benjamini-Hochberg.
MLflow (option 3) is rejected as the ledger: it is a new service with its own storage, backup and
auth, a second registry beside the consumer's, and the "models" these loops fit are small
parameter rows, not artifacts. An MLflow RunLedger adapter stays possible as an optional extra.
Consequences¶
- Good: one verdict type and one run record for every loop; a new loop is four methods; the core runs anywhere with a JSON-lines ledger and zero setup.
- Good: a loop that cannot decide, or whose action fails, leaves a row saying so.
- Bad: the package name says "llm" for a core that is not LLM-specific; the trigger to split it out is a consumer that objects to the name.
- Bad: a database ledger is the consumer's adapter to write (two methods) until a portable Postgres adapter ships as an extra.