The learning-loop core: one spine, one gate, one ledger¶
Status: DESIGN, 2026-11-07, revision 2 (answers the review). Supersedes the scope of EVAL-LABELS-CONSOLIDATION.md, which stays
as the detail of ONE loop kind (LLM labels, §LLM labels below). Owner of the decision: this file.
Facts read from origin/main of each repo on 2026-11-07 (trade-data 6e941731, trade-ui,
ci-kit, work-timeline) with git show / git grep; every load-bearing claim carries its
file:line. A claim not checked that way says UNVERIFIED.
What was asked, in priority order¶
- Centralize the core in one good home: portable, zero EDGE imports, storage behind a port, usable in an unrelated codebase, including a portable Postgres ledger (Decision 2).
- Make a new loop cheap: this doc states the cost in files and lines; the first new consumer (a multi-symbol daily trend loop) measures it.
- Then fan out: existing loops adopt it, judges and copilot read the shared store.
Hard limits: trade-data/docs/CARD-SYSTEM-GATE.md and docs/PATTERN-GATE.md are SEALED (never
edited; the core may only WRAP what they decide); pattern strategy_weight stays 0. trade-data's
existing learn engine is reused, never forked ("it must not end with two engines"). No fine-tuning
or training run; no Anthropic API (strong-model labelling = Gemini via the box gateway or llm-box
models).
The shape¶
Every loop in this platform is the same five moves:
evidence in labelled outcomes fit / calibrate gate act + audit
(series, events, ──▶ (outcome resolved, ──▶ (a model, a weight, ──▶ (evidence ──▶ ──▶ (promote at weight,
replies, picks) human / model / a threshold, a verdict) show, page) + one
judge label) calibration) ledger row per run
trade-data already names the stages predict / resolve / score / consume / surface
(service/platform/loops.py:9-16) and registers ~45 loops against them. The core does not
replace that vocabulary; it supplies the parts every loop re-wrote: the gate, the run record, the
labelled-example contract, the statistics.
Where the shape occurs (and what each is missing)¶
| loop | evidence | outcome / label | fit | gate today | missing |
|---|---|---|---|---|---|
| picker signal weights | picker runs | picker_outcomes (honest fill, EOD) |
signals/weight_learning.py IC-EWA |
signals/eval_metrics.py _blockers (:526) |
standard verdict row |
| prob calibration | p_up per pick | outcome | signals/prob_calibration.py Platt |
usecases/fit_prob_calibration.py earned() |
verdict lives in code, not ledger |
| hazard exit / hold duration | paths | MFE timing | fit_hazard_exit.py, learn_hold_duration.py |
concordance > 0.53 / eff_n ≥ 30 | same |
| meta-label | features | take/skip outcome | signals/meta_label.py |
identity under 40 samples | same |
| config / exit grids | replay | graded path | grid | signals/promotion_gate.py evaluate_gate (not wired live) |
wiring |
| level-source champion / challenger | level events | reaction hits | — | domain/promotion.py evaluate (promote / hold / reject) |
ledger row |
| level calibration | level reactions | reaction / no reaction | calibrate_level_reactions.py |
signals/level_calibration.py coarse gate cell |
standard verdict |
| weight ledger | learned weights | — | — | quant.weight_ledger (mig 074), versioned, operator opt-in to apply |
the APPLY stage of the picker loop: read, not replaced |
| pattern ladder | detectors | base rates | — | PATTERN-GATE.md ladder (SEALED) |
wrap only |
| Pattern Lexicon | detectors | base rates | — | PATTERN-GATE.md (SEALED) |
wrap only |
| Daily Card | card | session outcome | — | CARD-SYSTEM-GATE.md (SEALED) |
wrap only |
| Setup Radar | setups | forward return | — | setup_radar.py wilson_lb (:284) |
gate, ledger |
| event study | calendar events | forward return | usecases/event_study.py |
fdr_families (:215) |
ledger, gate |
| churn / news | articles | human-labelled fields (203) | — | usecases/churn_eval.py pass/fail |
already the precedent |
| LLM judge trust | judged replies | human verdicts (~20) | — | platform/llm_judge_trust.py assess |
labels (§LLM labels) |
| LLM roles (34) | replies | none | — | sampled 9B judge (kappa ~0.19) | labels, coverage |
| trend / "what's popping" (new) | quant.ohlcv_daily_mv + calendar |
forward return | rank / logistic | — | everything: the proof consumer |
Gates in use today, three shapes and no shared type: promotion_gate.GateVerdict(promote,
clauses, blocker) (signals/promotion_gate.py:69), domain/promotion.PromotionDecision
(domain/promotion.py:71: promote | hold | reject; a clause that could not be evaluated
gives HOLD, a clause that LOOKED and disqualified gives REJECT, and REJECT wins over HOLD),
llm_judge_trust.Trust(model, trusted, reason, kappa…) (platform/llm_judge_trust.py:53), plus
earned() booleans in each fit usecase. The Research Ledger (signals/research_ledger.py) then
re-derives an earn state per run kind from each kind's ad-hoc metrics, because no run states
its verdict the same way. llm-ports itself has two more: evals.calibration.verdict (a judge's
calibration verdict) and provenance.GateVerdict (a grounding gate's). Both stay: they are
LLM-shaped results, not loop gates; evals.calibration builds on learn.stats (Decision 1).
The ledger is already in use under the name this core must NOT take: metrics.gate holds
{promote, blocker, clauses: dict[str, bool]}, written by compiler/regrade.py:126 and
strategy/lever_grid.py:284, read by research_ledger._promote_gate (:125-129, AUTHORITATIVE),
regrade.verdict_of (:139), jobs/handlers/replay.py:315,366, strategy/hunt.py:150, copilot's
chat tune agent and trade-ui TuneChatPanel.tsx:62-65. And metrics.verdict is not one
vocabulary: supported | refuted | inconclusive (regrade), probe_pass | probe_fail
(rag_eval), free text (ou_meanrev_diag). Decision 3 therefore writes under a new key.
Decision 1 — the core's home: llm_ports.learn¶
A new subpackage of llm-ports, enforced standalone:
- Why llm-ports and not a new repo. llm-ports is already the estate's published, portable
library: PyPI release by tag, strict mypy +
pyright --verifytypes100%, import-linter layers, an executed docs site, atest-lowestleg, and a core that installs with no vendor. A new repo would rebuild all of that for ~600 lines and add a second pin to every consumer: "should probably not have two". The pure statistics it needs (cohen_kappa,gwet_ac1) already live here (evals/calibration.py:54,75). - Why a standalone subpackage. An import-linter
forbiddencontract:llm_ports.learnimports nothing from the rest ofllm_portsand no vendor; it depends on the standard library only. Splitting it into its own distribution later is a file move, not a refactor. The name "llm" is the only cost; the trigger to split is a consumer that wantslearnand objects to the name. - Its Protocols live in
learn/, notports/. Deliberate exception to "core depends on a port inports/":ports/carries LLM types, and the standalone contract forbids reaching it. ADR-0010 records this. - Layering.
learnsits in the bottom layer besideports;evals(LLM labels, judge trust) builds on it.evals.calibration.cohen_kappa/gwet_ac1move tolearn.statsand stay re-exported fromevals.calibration(no consumer change).
Decision 2 — the run ledger: quant.learn_runs, behind a port¶
| option | for | against | verdict |
|---|---|---|---|
extend quant.learn_runs |
exists (migrations/076_learn_runs.sql), ~20 kinds already write it (learn_runs_repo.py:19-33), run_kind is free text with no CHECK (:11), grants for app/copilot/grafana, research_registry FKs onto it, latest_with_verdict already reads metrics->>'verdict' (learn_runs_repo.py:106), the Research Ledger page and /compiler/runs render it |
Postgres-only; a schema-specific adapter | chosen |
| MLflow tracking + registry | real UI, model registry | a NEW service (Postgres backend, backup, auth), a second registry beside loops.py, mig076 already rejected it for a single operator ("Deliberately NOT MLflow/W&B", :6); EDGE "models" are small param rows, not artifacts |
rejected for EDGE; possible later optional adapter for portable users |
| Langfuse | already deployed; datasets, scores, annotation queues | LLM-shaped (traces, observations); an options fit is not a trace | rejected as the ledger; KEPT as the LLM label store (§LLM labels) |
Portability comes from the port, not the table: learn.RunLedger (append, latest) with three
implementations: MemoryRunLedger and JsonlRunLedger (stdlib, in the core, so the library works
anywhere with zero setup), and trade-data's LearnRunsLedger over LearnRunsRepo (adapter,
trade-data). An MLflow adapter is an optional extra when an outside user asks for it.
A portable Postgres ledger (requirement 6) is a later llm-ports adapter, adapters/postgres/
behind a postgres extra (psycopg), owning one table of the same shape as quant.learn_runs
(run_kind, subject, params jsonb, metrics jsonb, created_at; quant.learn_runs names the
subject column symbol and adds git_sha and notes, mig 076, so trade-data's adapter maps
subject to symbol). It is NOT in slice 1: the core
proves the port with two stdlib ledgers first, and the first outside user who wants Postgres
gets the adapter. trade-data keeps its own adapter over LearnRunsRepo either way: its table has
grants, FKs and readers a generic adapter must not own.
LearnRunsLedger.append answers ok only after its own commit. LearnRunsRepo.append
leaves the commit to its caller (learn_runs_repo.py:57-58) and the generic fit route opens a
session it never commits (picker_replay_router.py:310-311), so an ok before commit would be
a row that rolls back while Run.recorded.ok says it landed: the silent green this design
closes. The adapter writes on its OWN session (a session factory, not the dispatcher's session),
so its commit neither commits nor discards apply's pending writes on the dispatcher's session;
after apply_failed those writes are the dispatcher's to roll back, and the row still lands.
Decision 3 — ONE gate: evidence in, verdict + audit row out¶
Clause(name, passed: bool | None, value, threshold, op, role) passed None = not evaluable
role ∈ bar | floor | veto default bar
Verdict(gate, subject, decision, clauses, blocker, reason) frozen, JSON-able
decision ∈ pass | fail | insufficient | could_not_run
decide(gate, clauses, *, subject="", unevaluated="fail") -> Verdict
decide, in order (first rule that applies wins; the blocker is the first clause, in the order
given, that the rule names):
- no clauses →
could_not_run. - a
vetoclause evaluated False →fail(we looked and it disqualifies, whatever the sample). - a
floorclause not True (False or None) →insufficient(a sample floor not met: anything else measured on that sample is noise, so a failingbaris not yet afail). unevaluated="fail"(default): any clause not True →fail.unevaluated="insufficient": a clause evaluated False →fail, else a clause that is None →insufficient.-
otherwise
pass. -
Fails closed. A None clause is never a pass under either policy. The default (
fail) ispromotion_gate._ok(:78) and the strict AND of each fit'searned();insufficientis the domain/promotion law ("a conjunct that could not be evaluated has not been satisfied", HOLD). - promote / hold / reject map exactly:
domain/promotionadopts withmin_sample,reaction_liftwhen the lift is positive but not robust, anddeflated_barasfloor(failing = unproven = HOLD),displacementandreaction_liftwith no lift at all asveto(REJECT), andunevaluated="insufficient". Thenpass=promote,insufficient=hold,fail=reject, and REJECT-beats-HOLD is rule 2 before rule 3. The clause's role is set by the gate when it evaluates the clause, so one clause can be avetoin one branch and afloorin another, asreaction_liftis there. - Every clause is kept, not only the blocker.
Verdict.clausesholds every clause in order andVerdict.blockinglists every clause that is not True, so a surface renders all reasons (whatPromotionDecision.reasonslists) and the clause map that "cannot hide a skipped conjunct" (domain/promotion.py:79-88). - Its own key in the ledger:
metrics.learn_verdict. A recorded run writesmetrics.learn_verdict = Verdict.as_dict()(itsdecisionis the verdict) and never writesmetrics.gateormetrics.verdict, whose readers are listed above. A kind that already writes those keeps writing them; nothing reading them changes. The Research Ledger gains ONE generic branch: a row withlearn_verdictmapspass→earned,fail→not_earned,insufficient→insufficient,could_not_run→error. It sits IMMEDIATELY after the existing_promote_gatebranch and before the per-kind extractors and_gate_kind(research_ledger.py:241-256):runmerges fit metrics into the top level, so a core loop whose fit reportsfdr_pass,dsrorpbowould otherwise be read by_gate_kindinstead of its verdict. A fit metricstatus="failed"still preempts it, as today (:241-243). A kind with both keys keeps today's state (so it rendersinsufficientasnot_earned) until its slice-5 PR stops writinggate.rigor()(:198-228) gains the same one generic branch (alearn_verdictrow's rigor read from its clauses), so a new loop gets a rigor without a_RIGORedit. A row with neither key staysinformational.latest_with_verdictis unchanged;LearnRunsLedger.latest(decisions=...)filters onmetrics->'learn_verdict'->>'decision'. - Audit = the ledger row.
loop.run(andloop.recordfor a gate run outside aLoop) appends one row. A gate is a pure functionevidence -> clauses; there is no base class. - Existing gates adopt by returning clauses, thresholds unchanged:
promotion_gate,domain/promotion, each fit'searned(),eval_metrics._blockers,llm_judge_trust.assess, the level-calibration coarse gate cell. - Sealed gates are wrapped, never changed. The card and pattern gates (and the pattern
ladder) keep their code and documents; an adapter in trade-data reads their DECIDED outputs and
records them as
Verdicts (gate="card-system","pattern"), so they appear on the same ledger and page. The pattern code'sbh_fdr(signals/pattern_stats.py:57) is NOT replaced.
Decision 4 — the training-data contract and its read surface¶
Example(topic, subject, value, source, by, at, reason="", ref={})
topic what is being learned: "llm:narrative.research", "trend:1d", "picker:SPY"
subject the thing labelled (an observation id, "SPY@2026-11-06", a pick id)
value the label: "good" | "bad" for verdicts, a number for an outcome
source human | model | judge | outcome (outcome = resolved mechanically from the market)
by who: an operator id, a model id, a grader name
ref provenance: trace id, table + key, git sha
- One contract, many physical stores, no copying. Options outcomes stay in their tables
(
quant.picker_outcomes…); LLM verdict labels stay in Langfuse scores; the operator's trade truth stays inquant.autopsy_calibration. Each store is anExampleStoreadapter that READS its table asExamples;learn.union(stores)reads several as one. A copy would drift. ExampleStore(Protocol, never raises, named outcomes):examples(topics, *, since, source)over a sequence of topics; optionalput(examples)(a sequence) for stores that accept writes (Langfuse scores, a JSONL file).- Pure readers in the core:
coverage(examples, topics, target)(every topic appears, zero rows included: the gap is the point);agreement(examples, *, reference="human", measured=("judge", "model"), positive)(kappa + AC1 on shared subjects);few_shot(examples, k, *, topic, source="human")(deterministic, always balanced across both classes). - The read surface for judges and copilot ("all of this could be used by the judge"):
few_shothands an LLM judge graded examples of its own role; copilot's existingget_similar_setups(pgvector over decision reflections) stays the similarity search and gains anExampleStoreadapter, not a second index. Copilot's event tools (get_event_performance,get_calendar,get_structural_events,get_market_moments) are evidence READERS for the event-study and trend loops: a loop'sevidence()calls the same usecases they call, never a copy. Similarity search is NOT in the core (it needs a vector store); aSimilarExamplesProtocol is added when its first adapter is, not before.
Decision 5 — the engine: trade-data's learning plane, only agreement + FDR math lifted¶
"We have a training engine already": the engine IS trade-data's learning plane, and stays there:
FIT_REGISTRY dispatch (usecases/fit_registry.py, POST /admin/run/fit/{model}), admin-run
declarations with learn_kind (admin/admin_run_declarations_*.py), Windmill schedules, the loop
registry with its loop-stalled watchdog, the weight ledger, the run ledger. Nothing of that
moves; it gains the core's types. What lifts is only math that has ONE definition:
| function | today | in the core |
|---|---|---|
| kappa / AC1 | llm-ports evals/calibration.py:54,75 |
learn.stats.cohen_kappa, gwet_ac1; re-exported from evals.calibration |
| false-discovery control | signals/eval_metrics.py:272 bh_fdr, BY by default (dependent=True; 8 callers use the default, incl. the signal-trust gate learn_signal_weights.py:216; its comments at eval_metrics.py:39 and learn_signal_weights.py:212-215 mislabel it Benjamini-Hochberg). Stay: pattern_stats.py:57 (sealed) and eval_metrics.by_fdr (:257, scipy BY q-values for discovery.py, event_study.py) |
learn.stats.bh_fdr, Benjamini-Yekutieli by default (valid under any dependence between tests, the estate's correlated signals), dependent=False for plain Benjamini-Hochberg (independent tests). Byte-identical to the trade-data copy (alpha 0.10, None excluded from m, same tie order), so its callers keep the DEFAULT and their numbers do not change |
NOT lifted, on evidence: Wilson is already one scipy shim in trade-data (domain/stats.py;
level_calibration and setup_radar delegate to it, consolidation B3), so a lift would add a
second copy, not remove one. Effective n is three DIFFERENT estimators (consolidation F4:
autocorrelation-adjusted, Kish weights, cluster count), not one formula with three spellings.
Reliability / ECE / Brier, Platt, purged CV lift in the slice whose consumer needs them (the
trend loop), with the same test. A lift is byte-for-byte on numbers: trade-data's own tests run
against the lifted function before its copy is deleted. Fit usecases (they read tables) are NOT
lifted.
Decision 6 — ONE registry, ONE view of when training runs¶
- The registry is
service/platform/loops.pyand its generateddocs/LEARNING-LOOPS.md+GET /ops/loops(admin/ops_router.py:36). Not forked. A loop built on the core declares itsLoopSpecthe same way; a helper derives itsscoreslot (quant.learn_runswhererun_kind=<kind>, precedentloop_registrants_llm.py:53).docs/LEARNING-LOOPS.md(every loop),docs/design/LEARNING-LOOP-OVERVIEW.md(the picker loop in depth) anddocs/AUTOMATION-TIMELINE.md(what runs when) each gain one paragraph pointing here; none is replaced. - A loop's run kind rides its declaration, not its route.
AdminRunSpec.kindis the first path segment only (platform/admin_run.py:52), andspec_forresolves/admin/run/fit/{model}tofit(:74-82), so a route-levellearn_kindcannot name a per-model loop. EachFIT_REGISTRYentry therefore declares its ownlearn_kind(aFitSpec(name, fit, read, learn_kind)replacing the bare(fit, read)tuple), and the fit dispatcher stamps that kind only for a spec that is NOT a loop: aregister_loopspec's row is written byrun, so a run is one row, never a second verdict-less row thatLearnRunsRepo.landscape(DISTINCT ONrun_kind) would show asinformational. - A fit run's
ops.workname is alreadyfit/<model>and stays so. The sharedunit_namerule keeps the full remainder after/admin/run/(edge-platformtimeline.py:37-38, used byasgi_work.py:87-92; trade-quant uses the same rule), andwork_reap_repo.py:271'ssplit_part(name, '/', 1)relies on that slash. No rename. learn_kinds()is declared kinds UNION observed kinds: admin-runlearn_kinds andFIT_REGISTRYspecs, plus the distinctrun_kinds inquant.learn_runs. Today 16 kinds are stamped with literal kinds and declare none (research,card_grade,llm_gauntlet,paper_loop, thevalidate_*,cohort_*andenrich_*kinds, ...); each shows triggerundeclared, nevermanual, so the view names the gap instead of asserting a trigger, and slice 4 declares them.- "When does training run" is a JOIN of records that already exist, not a new store:
learn_kinds()names each kind and its trigger; the LIVE schedules come from Windmill throughadapters/windmill_ops.list_schedules(windmill_ops.py:75, already read byusecases/automation_status.py:80), never thewindmill/tree (it is not in the image); box timers are named as triggers but carry no next-run in this view;ops.workholds each run's start, end and status;quant.learn_runsholds its result and verdict. One read,GET /research/schedule, joins them per kind: last run, status, verdict, next run, and a MISSED flag (a scheduled run with noops.workrow in its window, the ruleSlot.work_namealready encodes,loops.py:107). Two join keys, both exact: loop to run is theops.workname itsLoopSpectrigger spells afterflow:(fit/<model>for a fit loop;Slot.work_name,loops.py:107-119, matched exactly at :256-262); schedule to loop is the Windmill flow path, which theLoopSpecdeclares andlist_schedulesreturns (windmill_ops.py:75-80returns schedule and flow paths, not the path the flow POSTs). A schedule that matches no loop is its ownunmatched schedulerow and a loop that declares a flow with no schedule saysno schedule: neither is silent. A declared kind with no schedule saysmanual, never blank; a declaredmanualthat a box cron in fact runs (loop_registrants.py:85-88, hazard exit) is corrected in slice 4. - The view is keyed by registered loop, not by run kind: a loop in
loops.pywith nolearn_runsstamp (hazard exit today) shows as a row withno ledger row, so it is visible instead of absent. - The view is the existing Research Ledger page (trade-ui
#/research,src/routes.tsx:462), which already lists every kind with its earn state. It gains last / next / missed columns. No new page.
Decision 7 — the ratchet: mapped, not adopted¶
The ratchet (ci-kit ci_kit/ratchet.py: RatchetSet, RatchetCount) IS a gate: findings in →
regression or not → the baseline file in git is the audit. It does not adopt the ledger: it runs
in CI, which cannot reach the box's Postgres, and its audit trail (the baseline's git history, with
a stated reason per re-freeze) is already append-only. If a ratchet ever reports into the view,
it does so by emitting a Verdict JSON as a CI artifact; nothing is built for it now. ci-kit stays
its home.
The core's public surface (llm_ports.learn)¶
| module | holds |
|---|---|
learn (types) |
Example, Clause, Verdict, Decision, LedgerRow, store results (ExamplesRead, RowRead, Written) |
learn.gate |
clause helpers (at_least, at_most, equals, check), decide, could_not_run (pure) |
learn.stats |
bh_fdr (BY default), cohen_kappa, gwet_ac1 |
learn.examples |
coverage, agreement, few_shot |
learn.stores |
MemoryRunLedger, JsonlRunLedger, MemoryExampleStore, JsonlExampleStore, union |
| ports | RunLedger, ExampleStore, WritableExampleStore, Loop, Fit Protocols / values |
learn.loop |
run(loop, ledger) -> Run; record(ledger, kind, verdict) for a gate run outside a loop |
The loop has four stages: evidence → fit → clauses → apply. run drives the first three,
decides the verdict, then calls loop.apply(verdict, fit), so what a loop does with its verdict
(store the fitted map on pass, the identity on anything else, as fit_prob_calibration does
inside compute today via earned()) is a named stage, not buried in the fit. apply returns
what it did ("stored", "held", ...), recorded in the same row. A stage that raises becomes
could_not_run with the stage named (an apply that raises after a pass is recorded as
apply_failed in the row, keeping the verdict). Exactly one ledger row per run, every exit
included; run never raises. A loop that wants no action implements apply as a no-op. The
legacy per-model routes keep their own path until slice 5 moves each onto run; until then they
stamp through stamp_run as today.
Async: the Protocols are async (the stores a loop reads are async; trade-data is asyncio),
fit and clauses stay synchronous (pure). A synchronous backend runs its calls in a thread.
Every function is pure or behind a Protocol; no clock without injection; no I/O outside the JSONL
adapters. Docs: guide docs/guides/learning-loops.md with an executed, self-asserting example
(a loop over synthetic data from evidence to recorded verdict), API page, ADR-0010.
The cost of a new loop (the adoption deliverable)¶
After slice 3 (below), a new loop in trade-data is:
| file | new / edit | about |
|---|---|---|
service/learn/<loop>.py |
new | the Loop: read evidence (a repo call), fit, return clauses: ≤ 120 lines |
tests/learn/test_<loop>.py |
new | pure tests of fit + clauses |
windmill/u/admin/<loop>.schedule.yaml + .flow |
new | the cron and the thin POST, copied |
service/usecases/fit_registry.py |
+1 line | register_loop(<Loop factory>): a slice-3 helper builds the FitSpec (fit = run(loop, ledger), read = the ledger's latest DECISIVE row, latest(decisions=("pass", "fail")), serving its learn_model only when that row is pass with learn_apply not apply_failed, else nothing, so a later fail revokes the model as fit_prob_calibration does today; learn_kind = loop.kind), dispatched by the existing POST /admin/run/fit/{model} |
service/usecases/loop_registrants.py |
+1 entry | the LoopSpec (score slot derived) |
Zero migrations, zero routes, zero UI code: the ledger is learn_runs, the route is the fit
route, the Research Ledger page lists any kind and its schedule. A cross-sectional loop (many
symbols, one verdict) ignores the dispatcher's symbol argument, as the scope-global slippage
fit already does (fit_registry.py _fit_slippage), and puts its universe in params. The
budget holds only for a loop that stores nothing beyond its ledger row (its fitted model lives
in metrics); a loop that needs a per-symbol table pays one migration and one repository, stated
in its PR. Budget: ≤ 7 files, ≤ 250 lines
including tests. The trend loop's PR reports its git diff --stat against this budget; a miss is
a core defect to fix, not a number to restate.
Portability (acceptance criterion, proved in CI)¶
- import-linter contract
learn-standalone:llm_ports.learnimports no otherllm_portsmodule. No third-party import comes from the no-vendor-in-core contract plus the subprocess test below. import llm_ports.learnstill runsllm_ports/__init__(pydantic, jinja2, pyyaml), so the contract alone does not prove portability.tests/contract/test_learn_standalone.pycopiessrc/llm_ports/learn/into a temporary directory under an emptyllm_ports/__init__.py(no import is rewritten) and runs it in a subprocess whosesys.modulesmust then hold no third-party module: the "lift it into another codebase" claim, executed.tests/contract/test_core_imports.py(thecore-onlyCI job, a bare install) walksllm_ports.learnwith the rest of the core and asserts it loads no vendor module;examples/learning_loop.pyruns on every pytest leg.- The deployment-vocabulary pre-commit hook already rejects EDGE names in
src/llm_ports/. - Docs say how to plug in a ledger: implement two methods.
Telemetry (what fails silently, where it lands, what watches it)¶
| silent failure | lands in | watched from outside by |
|---|---|---|
| a loop stops running | no learn_runs row / no ops.work row in window |
loop-stalled watchdog (exists) via the derived score slot; MISSED flag on the schedule view |
| a loop runs but cannot decide (data gone, store down) | metrics.learn_verdict.decision = could_not_run with reason |
Research Ledger shows error; a gauge on the service's watched_gauge tick (service/observability/learn_gauge.py, daemon=service, every 5 min; precedent day_brief_gauge.py) computes edge_learn_could_not_run_streak{kind} from the ledger, independent of any page being open; alert at ≥ 3 (trade-infra). An in-service gauge dies with the service, so it exports edge_learn_gauge_updated_ts and trade-infra alerts on its staleness (absent() or age > 15 min): the outside watcher for this row and the next two |
a loop never earns: stuck at insufficient forever (the sample never grows) |
learn_verdict.decision = insufficient, every run |
the same gauge exports edge_learn_days_insufficient{kind} (days since the kind's first consecutive insufficient); alert at ≥ 30 days, so a floor that cannot be reached is named rather than waited on |
an apply fails after a pass |
metrics.learn_apply = apply_failed |
the gauge's edge_learn_apply_failed_total{kind}; alert > 0 |
| a gate silently passes on missing evidence | impossible by construction: None fails closed | unit tests of decide both directions (a pass and every fail mode) |
| a ledger write fails | stamp_run swallows it today (learn_ledger.py:45-54) |
run() returns the ledger outcome; the trade-data runner counts edge_learn_ledger_write_failed_total{kind}; alert > 0 |
| a lifted statistic changes a number | n/a | trade-data's existing tests run against the lifted function before the copy is deleted |
Slices (PRs), in order¶
- llm-ports:
llm_ports.learncore (types, gate, stats, examples, stores, loop runner with itsapplystage, standalone contract, guide, ADR-0010, API page) → release 1.5.0. - llm-ports: LLM labels on the core: re-scope WIP
04678fd(below) → 1.6.0. - trade-data: adoption plumbing: pin;
LearnRunsLedgeradapter;FitSpecwithlearn_kind+register_loop(stamps only non-loop specs);learn_kinds()(declared UNION observed); Research Ledger readsmetrics.learn_verdictgenerically (andrigor()); derived score slot;eval_metrics's BY-defaultbh_fdrcopy replaced bylearn.stats.bh_fdrwith its DEFAULT (BY: numbers unchanged for its 8 callers;dependent=Falsewould loosen the signal-trust gate) and its comments that call it Benjamini-Hochberg corrected (notpattern_stats, notby_fdr); thelearn_gaugegauge and its staleness timestamp. - trade-data + trade-ui + trade-infra: schedule view:
GET /research/schedule(Windmill vialist_schedules, keyed by registered loop), Research Ledger columns, the three alerts. - trade-data: existing gates return
Verdicts: promotion_gate, domain/promotion, fitearned()s, llm_judge_trust; sealed-gate wrappers. Thresholds unchanged. - trade-data: LLM label loop (original Step 4): queue, coverage, LABELS card (§LLM labels).
- trade-data: the trend loop, first new consumer, weight 0, display-only on the Research
Ledger; measures the adoption budget. Reads
quant.ohlcv_daily_mvonbar_date(neverts) joined to the calendar; honest-fill forward returns perdocs/STRATEGY-SYSTEM.md. Roadmap line inplans/estate-roadmap.mdwhen this doc lands. - Fan-out: judges read
few_shot; copilot's similar-setups adapter; remaining loops.
Slices 1-2 are core, 3-5 make it easy, 6-8 go wide: the operator's order.
LLM labels (one loop kind)¶
EVAL-LABELS-CONSOLIDATION.md holds the inventory and the label decisions; they stand with these
changes:
ports.Label/ports.LabelStore(WIP04678fd) are NOT added: a label is alearn.Example(topicllm:<role>,evals.labels.topic; source human / model / judge; the trace id inref["trace"]) read throughlearn.ExampleStore.evals.labelskeeps only what is LLM-specific:topic/role_of,draw(stratified by the judge's verdict, keyed by the core'scoverage),label_of(JudgeVerdict, source="model"|"judge"), and theLabelQueueport (enqueue(topic, subjects), the review surface).coverageandagreementare the core's (learn.examples); the test double islearn.stores.MemoryExampleStore, not a new fake.adapters/langfuse/labels.pyisLangfuseExampleStore, aWritableExampleStoreand aLabelQueue(scoreslabel-<role>on the production generation,label.model-<role>andlabel.judge-<role>written byput, and since 1.12.0 a person'slabel-<role>relayed byputwithby, ADR-0014; one annotation queue per role; SDK calls in a worker thread).adapters/langfuse/experiments.pyships as drafted. Both in slice 2 (1.6.0).- The judge-trust gate becomes a
Loopin trade-data (slice 6): examples = human labels from the store, clauses =labelled >= 20(floor),kappa >= 0.6,seeded_bad_all_caught(viaevals.calibration.verdict, asllm_judge_trust.assessdoes today), recorded asrun_kind = 'llm_judge_trust';edge_llm_judge_trustedreads its latest verdict instead of a baked JSONL. - Langfuse stays the LLM label store (UI exists, the label sits on the reply it labels, already backed up). A replay read from a Langfuse export was cut at 4,000 characters (sibling notes 2026-11-07; source of the cut UNVERIFIED): the queue job records the length it enqueued and the label loop refuses a label whose stored reply length differs.
Corrections to the inputs¶
training-engine-inventory.md§6 saysllm_judge_trustis not on main. It is: trade-data#2823 merged as6e941731, andservice/platform/llm_judge_trust.py(109 lines) is on main.eval-labels-design-review.md(the prior reviewer's findings) is an empty file on disk; its findings are not recoverable. This doc got a fresh review (learning-loop-core-design-review.mdin the session notes, DO NOT SHIP, 11 findings); this revision answers each: 1 →learn_verdict(Decision 3), 2 + 3 →FitSpec.learn_kind,list_schedules(Decision 6), 4 → lift only kappa/AC1/BY (Decision 5), 5 → standalone lifted-copy test, other verdict types named, 6 → roles +unevaluatedpolicy, 7 →applystage, 8 → scrape-time collector, 9 → async stated, 10 → cross-sectional + budget scope, 11 → inventory rows, event tools, sibling docs, Postgres adapter, view keyed by loop.
Not in scope¶
Training or fine-tuning runs; a new judge model choice (measured by a sibling session); MLflow; editing either sealed gate; moving any fit usecase out of trade-data; a new UI page.