Bigger ML engines¶
The learning loop's storage is three ports, each with a standard-library backend. A bigger engine (MLflow first; W&B, Ray, a managed platform later) is an adapter that implements the same port, so a loop and its consumers do not change when the engine does. Why it is shaped this way: ADR-0016.
| Port | Standard library | MLflow ([mlflow]) |
Maps onto |
|---|---|---|---|
RunLedger |
MemoryRunLedger, JsonlRunLedger |
MlflowRunLedger |
tracking runs |
ModelRegistry |
MemoryModelRegistry, JsonlModelRegistry |
MlflowModelRegistry |
registry versions and aliases |
ExampleStore |
MemoryExampleStore, JsonlExampleStore |
a feature store's offline read |
The rest of the stack already speaks the standards an engine reads: estimators follow the
scikit-learn API, tuning is an Optuna study (Ray Tune wraps it with OptunaSearch), traces use
the OpenTelemetry GenAI conventions, provenance is PROV-O.
Lineage: run, version, alias¶
run(loop, ledger) returns the ledger's answer in Run.recorded; its ref is the row's id (a
UUID in the standard-library ledgers, the run id in MLflow). Register the fit with that ref, and
promotion is moving an alias:
"""A loop's run linked to a registered model version, promoted by alias, on swappable backends.
The run ledger ids the row it writes (Written.ref); the fit is registered as a numbered model
version that points back at that row (run_ref); promotion is moving the champion alias. Every
piece here is a port with a standard-library backend, and each backend passes the conformance
suite (llm_ports.testing.conformance), the check a bigger engine's adapter (MLflow:
llm_ports.adapters.mlflow) passes too. Swapping engines is changing the two constructors.
"""
import asyncio import tempfile from collections.abc import Mapping, Sequence from pathlib import Path from typing import Any
from llm_ports.learn import Clause, Fit, Verdict, gate, run from llm_ports.learn.registry import JsonlModelRegistry from llm_ports.learn.stores import JsonlRunLedger from llm_ports.testing.conformance import check_model_registry, check_run_ledger
class MeanLoop: """Learn a mean; the gate asks for at least three points."""
kind, subject = "mean", "demo"
params: Mapping[str, Any] = {"window": 3}
async def evidence(self) -> list[float]:
return [1.0, 2.0, 4.5]
def fit(self, xs: list[float]) -> Fit:
return Fit({"mean": sum(xs) / len(xs)}, {"n": len(xs)})
def clauses(self, fit: Fit, xs: list[float]) -> Sequence[Clause]:
return [gate.at_least("n", fit.metrics["n"], 3, role="floor")]
async def apply(self, verdict: Verdict, fit: Fit) -> str:
return "registered" if verdict.passed else "held"
async def main(root: Path) -> None: ledger = JsonlRunLedger(root / "runs.jsonl") # or MlflowRunLedger(MlflowClient(...)) registry = JsonlModelRegistry(root / "models.jsonl") # or MlflowModelRegistry(MlflowClient(...))
got = await run(MeanLoop(), ledger)
assert got.verdict.passed and got.fit is not None and got.recorded.ref
version = await registry.register("mean-demo", model=got.fit.model, run_ref=got.recorded.ref)
assert version.ok and version.ref == "1"
assert (await registry.set_alias("mean-demo", "champion", int(version.ref))).ok
champion = (await registry.by_alias("mean-demo", "champion")).version
assert champion is not None and champion.model == {"mean": 2.5}
row = (await ledger.latest("mean")).row
assert row is not None and row.ref == champion.run_ref # the champion's lineage, one hop
# The outside witness: both backends keep their port's contract.
assert len(await check_run_ledger(ledger)) == 9
assert len(await check_model_registry(registry)) == 9
print(f"champion v{champion.version} mean={champion.model['mean']} from run {champion.run_ref}")
if name == "main": with tempfile.TemporaryDirectory() as tmp: asyncio.run(main(Path(tmp)))
A model too big for JSON goes to a file or bucket first; set Fit.artifact_uri and pass the
same URI to register. The ledger row records it as learn_artifact_uri.
Swapping in MLflow¶
from mlflow import MlflowClient
from llm_ports.adapters.mlflow import MlflowModelRegistry, MlflowRunLedger
client = MlflowClient(tracking_uri="sqlite:///mlflow.db", registry_uri="sqlite:///mlflow.db")
ledger = MlflowRunLedger(client, prefix="loops/", artifact_location="file:///srv/mlflow/artifacts")
registry = MlflowModelRegistry(client)
Each row is one MLflow run in the experiment prefix + kind, tagged with its subject and
decision; scalar params and numeric metrics are logged so MLflow can chart them, and the whole
row is JSON in the run's tags (llm_ports.row/000, ...), read back with no artifact store. A
tracking server works the same way (tracking_uri="http://..."); there, an artifact_uri
passed to register must be a remote URI (s3://...), since MLflow refuses a local path as a
model version's source. A version registered with no artifact_uri gets a backing run in the
experiment llm_ports.registry as its source.
Adding a backend¶
A new backend passes the conformance suite before anything reads from it:
from llm_ports.testing.conformance import check_run_ledger, check_unavailable_run_ledger
assert await check_run_ledger(MyLedger(live_connection))
assert await check_unavailable_run_ledger(MyLedger(closed_connection))
Each check writes under fresh random names, so it runs against a shared or production store, and
raises ContractViolation naming the first invariant the backend breaks: a raise across the
port, an unreadable store read as empty, a filter ignored. Rules for an engine adapter:
- The core stays standard library; the engine's client is imported only in its adapter, behind its extra.
Loop.fitstays pure andFitplain data, so an engine can run the fit in a remote worker.- Map each port onto the engine's own concept (experiment, run, alias), not a private schema.