Forecasts: grade, calibrate, combine¶
Anything that says what the market, a metric or the world will do is a voice: a model role,
a human author a model transcribed, a rule-based assembler. llm_ports.forecast turns what each
voice said into a track record, fixes its confidence, and pools voices, with no human labels:
the world is the label.
The loop has four steps:
| step | what | where |
|---|---|---|
| record | Forecast: author, subject, kind, stated probability and/or hedge, verbatim text |
core |
| score | score(obs, fct): Brier score, log score, skill over the base rate; reliability(), ece(), mce() and decompose() (Murphy: reliability, resolution, uncertainty) |
core |
| calibrate | Calibrator: fit(X, y), then predict_proba(X); a refused fit records Insufficient |
core port; fitters in adapters.sklearn (isotonic) and adapters.venn_abers (small n) |
| combine | Combiner: predict_proba_one(x), learn_one(x, y); LogitPool is the fixed baseline |
core; adapters.river.RiverLogitPool learns online |
Verbal confidence is learned per source: HedgeMap gives "likely" from each author the
probability it has earned, shrunk toward the word's pooled rate when that author has few resolved
uses of it. Ranges and distributions score through adapters.scoringrules (interval score, CRPS).
Standard names at every seam¶
Each seam speaks a protocol an existing library already defines, so an outside model drops in without a wrapper and a reader who knows the library reads the code unchanged:
| seam | protocol | so this drops in |
|---|---|---|
Calibrator, HedgeMap |
scikit-learn estimator: fit(X, y) returns the estimator, predict_proba(X) gives (P(no), P(yes)) rows, get_params / set_params, fitted state ends in _ |
a scikit-learn, venn-abers or MAPIE calibrator; sklearn.base.clone |
Combiner |
River classifier: learn_one(x, y), predict_proba_one(x) with x as voice -> probability |
any River binary classifier |
score |
scoringrules: brier_score(obs, fct), log_score(obs, fct), observation first, lower is better |
scoringrules' vectorised scores in adapters.scoringrules |
Forecast, Resolved |
Metaculus / Good Judgment question fields, schema.org Claim / ClaimReview, W3C PROV |
a platform export or a fact-check feed maps field for field (the table is in llm_ports.forecast._types) |
The whole loop¶
"""Grade a voice, calibrate it, learn its hedge words, and pool it with another.
Standard library only: the fitting libraries (scikit-learn, venn-abers, River, scoringrules) sit
behind adapters, and what they fit is applied here without them.
"""
from __future__ import annotations
import random
from datetime import UTC, datetime, timedelta
from llm_ports.forecast import Forecast, HedgeMap, LogitPool, PiecewiseLinear
from llm_ports.forecast.score import ece, reliability, score
rng = random.Random(1)
# 1. A forecast is a record of what a voice said, kept with its evidence.
said = Forecast(
author="desk",
subject="SPY",
kind="range",
forecast_time=datetime(2026, 3, 2, 14, 30, tzinfo=UTC),
resolve_time=datetime(2026, 3, 2, 21, 0, tzinfo=UTC),
text="SPY holds 560-566 into the close",
probability=0.68,
)
# 2. Score it. This voice states 0.68 for every range and the ranges hold 38% of the time.
stated = [said.probability or 0.0] * 2000
happened = [rng.random() < 0.38 for _ in stated]
raw = score(happened, stated) # scoringrules order: observation first
assert raw.overconfidence is not None and raw.overconfidence > 0.25
assert raw.brier_skill_score is not None and raw.brier_skill_score < 0 # worse than stating the base rate
print(f"raw: n={raw.n} claims {raw.mean_p:.2f} hits {raw.hit_rate:.2f} brier {raw.brier_score:.3f}")
# 3. Calibrate. A fitted isotonic map (adapters.sklearn, `IsotonicCalibrator().fit(X, y).map_`)
# is a PiecewiseLinear; stored as JSON, it is applied with no fitting library. Here it is the
# map such a fit returns for this voice.
fitted = PiecewiseLinear.from_dict({"xs": [0.0, 0.68, 1.0], "ys": [0.0, 0.38, 0.38], "n": 2000})
calibrated = score(happened, fitted.predict(stated))
assert calibrated.brier_score is not None and raw.brier_score is not None
assert calibrated.brier_score < raw.brier_score
assert abs(fitted(said.probability or 0.0) - 0.38) < 1e-9
print(f"calibrated: states {fitted(0.68):.2f}, brier {calibrated.brier_score:.3f}")
# 4. Verbal confidence. "likely" from author a has meant ~0.75; from author b ~0.45.
# HedgeMap is a scikit-learn estimator: X is (source, hedge) rows, y what happened.
words = [("a", "likely")] * 300 + [("b", "Likely")] * 300 + [("c", "likely")] * 2
outcomes = [rng.random() < 0.75 for _ in range(300)] + [rng.random() < 0.45 for _ in range(300)]
outcomes += [True, True] # two lucky calls from c
hedges = HedgeMap().fit(words, outcomes)
pa, pb, pc = (hedges.probability(src, "likely") for src in ("a", "b", "c"))
assert pa is not None and pb is not None and pc is not None
assert pa > pb and pc < 0.9 # c is shrunk toward the pool, not trusted on two calls
print(f"'likely': a={pa:.2f} b={pb:.2f} c={pc:.2f} (pooled {hedges.pooled_['likely']:.2f})")
# 5. Pool two voices in log-odds; the equal-weight pool is the baseline a learned combiner
# (adapters.river) has to beat. Combiners speak River's protocol: x is voice -> probability.
pool = LogitPool()
both = pool.predict_proba_one({"desk": fitted(0.68), "author-a": pa}).get(True)
assert both is not None and min(fitted(0.68), pa) < both < max(fitted(0.68), pa)
print(f"pooled: {both:.2f}; ece of the calibrated voice {ece(reliability(happened, fitted.predict(stated))):.3f}")
assert said.resolve_time - said.forecast_time < timedelta(days=1)
Choosing a calibrator¶
- Isotonic (
IsotonicCalibrator, extra[sklearn]): a monotone map, from about 30 resolved forecasts up (MIN_PAIRS); the fitted map is aPiecewiseLinearinmap_, stored as JSON and applied without scikit-learn. - Venn-ABERS (
VennAbersCalibrator, extra[venn-abers]): valid at any sample size; it returns a bracket(p0, p1)whose width says how much a short history can say. Use it for a voice with tens of resolved forecasts, isotonic once it has hundreds.
Neither raises: for too few pairs or a history where every outcome was the same, fit records
Insufficient in insufficient_ and predict_proba returns None.
Combining earns weight¶
A learned combiner reaches a decision the same way any learning loop does: logged in shadow
against the equal-weight LogitPool and the best single voice, and promoted only through a gate
(llm_ports.learn.gate). RiverLogitPool returns {} until it has heard warmup outcomes, so
a caller falls back to its baseline instead of reading an untrained model.
Replaying in time order, against the baselines¶
walk_forward(combiner, events) replays Events as the combiner would have run: each forecast
is made at its forecast_time, and an outcome is learned only once its resolve_time has
passed. compare runs several combiners over the same events and scores them on the same
resolved, answered subset, so a learned pool is always read beside the equal-weight
LogitPool and BestVoice. PriorPool starts every combiner from a prior voice (a base rate for
the subject) instead of 0.5:
import random
from datetime import datetime, timedelta
from llm_ports.forecast import BestVoice, Event, LogitPool, PriorPool
from llm_ports.forecast.walk import compare, holdout_start
rng = random.Random(2)
t0 = datetime(2026, 1, 1)
events = []
for h in range(600):
y = rng.random() < 0.55
x = {"sharp": 0.75 if y == (rng.random() < 0.8) else 0.25, "noise": rng.random(), "base": 0.55}
t = t0 + timedelta(hours=h)
events.append(Event(x=x, forecast_time=t, resolve_time=t + timedelta(hours=3), resolution=y))
scores = compare(
{
"equal": PriorPool(inner=LogitPool(), prior_key="base"),
"best": PriorPool(inner=BestVoice(min_n=20), prior_key="base"),
"prior": PriorPool(inner=BestVoice(min_n=10**9), prior_key="base"),
},
events,
since=holdout_start(events, 0.2),
)
assert scores["best"].brier_score < scores["prior"].brier_score
adapters.optuna.tune(build, events, space, study_name=..., storage=...) searches a combiner's
parameters (a River pool's learning rate, alpha, shrink) on the train split only and reports
the best parameters' holdout Score; the study persists to any Optuna storage, where
optuna-dashboard reads it.
Behind a transaction-mode pooler (PgBouncer), Optuna's RDB storage cannot be kept in one schema:
it creates its tables on the connection's search path and Alembic checks them in the default
schema. adapters.optuna.journal_storage(engine, "schema.table") is the storage for that case:
Optuna's JournalStorage over SqlJournal, one append-only (n bigint primary key, log text)
table that the caller's migration creates and every statement names in full.