llm_ports.forecast¶
llm_ports.forecast
¶
Forecasts: grade what a voice said would happen, calibrate it, combine voices.
A voice is anything that asserts a future event: a model role, a human author transcribed by a
model, an assembler. Each assertion is a Forecast; once the world settles it, a Resolved.
The seams speak standard protocols, so outside models drop in unwrapped: calibrators and
HedgeMap are scikit-learn estimators (fit(X, y), predict_proba(X), get_params), combiners
are River classifiers (learn_one(x, y), predict_proba_one(x)), scores take scoringrules'
names and order (brier_score(obs, fct)), and the values take forecasting-platform, schema.org
and W3C PROV field names (_types maps them).
This package holds the parts that do not depend on who the voices are:
score: proper scores (Brier, log score, skill), the reliability curve and its error.skill:brier_skill, skill over a per-event REFERENCE forecast (a base rate, a prior), with a cluster-bootstrap interval.calibrate: theCalibratorport andPiecewiseLinear, a fitted map applied without a fitting library (fitters:adapters.sklearn,adapters.venn_abers).platt:PlattCalibrator, a two-parameter logistic map fitted and applied with the standard library; it refuses a slope that inverts the voice's ranking.hedge:HedgeMap, the probability each source's verbal confidence has earned.combine: theCombinerport,LogitPool(the fixed baseline),PriorPool(shrink toward and fall back to a prior voice, extremize),BestVoice(the best-single-voice baseline) and the log-odds transform (a learned combiner:adapters.river).walk:Eventandwalk_forward, a combiner replayed in time order that learns an outcome only once it was known;comparescores several on the same events (tuning:adapters.optuna).
Invariants
- Standard library only: imports nothing else from
llm_portsand no third-party package (enforced by an import-linter contract), likellm_ports.learn.
Rationale: ADR-0013.
Params
¶
Mixin: get_params / set_params over the __init__ parameters.
get_params
¶
get_params(deep: bool = True) -> dict[str, Any]
Return the constructor parameters by name.
Parameters:
-
deep(bool, default:True) –Accepted for the scikit-learn contract; these estimators hold no sub-estimators.
Returns:
-
dict[str, Any]–name -> valuefor every__init__parameter. Never raises.
set_params
¶
set_params(**params: Any) -> Self
Set constructor parameters by name; fitted state is untouched until the next fit.
Returns:
-
Self–The estimator.
Raises:
-
ValueError–For a name that is not an
__init__parameter (scikit-learn's contract).
Forecast
dataclass
¶
Forecast(
author: str,
subject: str,
kind: str,
forecast_time: datetime,
resolve_time: datetime,
text: str,
probability: float | None = None,
hedge: str | None = None,
resolution_criteria: str | None = None,
was_generated_by: str | None = None,
used: tuple[str, ...] = (),
)
One gradeable statement a voice made about a future event.
A forecast is a record of what was said, not an input to a decision: its probability is the
voice's own (probability), its verbal confidence is kept as written (hedge), and a
calibrator or a HedgeMap turns either into a probability the voice has earned.
Attributes:
-
author(str) –Who asserted it (an author, a model role, an assembler).
-
subject(str) –What it is about (an instrument, an event key).
-
kind(str) –What shape of claim it is (
direction,level,range,event, ...). -
forecast_time(datetime) –When it was asserted; only text available then may back it.
-
resolve_time(datetime) –When the world settles it; never before
forecast_time. -
text(str) –The span the forecast was read from, verbatim.
-
probability(float | None) –The probability the voice stated, or None when it stated only words.
-
hedge(str | None) –The verbal confidence as written (
likely,could), or None. -
resolution_criteria(str | None) –How the outcome is read, or None when
kindimplies it. -
was_generated_by(str | None) –What transcribed it (a prompt pack and its sha), or None when the author wrote it directly.
-
used(tuple[str, ...]) –What the transcription read (source document ids); empty when none are recorded.
Raises:
-
ValueError–At construction, for a
probabilityoutside [0, 1] or not finite, aresolve_timebeforeforecast_time(or one naive and one aware), an emptytext,author,subjectorkind, or a forecast with neitherprobabilitynorhedge.
Resolved
dataclass
¶
Resolved(
claim_reviewed: Forecast,
resolution: Resolution,
actual_resolve_time: datetime,
)
A forecast and what the world did.
Attributes:
-
claim_reviewed(Forecast) –The forecast.
-
resolution(Resolution) –yeswhen the forecast event happened,nowhen it did not,annulledwhen it can no longer be graded (the question was voided). -
actual_resolve_time(datetime) –When the outcome was read.
Raises:
-
ValueError–At construction, for a
resolutionthat is notyes,noorannulled.
outcome
property
¶
outcome: bool | None
True for yes, False for no, None for annulled (not gradeable).
Calibrator
¶
Bases: Protocol
Fits a calibration map from resolved forecasts (the scikit-learn estimator protocol).
get_params
¶
get_params(deep: bool = True) -> dict[str, Any]
Return the constructor parameters by name.
set_params
¶
set_params(**params: Any) -> Self
Set constructor parameters by name and return the estimator.
fit
¶
fit(X: Sequence[float], y: Sequence[bool]) -> Self
Fit a map from stated probabilities and what happened.
Parameters:
-
X(Sequence[float]) –Stated probabilities.
-
y(Sequence[bool]) –Whether each event happened, aligned with
X.
Returns:
-
Self–The estimator. A llm-ports calibrator never raises; too little data is recorded in
-
Self–insufficient_.
predict_proba
¶
predict_proba(
X: Sequence[float],
) -> list[tuple[float, float]] | None
Return (P(no), P(yes)) per stated probability, or None with no usable fit.
Insufficient
dataclass
¶
Insufficient(n: int, need: int, reason: str)
Too few resolved forecasts to fit a map.
Attributes:
-
n(int) –Usable pairs given.
-
need(int) –The fitter's floor.
-
reason(str) –Why, in words (
n < need,one outcome only).
PiecewiseLinear
dataclass
¶
PiecewiseLinear(
xs: tuple[float, ...], ys: tuple[float, ...], n: int = 0
)
A monotone non-decreasing calibration map held as knots.
Between knots the map interpolates linearly; outside them it holds the end value. It is the
fitted map, not a fitter: IsotonicCalibrator.fit returns one in map_.
Attributes:
-
xs(tuple[float, ...]) –Knot inputs, strictly increasing, in [0, 1].
-
ys(tuple[float, ...]) –Knot outputs, non-decreasing, in [0, 1].
-
n(int) –The pairs it was fitted on (0 when unknown).
Raises:
-
ValueError–At construction, for no knots, unequal lengths, inputs not strictly increasing, outputs decreasing, or a knot outside [0, 1].
predict
¶
predict(X: Sequence[float]) -> list[float]
Return the calibrated probability per stated probability. Never raises.
predict_proba
¶
predict_proba(
X: Sequence[float],
) -> list[tuple[float, float]]
Return (P(no), P(yes)) per stated probability. Never raises.
from_dict
classmethod
¶
from_dict(data: Mapping[str, Any]) -> PiecewiseLinear
Build a map from as_dict output.
Raises:
-
ValueError–As the constructor, or for a missing key.
BestVoice
dataclass
¶
BestVoice(
min_n: int = 20,
n: dict[str, int] = dict(),
brier_sum: dict[str, float] = dict(),
)
The best single voice so far: answers with the speaking voice of lowest running Brier score.
A voice is eligible once it has been heard on min_n outcomes; among the eligible voices in
x, the one with the lowest mean Brier score answers (ties: the voice name, so the choice is
deterministic). With no eligible voice it abstains ({}).
Attributes:
-
min_n(int) –Outcomes a voice must have been scored on before it can answer.
-
n(dict[str, int]) –voice -> outcomes it was scored on. -
brier_sum(dict[str, float]) –voice -> sum of its Brier scores.
best
¶
best(x: Mapping[str, float]) -> str | None
Return the voice that would answer for x, or None. Never raises.
predict_proba_one
¶
predict_proba_one(
x: Mapping[str, float],
) -> dict[bool, float]
Return the best eligible voice's {False: P(no), True: P(yes)}, or {}. Never raises.
learn_one
¶
learn_one(x: Mapping[str, float], y: bool) -> None
Score every usable voice in x against y. Never raises.
Combiner
¶
Bases: Protocol
Combines voices' probabilities for one event (River's classifier protocol).
predict_proba_one
¶
predict_proba_one(
x: Mapping[str, float],
) -> dict[bool, float]
Return the combined probability as {False: P(no), True: P(yes)}.
Parameters:
-
x(Mapping[str, float]) –voice -> probabilityfor the voices that spoke; a silent voice is absent.
Returns:
-
dict[bool, float]–The two class probabilities, or
{}when no voice is usable or the combiner cannot -
dict[bool, float]–combine yet. Never raises.
learn_one
¶
learn_one(x: Mapping[str, float], y: bool) -> None
Hear the outcome y of an event predict_proba_one was asked about. Never raises.
LogitPool
dataclass
¶
LogitPool(
weights: Mapping[str, float] = dict(),
alpha: float = 1.0,
)
A weighted mean in log-odds, extremized by alpha; a fixed baseline that never learns.
Attributes:
-
weights(Mapping[str, float]) –voice -> weight; a voice absent here weighs 1.0. Weights are renormalised over the voices that spoke, and a voice whose weight is not finite and > 0 is ignored. -
alpha(float) –The extremizing factor; 1.0 (default) leaves the pool as is. A non-finite
alphamakespredict_proba_onereturn{}.
PriorPool
dataclass
¶
PriorPool(
inner: Combiner,
prior_key: str = "prior",
alpha: float = 1.0,
shrink: float = 0.0,
)
Shrink a combiner toward a prior voice, then extremize: expit(alpha * z).
z = (1 - shrink) * logit(inner) + shrink * logit(prior), where inner is the wrapped
combiner's probability and prior is x[prior_key]. The prior also reaches the inner
combiner as an ordinary voice, so a learned combiner can weigh it. While the inner combiner
abstains ({}, e.g. before its warmup), the pool answers with the prior alone, extremized.
Without a prior in x it is the inner combiner, extremized.
Attributes:
-
inner(Combiner) –The wrapped combiner; it hears every outcome
learn_onehears. -
prior_key(str) –The voice in
xthat carries the prior. -
alpha(float) –The extremizing factor; 1.0 leaves the pool as is.
-
shrink(float) –The weight on the prior's log-odds, in [0, 1]; 0 ignores it beyond the fallback. A non-finite
alphaor ashrinkoutside [0, 1] makespredict_proba_onereturn{}.
predict_proba_one
¶
predict_proba_one(
x: Mapping[str, float],
) -> dict[bool, float]
Return {False: P(no), True: P(yes)}, or {} when neither the inner combiner nor the prior answers. Never raises.
learn_one
¶
learn_one(x: Mapping[str, float], y: bool) -> None
Pass the outcome to the inner combiner. Never raises.
HedgeMap
¶
HedgeMap(strength: float | None = None)
Bases: Params
The probability each (source, hedge) has earned: a scikit-learn estimator.
fit(X, y) learns from resolved forecasts, X rows of (source, hedge) and y whether each
happened; probability(source, hedge) and predict_proba(X) read the fitted map.
Parameters:
-
strength(float | None, default:None) –A fixed prior strength for every word (any finite real > 0, stored as float); None (default) estimates it per word.
Attributes:
-
strength–As given.
-
cells_(dict[tuple[str, str], tuple[int, int]]) –(source, hedge) -> (hits, n). Set byfitorfrom_dict. -
pooled_(dict[str, float]) –hedge -> pooled hit rate(Jeffreys-smoothed). -
strengths_(dict[str, float]) –hedge -> prior strengthused for that word.
fit
¶
fit(
X: Iterable[tuple[str, str]], y: Iterable[bool]
) -> Self
Learn the map from resolved forecasts.
Parameters:
-
X(Iterable[tuple[str, str]]) –(source, hedge)rows; a row whose hedge is empty or not a string is skipped. -
y(Iterable[bool]) –Whether each event happened, aligned with
X.
Returns:
-
Self–The estimator.
Raises:
-
ValueError–For a
strengththat is not finite and > 0, orXandyof different lengths.
probability
¶
probability(source: str, hedge: str) -> float | None
Return the probability hedge from source has earned.
A source that never used the word resolved gets the word's pooled rate.
Returns:
-
float | None–The probability, or None for a word no source has used resolved, a hedge that is
-
float | None–not a string, or an unfitted map. Never raises.
predict_proba
¶
predict_proba(
X: Iterable[tuple[str, str]],
) -> list[tuple[float, float] | None]
Return (P(no), P(yes)) per (source, hedge) row; None for a word with no probability.
Never raises.
rows
¶
rows() -> tuple[HedgeRow, ...]
Return every fitted cell, by source then hedge, for a readout; empty when unfitted.
as_dict
¶
as_dict() -> dict[str, Any]
Return the fitted map as a JSON-ready dict; from_dict reads it back.
HedgeRow
dataclass
¶
HedgeRow(
source: str,
hedge: str,
n: int,
hits: int,
p: float,
pooled: float,
)
One (source, hedge) cell of a fitted map.
Attributes:
-
source(str) –The voice.
-
hedge(str) –The normalised word or phrase.
-
n(int) –Resolved forecasts that used it.
-
hits(int) –How many happened.
-
p(float) –The shrunk probability it has earned.
-
pooled(float) –The word's pooled rate across every source (the prior mean).
Platt
dataclass
¶
Platt(a: float, b: float, n: int = 0)
A fitted Platt map: p -> sigmoid(a * logit(p) + b).
Attributes:
-
a(float) –Slope on the logit of the stated probability; finite and > 0.
-
b(float) –Intercept; finite.
-
n(int) –The pairs it was fitted on (0 when unknown).
Raises:
-
ValueError–At construction, for a slope that is not finite and > 0 or an intercept that is not finite.
predict
¶
predict(X: Sequence[float]) -> list[float]
Return the calibrated probability per stated probability.
PlattCalibrator
¶
PlattCalibrator(
min_pairs: int = MIN_PAIRS,
ridge: float = 0.001,
max_iter: int = 50,
tol: float = 1e-08,
)
Bases: Params
Fits a Platt calibration map from stated probabilities and outcomes.
Parameters:
-
min_pairs(int, default:MIN_PAIRS) –The floor of usable pairs;
MIN_PAIRSby default. -
ridge(float, default:0.001) –L2 penalty on
(a, b), which keeps a Newton step finite on separable or thin data. -
max_iter(int, default:50) –Newton steps at most.
-
tol(float, default:1e-08) –Stop once both parameter steps are below this.
Attributes:
-
map_(Platt | None) –The fitted
Platt, or None when the last fit produced none. Set byfit. -
insufficient_(Insufficient | None) –Why the last fit produced no map, or None.
-
fitted_(tuple[float, float] | None) –The
(a, b)the last fit reached, kept even when refused; None before a fit or when there was too little data to fit.
fit
¶
fit(X: Sequence[float], y: Sequence[bool]) -> Self
Fit the map; never raises (the reason a fit produced no map lands in insufficient_).
Returns:
-
Self–The estimator.
predict_proba
¶
predict_proba(
X: Sequence[float],
) -> list[tuple[float, float]] | None
Return (P(no), P(yes)) per stated probability, or None with no usable fit.
Bin
dataclass
¶
Bin(
lo: float,
hi: float,
n: int,
mean_p: float,
hit_rate: float,
)
One bin of a reliability curve.
Attributes:
-
lo(float) –The bin's lower edge (inclusive).
-
hi(float) –The bin's upper edge (exclusive, except the last bin, which includes 1.0).
-
n(int) –Forecasts in the bin.
-
mean_p(float) –Their mean probability.
-
hit_rate(float) –The share that happened; on a calibrated voice it tracks
mean_p.
Decomposition
dataclass
¶
Decomposition(
n: int,
base_rate: float,
reliability: float,
resolution: float,
uncertainty: float,
)
Murphy's decomposition of the Brier score over a reliability curve.
brier = reliability - resolution + uncertainty. It is exact when every forecast in a bin
states the same probability (a curve over cells, each forecasting one rate); otherwise it
approximates the mean Brier score by the within-bin spread.
Attributes:
-
n(int) –Forecasts on the curve.
-
base_rate(float) –The share that happened across every bin.
-
reliability(float) –The n-weighted mean squared bin gap; the calibration error (lower is better).
-
resolution(float) –The n-weighted mean squared distance of each bin's hit rate from the base rate; how far the forecasts separate outcomes (higher is better).
-
uncertainty(float) –base_rate * (1 - base_rate); the variance no forecast can remove.
Score
dataclass
¶
Score(
n: int,
mean_p: float | None,
hit_rate: float | None,
brier_score: float | None,
log_score: float | None,
brier_skill_score: float | None,
)
How a set of binary forecasts scored.
Attributes:
-
n(int) –Pairs scored.
-
mean_p(float | None) –The mean forecast probability; None at n = 0.
-
hit_rate(float | None) –The share that happened (the base rate); None at n = 0.
-
brier_score(float | None) –The mean Brier score; None at n = 0.
-
log_score(float | None) –The mean log score; None at n = 0.
-
brier_skill_score(float | None) –1 - brier_score / (hit_rate * (1 - hit_rate)), skill over always forecasting the base rate (> 0 beats it); None when undefined (n = 0, or every outcome the same).
overconfidence
property
¶
overconfidence: float | None
mean_p - hit_rate: positive when the voice claims more than it delivers; None at n = 0.
Skill
dataclass
¶
Skill(
n: int,
clusters: int,
brier_score: float | None,
ref_brier_score: float | None,
skill: float | None,
lo: float | None,
hi: float | None,
)
How a voice scored against a reference forecast on the same events.
Attributes:
-
n(int) –Pairs scored (both probabilities valid).
-
clusters(int) –Distinct clusters among them.
-
brier_score(float | None) –The voice's mean Brier score; None at n = 0.
-
ref_brier_score(float | None) –The reference's mean Brier score on the same pairs; None at n = 0.
-
skill(float | None) –1 - brier_score / ref_brier_score; None at n = 0 or when the reference is perfect. -
lo(float | None) –Lower bound of the cluster-bootstrap interval on
skill; None below MIN_CLUSTERS. -
hi(float | None) –Upper bound, as
lo.
Event
dataclass
¶
Event(
x: Mapping[str, float],
forecast_time: datetime,
resolve_time: datetime | None = None,
resolution: bool | None = None,
)
One question several voices forecast.
Attributes:
-
x(Mapping[str, float]) –voice -> probabilityof yes, for the voices that spoke. -
forecast_time(datetime) –When the combined forecast is made (the newest voice's time, at earliest).
-
resolve_time(datetime | None) –When the outcome became known; None while open.
-
resolution(bool | None) –True if it happened, False if not, None while open.
finite_real
¶
finite_real(v: object) -> bool
Return True for a finite numbers.Real that is not a bool (a weight, a strength). Never raises.
valid_p
¶
valid_p(p: object) -> bool
Return True when p is a finite real probability in [0, 1] (a bool is not a probability).
Any numbers.Real counts (int, float, NumPy floating scalars, Fraction); Decimal
is not a numbers.Real and is refused. A caller that accepts p reads it through float.
Never raises.
brier_skill
¶
brier_skill(
obs: Sequence[bool],
fct: Sequence[float],
ref: Sequence[float],
clusters: Sequence[Hashable] | None = None,
*,
level: float = 0.95,
n_resamples: int = N_RESAMPLES,
seed: int = 0
) -> Skill
Score forecasts against a per-event reference forecast.
Parameters:
-
obs(Sequence[bool]) –Whether each event happened.
-
fct(Sequence[float]) –The voice's probabilities, aligned with
obs. -
ref(Sequence[float]) –The reference's probabilities for the same events.
-
clusters(Sequence[Hashable] | None, default:None) –A cluster id per event (events sharing evidence share an id); None = each event is its own cluster.
-
level(float, default:0.95) –Two-sided confidence level of the interval, in (0, 1).
-
n_resamples(int, default:N_RESAMPLES) –Bootstrap resamples.
-
seed(int, default:0) –The generator seed.
Returns:
-
Skill–The
Skill. Never raises on data; a sequence of a different length scores nothing.
Raises:
-
ValueError–leveloutside (0, 1) orn_resamples< 1.
walk_forward
¶
Replay events through combiner in time order; return each event's probability of yes.
Parameters:
-
combiner(Combiner) –A fresh combiner; it is trained in place.
-
events(Sequence[Event]) –The events, in any order.
Returns:
-
list[float | None]–One probability per event, aligned with
events; None where the combiner abstained. -
list[float | None]–Never raises for a combiner that keeps its contract.
llm_ports.forecast.score¶
Proper scores for binary forecasts: Brier, log score, skill against the base rate, reliability.
A proper scoring rule is minimised in expectation only by reporting the true probability, so a
voice cannot improve its score by hedging toward 0.5 or by overclaiming. Names and argument order
follow scoringrules: observation first, forecast second (brier_score(obs, fct)), and lower is
better.
Invariants
- Pure, standard library only.
- A pair whose forecast is not a finite probability is skipped, never scored.
- Observations and forecasts of different lengths are not aligned, so nothing is scored.
- A statistic the data cannot define is None, never a number.
Rationale: ADR-0013.
Score
dataclass
¶
Score(
n: int,
mean_p: float | None,
hit_rate: float | None,
brier_score: float | None,
log_score: float | None,
brier_skill_score: float | None,
)
How a set of binary forecasts scored.
Attributes:
-
n(int) –Pairs scored.
-
mean_p(float | None) –The mean forecast probability; None at n = 0.
-
hit_rate(float | None) –The share that happened (the base rate); None at n = 0.
-
brier_score(float | None) –The mean Brier score; None at n = 0.
-
log_score(float | None) –The mean log score; None at n = 0.
-
brier_skill_score(float | None) –1 - brier_score / (hit_rate * (1 - hit_rate)), skill over always forecasting the base rate (> 0 beats it); None when undefined (n = 0, or every outcome the same).
overconfidence
property
¶
overconfidence: float | None
mean_p - hit_rate: positive when the voice claims more than it delivers; None at n = 0.
Bin
dataclass
¶
Bin(
lo: float,
hi: float,
n: int,
mean_p: float,
hit_rate: float,
)
One bin of a reliability curve.
Attributes:
-
lo(float) –The bin's lower edge (inclusive).
-
hi(float) –The bin's upper edge (exclusive, except the last bin, which includes 1.0).
-
n(int) –Forecasts in the bin.
-
mean_p(float) –Their mean probability.
-
hit_rate(float) –The share that happened; on a calibrated voice it tracks
mean_p.
Decomposition
dataclass
¶
Decomposition(
n: int,
base_rate: float,
reliability: float,
resolution: float,
uncertainty: float,
)
Murphy's decomposition of the Brier score over a reliability curve.
brier = reliability - resolution + uncertainty. It is exact when every forecast in a bin
states the same probability (a curve over cells, each forecasting one rate); otherwise it
approximates the mean Brier score by the within-bin spread.
Attributes:
-
n(int) –Forecasts on the curve.
-
base_rate(float) –The share that happened across every bin.
-
reliability(float) –The n-weighted mean squared bin gap; the calibration error (lower is better).
-
resolution(float) –The n-weighted mean squared distance of each bin's hit rate from the base rate; how far the forecasts separate outcomes (higher is better).
-
uncertainty(float) –base_rate * (1 - base_rate); the variance no forecast can remove.
brier_score
¶
brier_score(obs: bool, fct: float) -> float
Return the Brier score of one binary forecast, (fct - obs) ** 2; 0 is perfect.
Parameters:
-
obs(bool) –Whether the event happened.
-
fct(float) –The forecast probability.
log_score
¶
log_score(obs: bool, fct: float) -> float
Return the log score of one binary forecast, -ln(fct) if it happened else -ln(1 - fct).
fct is clipped to [EPS, 1 - EPS], so the score is finite.
Parameters:
-
obs(bool) –Whether the event happened.
-
fct(float) –The forecast probability.
score
¶
score(obs: Sequence[bool], fct: Sequence[float]) -> Score
Score binary forecasts.
Parameters:
-
obs(Sequence[bool]) –Whether each event happened.
-
fct(Sequence[float]) –The forecast probabilities, aligned with
obs; a pair with an invalid forecast is skipped, and sequences of different lengths score nothing (n = 0).
Returns:
-
Score–The
Score. Never raises.
reliability
¶
reliability(
obs: Sequence[bool],
fct: Sequence[float],
bins: int = 10,
) -> tuple[Bin, ...]
Return the reliability curve: forecasts grouped into equal-width probability bins.
Parameters:
-
obs(Sequence[bool]) –Whether each event happened.
-
fct(Sequence[float]) –The forecast probabilities, aligned with
obs; a pair with an invalid forecast is skipped, and sequences of different lengths give an empty curve. -
bins(int, default:10) –The number of equal-width bins over [0, 1]; values below 1 are read as 1.
Returns:
-
tuple[Bin, ...]–The non-empty bins, lowest first. Never raises.
ece
¶
ece(curve: Sequence[Bin]) -> float | None
Return the expected calibration error of a reliability curve.
The n-weighted mean of |mean_p - hit_rate| over the bins.
Returns:
-
float | None–The error, or None for an empty curve.
mce
¶
mce(curve: Sequence[Bin]) -> float | None
Return the maximum calibration error of a reliability curve: the largest bin gap.
Returns:
-
float | None–The largest
|mean_p - hit_rate|over the bins, or None for an empty curve.
decompose
¶
decompose(curve: Sequence[Bin]) -> Decomposition | None
Return Murphy's decomposition of a reliability curve.
Returns:
-
Decomposition | None–The
Decomposition, or None for an empty curve.
llm_ports.forecast.skill¶
Paired skill: how much a voice's probabilities beat a REFERENCE forecast on the same events.
score.Score.brier_skill_score measures skill over climatology, the constant forecast of the
sample's own hit rate. That is the wrong yardstick when a better reference exists for each event:
a symbol's base rate, a model's prior, yesterday's forecast. A voice that says "up 60%" in a tape
where the reference already says "up 60%" knows nothing the reference did not. Here the reference
is given per event, and skill is 1 - BS(fct) / BS(ref) over the pairs both scored (> 0 beats the
reference, 0 ties it, < 0 loses to it).
The interval resamples whole CLUSTERS, never single pairs: events that share evidence (one day's tape, overlapping horizons) are not independent, and an iid resample would be too narrow. The percentile bootstrap recomputes both Brier sums in every resample, so the reference's own noise is in the interval. Seeded, so a re-run over the same pairs is identical.
Invariants
- Pure, standard library only.
- A pair is scored only when BOTH probabilities are valid; others are skipped.
- Inputs of different lengths are not aligned, so nothing is scored.
- A statistic the data cannot define is None, never a number.
Rationale: ADR-0013.
Skill
dataclass
¶
Skill(
n: int,
clusters: int,
brier_score: float | None,
ref_brier_score: float | None,
skill: float | None,
lo: float | None,
hi: float | None,
)
How a voice scored against a reference forecast on the same events.
Attributes:
-
n(int) –Pairs scored (both probabilities valid).
-
clusters(int) –Distinct clusters among them.
-
brier_score(float | None) –The voice's mean Brier score; None at n = 0.
-
ref_brier_score(float | None) –The reference's mean Brier score on the same pairs; None at n = 0.
-
skill(float | None) –1 - brier_score / ref_brier_score; None at n = 0 or when the reference is perfect. -
lo(float | None) –Lower bound of the cluster-bootstrap interval on
skill; None below MIN_CLUSTERS. -
hi(float | None) –Upper bound, as
lo.
brier_skill
¶
brier_skill(
obs: Sequence[bool],
fct: Sequence[float],
ref: Sequence[float],
clusters: Sequence[Hashable] | None = None,
*,
level: float = 0.95,
n_resamples: int = N_RESAMPLES,
seed: int = 0
) -> Skill
Score forecasts against a per-event reference forecast.
Parameters:
-
obs(Sequence[bool]) –Whether each event happened.
-
fct(Sequence[float]) –The voice's probabilities, aligned with
obs. -
ref(Sequence[float]) –The reference's probabilities for the same events.
-
clusters(Sequence[Hashable] | None, default:None) –A cluster id per event (events sharing evidence share an id); None = each event is its own cluster.
-
level(float, default:0.95) –Two-sided confidence level of the interval, in (0, 1).
-
n_resamples(int, default:N_RESAMPLES) –Bootstrap resamples.
-
seed(int, default:0) –The generator seed.
Returns:
-
Skill–The
Skill. Never raises on data; a sequence of a different length scores nothing.
Raises:
-
ValueError–leveloutside (0, 1) orn_resamples< 1.
llm_ports.forecast.calibrate¶
Calibration: map a voice's stated probability to the probability it has earned.
Calibrator: the port, the scikit-learn estimator protocol.fit(X, y)returns the estimator;predict_proba(X)returns one(P(no), P(yes))row per stated probability, the column order of a binary scikit-learn classifier. A scikit-learn, venn-abers or MAPIE calibrator satisfies it unwrapped.Insufficient: why a fit produced no map (too few resolved forecasts, one outcome only); a calibrator here records it ininsufficient_and itspredict_probareturns None.PiecewiseLinear: a monotone map held as knots, applied with the standard library and serialised as JSON, so a map fitted offline (an isotonic fit,adapters.sklearn) is applied where no fitting library is installed.
Invariants
- Pure, standard library only; the fitting libraries live in adapters.
- A calibrated probability is always in [0, 1].
Rationale: ADR-0013.
Insufficient
dataclass
¶
Insufficient(n: int, need: int, reason: str)
Too few resolved forecasts to fit a map.
Attributes:
-
n(int) –Usable pairs given.
-
need(int) –The fitter's floor.
-
reason(str) –Why, in words (
n < need,one outcome only).
Calibrator
¶
Bases: Protocol
Fits a calibration map from resolved forecasts (the scikit-learn estimator protocol).
get_params
¶
get_params(deep: bool = True) -> dict[str, Any]
Return the constructor parameters by name.
set_params
¶
set_params(**params: Any) -> Self
Set constructor parameters by name and return the estimator.
fit
¶
fit(X: Sequence[float], y: Sequence[bool]) -> Self
Fit a map from stated probabilities and what happened.
Parameters:
-
X(Sequence[float]) –Stated probabilities.
-
y(Sequence[bool]) –Whether each event happened, aligned with
X.
Returns:
-
Self–The estimator. A llm-ports calibrator never raises; too little data is recorded in
-
Self–insufficient_.
predict_proba
¶
predict_proba(
X: Sequence[float],
) -> list[tuple[float, float]] | None
Return (P(no), P(yes)) per stated probability, or None with no usable fit.
PiecewiseLinear
dataclass
¶
PiecewiseLinear(
xs: tuple[float, ...], ys: tuple[float, ...], n: int = 0
)
A monotone non-decreasing calibration map held as knots.
Between knots the map interpolates linearly; outside them it holds the end value. It is the
fitted map, not a fitter: IsotonicCalibrator.fit returns one in map_.
Attributes:
-
xs(tuple[float, ...]) –Knot inputs, strictly increasing, in [0, 1].
-
ys(tuple[float, ...]) –Knot outputs, non-decreasing, in [0, 1].
-
n(int) –The pairs it was fitted on (0 when unknown).
Raises:
-
ValueError–At construction, for no knots, unequal lengths, inputs not strictly increasing, outputs decreasing, or a knot outside [0, 1].
predict
¶
predict(X: Sequence[float]) -> list[float]
Return the calibrated probability per stated probability. Never raises.
predict_proba
¶
predict_proba(
X: Sequence[float],
) -> list[tuple[float, float]]
Return (P(no), P(yes)) per stated probability. Never raises.
from_dict
classmethod
¶
from_dict(data: Mapping[str, Any]) -> PiecewiseLinear
Build a map from as_dict output.
Raises:
-
ValueError–As the constructor, or for a missing key.
usable
¶
usable(
X: Sequence[float], y: Sequence[bool], need: int
) -> tuple[list[float], list[bool]] | Insufficient
Return the aligned pairs a fitter may use, or why it may not.
A pair with an invalid probability is dropped. A fit needs at least need pairs and both
outcomes present.
Parameters:
-
X(Sequence[float]) –Stated probabilities.
-
y(Sequence[bool]) –Outcomes aligned with
X; sequences of different lengths are not aligned, so nothing is usable. -
need(int) –The fitter's floor.
Returns:
-
tuple[list[float], list[bool]] | Insufficient–(probabilities, outcomes), orInsufficient. Never raises.
proba_rows
¶
proba_rows(
ps: Sequence[float],
) -> list[tuple[float, float]]
Return (1 - p, p) per probability: the two-column predict_proba layout.
llm_ports.forecast.hedge¶
Verbal confidence, learned per source: what "likely" from one voice has meant.
A voice that writes "likely" does not state a number, and two voices mean different things by
the same word. HedgeMap (a scikit-learn estimator) learns, from resolved forecasts, the probability each (source, hedge)
has earned. A source with few resolved uses of a word is shrunk toward the word's pooled rate
across every source (beta-binomial shrinkage), so one lucky call does not make "could" mean 0.9.
The prior for a word is Beta(m * s, (1 - m) * s): m is the word's pooled hit rate, s its
strength in pseudo-observations. s is estimated by the method of moments from how much the
sources disagree beyond binomial noise (Kleinman's weighted estimator). It falls back to
DEFAULT_STRENGTH when it cannot be estimated (fewer than two sources, a pooled rate of 0 or 1,
or sources that each used the word once), and to MAX_STRENGTH (near-full pooling) when the
sources agree within noise.
Invariants
- Pure, standard library only.
- Hedge text is compared normalised: case-folded, whitespace collapsed.
- A word never seen resolved has no probability (None), never a guess.
Rationale: ADR-0013.
HedgeRow
dataclass
¶
HedgeRow(
source: str,
hedge: str,
n: int,
hits: int,
p: float,
pooled: float,
)
One (source, hedge) cell of a fitted map.
Attributes:
-
source(str) –The voice.
-
hedge(str) –The normalised word or phrase.
-
n(int) –Resolved forecasts that used it.
-
hits(int) –How many happened.
-
p(float) –The shrunk probability it has earned.
-
pooled(float) –The word's pooled rate across every source (the prior mean).
HedgeMap
¶
HedgeMap(strength: float | None = None)
Bases: Params
The probability each (source, hedge) has earned: a scikit-learn estimator.
fit(X, y) learns from resolved forecasts, X rows of (source, hedge) and y whether each
happened; probability(source, hedge) and predict_proba(X) read the fitted map.
Parameters:
-
strength(float | None, default:None) –A fixed prior strength for every word (any finite real > 0, stored as float); None (default) estimates it per word.
Attributes:
-
strength–As given.
-
cells_(dict[tuple[str, str], tuple[int, int]]) –(source, hedge) -> (hits, n). Set byfitorfrom_dict. -
pooled_(dict[str, float]) –hedge -> pooled hit rate(Jeffreys-smoothed). -
strengths_(dict[str, float]) –hedge -> prior strengthused for that word.
fit
¶
fit(
X: Iterable[tuple[str, str]], y: Iterable[bool]
) -> Self
Learn the map from resolved forecasts.
Parameters:
-
X(Iterable[tuple[str, str]]) –(source, hedge)rows; a row whose hedge is empty or not a string is skipped. -
y(Iterable[bool]) –Whether each event happened, aligned with
X.
Returns:
-
Self–The estimator.
Raises:
-
ValueError–For a
strengththat is not finite and > 0, orXandyof different lengths.
probability
¶
probability(source: str, hedge: str) -> float | None
Return the probability hedge from source has earned.
A source that never used the word resolved gets the word's pooled rate.
Returns:
-
float | None–The probability, or None for a word no source has used resolved, a hedge that is
-
float | None–not a string, or an unfitted map. Never raises.
predict_proba
¶
predict_proba(
X: Iterable[tuple[str, str]],
) -> list[tuple[float, float] | None]
Return (P(no), P(yes)) per (source, hedge) row; None for a word with no probability.
Never raises.
rows
¶
rows() -> tuple[HedgeRow, ...]
Return every fitted cell, by source then hedge, for a readout; empty when unfitted.
as_dict
¶
as_dict() -> dict[str, Any]
Return the fitted map as a JSON-ready dict; from_dict reads it back.
normalise
¶
normalise(hedge: str) -> str
Return hedge case-folded with whitespace collapsed, the key a HedgeMap compares.
llm_ports.forecast.combine¶
Combining voices: one probability from several forecasts of the same event.
Combiner: the port, River's classifier protocol.predict_proba_one(x)reads the voices' probabilities for one event (xisvoice -> probability) and returns{False: P(no), True: P(yes)};learn_one(x, y)hears the outcome, so an online combiner can move its weights. A River binary classifier satisfies it unwrapped.LogitPool: the fixed baseline, a weighted mean in log-odds, optionally extremized (Satopää et al.: averaging forecasts that share information pulls the pool toward 0.5, and an alpha above 1 pushes it back). With no weights it is the equal-weight pool; it never learns.PriorPool: wraps any combiner with a prior voice: shrinks its log-odds toward the prior's, extremizes the result, and answers with the prior alone while the inner combiner abstains.BestVoice: the best-single-voice baseline, online: it answers with whichever speaking voice has the lowest running Brier score so far.logit/expit: the log-odds transform and its inverse, clipped so a 0 or 1 stays finite.
Invariants
- Pure, standard library only; learned combiners live in adapters (
adapters.river). - A combined probability is in (0, 1); no usable voice is an empty dict, never 0.5, so
predict_proba_one(x).get(True)is None.
Rationale: ADR-0013.
Combiner
¶
Bases: Protocol
Combines voices' probabilities for one event (River's classifier protocol).
predict_proba_one
¶
predict_proba_one(
x: Mapping[str, float],
) -> dict[bool, float]
Return the combined probability as {False: P(no), True: P(yes)}.
Parameters:
-
x(Mapping[str, float]) –voice -> probabilityfor the voices that spoke; a silent voice is absent.
Returns:
-
dict[bool, float]–The two class probabilities, or
{}when no voice is usable or the combiner cannot -
dict[bool, float]–combine yet. Never raises.
learn_one
¶
learn_one(x: Mapping[str, float], y: bool) -> None
Hear the outcome y of an event predict_proba_one was asked about. Never raises.
LogitPool
dataclass
¶
LogitPool(
weights: Mapping[str, float] = dict(),
alpha: float = 1.0,
)
A weighted mean in log-odds, extremized by alpha; a fixed baseline that never learns.
Attributes:
-
weights(Mapping[str, float]) –voice -> weight; a voice absent here weighs 1.0. Weights are renormalised over the voices that spoke, and a voice whose weight is not finite and > 0 is ignored. -
alpha(float) –The extremizing factor; 1.0 (default) leaves the pool as is. A non-finite
alphamakespredict_proba_onereturn{}.
PriorPool
dataclass
¶
PriorPool(
inner: Combiner,
prior_key: str = "prior",
alpha: float = 1.0,
shrink: float = 0.0,
)
Shrink a combiner toward a prior voice, then extremize: expit(alpha * z).
z = (1 - shrink) * logit(inner) + shrink * logit(prior), where inner is the wrapped
combiner's probability and prior is x[prior_key]. The prior also reaches the inner
combiner as an ordinary voice, so a learned combiner can weigh it. While the inner combiner
abstains ({}, e.g. before its warmup), the pool answers with the prior alone, extremized.
Without a prior in x it is the inner combiner, extremized.
Attributes:
-
inner(Combiner) –The wrapped combiner; it hears every outcome
learn_onehears. -
prior_key(str) –The voice in
xthat carries the prior. -
alpha(float) –The extremizing factor; 1.0 leaves the pool as is.
-
shrink(float) –The weight on the prior's log-odds, in [0, 1]; 0 ignores it beyond the fallback. A non-finite
alphaor ashrinkoutside [0, 1] makespredict_proba_onereturn{}.
predict_proba_one
¶
predict_proba_one(
x: Mapping[str, float],
) -> dict[bool, float]
Return {False: P(no), True: P(yes)}, or {} when neither the inner combiner nor the prior answers. Never raises.
learn_one
¶
learn_one(x: Mapping[str, float], y: bool) -> None
Pass the outcome to the inner combiner. Never raises.
BestVoice
dataclass
¶
BestVoice(
min_n: int = 20,
n: dict[str, int] = dict(),
brier_sum: dict[str, float] = dict(),
)
The best single voice so far: answers with the speaking voice of lowest running Brier score.
A voice is eligible once it has been heard on min_n outcomes; among the eligible voices in
x, the one with the lowest mean Brier score answers (ties: the voice name, so the choice is
deterministic). With no eligible voice it abstains ({}).
Attributes:
-
min_n(int) –Outcomes a voice must have been scored on before it can answer.
-
n(dict[str, int]) –voice -> outcomes it was scored on. -
brier_sum(dict[str, float]) –voice -> sum of its Brier scores.
best
¶
best(x: Mapping[str, float]) -> str | None
Return the voice that would answer for x, or None. Never raises.
predict_proba_one
¶
predict_proba_one(
x: Mapping[str, float],
) -> dict[bool, float]
Return the best eligible voice's {False: P(no), True: P(yes)}, or {}. Never raises.
learn_one
¶
learn_one(x: Mapping[str, float], y: bool) -> None
Score every usable voice in x against y. Never raises.
logit
¶
logit(p: float, eps: float = EPS) -> float
Return ln(p / (1 - p)), with p clipped to [eps, 1 - eps].
proba_one
¶
proba_one(p: float | None) -> dict[bool, float]
Return {False: 1 - p, True: p}, River's predict_proba_one layout; {} for None.
llm_ports.forecast.walk¶
Replaying a combiner in time order, as it would have run: predict first, learn once resolved.
Event: one question the voices spoke on:x(voice -> probability), when the combined forecast is made (forecast_time), when the outcome became known (resolve_time) and the outcome (resolution: True / False, None while open).walk_forward(combiner, events): replays the events inforecast_timeorder. Before each prediction the combiner hears every outcome whoseresolve_timeis at or before thatforecast_time, so it never learns from an outcome before the outcome was known. It returns one probability (None where the combiner abstained) per event, in the input's order. An open event is predicted and never learned, so the same replay yields the shadow forecasts.holdout_start(events, share): theforecast_timefrom which the lastshareof the resolved events falls, the split a tuner fits before and is judged after.compare(combiners, events, since=): walks each combiner over the same events and scores each on the same subset: resolved, forecast at or aftersince, and answered by every combiner.
Invariants
- Pure, standard library only.
- Never raises for a combiner that keeps the
Combinercontract. - Ties in time keep the input order; an outcome resolving exactly at a forecast time is heard before that forecast.
Rationale: ADR-0015.
Event
dataclass
¶
Event(
x: Mapping[str, float],
forecast_time: datetime,
resolve_time: datetime | None = None,
resolution: bool | None = None,
)
One question several voices forecast.
Attributes:
-
x(Mapping[str, float]) –voice -> probabilityof yes, for the voices that spoke. -
forecast_time(datetime) –When the combined forecast is made (the newest voice's time, at earliest).
-
resolve_time(datetime | None) –When the outcome became known; None while open.
-
resolution(bool | None) –True if it happened, False if not, None while open.
walk_forward
¶
Replay events through combiner in time order; return each event's probability of yes.
Parameters:
-
combiner(Combiner) –A fresh combiner; it is trained in place.
-
events(Sequence[Event]) –The events, in any order.
Returns:
-
list[float | None]–One probability per event, aligned with
events; None where the combiner abstained. -
list[float | None]–Never raises for a combiner that keeps its contract.
holdout_start
¶
holdout_start(
events: Sequence[Event], share: float = 0.2
) -> datetime | None
Return the forecast_time from which the last share of the resolved events falls.
Parameters:
-
events(Sequence[Event]) –The events.
-
share(float, default:0.2) –The holdout's share of the resolved events, in (0, 1).
Returns:
-
datetime | None–The first holdout event's
forecast_time, or None when there are fewer than two resolved -
datetime | None–events or
shareis outside (0, 1). Never raises.
compare
¶
compare(
combiners: Mapping[str, Combiner],
events: Sequence[Event],
since: datetime | None = None,
) -> dict[str, Score]
Walk each combiner over events and score all of them on the same events.
Parameters:
-
combiners(Mapping[str, Combiner]) –name -> fresh combiner; each is trained in place. -
events(Sequence[Event]) –The events, in any order.
-
since(datetime | None, default:None) –Score only events forecast at or after this time (a holdout); None scores all.
Returns: