Skip to content

llm_ports.forecast

llm_ports.forecast

Forecasts: grade what a voice said would happen, calibrate it, combine voices.

A voice is anything that asserts a future event: a model role, a human author transcribed by a model, an assembler. Each assertion is a Forecast; once the world settles it, a Resolved. The seams speak standard protocols, so outside models drop in unwrapped: calibrators and HedgeMap are scikit-learn estimators (fit(X, y), predict_proba(X), get_params), combiners are River classifiers (learn_one(x, y), predict_proba_one(x)), scores take scoringrules' names and order (brier_score(obs, fct)), and the values take forecasting-platform, schema.org and W3C PROV field names (_types maps them). This package holds the parts that do not depend on who the voices are:

  • score: proper scores (Brier, log score, skill), the reliability curve and its error.
  • skill: brier_skill, skill over a per-event REFERENCE forecast (a base rate, a prior), with a cluster-bootstrap interval.
  • calibrate: the Calibrator port and PiecewiseLinear, a fitted map applied without a fitting library (fitters: adapters.sklearn, adapters.venn_abers).
  • platt: PlattCalibrator, a two-parameter logistic map fitted and applied with the standard library; it refuses a slope that inverts the voice's ranking.
  • hedge: HedgeMap, the probability each source's verbal confidence has earned.
  • combine: the Combiner port, LogitPool (the fixed baseline), PriorPool (shrink toward and fall back to a prior voice, extremize), BestVoice (the best-single-voice baseline) and the log-odds transform (a learned combiner: adapters.river).
  • walk: Event and walk_forward, a combiner replayed in time order that learns an outcome only once it was known; compare scores several on the same events (tuning: adapters.optuna).
Invariants
  • Standard library only: imports nothing else from llm_ports and no third-party package (enforced by an import-linter contract), like llm_ports.learn.

Rationale: ADR-0013.

Params

Mixin: get_params / set_params over the __init__ parameters.

get_params

get_params(deep: bool = True) -> dict[str, Any]

Return the constructor parameters by name.

Parameters:

  • deep (bool, default: True ) –

    Accepted for the scikit-learn contract; these estimators hold no sub-estimators.

Returns:

  • dict[str, Any] –

    name -> value for every __init__ parameter. Never raises.

set_params

set_params(**params: Any) -> Self

Set constructor parameters by name; fitted state is untouched until the next fit.

Returns:

  • Self –

    The estimator.

Raises:

  • ValueError –

    For a name that is not an __init__ parameter (scikit-learn's contract).

Forecast dataclass

Forecast(
    author: str,
    subject: str,
    kind: str,
    forecast_time: datetime,
    resolve_time: datetime,
    text: str,
    probability: float | None = None,
    hedge: str | None = None,
    resolution_criteria: str | None = None,
    was_generated_by: str | None = None,
    used: tuple[str, ...] = (),
)

One gradeable statement a voice made about a future event.

A forecast is a record of what was said, not an input to a decision: its probability is the voice's own (probability), its verbal confidence is kept as written (hedge), and a calibrator or a HedgeMap turns either into a probability the voice has earned.

Attributes:

  • author (str) –

    Who asserted it (an author, a model role, an assembler).

  • subject (str) –

    What it is about (an instrument, an event key).

  • kind (str) –

    What shape of claim it is (direction, level, range, event, ...).

  • forecast_time (datetime) –

    When it was asserted; only text available then may back it.

  • resolve_time (datetime) –

    When the world settles it; never before forecast_time.

  • text (str) –

    The span the forecast was read from, verbatim.

  • probability (float | None) –

    The probability the voice stated, or None when it stated only words.

  • hedge (str | None) –

    The verbal confidence as written (likely, could), or None.

  • resolution_criteria (str | None) –

    How the outcome is read, or None when kind implies it.

  • was_generated_by (str | None) –

    What transcribed it (a prompt pack and its sha), or None when the author wrote it directly.

  • used (tuple[str, ...]) –

    What the transcription read (source document ids); empty when none are recorded.

Raises:

  • ValueError –

    At construction, for a probability outside [0, 1] or not finite, a resolve_time before forecast_time (or one naive and one aware), an empty text, author, subject or kind, or a forecast with neither probability nor hedge.

Resolved dataclass

Resolved(
    claim_reviewed: Forecast,
    resolution: Resolution,
    actual_resolve_time: datetime,
)

A forecast and what the world did.

Attributes:

  • claim_reviewed (Forecast) –

    The forecast.

  • resolution (Resolution) –

    yes when the forecast event happened, no when it did not, annulled when it can no longer be graded (the question was voided).

  • actual_resolve_time (datetime) –

    When the outcome was read.

Raises:

  • ValueError –

    At construction, for a resolution that is not yes, no or annulled.

outcome property

outcome: bool | None

True for yes, False for no, None for annulled (not gradeable).

Calibrator

Bases: Protocol

Fits a calibration map from resolved forecasts (the scikit-learn estimator protocol).

get_params

get_params(deep: bool = True) -> dict[str, Any]

Return the constructor parameters by name.

set_params

set_params(**params: Any) -> Self

Set constructor parameters by name and return the estimator.

fit

fit(X: Sequence[float], y: Sequence[bool]) -> Self

Fit a map from stated probabilities and what happened.

Parameters:

  • X (Sequence[float]) –

    Stated probabilities.

  • y (Sequence[bool]) –

    Whether each event happened, aligned with X.

Returns:

  • Self –

    The estimator. A llm-ports calibrator never raises; too little data is recorded in

  • Self –

    insufficient_.

predict_proba

predict_proba(
    X: Sequence[float],
) -> list[tuple[float, float]] | None

Return (P(no), P(yes)) per stated probability, or None with no usable fit.

Insufficient dataclass

Insufficient(n: int, need: int, reason: str)

Too few resolved forecasts to fit a map.

Attributes:

  • n (int) –

    Usable pairs given.

  • need (int) –

    The fitter's floor.

  • reason (str) –

    Why, in words (n < need, one outcome only).

PiecewiseLinear dataclass

PiecewiseLinear(
    xs: tuple[float, ...], ys: tuple[float, ...], n: int = 0
)

A monotone non-decreasing calibration map held as knots.

Between knots the map interpolates linearly; outside them it holds the end value. It is the fitted map, not a fitter: IsotonicCalibrator.fit returns one in map_.

Attributes:

  • xs (tuple[float, ...]) –

    Knot inputs, strictly increasing, in [0, 1].

  • ys (tuple[float, ...]) –

    Knot outputs, non-decreasing, in [0, 1].

  • n (int) –

    The pairs it was fitted on (0 when unknown).

Raises:

  • ValueError –

    At construction, for no knots, unequal lengths, inputs not strictly increasing, outputs decreasing, or a knot outside [0, 1].

predict

predict(X: Sequence[float]) -> list[float]

Return the calibrated probability per stated probability. Never raises.

predict_proba

predict_proba(
    X: Sequence[float],
) -> list[tuple[float, float]]

Return (P(no), P(yes)) per stated probability. Never raises.

as_dict

as_dict() -> dict[str, Any]

Return the map as a JSON-ready dict; from_dict reads it back.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> PiecewiseLinear

Build a map from as_dict output.

Raises:

  • ValueError –

    As the constructor, or for a missing key.

BestVoice dataclass

BestVoice(
    min_n: int = 20,
    n: dict[str, int] = dict(),
    brier_sum: dict[str, float] = dict(),
)

The best single voice so far: answers with the speaking voice of lowest running Brier score.

A voice is eligible once it has been heard on min_n outcomes; among the eligible voices in x, the one with the lowest mean Brier score answers (ties: the voice name, so the choice is deterministic). With no eligible voice it abstains ({}).

Attributes:

  • min_n (int) –

    Outcomes a voice must have been scored on before it can answer.

  • n (dict[str, int]) –

    voice -> outcomes it was scored on.

  • brier_sum (dict[str, float]) –

    voice -> sum of its Brier scores.

best

best(x: Mapping[str, float]) -> str | None

Return the voice that would answer for x, or None. Never raises.

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return the best eligible voice's {False: P(no), True: P(yes)}, or {}. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Score every usable voice in x against y. Never raises.

Combiner

Bases: Protocol

Combines voices' probabilities for one event (River's classifier protocol).

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return the combined probability as {False: P(no), True: P(yes)}.

Parameters:

  • x (Mapping[str, float]) –

    voice -> probability for the voices that spoke; a silent voice is absent.

Returns:

  • dict[bool, float] –

    The two class probabilities, or {} when no voice is usable or the combiner cannot

  • dict[bool, float] –

    combine yet. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Hear the outcome y of an event predict_proba_one was asked about. Never raises.

LogitPool dataclass

LogitPool(
    weights: Mapping[str, float] = dict(),
    alpha: float = 1.0,
)

A weighted mean in log-odds, extremized by alpha; a fixed baseline that never learns.

Attributes:

  • weights (Mapping[str, float]) –

    voice -> weight; a voice absent here weighs 1.0. Weights are renormalised over the voices that spoke, and a voice whose weight is not finite and > 0 is ignored.

  • alpha (float) –

    The extremizing factor; 1.0 (default) leaves the pool as is. A non-finite alpha makes predict_proba_one return {}.

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return the pooled {False: P(no), True: P(yes)}, or {} when no voice is usable. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Do nothing: the pool is fixed. Never raises.

PriorPool dataclass

PriorPool(
    inner: Combiner,
    prior_key: str = "prior",
    alpha: float = 1.0,
    shrink: float = 0.0,
)

Shrink a combiner toward a prior voice, then extremize: expit(alpha * z).

z = (1 - shrink) * logit(inner) + shrink * logit(prior), where inner is the wrapped combiner's probability and prior is x[prior_key]. The prior also reaches the inner combiner as an ordinary voice, so a learned combiner can weigh it. While the inner combiner abstains ({}, e.g. before its warmup), the pool answers with the prior alone, extremized. Without a prior in x it is the inner combiner, extremized.

Attributes:

  • inner (Combiner) –

    The wrapped combiner; it hears every outcome learn_one hears.

  • prior_key (str) –

    The voice in x that carries the prior.

  • alpha (float) –

    The extremizing factor; 1.0 leaves the pool as is.

  • shrink (float) –

    The weight on the prior's log-odds, in [0, 1]; 0 ignores it beyond the fallback. A non-finite alpha or a shrink outside [0, 1] makes predict_proba_one return {}.

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return {False: P(no), True: P(yes)}, or {} when neither the inner combiner nor the prior answers. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Pass the outcome to the inner combiner. Never raises.

HedgeMap

HedgeMap(strength: float | None = None)

Bases: Params

The probability each (source, hedge) has earned: a scikit-learn estimator.

fit(X, y) learns from resolved forecasts, X rows of (source, hedge) and y whether each happened; probability(source, hedge) and predict_proba(X) read the fitted map.

Parameters:

  • strength (float | None, default: None ) –

    A fixed prior strength for every word (any finite real > 0, stored as float); None (default) estimates it per word.

Attributes:

  • strength –

    As given.

  • cells_ (dict[tuple[str, str], tuple[int, int]]) –

    (source, hedge) -> (hits, n). Set by fit or from_dict.

  • pooled_ (dict[str, float]) –

    hedge -> pooled hit rate (Jeffreys-smoothed).

  • strengths_ (dict[str, float]) –

    hedge -> prior strength used for that word.

fit

fit(
    X: Iterable[tuple[str, str]], y: Iterable[bool]
) -> Self

Learn the map from resolved forecasts.

Parameters:

  • X (Iterable[tuple[str, str]]) –

    (source, hedge) rows; a row whose hedge is empty or not a string is skipped.

  • y (Iterable[bool]) –

    Whether each event happened, aligned with X.

Returns:

  • Self –

    The estimator.

Raises:

  • ValueError –

    For a strength that is not finite and > 0, or X and y of different lengths.

probability

probability(source: str, hedge: str) -> float | None

Return the probability hedge from source has earned.

A source that never used the word resolved gets the word's pooled rate.

Returns:

  • float | None –

    The probability, or None for a word no source has used resolved, a hedge that is

  • float | None –

    not a string, or an unfitted map. Never raises.

predict_proba

predict_proba(
    X: Iterable[tuple[str, str]],
) -> list[tuple[float, float] | None]

Return (P(no), P(yes)) per (source, hedge) row; None for a word with no probability.

Never raises.

rows

rows() -> tuple[HedgeRow, ...]

Return every fitted cell, by source then hedge, for a readout; empty when unfitted.

as_dict

as_dict() -> dict[str, Any]

Return the fitted map as a JSON-ready dict; from_dict reads it back.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> HedgeMap

Build a fitted map from as_dict output.

Raises:

  • ValueError –

    For a dict that is not as_dict output, or counts, rates or strengths out of range.

HedgeRow dataclass

HedgeRow(
    source: str,
    hedge: str,
    n: int,
    hits: int,
    p: float,
    pooled: float,
)

One (source, hedge) cell of a fitted map.

Attributes:

  • source (str) –

    The voice.

  • hedge (str) –

    The normalised word or phrase.

  • n (int) –

    Resolved forecasts that used it.

  • hits (int) –

    How many happened.

  • p (float) –

    The shrunk probability it has earned.

  • pooled (float) –

    The word's pooled rate across every source (the prior mean).

Platt dataclass

Platt(a: float, b: float, n: int = 0)

A fitted Platt map: p -> sigmoid(a * logit(p) + b).

Attributes:

  • a (float) –

    Slope on the logit of the stated probability; finite and > 0.

  • b (float) –

    Intercept; finite.

  • n (int) –

    The pairs it was fitted on (0 when unknown).

Raises:

  • ValueError –

    At construction, for a slope that is not finite and > 0 or an intercept that is not finite.

predict

predict(X: Sequence[float]) -> list[float]

Return the calibrated probability per stated probability.

as_dict

as_dict() -> dict[str, Any]

Return the map as plain JSON-serialisable values.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> Platt

Rebuild a map from as_dict output.

Raises:

  • ValueError –

    For a missing key or values that fail the construction checks.

PlattCalibrator

PlattCalibrator(
    min_pairs: int = MIN_PAIRS,
    ridge: float = 0.001,
    max_iter: int = 50,
    tol: float = 1e-08,
)

Bases: Params

Fits a Platt calibration map from stated probabilities and outcomes.

Parameters:

  • min_pairs (int, default: MIN_PAIRS ) –

    The floor of usable pairs; MIN_PAIRS by default.

  • ridge (float, default: 0.001 ) –

    L2 penalty on (a, b), which keeps a Newton step finite on separable or thin data.

  • max_iter (int, default: 50 ) –

    Newton steps at most.

  • tol (float, default: 1e-08 ) –

    Stop once both parameter steps are below this.

Attributes:

  • map_ (Platt | None) –

    The fitted Platt, or None when the last fit produced none. Set by fit.

  • insufficient_ (Insufficient | None) –

    Why the last fit produced no map, or None.

  • fitted_ (tuple[float, float] | None) –

    The (a, b) the last fit reached, kept even when refused; None before a fit or when there was too little data to fit.

fit

fit(X: Sequence[float], y: Sequence[bool]) -> Self

Fit the map; never raises (the reason a fit produced no map lands in insufficient_).

Returns:

  • Self –

    The estimator.

predict_proba

predict_proba(
    X: Sequence[float],
) -> list[tuple[float, float]] | None

Return (P(no), P(yes)) per stated probability, or None with no usable fit.

Bin dataclass

Bin(
    lo: float,
    hi: float,
    n: int,
    mean_p: float,
    hit_rate: float,
)

One bin of a reliability curve.

Attributes:

  • lo (float) –

    The bin's lower edge (inclusive).

  • hi (float) –

    The bin's upper edge (exclusive, except the last bin, which includes 1.0).

  • n (int) –

    Forecasts in the bin.

  • mean_p (float) –

    Their mean probability.

  • hit_rate (float) –

    The share that happened; on a calibrated voice it tracks mean_p.

Decomposition dataclass

Decomposition(
    n: int,
    base_rate: float,
    reliability: float,
    resolution: float,
    uncertainty: float,
)

Murphy's decomposition of the Brier score over a reliability curve.

brier = reliability - resolution + uncertainty. It is exact when every forecast in a bin states the same probability (a curve over cells, each forecasting one rate); otherwise it approximates the mean Brier score by the within-bin spread.

Attributes:

  • n (int) –

    Forecasts on the curve.

  • base_rate (float) –

    The share that happened across every bin.

  • reliability (float) –

    The n-weighted mean squared bin gap; the calibration error (lower is better).

  • resolution (float) –

    The n-weighted mean squared distance of each bin's hit rate from the base rate; how far the forecasts separate outcomes (higher is better).

  • uncertainty (float) –

    base_rate * (1 - base_rate); the variance no forecast can remove.

brier property

brier: float

reliability - resolution + uncertainty.

Score dataclass

Score(
    n: int,
    mean_p: float | None,
    hit_rate: float | None,
    brier_score: float | None,
    log_score: float | None,
    brier_skill_score: float | None,
)

How a set of binary forecasts scored.

Attributes:

  • n (int) –

    Pairs scored.

  • mean_p (float | None) –

    The mean forecast probability; None at n = 0.

  • hit_rate (float | None) –

    The share that happened (the base rate); None at n = 0.

  • brier_score (float | None) –

    The mean Brier score; None at n = 0.

  • log_score (float | None) –

    The mean log score; None at n = 0.

  • brier_skill_score (float | None) –

    1 - brier_score / (hit_rate * (1 - hit_rate)), skill over always forecasting the base rate (> 0 beats it); None when undefined (n = 0, or every outcome the same).

overconfidence property

overconfidence: float | None

mean_p - hit_rate: positive when the voice claims more than it delivers; None at n = 0.

Skill dataclass

Skill(
    n: int,
    clusters: int,
    brier_score: float | None,
    ref_brier_score: float | None,
    skill: float | None,
    lo: float | None,
    hi: float | None,
)

How a voice scored against a reference forecast on the same events.

Attributes:

  • n (int) –

    Pairs scored (both probabilities valid).

  • clusters (int) –

    Distinct clusters among them.

  • brier_score (float | None) –

    The voice's mean Brier score; None at n = 0.

  • ref_brier_score (float | None) –

    The reference's mean Brier score on the same pairs; None at n = 0.

  • skill (float | None) –

    1 - brier_score / ref_brier_score; None at n = 0 or when the reference is perfect.

  • lo (float | None) –

    Lower bound of the cluster-bootstrap interval on skill; None below MIN_CLUSTERS.

  • hi (float | None) –

    Upper bound, as lo.

Event dataclass

Event(
    x: Mapping[str, float],
    forecast_time: datetime,
    resolve_time: datetime | None = None,
    resolution: bool | None = None,
)

One question several voices forecast.

Attributes:

  • x (Mapping[str, float]) –

    voice -> probability of yes, for the voices that spoke.

  • forecast_time (datetime) –

    When the combined forecast is made (the newest voice's time, at earliest).

  • resolve_time (datetime | None) –

    When the outcome became known; None while open.

  • resolution (bool | None) –

    True if it happened, False if not, None while open.

resolved property

resolved: bool

True when both the outcome and its time are known.

finite_real

finite_real(v: object) -> bool

Return True for a finite numbers.Real that is not a bool (a weight, a strength). Never raises.

valid_p

valid_p(p: object) -> bool

Return True when p is a finite real probability in [0, 1] (a bool is not a probability).

Any numbers.Real counts (int, float, NumPy floating scalars, Fraction); Decimal is not a numbers.Real and is refused. A caller that accepts p reads it through float. Never raises.

brier_skill

brier_skill(
    obs: Sequence[bool],
    fct: Sequence[float],
    ref: Sequence[float],
    clusters: Sequence[Hashable] | None = None,
    *,
    level: float = 0.95,
    n_resamples: int = N_RESAMPLES,
    seed: int = 0
) -> Skill

Score forecasts against a per-event reference forecast.

Parameters:

  • obs (Sequence[bool]) –

    Whether each event happened.

  • fct (Sequence[float]) –

    The voice's probabilities, aligned with obs.

  • ref (Sequence[float]) –

    The reference's probabilities for the same events.

  • clusters (Sequence[Hashable] | None, default: None ) –

    A cluster id per event (events sharing evidence share an id); None = each event is its own cluster.

  • level (float, default: 0.95 ) –

    Two-sided confidence level of the interval, in (0, 1).

  • n_resamples (int, default: N_RESAMPLES ) –

    Bootstrap resamples.

  • seed (int, default: 0 ) –

    The generator seed.

Returns:

  • Skill –

    The Skill. Never raises on data; a sequence of a different length scores nothing.

Raises:

  • ValueError –

    level outside (0, 1) or n_resamples < 1.

walk_forward

walk_forward(
    combiner: Combiner, events: Sequence[Event]
) -> list[float | None]

Replay events through combiner in time order; return each event's probability of yes.

Parameters:

  • combiner (Combiner) –

    A fresh combiner; it is trained in place.

  • events (Sequence[Event]) –

    The events, in any order.

Returns:

  • list[float | None] –

    One probability per event, aligned with events; None where the combiner abstained.

  • list[float | None] –

    Never raises for a combiner that keeps its contract.

llm_ports.forecast.score

Proper scores for binary forecasts: Brier, log score, skill against the base rate, reliability.

A proper scoring rule is minimised in expectation only by reporting the true probability, so a voice cannot improve its score by hedging toward 0.5 or by overclaiming. Names and argument order follow scoringrules: observation first, forecast second (brier_score(obs, fct)), and lower is better.

Invariants
  • Pure, standard library only.
  • A pair whose forecast is not a finite probability is skipped, never scored.
  • Observations and forecasts of different lengths are not aligned, so nothing is scored.
  • A statistic the data cannot define is None, never a number.

Rationale: ADR-0013.

Score dataclass

Score(
    n: int,
    mean_p: float | None,
    hit_rate: float | None,
    brier_score: float | None,
    log_score: float | None,
    brier_skill_score: float | None,
)

How a set of binary forecasts scored.

Attributes:

  • n (int) –

    Pairs scored.

  • mean_p (float | None) –

    The mean forecast probability; None at n = 0.

  • hit_rate (float | None) –

    The share that happened (the base rate); None at n = 0.

  • brier_score (float | None) –

    The mean Brier score; None at n = 0.

  • log_score (float | None) –

    The mean log score; None at n = 0.

  • brier_skill_score (float | None) –

    1 - brier_score / (hit_rate * (1 - hit_rate)), skill over always forecasting the base rate (> 0 beats it); None when undefined (n = 0, or every outcome the same).

overconfidence property

overconfidence: float | None

mean_p - hit_rate: positive when the voice claims more than it delivers; None at n = 0.

Bin dataclass

Bin(
    lo: float,
    hi: float,
    n: int,
    mean_p: float,
    hit_rate: float,
)

One bin of a reliability curve.

Attributes:

  • lo (float) –

    The bin's lower edge (inclusive).

  • hi (float) –

    The bin's upper edge (exclusive, except the last bin, which includes 1.0).

  • n (int) –

    Forecasts in the bin.

  • mean_p (float) –

    Their mean probability.

  • hit_rate (float) –

    The share that happened; on a calibrated voice it tracks mean_p.

Decomposition dataclass

Decomposition(
    n: int,
    base_rate: float,
    reliability: float,
    resolution: float,
    uncertainty: float,
)

Murphy's decomposition of the Brier score over a reliability curve.

brier = reliability - resolution + uncertainty. It is exact when every forecast in a bin states the same probability (a curve over cells, each forecasting one rate); otherwise it approximates the mean Brier score by the within-bin spread.

Attributes:

  • n (int) –

    Forecasts on the curve.

  • base_rate (float) –

    The share that happened across every bin.

  • reliability (float) –

    The n-weighted mean squared bin gap; the calibration error (lower is better).

  • resolution (float) –

    The n-weighted mean squared distance of each bin's hit rate from the base rate; how far the forecasts separate outcomes (higher is better).

  • uncertainty (float) –

    base_rate * (1 - base_rate); the variance no forecast can remove.

brier property

brier: float

reliability - resolution + uncertainty.

brier_score

brier_score(obs: bool, fct: float) -> float

Return the Brier score of one binary forecast, (fct - obs) ** 2; 0 is perfect.

Parameters:

  • obs (bool) –

    Whether the event happened.

  • fct (float) –

    The forecast probability.

log_score

log_score(obs: bool, fct: float) -> float

Return the log score of one binary forecast, -ln(fct) if it happened else -ln(1 - fct).

fct is clipped to [EPS, 1 - EPS], so the score is finite.

Parameters:

  • obs (bool) –

    Whether the event happened.

  • fct (float) –

    The forecast probability.

score

score(obs: Sequence[bool], fct: Sequence[float]) -> Score

Score binary forecasts.

Parameters:

  • obs (Sequence[bool]) –

    Whether each event happened.

  • fct (Sequence[float]) –

    The forecast probabilities, aligned with obs; a pair with an invalid forecast is skipped, and sequences of different lengths score nothing (n = 0).

Returns:

  • Score –

    The Score. Never raises.

reliability

reliability(
    obs: Sequence[bool],
    fct: Sequence[float],
    bins: int = 10,
) -> tuple[Bin, ...]

Return the reliability curve: forecasts grouped into equal-width probability bins.

Parameters:

  • obs (Sequence[bool]) –

    Whether each event happened.

  • fct (Sequence[float]) –

    The forecast probabilities, aligned with obs; a pair with an invalid forecast is skipped, and sequences of different lengths give an empty curve.

  • bins (int, default: 10 ) –

    The number of equal-width bins over [0, 1]; values below 1 are read as 1.

Returns:

  • tuple[Bin, ...] –

    The non-empty bins, lowest first. Never raises.

ece

ece(curve: Sequence[Bin]) -> float | None

Return the expected calibration error of a reliability curve.

The n-weighted mean of |mean_p - hit_rate| over the bins.

Returns:

  • float | None –

    The error, or None for an empty curve.

mce

mce(curve: Sequence[Bin]) -> float | None

Return the maximum calibration error of a reliability curve: the largest bin gap.

Returns:

  • float | None –

    The largest |mean_p - hit_rate| over the bins, or None for an empty curve.

decompose

decompose(curve: Sequence[Bin]) -> Decomposition | None

Return Murphy's decomposition of a reliability curve.

Returns:

  • Decomposition | None –

    The Decomposition, or None for an empty curve.

llm_ports.forecast.skill

Paired skill: how much a voice's probabilities beat a REFERENCE forecast on the same events.

score.Score.brier_skill_score measures skill over climatology, the constant forecast of the sample's own hit rate. That is the wrong yardstick when a better reference exists for each event: a symbol's base rate, a model's prior, yesterday's forecast. A voice that says "up 60%" in a tape where the reference already says "up 60%" knows nothing the reference did not. Here the reference is given per event, and skill is 1 - BS(fct) / BS(ref) over the pairs both scored (> 0 beats the reference, 0 ties it, < 0 loses to it).

The interval resamples whole CLUSTERS, never single pairs: events that share evidence (one day's tape, overlapping horizons) are not independent, and an iid resample would be too narrow. The percentile bootstrap recomputes both Brier sums in every resample, so the reference's own noise is in the interval. Seeded, so a re-run over the same pairs is identical.

Invariants
  • Pure, standard library only.
  • A pair is scored only when BOTH probabilities are valid; others are skipped.
  • Inputs of different lengths are not aligned, so nothing is scored.
  • A statistic the data cannot define is None, never a number.

Rationale: ADR-0013.

Skill dataclass

Skill(
    n: int,
    clusters: int,
    brier_score: float | None,
    ref_brier_score: float | None,
    skill: float | None,
    lo: float | None,
    hi: float | None,
)

How a voice scored against a reference forecast on the same events.

Attributes:

  • n (int) –

    Pairs scored (both probabilities valid).

  • clusters (int) –

    Distinct clusters among them.

  • brier_score (float | None) –

    The voice's mean Brier score; None at n = 0.

  • ref_brier_score (float | None) –

    The reference's mean Brier score on the same pairs; None at n = 0.

  • skill (float | None) –

    1 - brier_score / ref_brier_score; None at n = 0 or when the reference is perfect.

  • lo (float | None) –

    Lower bound of the cluster-bootstrap interval on skill; None below MIN_CLUSTERS.

  • hi (float | None) –

    Upper bound, as lo.

brier_skill

brier_skill(
    obs: Sequence[bool],
    fct: Sequence[float],
    ref: Sequence[float],
    clusters: Sequence[Hashable] | None = None,
    *,
    level: float = 0.95,
    n_resamples: int = N_RESAMPLES,
    seed: int = 0
) -> Skill

Score forecasts against a per-event reference forecast.

Parameters:

  • obs (Sequence[bool]) –

    Whether each event happened.

  • fct (Sequence[float]) –

    The voice's probabilities, aligned with obs.

  • ref (Sequence[float]) –

    The reference's probabilities for the same events.

  • clusters (Sequence[Hashable] | None, default: None ) –

    A cluster id per event (events sharing evidence share an id); None = each event is its own cluster.

  • level (float, default: 0.95 ) –

    Two-sided confidence level of the interval, in (0, 1).

  • n_resamples (int, default: N_RESAMPLES ) –

    Bootstrap resamples.

  • seed (int, default: 0 ) –

    The generator seed.

Returns:

  • Skill –

    The Skill. Never raises on data; a sequence of a different length scores nothing.

Raises:

  • ValueError –

    level outside (0, 1) or n_resamples < 1.

llm_ports.forecast.calibrate

Calibration: map a voice's stated probability to the probability it has earned.

  • Calibrator: the port, the scikit-learn estimator protocol. fit(X, y) returns the estimator; predict_proba(X) returns one (P(no), P(yes)) row per stated probability, the column order of a binary scikit-learn classifier. A scikit-learn, venn-abers or MAPIE calibrator satisfies it unwrapped.
  • Insufficient: why a fit produced no map (too few resolved forecasts, one outcome only); a calibrator here records it in insufficient_ and its predict_proba returns None.
  • PiecewiseLinear: a monotone map held as knots, applied with the standard library and serialised as JSON, so a map fitted offline (an isotonic fit, adapters.sklearn) is applied where no fitting library is installed.
Invariants
  • Pure, standard library only; the fitting libraries live in adapters.
  • A calibrated probability is always in [0, 1].

Rationale: ADR-0013.

Insufficient dataclass

Insufficient(n: int, need: int, reason: str)

Too few resolved forecasts to fit a map.

Attributes:

  • n (int) –

    Usable pairs given.

  • need (int) –

    The fitter's floor.

  • reason (str) –

    Why, in words (n < need, one outcome only).

Calibrator

Bases: Protocol

Fits a calibration map from resolved forecasts (the scikit-learn estimator protocol).

get_params

get_params(deep: bool = True) -> dict[str, Any]

Return the constructor parameters by name.

set_params

set_params(**params: Any) -> Self

Set constructor parameters by name and return the estimator.

fit

fit(X: Sequence[float], y: Sequence[bool]) -> Self

Fit a map from stated probabilities and what happened.

Parameters:

  • X (Sequence[float]) –

    Stated probabilities.

  • y (Sequence[bool]) –

    Whether each event happened, aligned with X.

Returns:

  • Self –

    The estimator. A llm-ports calibrator never raises; too little data is recorded in

  • Self –

    insufficient_.

predict_proba

predict_proba(
    X: Sequence[float],
) -> list[tuple[float, float]] | None

Return (P(no), P(yes)) per stated probability, or None with no usable fit.

PiecewiseLinear dataclass

PiecewiseLinear(
    xs: tuple[float, ...], ys: tuple[float, ...], n: int = 0
)

A monotone non-decreasing calibration map held as knots.

Between knots the map interpolates linearly; outside them it holds the end value. It is the fitted map, not a fitter: IsotonicCalibrator.fit returns one in map_.

Attributes:

  • xs (tuple[float, ...]) –

    Knot inputs, strictly increasing, in [0, 1].

  • ys (tuple[float, ...]) –

    Knot outputs, non-decreasing, in [0, 1].

  • n (int) –

    The pairs it was fitted on (0 when unknown).

Raises:

  • ValueError –

    At construction, for no knots, unequal lengths, inputs not strictly increasing, outputs decreasing, or a knot outside [0, 1].

predict

predict(X: Sequence[float]) -> list[float]

Return the calibrated probability per stated probability. Never raises.

predict_proba

predict_proba(
    X: Sequence[float],
) -> list[tuple[float, float]]

Return (P(no), P(yes)) per stated probability. Never raises.

as_dict

as_dict() -> dict[str, Any]

Return the map as a JSON-ready dict; from_dict reads it back.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> PiecewiseLinear

Build a map from as_dict output.

Raises:

  • ValueError –

    As the constructor, or for a missing key.

usable

usable(
    X: Sequence[float], y: Sequence[bool], need: int
) -> tuple[list[float], list[bool]] | Insufficient

Return the aligned pairs a fitter may use, or why it may not.

A pair with an invalid probability is dropped. A fit needs at least need pairs and both outcomes present.

Parameters:

  • X (Sequence[float]) –

    Stated probabilities.

  • y (Sequence[bool]) –

    Outcomes aligned with X; sequences of different lengths are not aligned, so nothing is usable.

  • need (int) –

    The fitter's floor.

Returns:

  • tuple[list[float], list[bool]] | Insufficient –

    (probabilities, outcomes), or Insufficient. Never raises.

proba_rows

proba_rows(
    ps: Sequence[float],
) -> list[tuple[float, float]]

Return (1 - p, p) per probability: the two-column predict_proba layout.

llm_ports.forecast.hedge

Verbal confidence, learned per source: what "likely" from one voice has meant.

A voice that writes "likely" does not state a number, and two voices mean different things by the same word. HedgeMap (a scikit-learn estimator) learns, from resolved forecasts, the probability each (source, hedge) has earned. A source with few resolved uses of a word is shrunk toward the word's pooled rate across every source (beta-binomial shrinkage), so one lucky call does not make "could" mean 0.9.

The prior for a word is Beta(m * s, (1 - m) * s): m is the word's pooled hit rate, s its strength in pseudo-observations. s is estimated by the method of moments from how much the sources disagree beyond binomial noise (Kleinman's weighted estimator). It falls back to DEFAULT_STRENGTH when it cannot be estimated (fewer than two sources, a pooled rate of 0 or 1, or sources that each used the word once), and to MAX_STRENGTH (near-full pooling) when the sources agree within noise.

Invariants
  • Pure, standard library only.
  • Hedge text is compared normalised: case-folded, whitespace collapsed.
  • A word never seen resolved has no probability (None), never a guess.

Rationale: ADR-0013.

HedgeRow dataclass

HedgeRow(
    source: str,
    hedge: str,
    n: int,
    hits: int,
    p: float,
    pooled: float,
)

One (source, hedge) cell of a fitted map.

Attributes:

  • source (str) –

    The voice.

  • hedge (str) –

    The normalised word or phrase.

  • n (int) –

    Resolved forecasts that used it.

  • hits (int) –

    How many happened.

  • p (float) –

    The shrunk probability it has earned.

  • pooled (float) –

    The word's pooled rate across every source (the prior mean).

HedgeMap

HedgeMap(strength: float | None = None)

Bases: Params

The probability each (source, hedge) has earned: a scikit-learn estimator.

fit(X, y) learns from resolved forecasts, X rows of (source, hedge) and y whether each happened; probability(source, hedge) and predict_proba(X) read the fitted map.

Parameters:

  • strength (float | None, default: None ) –

    A fixed prior strength for every word (any finite real > 0, stored as float); None (default) estimates it per word.

Attributes:

  • strength –

    As given.

  • cells_ (dict[tuple[str, str], tuple[int, int]]) –

    (source, hedge) -> (hits, n). Set by fit or from_dict.

  • pooled_ (dict[str, float]) –

    hedge -> pooled hit rate (Jeffreys-smoothed).

  • strengths_ (dict[str, float]) –

    hedge -> prior strength used for that word.

fit

fit(
    X: Iterable[tuple[str, str]], y: Iterable[bool]
) -> Self

Learn the map from resolved forecasts.

Parameters:

  • X (Iterable[tuple[str, str]]) –

    (source, hedge) rows; a row whose hedge is empty or not a string is skipped.

  • y (Iterable[bool]) –

    Whether each event happened, aligned with X.

Returns:

  • Self –

    The estimator.

Raises:

  • ValueError –

    For a strength that is not finite and > 0, or X and y of different lengths.

probability

probability(source: str, hedge: str) -> float | None

Return the probability hedge from source has earned.

A source that never used the word resolved gets the word's pooled rate.

Returns:

  • float | None –

    The probability, or None for a word no source has used resolved, a hedge that is

  • float | None –

    not a string, or an unfitted map. Never raises.

predict_proba

predict_proba(
    X: Iterable[tuple[str, str]],
) -> list[tuple[float, float] | None]

Return (P(no), P(yes)) per (source, hedge) row; None for a word with no probability.

Never raises.

rows

rows() -> tuple[HedgeRow, ...]

Return every fitted cell, by source then hedge, for a readout; empty when unfitted.

as_dict

as_dict() -> dict[str, Any]

Return the fitted map as a JSON-ready dict; from_dict reads it back.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> HedgeMap

Build a fitted map from as_dict output.

Raises:

  • ValueError –

    For a dict that is not as_dict output, or counts, rates or strengths out of range.

normalise

normalise(hedge: str) -> str

Return hedge case-folded with whitespace collapsed, the key a HedgeMap compares.

llm_ports.forecast.combine

Combining voices: one probability from several forecasts of the same event.

  • Combiner: the port, River's classifier protocol. predict_proba_one(x) reads the voices' probabilities for one event (x is voice -> probability) and returns {False: P(no), True: P(yes)}; learn_one(x, y) hears the outcome, so an online combiner can move its weights. A River binary classifier satisfies it unwrapped.
  • LogitPool: the fixed baseline, a weighted mean in log-odds, optionally extremized (Satopää et al.: averaging forecasts that share information pulls the pool toward 0.5, and an alpha above 1 pushes it back). With no weights it is the equal-weight pool; it never learns.
  • PriorPool: wraps any combiner with a prior voice: shrinks its log-odds toward the prior's, extremizes the result, and answers with the prior alone while the inner combiner abstains.
  • BestVoice: the best-single-voice baseline, online: it answers with whichever speaking voice has the lowest running Brier score so far.
  • logit / expit: the log-odds transform and its inverse, clipped so a 0 or 1 stays finite.
Invariants
  • Pure, standard library only; learned combiners live in adapters (adapters.river).
  • A combined probability is in (0, 1); no usable voice is an empty dict, never 0.5, so predict_proba_one(x).get(True) is None.

Rationale: ADR-0013.

Combiner

Bases: Protocol

Combines voices' probabilities for one event (River's classifier protocol).

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return the combined probability as {False: P(no), True: P(yes)}.

Parameters:

  • x (Mapping[str, float]) –

    voice -> probability for the voices that spoke; a silent voice is absent.

Returns:

  • dict[bool, float] –

    The two class probabilities, or {} when no voice is usable or the combiner cannot

  • dict[bool, float] –

    combine yet. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Hear the outcome y of an event predict_proba_one was asked about. Never raises.

LogitPool dataclass

LogitPool(
    weights: Mapping[str, float] = dict(),
    alpha: float = 1.0,
)

A weighted mean in log-odds, extremized by alpha; a fixed baseline that never learns.

Attributes:

  • weights (Mapping[str, float]) –

    voice -> weight; a voice absent here weighs 1.0. Weights are renormalised over the voices that spoke, and a voice whose weight is not finite and > 0 is ignored.

  • alpha (float) –

    The extremizing factor; 1.0 (default) leaves the pool as is. A non-finite alpha makes predict_proba_one return {}.

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return the pooled {False: P(no), True: P(yes)}, or {} when no voice is usable. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Do nothing: the pool is fixed. Never raises.

PriorPool dataclass

PriorPool(
    inner: Combiner,
    prior_key: str = "prior",
    alpha: float = 1.0,
    shrink: float = 0.0,
)

Shrink a combiner toward a prior voice, then extremize: expit(alpha * z).

z = (1 - shrink) * logit(inner) + shrink * logit(prior), where inner is the wrapped combiner's probability and prior is x[prior_key]. The prior also reaches the inner combiner as an ordinary voice, so a learned combiner can weigh it. While the inner combiner abstains ({}, e.g. before its warmup), the pool answers with the prior alone, extremized. Without a prior in x it is the inner combiner, extremized.

Attributes:

  • inner (Combiner) –

    The wrapped combiner; it hears every outcome learn_one hears.

  • prior_key (str) –

    The voice in x that carries the prior.

  • alpha (float) –

    The extremizing factor; 1.0 leaves the pool as is.

  • shrink (float) –

    The weight on the prior's log-odds, in [0, 1]; 0 ignores it beyond the fallback. A non-finite alpha or a shrink outside [0, 1] makes predict_proba_one return {}.

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return {False: P(no), True: P(yes)}, or {} when neither the inner combiner nor the prior answers. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Pass the outcome to the inner combiner. Never raises.

BestVoice dataclass

BestVoice(
    min_n: int = 20,
    n: dict[str, int] = dict(),
    brier_sum: dict[str, float] = dict(),
)

The best single voice so far: answers with the speaking voice of lowest running Brier score.

A voice is eligible once it has been heard on min_n outcomes; among the eligible voices in x, the one with the lowest mean Brier score answers (ties: the voice name, so the choice is deterministic). With no eligible voice it abstains ({}).

Attributes:

  • min_n (int) –

    Outcomes a voice must have been scored on before it can answer.

  • n (dict[str, int]) –

    voice -> outcomes it was scored on.

  • brier_sum (dict[str, float]) –

    voice -> sum of its Brier scores.

best

best(x: Mapping[str, float]) -> str | None

Return the voice that would answer for x, or None. Never raises.

predict_proba_one

predict_proba_one(
    x: Mapping[str, float],
) -> dict[bool, float]

Return the best eligible voice's {False: P(no), True: P(yes)}, or {}. Never raises.

learn_one

learn_one(x: Mapping[str, float], y: bool) -> None

Score every usable voice in x against y. Never raises.

logit

logit(p: float, eps: float = EPS) -> float

Return ln(p / (1 - p)), with p clipped to [eps, 1 - eps].

expit

expit(x: float) -> float

Return 1 / (1 + exp(-x)), the inverse of logit, without overflow.

proba_one

proba_one(p: float | None) -> dict[bool, float]

Return {False: 1 - p, True: p}, River's predict_proba_one layout; {} for None.

llm_ports.forecast.walk

Replaying a combiner in time order, as it would have run: predict first, learn once resolved.

  • Event: one question the voices spoke on: x (voice -> probability), when the combined forecast is made (forecast_time), when the outcome became known (resolve_time) and the outcome (resolution: True / False, None while open).
  • walk_forward(combiner, events): replays the events in forecast_time order. Before each prediction the combiner hears every outcome whose resolve_time is at or before that forecast_time, so it never learns from an outcome before the outcome was known. It returns one probability (None where the combiner abstained) per event, in the input's order. An open event is predicted and never learned, so the same replay yields the shadow forecasts.
  • holdout_start(events, share): the forecast_time from which the last share of the resolved events falls, the split a tuner fits before and is judged after.
  • compare(combiners, events, since=): walks each combiner over the same events and scores each on the same subset: resolved, forecast at or after since, and answered by every combiner.
Invariants
  • Pure, standard library only.
  • Never raises for a combiner that keeps the Combiner contract.
  • Ties in time keep the input order; an outcome resolving exactly at a forecast time is heard before that forecast.

Rationale: ADR-0015.

Event dataclass

Event(
    x: Mapping[str, float],
    forecast_time: datetime,
    resolve_time: datetime | None = None,
    resolution: bool | None = None,
)

One question several voices forecast.

Attributes:

  • x (Mapping[str, float]) –

    voice -> probability of yes, for the voices that spoke.

  • forecast_time (datetime) –

    When the combined forecast is made (the newest voice's time, at earliest).

  • resolve_time (datetime | None) –

    When the outcome became known; None while open.

  • resolution (bool | None) –

    True if it happened, False if not, None while open.

resolved property

resolved: bool

True when both the outcome and its time are known.

walk_forward

walk_forward(
    combiner: Combiner, events: Sequence[Event]
) -> list[float | None]

Replay events through combiner in time order; return each event's probability of yes.

Parameters:

  • combiner (Combiner) –

    A fresh combiner; it is trained in place.

  • events (Sequence[Event]) –

    The events, in any order.

Returns:

  • list[float | None] –

    One probability per event, aligned with events; None where the combiner abstained.

  • list[float | None] –

    Never raises for a combiner that keeps its contract.

holdout_start

holdout_start(
    events: Sequence[Event], share: float = 0.2
) -> datetime | None

Return the forecast_time from which the last share of the resolved events falls.

Parameters:

  • events (Sequence[Event]) –

    The events.

  • share (float, default: 0.2 ) –

    The holdout's share of the resolved events, in (0, 1).

Returns:

  • datetime | None –

    The first holdout event's forecast_time, or None when there are fewer than two resolved

  • datetime | None –

    events or share is outside (0, 1). Never raises.

compare

compare(
    combiners: Mapping[str, Combiner],
    events: Sequence[Event],
    since: datetime | None = None,
) -> dict[str, Score]

Walk each combiner over events and score all of them on the same events.

Parameters:

  • combiners (Mapping[str, Combiner]) –

    name -> fresh combiner; each is trained in place.

  • events (Sequence[Event]) –

    The events, in any order.

  • since (datetime | None, default: None ) –

    Score only events forecast at or after this time (a holdout); None scores all.

Returns:

  • dict[str, Score] –

    name -> Score over the resolved events every combiner answered (each Score.n is the

  • dict[str, Score] –

    same). Never raises for combiners that keep their contract.