Skip to content

llm_ports.evals

llm_ports.evals

Evaluation: the board, the sampled judge, judge calibration, golden-set runs, labelled data and claim refutation.

GauntletEvaluator, judge, calibration, golden, labels, refute.

GauntletEvaluator

An Evaluator that runs deterministic graders over a set of rows into one board.

Every reader of a surface's quality (a CI golden gate, a scheduled scorer, an ad-hoc script) gets the same numbers. The board's shape:

{"surface": ..., "n": ..., "rows_failed": N,
 "by_grader": {grader: {"pass": n, "fail": n, "examples": [...]}},
 "monitors": {...}}

examples carries at most BOARD_EXAMPLES (id, reason) pairs per grader.

run

run(
    surface: str,
    rows: list[dict[str, Any]],
    graders: Mapping[str, Grader],
    *,
    id_field: str = "id",
    monitors: _Monitors | None = None
) -> dict[str, Any]

Grade every row with every grader and return the board.

id_field names the row key reported in examples; monitors, when given, computes the board's monitors entry over all rows. A grader that raises is a bug in the consumer's code: the exception propagates unchanged, and where it lands is the caller's concern.

llm_ports.evals.judge

A sampled model judge: what it reads, which reply it draws, how it grades, and the lane that runs it.

  • Seen / seen_for: what the judged call put in front of its model beside the prompt (system prompt, output grammar, images, a web tier). The judge reads the same.
  • judge_input: the judge's Input text, each part under a heading the rubric names.
  • reply_text: the reply as the judge reads it.
  • sample_key / selected: a deterministic stamp of one generation and a 1-in-N draw on it.
  • rubric: the four laws the deterministic graders (call.graders) encode, stated for a model: schema answered honestly, no number absent from the Input (numbers.law_text), no echo, grounded.
  • verdict_label: pass, fail or could_not_run.
  • JudgeLane: schedules the judging of a drawn reply as a background task, off the caller's await path; its start is the body of a call.Hooks.on_ok hook. Sample is what one task judges.

The consumer passes in, and this module never assumes: whether a role is eligible for judging, the role's grounding mode, the platform named in the rubric, today's date with its clock label, and any domain words the number law should name (numbers.LawExamples). Where a verdict is stored is the consumer's concern.

Invariants
  • Everything but JudgeLane is pure: no model, no database, no clock.
  • JudgeLane.start never raises and never awaits; a failure inside it is an event.
  • A draw is a function of (role, prompt, reply), so whether a reply was judged is answerable after the fact.

Seen dataclass

Seen(
    system: str = "",
    schema: Mapping[str, Any] | None = None,
    images: tuple[tuple[bytes, str], ...] = (),
    web: bool = False,
)

What the judged call put in front of its model beside the prompt.

The judge reads the same context, so a rule the model was given is not an invention and a figure on an attached image is not absent.

Attributes:

  • system (str) –

    The judged call's system prompt.

  • schema (Mapping[str, Any] | None) –

    The JSON schema the reply was constrained to, or None for prose.

  • images (tuple[tuple[bytes, str], ...]) –

    (bytes, media_type) pairs sent beside the prompt.

  • web (bool) –

    The call's tier fetched web pages the judge cannot see.

Sample dataclass

Sample(
    role: Role,
    prompt: str,
    reply: str,
    key: str,
    seen: Seen,
    result: Generated[Any],
)

One drawn reply, as a JudgeLane task receives it.

Attributes:

  • role (Role) –

    The judged role.

  • prompt (str) –

    The judged call's prompt.

  • reply (str) –

    The reply as the judge reads it (reply_text).

  • key (str) –

    The sample's stamp (sample_key).

  • seen (Seen) –

    What the judged call saw beside its prompt.

  • result (Generated[Any]) –

    The judged call's result; its stamp carries any trace handle to score onto.

JudgeLane

JudgeLane(
    *,
    name: str = "llm_judge",
    on_event: EventHook | None = None
)

Run a sampled judge as background tasks, off the caller's await path.

start decides and schedules; the consumer's run judges and stores. The lane holds every task it created by a strong reference until the task finishes (an event loop keeps tasks only weakly) and starts each in a fresh contextvars.Context, so a task never inherits the caller's context-scoped state after the caller has moved on.

Parameters:

  • name (str, default: 'llm_judge' ) –

    The prefix of the lane's event names (<name>_sampled, <name>_seen_failed, <name>_sampler_failed).

  • on_event (EventHook | None, default: None ) –

    Receives the lane's events; without one they go to the stdlib logger llm_ports.evals.judge.

start

start(
    role: Role,
    prompt: str,
    result: Generated[Any],
    *,
    mod: int | Callable[[], int],
    run: JudgeRun,
    eligible: Eligible | None = None,
    seen: Seen | Callable[[], Seen] | None = None
) -> str | None

Draw result and, when drawn, schedule run on it.

Nothing is scheduled without a running event loop, for a non-ok outcome, when eligible refuses, or when selected(key, mod) does not draw the reply. mod and seen may be callables, resolved only here: mod once per call, seen only for a drawn reply. A seen callable that raises is judged with an empty Seen and reported as <name>_seen_failed.

Parameters:

  • role (Role) –

    The judged role.

  • prompt (str) –

    The judged call's prompt.

  • result (Generated[Any]) –

    The judged call's result.

  • mod (int | Callable[[], int]) –

    The 1-in-N draw (selected); <= 0 is off.

  • run (JudgeRun) –

    The task body, called with the Sample.

  • eligible (Eligible | None, default: None ) –

    Whether the role may be drawn; None admits every role.

  • seen (Seen | Callable[[], Seen] | None, default: None ) –

    What the judged call saw beside its prompt; None reads as nothing.

Returns:

  • str | None –

    The sample key when a task was scheduled, else None. Never raises: any failure is

  • str | None –

    reported as <name>_sampler_failed and returns None.

pending

pending() -> int

Return the number of lane tasks not yet finished.

drain async

drain() -> None

Await every lane task that belongs to the running loop.

A task created on another loop cannot be awaited here; it is dropped from the lane. Never raises for a task's own exception.

schema_of

schema_of(output: Any) -> Mapping[str, Any] | None

Return the JSON schema an output= type sends as the grammar.

Returns:

  • Mapping[str, Any] | None –

    A json_output type's own schema, else a pydantic model's, else None (the reply is prose,

  • Mapping[str, Any] | None –

    and so is a type whose schema cannot be rendered).

seen_for

seen_for(
    *,
    system: str = "",
    output: Any = None,
    images: Sequence[tuple[bytes, str]] = (),
    web: bool = False
) -> Seen

Build the Seen for a call from its system prompt, output= type, images and web flag.

judge_input

judge_input(prompt: str, seen: Seen) -> str

Return the judge's Input text.

The judged call's system prompt (cut to SYSTEM_CAP), output grammar (cut to SCHEMA_CAP), a note for any images, and the prompt, each under a heading the rubric names. A call that saw nothing beside its prompt yields the bare prompt.

reply_text

reply_text(value: Any) -> str

Return the reply as the judge reads it: a pydantic model's JSON, a mapping's JSON, else str.

sample_key

sample_key(
    role: Role, prompt: str, output_text: str
) -> str

Return a deterministic 16-hex stamp of one generation: a sha over role name, prompt and reply.

selected

selected(key: str, mod: int) -> bool

Return whether key is drawn: 1-in-mod on its first 32 bits.

mod <= 0 selects nothing, which is the off switch.

rubric

rubric(
    role: Role,
    seen: Seen | None = None,
    *,
    platform: str,
    mode: str,
    today: date,
    tz_label: str,
    examples: LawExamples | None = None
) -> str

Return the judge's instruction for one reply from role.

The laws are the ones call.graders enforce, so the rule and the reader disagree only about judgement. A convention the role's own instructions mandate is the law kept, not broken.

Parameters:

  • role (Role) –

    The judged role; its name and whether it is structured shape the first line.

  • seen (Seen | None, default: None ) –

    What the judged call saw beside its prompt; None reads as nothing.

  • platform (str) –

    Whose reply it is, as the first line names it ("the X platform").

  • mode (str) –

    The role's grounding mode word. "slots" changes laws 1 and 2 (numbers.law_text).

  • today (date) –

    The judge's calendar date.

  • tz_label (str) –

    The label of the clock today was read on.

  • examples (LawExamples | None, default: None ) –

    Domain words the number law names as not figures; none by default.

Returns:

  • str –

    The rubric text.

verdict_label

verdict_label(v: JudgeVerdict | None) -> str

Return pass, fail, or could_not_run when the judge never answered (None).

A missing verdict is never guessed as a pass or a fail.

llm_ports.evals.refute

The adversarial pass's pure half: one grammar builder and one rule for attributing its verdicts.

The grammar is verdicts[<= max_items]{idx, claim <= claim_cap, verdict uphold|refute, reason <= reason_cap}.

llama.cpp compiles maxLength into GBNF and closes a string at the cap with valid JSON and finish_reason=stop, so a cut reason arrives looking whole. A reason at the cap is flagged reason_truncated on its refuted entry and never discarded: the verdict is an enum the grammar cannot cut, so the refutation stands and only its explanation is partial.

The flag has a one-sided blind spot. The grammar counts a \uXXXX escape as one character, so an escaped surrogate pair is two grammar characters but one Python character, and a cut reason holding k of them reads cap - k characters and escapes the flag. The flag can miss; it never fires early.

Attribution is by idx, with the claim text only as a legacy fallback: the grammar forces claim to be a string, not the claim's own string, and a capped echo cannot carry a longer claim at all.

The pass (refute) is one prompt, one grammar, any number of callers. Each claim is echoed with the 1-based idx its verdict must return (refute_slots); the consumer's refute pack renders it (refute_prompt; the pack, the role its manifest names, and its store are the consumer's); and the consumer's own generate runs it under its hooks (gate, trace, graders). Attribute the reply with match_verdicts(texts, value["verdicts"]).

verdicts_schema

verdicts_schema(
    *,
    max_items: int,
    claim_cap: int,
    reason_cap: int = REASON_MAX_CHARS
) -> dict[str, Any]

Return the hand-written JSON schema for one refute call, to pass through ports.json_output.

Pure; returns a fresh dict per call.

match_verdicts

match_verdicts(
    claims: list[str], raw_verdicts: Any
) -> list[dict[str, Any] | None]

Return one verdict per claim, or None where the claim went unexamined.

Pure. Verdicts attach by idx (1-based, matching the prompt), with the claim text only as a legacy fallback. An out-of-range or duplicate idx is ignored rather than misattributed, and a verdict taken by idx never doubles as a text match for another claim.

refutation

refutation(
    claim: str,
    reason: Any,
    cap: int = REASON_MAX_CHARS,
    **extra: Any
) -> dict[str, Any]

Return one adversary refutation as stored.

A reason at cap was closed by the grammar, not finished by the model, and is flagged reason_truncated. The key is present only when true; otherwise the entry's shape is unchanged.

refute_slots

refute_slots(
    *,
    claims: Sequence[RefuteClaim],
    evidence: str,
    noun: str,
    subject: str,
    evidence_label: str,
    reason_cap: int = REASON_MAX_CHARS
) -> dict[str, Any]

Return the refute pack's slots.

Pure. Each claim gets a 1-based idx its verdict must return: a string becomes {idx, claim}, and a mapping keeps its extra keys. The claims slot is that list as JSON; the evidence slot is the caller's JSON text, verbatim.

refute_prompt

refute_prompt(
    pack: Pack,
    *,
    claims: Sequence[RefuteClaim],
    evidence: str,
    noun: str,
    subject: str,
    evidence_label: str,
    reason_cap: int = REASON_MAX_CHARS
) -> str

Return the adversary's prompt: pack's primary template rendered with refute_slots. Pure.

refute async

refute(
    generate: GenerateFn,
    role: Role,
    *,
    pack: Pack,
    claims: Sequence[RefuteClaim],
    evidence: str,
    max_items: int,
    claim_cap: int,
    noun: str,
    subject: str,
    evidence_label: str,
    name: str,
    reason_cap: int = REASON_MAX_CHARS
) -> Generated[Any]

Run the adversarial pass through the consumer's generate.

The prompt is refute_prompt and the grammar verdicts_schema. name is the output-schema name (json_output) a test stub dispatches on. The outcome is the caller's to read; nothing here catches.

llm_ports.evals.golden

Golden-set evaluation of a prompt pack: fire each item through the call path, grade on one board.

A golden item is a mapping with an id and the pack's template context under slots (omitted for a pack with no slots). An item that also carries user fires as a system and user pair: the rendered pack is the system prompt and user is the prompt, the shape of a producer whose fixed instructions ride as the system so a prefix cache holds them. Every other key of the item (a stratum, the keys a reply may cite) rides onto the graded row, where a grader can read it. The flow:

fire:   rendered = pack.render_source(source, **item.get("slots", {}))
        result   = generate(role, prompt=rendered, output=output(item), generator=generator)
                   or, with `user`: generate(role, prompt=user, system=rendered, ...)
grade:  the role's call graders (`call.graders.board_graders`) plus any extra row graders,
        through one `Evaluator`, over `{**item, prompt, value, outcome, reason, retries}` rows

fire_pack(resolved, ...) is the whole run for a resolved pack (prompts.resolve, or resolve_pack here, which falls back to the built-in call graders): it reads the golden set the manifest names (a JSON list of items, a JSON object with an items list, or JSON lines), fires it through the pack's own source and output type, grades it on the pack's graders, and stamps the run and every row with the pack's identity. A run that cannot start (no golden set or no graders, a missing, unreadable or empty file, an item with no id or whose slots do not render) is could_not_run with the reason, never a pass over zero items.

source is the template text under evaluation, so a candidate revision is graded before it is installed. output(item) is the output= type the producer itself compiles: a grammar that differs from the live one grades a different model. generator_for() returns a fresh Generator per item and the list its raw attempts land in (adapters.pydantic_ai.replay.recording builds one), so a recording keeps every attempt and not only the last.

Invariants
  • fire and fire_pack never raise for a model or wire failure: the item's outcome names it.
  • A row whose model never answered fails every call grader with its reason; the board names the outage instead of grading an absence.

GoldenRun dataclass

GoldenRun(
    status: Literal["ran", "could_not_run"],
    reason: str,
    stamp: Mapping[str, str | None],
    rows: list[dict[str, Any]] = list(),
    outs: Mapping[str, Mapping[str, Any]] = dict(),
    board: Mapping[str, Any] | None = None,
)

One golden run of a resolved pack.

Attributes:

  • status (Literal['ran', 'could_not_run']) –

    ran, or could_not_run when nothing was fired.

  • reason (str) –

    Why the run could not run; empty when it ran.

  • stamp (Mapping[str, str | None]) –

    The pack's stamp (pack, version, sha, identity), the role's tier and model, table_fingerprint (None when the caller passed none) and output_schema_sha (None for a prose pack; identity covers the manifest's pointer, not the model it names). Two runs are comparable only when identity and output_schema_sha match.

  • rows (list[dict[str, Any]]) –

    The graded rows (rows_of), each carrying identity.

  • outs (Mapping[str, Mapping[str, Any]]) –

    fire's results by item id.

  • board (Mapping[str, Any] | None) –

    The evaluator's board; None when the run could not run.

fire_one async

fire_one(
    item: Mapping[str, Any],
    *,
    role: Role,
    pack: Pack,
    source: str,
    output: OutputFor,
    generator_for: GeneratorFor,
    gate: Semaphore,
    generate: Generate = generate,
    on_event: EventHook | None = None
) -> dict[str, Any]

Fire one golden item and return its result.

Returns:

  • dict[str, Any] –

    {id, verdict, outcome, reason, retries, prompt, system, value, attempts, scores}:

  • dict[str, Any] –

    verdict is ok, gated or error; prompt and system are what the model was sent

  • dict[str, Any] –

    (system empty unless the item carries user); scores holds each call grader's score

  • dict[str, Any] –

    as a dict.

fire async

fire(
    items: Sequence[Mapping[str, Any]],
    *,
    role: Role,
    pack: Pack,
    source: str,
    output: OutputFor,
    generator_for: GeneratorFor,
    generate: Generate = generate,
    parallel: int = PARALLEL,
    on_event: EventHook | None = None
) -> dict[str, dict[str, Any]]

Fire every item, at most parallel at a time; return each result (fire_one) by item id.

rows_of

rows_of(
    items: Sequence[Mapping[str, Any]],
    outs: Mapping[str, Mapping[str, Any]],
) -> list[dict[str, Any]]

Return the graded row per item: the item's own keys except slots, under the reply.

An item with no result is outcome="not_run" with no value.

monitors

monitors(
    rows: Sequence[Mapping[str, Any]],
) -> dict[str, Any]

Return the board's rates: rows answered, and their mean_retries (None when none answered).

grade

grade(
    items: Sequence[Mapping[str, Any]],
    outs: Mapping[str, Mapping[str, Any]],
    role: Role,
    evaluator: Evaluator,
    *,
    extra: Mapping[str, Grader] | None = None,
    table: Mapping[str, CallGrader] | None = None
) -> tuple[dict[str, Any], list[dict[str, Any]]]

Grade fired items on one board: the role's call graders plus extra row graders.

Parameters:

  • items (Sequence[Mapping[str, Any]]) –

    The golden items.

  • outs (Mapping[str, Mapping[str, Any]]) –

    fire's results by item id.

  • role (Role) –

    The judged role; its graders name the call graders, its pack (else its name) the board's surface.

  • evaluator (Evaluator) –

    The board's Evaluator.

  • extra (Mapping[str, Grader] | None, default: None ) –

    Row graders beyond the call graders, by name.

  • table (Mapping[str, CallGrader] | None, default: None ) –

    The call-grader table; None means call.graders.GRADERS.

Returns:

  • tuple[dict[str, Any], list[dict[str, Any]]] –

    The board and the graded rows.

citations_resolve

citations_resolve(cited: str, known: str) -> Grader

Return a row grader: every key the reply lists under cited is one the row lists under known.

It reads the raw reply, before any production step drops unknown citations, so a reply that cites what its evidence never carried fails here.

resolve_pack

resolve_pack(
    pack: Pack,
    *,
    registry: GraderRegistry | Registry | None = None,
    import_roots: Sequence[str] = ()
) -> ResolvedPack

Return prompts.resolve(pack, ...) with call.graders.GRADERS behind registry.

A grader name the caller's registry lacks resolves to the built-in grader of that name.

Raises:

  • PromptPackInvalid –

    A manifest pointer does not resolve; the message names the field.

read_golden

read_golden(path: Path) -> list[dict[str, Any]]

Read a golden set: a JSON list of items, a JSON object with an items list, or JSON lines (.jsonl).

Returns:

  • list[dict[str, Any]] –

    The items, in file order.

Raises:

  • ValueError –

    The file is not one of those shapes, or an item is not a mapping with an id.

  • OSError –

    The file cannot be read.

fire_pack async

fire_pack(
    resolved: ResolvedPack,
    *,
    role: Role,
    generator_for: GeneratorFor,
    generate: Generate = generate,
    parallel: int = PARALLEL,
    on_event: EventHook | None = None,
    evaluator: Evaluator | None = None,
    table_fingerprint: str | None = None
) -> GoldenRun

Fire a resolved pack's golden set through its own source and output type, and grade it.

Parameters:

  • resolved (ResolvedPack) –

    The pack with its manifest resolved (prompts.resolve or resolve_pack).

  • role (Role) –

    The role the items fire under; its tier and model are stamped on the run.

  • generator_for (GeneratorFor) –

    A fresh generator per item and the list its attempts land in.

  • generate (Generate, default: generate ) –

    The call path.

  • parallel (int, default: PARALLEL ) –

    Items fired concurrently.

  • on_event (EventHook | None, default: None ) –

    The event sink; defaults to the stdlib logger llm_ports.evals.golden. Emits one golden_item per item and one golden_run per run (a WARNING when it could not run).

  • evaluator (Evaluator | None, default: None ) –

    The board's evaluator; defaults to GauntletEvaluator().

  • table_fingerprint (str | None, default: None ) –

    The tier table's fingerprint, stamped on the run; None when unknown.

Returns:

  • GoldenRun –

    The run. No golden set or no graders declared, a missing, unreadable or empty golden file,

  • GoldenRun –

    or an item whose slots do not render the pack's template is could_not_run with the

  • GoldenRun –

    reason, never a pass over zero items or zero graders. The board grades the pack's own

  • GoldenRun –

    graders.

Invariants
  • Never raises for a model, wire or golden-file failure.

llm_ports.evals.labels

Labelled LLM replies on the learning-loop core: topics, the review-queue draw, and judge labels.

A label on a model reply is a learn.Example whose topic is llm:<role> (topic), whose subject is the reply's id in the label store, and whose source is human (ground truth), model (a stronger model labelling offline) or judge (the sampled production judge). Counting and agreement are the core's (learn.examples.coverage, learn.examples.agreement); this module holds only what is specific to model replies:

  • draw: which unlabelled production replies to put in front of a human next. Per role it takes only what closes the role's coverage gap, deterministically, and stratifies by the judge's verdict so both classes reach the human: an agreement statistic needs both, and a judge that fails most replies would otherwise fill the queue with its fails.
  • label_of: any Judge's verdict as an example. The offline model labeller is a Judge with a stronger model behind it, so no second judging path exists.
  • LabelQueue: the port a review surface implements to receive the drawn replies.
Invariants
  • Pure, except LabelQueue implementations: no model, no store, no clock.
  • A model's verdict is never a human example.

Rationale: ADR-0010.

Candidate dataclass

Candidate(
    role: str,
    subject: str,
    trace: str = "",
    judged: str | None = None,
)

An unlabelled production reply that could be put in front of a human.

Attributes:

  • role (str) –

    The role that produced it.

  • subject (str) –

    Its id in the label store.

  • trace (str) –

    The trace it belongs to ("" where the store does not scope ids by trace).

  • judged (str | None) –

    The judge's verdict on it (pass, fail), or None when it was not judged.

Queued dataclass

Queued(
    pending: frozenset[str] = frozenset(),
    done: frozenset[str] = frozenset(),
    outcome: StoreOutcome = "ok",
    reason: str | None = None,
)

What LabelQueue.queued returns: the subjects a topic's queue holds.

Attributes:

  • pending (frozenset[str]) –

    Subjects waiting for a human.

  • done (frozenset[str]) –

    Subjects a human has completed.

  • outcome (StoreOutcome) –

    ok, unavailable when the queue could not be read, or request_fault for a topic that is not a model reply's.

  • reason (str | None) –

    One line naming the failure; None on ok.

Invariants
  • An unreadable queue is unavailable, never an empty one.
  • A topic with no queue yet is ok and empty.

ok property

ok: bool

True when outcome == "ok".

seen property

seen: frozenset[str]

Every subject the queue holds, pending or done: what draw should skip.

LabelQueue

Bases: Protocol

A review surface where a human labels queued replies (an annotation queue).

enqueue async

enqueue(
    topic: str, subjects: Sequence[tuple[str, str]]
) -> Written

Put replies in front of a human labeller.

Parameters:

  • topic (str) –

    The replies' topic (topic(role)); the queue is per topic.

  • subjects (Sequence[tuple[str, str]]) –

    (subject, trace) pairs.

Returns:

  • Written –

    How many were newly queued; a subject the queue already holds is skipped, not added

  • Written –

    twice and not an error. Never raises.

queued async

queued(topic: str) -> Queued

Read the subjects a topic's queue holds, so a caller can skip them and bound the backlog.

Returns:

  • Queued –

    The pending and done subjects, or an unavailable outcome. Never raises.

topic

topic(role: str) -> str

Return the topic of role's replies (llm:<role>).

role_of

role_of(name: str) -> str | None

Return the role a topic names, or None for a topic that is not a model reply's.

draw

draw(
    candidates: Iterable[Candidate],
    coverage_now: Mapping[str, Coverage],
    *,
    per_run: int,
    seen: Iterable[str] = ()
) -> list[Candidate]

Return the replies to queue for a human next.

Per role, at most min(per_run, gap) replies, none in seen (already labelled or queued). A role whose topic is absent from coverage_now is not drawn. Within a role the draw is deterministic (a hash of role and subject) and round-robin across the judge's verdicts, rarest first, so a queue holds both classes whenever the candidates do.

Parameters:

  • candidates (Iterable[Candidate]) –

    Unlabelled production replies.

  • coverage_now (Mapping[str, Coverage]) –

    learn.examples.coverage keyed by topic, which carries each role's gap.

  • per_run (int) –

    The most replies one run queues per role.

  • seen (Iterable[str], default: () ) –

    Subjects to skip.

Returns:

  • list[Candidate] –

    The replies to queue, grouped by role in the order of coverage_now. Never raises.

label_of

label_of(
    verdict: JudgeVerdict,
    *,
    role: str,
    subject: str,
    trace: str = "",
    by: str = "",
    source: Literal["model", "judge"] = "model",
    at: datetime | None = None
) -> Example

Return a Judge's verdict on one reply as an example.

Parameters:

  • verdict (JudgeVerdict) –

    Any Judge's verdict.

  • role (str) –

    The labelled reply's role.

  • subject (str) –

    The reply's id in the label store.

  • trace (str, default: '' ) –

    The reply's trace id, kept in ref["trace"] when given.

  • by (str, default: '' ) –

    The judging model's name; the verdict's judge when "".

  • source (Literal['model', 'judge'], default: 'model' ) –

    model for an offline labeller, judge for the production judge. Never human.

  • at (datetime | None, default: None ) –

    When it was given.

Returns:

  • Example –

    The example, good when the verdict passed, bad otherwise.

Raises:

  • ValueError –

    source is not model or judge: a model's verdict is never ground truth.

llm_ports.evals.calibration

Calibrate a judge in both directions: agreement with human labels, and replies it must fail.

A judge earns trust two ways, and each measures something the other cannot:

  • AGREEMENT with a human labeller on the same items. cohen_kappa is the chance-corrected rate to read; gwet_ac1 is its prevalence-robust companion, for an item whose labels are nearly all one answer (where kappa's chance correction degenerates). Raw agreement flatters a lopsided item.
  • SEEDED-BAD replies: corrupted copies of real replies that any honest judge must fail. seed_placeholder blanks every string (the schema-answered-honestly law), seed_figure appends a figure no input carried (the number law). A judge that passes these passes everything.

trust joins the two into one verdict a judge's verdicts carry: trusted only when it agrees with the human on enough items (kappa, or its interval's lower end, at a floor) AND its seeded-bad replay is calibrated. A judge with no record is untrusted, so swapping the model never inherits the old one's trust.

tally and verdict read the rows of a replay that judged good samples beside their seeded-bad copies. A row is a mapping with:

  • role: the judged role's name.
  • kind: "good" for a real reply, else the seed's name ("tbd", "figure", ...).
  • after: the replayed verdict (pass, fail or could_not_run, as judge.verdict_label).
  • before: the verdict previously stored for a good sample; None on a seeded row.
  • held_out: whether the good sample was outside the rows read while building the judge.
Invariants
  • Pure: no model, no database, no clock.
  • A statistic that the data cannot define is None, never a number.
  • A could_not_run row is never a catch: a judge that did not answer measured nothing.

Trust dataclass

Trust(
    trusted: bool,
    why: str,
    agreement: Agreement,
    false_fail: int = 0,
    human_good: int = 0,
    seeded: int | None = None,
)

Whether a judge's verdicts may be acted on, with the numbers that decided it.

Attributes:

  • trusted (bool) –

    True only when every condition of trust held.

  • why (str) –

    One line naming the deciding condition and its numbers.

  • agreement (Agreement) –

    The judge against the human on the shared items (n, kappa and its interval, AC1).

  • false_fail (int) –

    Items the human labelled good that the judge failed.

  • human_good (int) –

    Items the human labelled good.

  • seeded (int | None) –

    The seeded-bad replay's verdict code; None when no replay was given.

cohen_kappa

cohen_kappa(pairs: Pairs) -> float | None

Return Cohen's kappa over binary (a, b) label pairs.

Both labels are "yes" or anything else (read as no); abstentions belong outside pairs.

Returns:

  • float | None –

    Kappa, or None when it is undefined: under two pairs, or both labellers one-sided on the

  • float | None –

    same answer (chance agreement 1.0).

gwet_ac1

gwet_ac1(pairs: Pairs) -> float | None

Return Gwet's AC1 over binary (a, b) label pairs.

Chance agreement is 2 * pi * (1 - pi), with pi the pooled share of "yes", so the statistic stays defined and bounded where one answer dominates.

Returns:

  • float | None –

    AC1, or None under two pairs.

seed_placeholder

seed_placeholder(
    reply: str, placeholder: str = PLACEHOLDER
) -> str | None

Return a copy of the reply with every string replaced by placeholder.

A JSON object or list keeps its shape; any other reply becomes placeholder itself.

Returns:

  • str | None –

    The seeded reply, or None for a JSON reply with no string to replace.

seed_figure

seed_figure(reply: str, figure: str = FIGURE) -> str | None

Return a copy of the reply with figure appended to its longest free-text string.

A free-text string holds a space; a non-JSON reply has figure appended to the whole text.

Returns:

  • str | None –

    The seeded reply, or None for a JSON reply with no free text to carry the figure.

tally

tally(
    results: Sequence[Mapping[str, Any]],
) -> dict[str, dict[str, Any]]

Count a replay's rows per role, sorted by role.

Per role: n good samples, before_pass and after_pass among them, after_could_not_run, held_out, and per seed kind <kind>_n and <kind>_fail (a could_not_run is not a fail).

verdict

verdict(
    results: Sequence[Mapping[str, Any]],
    *,
    floor: float = 0.0
) -> tuple[int, str]

Return the replay's exit code and the reason.

  • BLIND (2), did not finish looking: nothing was replayed, or some role has no answered good sample or no answered seeded-bad one. Such a replay proves nothing in that direction.
  • FINDING (1): a seeded-bad reply passed, or a role's pass rate over its answered good samples is under floor.
  • CALIBRATED (0): measured and clean in both directions.

trust

trust(
    pairs: Pairs,
    *,
    replay: Sequence[Mapping[str, Any]] | None = None,
    min_labelled: int = MIN_LABELLED,
    min_kappa: float = MIN_KAPPA,
    bound: Literal["point", "low"] = "point",
    require_seeded: bool = True
) -> Trust

Return whether a judge is trusted, from its agreement with a human and its seeded-bad replay.

Checked in order, the first failure deciding: at least min_labelled pairs; the seeded-bad replay CALIBRATED (or absent with require_seeded=False); kappa at least min_kappa, read as the point estimate (bound="point") or the 95% interval's lower end (bound="low").

Parameters:

  • pairs (Pairs) –

    (judge, human) labels of the same real replies, "yes" for pass / good (learn.examples.pairs builds them from a label store; abstentions left out).

  • replay (Sequence[Mapping[str, Any]] | None, default: None ) –

    Rows of a seeded-bad replay, as verdict reads them; None when there is none.

  • min_labelled (int, default: MIN_LABELLED ) –

    The fewest pairs that can earn trust.

  • min_kappa (float, default: MIN_KAPPA ) –

    The agreement floor.

  • bound (Literal['point', 'low'], default: 'point' ) –

    Which kappa the floor reads.

  • require_seeded (bool, default: True ) –

    Whether a missing replay leaves the judge untrusted.

Returns:

  • Trust –

    The Trust. Never raises.

Invariants
  • No pairs is untrusted, never an error: a new judge starts untrusted.
  • An undefined kappa (one-sided labels) never clears the floor.
  • A replay row verdict cannot read (a missing key) is a BLIND replay: untrusted.