llm_ports.evals¶
llm_ports.evals
¶
Evaluation: the board, the sampled judge, judge calibration, golden-set runs, labelled data and claim refutation.
GauntletEvaluator, judge, calibration, golden, labels, refute.
GauntletEvaluator
¶
An Evaluator that runs deterministic graders over a set of rows into one board.
Every reader of a surface's quality (a CI golden gate, a scheduled scorer, an ad-hoc script) gets the same numbers. The board's shape:
{"surface": ..., "n": ..., "rows_failed": N,
"by_grader": {grader: {"pass": n, "fail": n, "examples": [...]}},
"monitors": {...}}
examples carries at most BOARD_EXAMPLES (id, reason) pairs per grader.
run
¶
run(
surface: str,
rows: list[dict[str, Any]],
graders: Mapping[str, Grader],
*,
id_field: str = "id",
monitors: _Monitors | None = None
) -> dict[str, Any]
Grade every row with every grader and return the board.
id_field names the row key reported in examples; monitors, when given, computes the
board's monitors entry over all rows. A grader that raises is a bug in the consumer's
code: the exception propagates unchanged, and where it lands is the caller's concern.
llm_ports.evals.judge¶
A sampled model judge: what it reads, which reply it draws, how it grades, and the lane that runs it.
Seen/seen_for: what the judged call put in front of its model beside the prompt (system prompt, output grammar, images, a web tier). The judge reads the same.judge_input: the judge's Input text, each part under a heading the rubric names.reply_text: the reply as the judge reads it.sample_key/selected: a deterministic stamp of one generation and a 1-in-N draw on it.rubric: the four laws the deterministic graders (call.graders) encode, stated for a model: schema answered honestly, no number absent from the Input (numbers.law_text), no echo, grounded.verdict_label:pass,failorcould_not_run.JudgeLane: schedules the judging of a drawn reply as a background task, off the caller's await path; itsstartis the body of acall.Hooks.on_okhook.Sampleis what one task judges.
The consumer passes in, and this module never assumes: whether a role is eligible for judging,
the role's grounding mode, the platform named in the rubric, today's date with its clock label,
and any domain words the number law should name (numbers.LawExamples). Where a verdict is stored
is the consumer's concern.
Invariants
- Everything but
JudgeLaneis pure: no model, no database, no clock. JudgeLane.startnever raises and never awaits; a failure inside it is an event.- A draw is a function of (role, prompt, reply), so whether a reply was judged is answerable after the fact.
Seen
dataclass
¶
Seen(
system: str = "",
schema: Mapping[str, Any] | None = None,
images: tuple[tuple[bytes, str], ...] = (),
web: bool = False,
)
What the judged call put in front of its model beside the prompt.
The judge reads the same context, so a rule the model was given is not an invention and a figure on an attached image is not absent.
Attributes:
-
system(str) –The judged call's system prompt.
-
schema(Mapping[str, Any] | None) –The JSON schema the reply was constrained to, or None for prose.
-
images(tuple[tuple[bytes, str], ...]) –(bytes, media_type)pairs sent beside the prompt. -
web(bool) –The call's tier fetched web pages the judge cannot see.
Sample
dataclass
¶
One drawn reply, as a JudgeLane task receives it.
Attributes:
-
role(Role) –The judged role.
-
prompt(str) –The judged call's prompt.
-
reply(str) –The reply as the judge reads it (
reply_text). -
key(str) –The sample's stamp (
sample_key). -
seen(Seen) –What the judged call saw beside its prompt.
-
result(Generated[Any]) –The judged call's result; its
stampcarries any trace handle to score onto.
JudgeLane
¶
JudgeLane(
*,
name: str = "llm_judge",
on_event: EventHook | None = None
)
Run a sampled judge as background tasks, off the caller's await path.
start decides and schedules; the consumer's run judges and stores. The lane holds every task
it created by a strong reference until the task finishes (an event loop keeps tasks only weakly)
and starts each in a fresh contextvars.Context, so a task never inherits the caller's
context-scoped state after the caller has moved on.
Parameters:
-
name(str, default:'llm_judge') –The prefix of the lane's event names (
<name>_sampled,<name>_seen_failed,<name>_sampler_failed). -
on_event(EventHook | None, default:None) –Receives the lane's events; without one they go to the stdlib logger
llm_ports.evals.judge.
start
¶
start(
role: Role,
prompt: str,
result: Generated[Any],
*,
mod: int | Callable[[], int],
run: JudgeRun,
eligible: Eligible | None = None,
seen: Seen | Callable[[], Seen] | None = None
) -> str | None
Draw result and, when drawn, schedule run on it.
Nothing is scheduled without a running event loop, for a non-ok outcome, when eligible
refuses, or when selected(key, mod) does not draw the reply. mod and seen may be
callables, resolved only here: mod once per call, seen only for a drawn reply. A seen
callable that raises is judged with an empty Seen and reported as <name>_seen_failed.
Parameters:
-
role(Role) –The judged role.
-
prompt(str) –The judged call's prompt.
-
result(Generated[Any]) –The judged call's result.
-
mod(int | Callable[[], int]) –The 1-in-N draw (
selected);<= 0is off. -
run(JudgeRun) –The task body, called with the
Sample. -
eligible(Eligible | None, default:None) –Whether the role may be drawn; None admits every role.
-
seen(Seen | Callable[[], Seen] | None, default:None) –What the judged call saw beside its prompt; None reads as nothing.
Returns:
-
str | None–The sample key when a task was scheduled, else None. Never raises: any failure is
-
str | None–reported as
<name>_sampler_failedand returns None.
drain
async
¶
drain() -> None
Await every lane task that belongs to the running loop.
A task created on another loop cannot be awaited here; it is dropped from the lane. Never raises for a task's own exception.
schema_of
¶
schema_of(output: Any) -> Mapping[str, Any] | None
Return the JSON schema an output= type sends as the grammar.
Returns:
-
Mapping[str, Any] | None–A
json_outputtype's own schema, else a pydantic model's, else None (the reply is prose, -
Mapping[str, Any] | None–and so is a type whose schema cannot be rendered).
seen_for
¶
seen_for(
*,
system: str = "",
output: Any = None,
images: Sequence[tuple[bytes, str]] = (),
web: bool = False
) -> Seen
Build the Seen for a call from its system prompt, output= type, images and web flag.
judge_input
¶
judge_input(prompt: str, seen: Seen) -> str
Return the judge's Input text.
The judged call's system prompt (cut to SYSTEM_CAP), output grammar (cut to SCHEMA_CAP), a
note for any images, and the prompt, each under a heading the rubric names. A call that saw
nothing beside its prompt yields the bare prompt.
reply_text
¶
reply_text(value: Any) -> str
Return the reply as the judge reads it: a pydantic model's JSON, a mapping's JSON, else str.
sample_key
¶
sample_key(
role: Role, prompt: str, output_text: str
) -> str
Return a deterministic 16-hex stamp of one generation: a sha over role name, prompt and reply.
selected
¶
selected(key: str, mod: int) -> bool
Return whether key is drawn: 1-in-mod on its first 32 bits.
mod <= 0 selects nothing, which is the off switch.
rubric
¶
rubric(
role: Role,
seen: Seen | None = None,
*,
platform: str,
mode: str,
today: date,
tz_label: str,
examples: LawExamples | None = None
) -> str
Return the judge's instruction for one reply from role.
The laws are the ones call.graders enforce, so the rule and the reader disagree only about
judgement. A convention the role's own instructions mandate is the law kept, not broken.
Parameters:
-
role(Role) –The judged role; its name and whether it is structured shape the first line.
-
seen(Seen | None, default:None) –What the judged call saw beside its prompt; None reads as nothing.
-
platform(str) –Whose reply it is, as the first line names it ("the X platform").
-
mode(str) –The role's grounding mode word.
"slots"changes laws 1 and 2 (numbers.law_text). -
today(date) –The judge's calendar date.
-
tz_label(str) –The label of the clock
todaywas read on. -
examples(LawExamples | None, default:None) –Domain words the number law names as not figures; none by default.
Returns:
-
str–The rubric text.
verdict_label
¶
verdict_label(v: JudgeVerdict | None) -> str
Return pass, fail, or could_not_run when the judge never answered (None).
A missing verdict is never guessed as a pass or a fail.
llm_ports.evals.refute¶
The adversarial pass's pure half: one grammar builder and one rule for attributing its verdicts.
The grammar is verdicts[<= max_items]{idx, claim <= claim_cap, verdict uphold|refute, reason <=
reason_cap}.
llama.cpp compiles maxLength into GBNF and closes a string at the cap with valid JSON and
finish_reason=stop, so a cut reason arrives looking whole. A reason at the cap is flagged
reason_truncated on its refuted entry and never discarded: the verdict is an enum the grammar
cannot cut, so the refutation stands and only its explanation is partial.
The flag has a one-sided blind spot. The grammar counts a \uXXXX escape as one character, so an
escaped surrogate pair is two grammar characters but one Python character, and a cut reason
holding k of them reads cap - k characters and escapes the flag. The flag can miss; it never
fires early.
Attribution is by idx, with the claim text only as a legacy fallback: the grammar forces claim
to be a string, not the claim's own string, and a capped echo cannot carry a longer claim at all.
The pass (refute) is one prompt, one grammar, any number of callers. Each claim is echoed with
the 1-based idx its verdict must return (refute_slots); the consumer's refute pack renders it
(refute_prompt; the pack, the role its manifest names, and its store are the consumer's); and
the consumer's own generate runs it under its hooks (gate, trace, graders). Attribute the reply
with match_verdicts(texts, value["verdicts"]).
verdicts_schema
¶
verdicts_schema(
*,
max_items: int,
claim_cap: int,
reason_cap: int = REASON_MAX_CHARS
) -> dict[str, Any]
Return the hand-written JSON schema for one refute call, to pass through ports.json_output.
Pure; returns a fresh dict per call.
match_verdicts
¶
match_verdicts(
claims: list[str], raw_verdicts: Any
) -> list[dict[str, Any] | None]
Return one verdict per claim, or None where the claim went unexamined.
Pure. Verdicts attach by idx (1-based, matching the prompt), with the claim text only as a
legacy fallback. An out-of-range or duplicate idx is ignored rather than misattributed, and a
verdict taken by idx never doubles as a text match for another claim.
refutation
¶
refutation(
claim: str,
reason: Any,
cap: int = REASON_MAX_CHARS,
**extra: Any
) -> dict[str, Any]
Return one adversary refutation as stored.
A reason at cap was closed by the grammar, not finished by the model, and is flagged
reason_truncated. The key is present only when true; otherwise the entry's shape is unchanged.
refute_slots
¶
refute_slots(
*,
claims: Sequence[RefuteClaim],
evidence: str,
noun: str,
subject: str,
evidence_label: str,
reason_cap: int = REASON_MAX_CHARS
) -> dict[str, Any]
Return the refute pack's slots.
Pure. Each claim gets a 1-based idx its verdict must return: a string becomes {idx, claim},
and a mapping keeps its extra keys. The claims slot is that list as JSON; the evidence slot
is the caller's JSON text, verbatim.
refute_prompt
¶
refute_prompt(
pack: Pack,
*,
claims: Sequence[RefuteClaim],
evidence: str,
noun: str,
subject: str,
evidence_label: str,
reason_cap: int = REASON_MAX_CHARS
) -> str
Return the adversary's prompt: pack's primary template rendered with refute_slots. Pure.
refute
async
¶
refute(
generate: GenerateFn,
role: Role,
*,
pack: Pack,
claims: Sequence[RefuteClaim],
evidence: str,
max_items: int,
claim_cap: int,
noun: str,
subject: str,
evidence_label: str,
name: str,
reason_cap: int = REASON_MAX_CHARS
) -> Generated[Any]
Run the adversarial pass through the consumer's generate.
The prompt is refute_prompt and the grammar verdicts_schema. name is the output-schema
name (json_output) a test stub dispatches on. The outcome is the caller's to read; nothing
here catches.
llm_ports.evals.golden¶
Golden-set evaluation of a prompt pack: fire each item through the call path, grade on one board.
A golden item is a mapping with an id and the pack's template context under slots (omitted
for a pack with no slots). An item that also carries user fires as a system and user pair: the
rendered pack is the system prompt and user is the prompt, the shape of a producer whose fixed
instructions ride as the system so a prefix cache holds them. Every other key of the item (a
stratum, the keys a reply may cite) rides onto the graded row, where a grader can read it. The
flow:
fire: rendered = pack.render_source(source, **item.get("slots", {}))
result = generate(role, prompt=rendered, output=output(item), generator=generator)
or, with `user`: generate(role, prompt=user, system=rendered, ...)
grade: the role's call graders (`call.graders.board_graders`) plus any extra row graders,
through one `Evaluator`, over `{**item, prompt, value, outcome, reason, retries}` rows
fire_pack(resolved, ...) is the whole run for a resolved pack (prompts.resolve, or
resolve_pack here, which falls back to the built-in call graders): it reads the golden set the
manifest names (a JSON list of items, a JSON object with an items list, or JSON lines), fires it
through the pack's own source and output type, grades it on the pack's graders, and stamps the run
and every row with the pack's identity. A run that cannot start (no golden set or no graders, a
missing, unreadable or empty file, an item with no id or whose slots do not render) is
could_not_run with the reason, never a pass over zero items.
source is the template text under evaluation, so a candidate revision is graded before it is
installed. output(item) is the output= type the producer itself compiles: a grammar that
differs from the live one grades a different model. generator_for() returns a fresh Generator
per item and the list its raw attempts land in (adapters.pydantic_ai.replay.recording builds
one), so a recording keeps every attempt and not only the last.
Invariants
fireandfire_packnever raise for a model or wire failure: the item's outcome names it.- A row whose model never answered fails every call grader with its reason; the board names the outage instead of grading an absence.
GoldenRun
dataclass
¶
GoldenRun(
status: Literal["ran", "could_not_run"],
reason: str,
stamp: Mapping[str, str | None],
rows: list[dict[str, Any]] = list(),
outs: Mapping[str, Mapping[str, Any]] = dict(),
board: Mapping[str, Any] | None = None,
)
One golden run of a resolved pack.
Attributes:
-
status(Literal['ran', 'could_not_run']) –ran, orcould_not_runwhen nothing was fired. -
reason(str) –Why the run could not run; empty when it ran.
-
stamp(Mapping[str, str | None]) –The pack's stamp (
pack,version,sha,identity), the role'stierandmodel,table_fingerprint(None when the caller passed none) andoutput_schema_sha(None for a prose pack;identitycovers the manifest's pointer, not the model it names). Two runs are comparable only whenidentityandoutput_schema_shamatch. -
rows(list[dict[str, Any]]) –The graded rows (
rows_of), each carryingidentity. -
outs(Mapping[str, Mapping[str, Any]]) –fire's results by item id. -
board(Mapping[str, Any] | None) –The evaluator's board; None when the run could not run.
fire_one
async
¶
fire_one(
item: Mapping[str, Any],
*,
role: Role,
pack: Pack,
source: str,
output: OutputFor,
generator_for: GeneratorFor,
gate: Semaphore,
generate: Generate = generate,
on_event: EventHook | None = None
) -> dict[str, Any]
Fire one golden item and return its result.
Returns:
-
dict[str, Any]–{id, verdict, outcome, reason, retries, prompt, system, value, attempts, scores}: -
dict[str, Any]–verdictisok,gatedorerror;promptandsystemare what the model was sent -
dict[str, Any]–(
systemempty unless the item carriesuser);scoresholds each call grader's score -
dict[str, Any]–as a dict.
fire
async
¶
fire(
items: Sequence[Mapping[str, Any]],
*,
role: Role,
pack: Pack,
source: str,
output: OutputFor,
generator_for: GeneratorFor,
generate: Generate = generate,
parallel: int = PARALLEL,
on_event: EventHook | None = None
) -> dict[str, dict[str, Any]]
Fire every item, at most parallel at a time; return each result (fire_one) by item id.
rows_of
¶
rows_of(
items: Sequence[Mapping[str, Any]],
outs: Mapping[str, Mapping[str, Any]],
) -> list[dict[str, Any]]
Return the graded row per item: the item's own keys except slots, under the reply.
An item with no result is outcome="not_run" with no value.
monitors
¶
monitors(
rows: Sequence[Mapping[str, Any]],
) -> dict[str, Any]
Return the board's rates: rows answered, and their mean_retries (None when none answered).
grade
¶
grade(
items: Sequence[Mapping[str, Any]],
outs: Mapping[str, Mapping[str, Any]],
role: Role,
evaluator: Evaluator,
*,
extra: Mapping[str, Grader] | None = None,
table: Mapping[str, CallGrader] | None = None
) -> tuple[dict[str, Any], list[dict[str, Any]]]
Grade fired items on one board: the role's call graders plus extra row graders.
Parameters:
-
items(Sequence[Mapping[str, Any]]) –The golden items.
-
outs(Mapping[str, Mapping[str, Any]]) –fire's results by item id. -
role(Role) –The judged role; its graders name the call graders, its pack (else its name) the board's surface.
-
evaluator(Evaluator) –The board's
Evaluator. -
extra(Mapping[str, Grader] | None, default:None) –Row graders beyond the call graders, by name.
-
table(Mapping[str, CallGrader] | None, default:None) –The call-grader table; None means
call.graders.GRADERS.
Returns:
-
tuple[dict[str, Any], list[dict[str, Any]]]–The board and the graded rows.
citations_resolve
¶
citations_resolve(cited: str, known: str) -> Grader
Return a row grader: every key the reply lists under cited is one the row lists under known.
It reads the raw reply, before any production step drops unknown citations, so a reply that cites what its evidence never carried fails here.
resolve_pack
¶
resolve_pack(
pack: Pack,
*,
registry: GraderRegistry | Registry | None = None,
import_roots: Sequence[str] = ()
) -> ResolvedPack
Return prompts.resolve(pack, ...) with call.graders.GRADERS behind registry.
A grader name the caller's registry lacks resolves to the built-in grader of that name.
Raises:
-
PromptPackInvalid–A manifest pointer does not resolve; the message names the field.
read_golden
¶
read_golden(path: Path) -> list[dict[str, Any]]
Read a golden set: a JSON list of items, a JSON object with an items list, or JSON lines (.jsonl).
Returns:
-
list[dict[str, Any]]–The items, in file order.
Raises:
-
ValueError–The file is not one of those shapes, or an item is not a mapping with an
id. -
OSError–The file cannot be read.
fire_pack
async
¶
fire_pack(
resolved: ResolvedPack,
*,
role: Role,
generator_for: GeneratorFor,
generate: Generate = generate,
parallel: int = PARALLEL,
on_event: EventHook | None = None,
evaluator: Evaluator | None = None,
table_fingerprint: str | None = None
) -> GoldenRun
Fire a resolved pack's golden set through its own source and output type, and grade it.
Parameters:
-
resolved(ResolvedPack) –The pack with its manifest resolved (
prompts.resolveorresolve_pack). -
role(Role) –The role the items fire under; its tier and model are stamped on the run.
-
generator_for(GeneratorFor) –A fresh generator per item and the list its attempts land in.
-
generate(Generate, default:generate) –The call path.
-
parallel(int, default:PARALLEL) –Items fired concurrently.
-
on_event(EventHook | None, default:None) –The event sink; defaults to the stdlib logger
llm_ports.evals.golden. Emits onegolden_itemper item and onegolden_runper run (a WARNING when it could not run). -
evaluator(Evaluator | None, default:None) –The board's evaluator; defaults to
GauntletEvaluator(). -
table_fingerprint(str | None, default:None) –The tier table's fingerprint, stamped on the run; None when unknown.
Returns:
-
GoldenRun–The run. No golden set or no graders declared, a missing, unreadable or empty golden file,
-
GoldenRun–or an item whose slots do not render the pack's template is
could_not_runwith the -
GoldenRun–reason, never a pass over zero items or zero graders. The board grades the pack's own
-
GoldenRun–graders.
Invariants
- Never raises for a model, wire or golden-file failure.
llm_ports.evals.labels¶
Labelled LLM replies on the learning-loop core: topics, the review-queue draw, and judge labels.
A label on a model reply is a learn.Example whose topic is llm:<role> (topic), whose
subject is the reply's id in the label store, and whose source is human (ground truth),
model (a stronger model labelling offline) or judge (the sampled production judge). Counting
and agreement are the core's (learn.examples.coverage, learn.examples.agreement); this module
holds only what is specific to model replies:
draw: which unlabelled production replies to put in front of a human next. Per role it takes only what closes the role's coverage gap, deterministically, and stratifies by the judge's verdict so both classes reach the human: an agreement statistic needs both, and a judge that fails most replies would otherwise fill the queue with its fails.label_of: anyJudge's verdict as an example. The offline model labeller is aJudgewith a stronger model behind it, so no second judging path exists.LabelQueue: the port a review surface implements to receive the drawn replies.
Invariants
- Pure, except
LabelQueueimplementations: no model, no store, no clock. - A model's verdict is never a
humanexample.
Rationale: ADR-0010.
Candidate
dataclass
¶
Candidate(
role: str,
subject: str,
trace: str = "",
judged: str | None = None,
)
An unlabelled production reply that could be put in front of a human.
Attributes:
-
role(str) –The role that produced it.
-
subject(str) –Its id in the label store.
-
trace(str) –The trace it belongs to ("" where the store does not scope ids by trace).
-
judged(str | None) –The judge's verdict on it (
pass,fail), or None when it was not judged.
Queued
dataclass
¶
Queued(
pending: frozenset[str] = frozenset(),
done: frozenset[str] = frozenset(),
outcome: StoreOutcome = "ok",
reason: str | None = None,
)
What LabelQueue.queued returns: the subjects a topic's queue holds.
Attributes:
-
pending(frozenset[str]) –Subjects waiting for a human.
-
done(frozenset[str]) –Subjects a human has completed.
-
outcome(StoreOutcome) –ok,unavailablewhen the queue could not be read, orrequest_faultfor a topic that is not a model reply's. -
reason(str | None) –One line naming the failure; None on
ok.
Invariants
- An unreadable queue is
unavailable, never an empty one. - A topic with no queue yet is
okand empty.
LabelQueue
¶
Bases: Protocol
A review surface where a human labels queued replies (an annotation queue).
role_of
¶
role_of(name: str) -> str | None
Return the role a topic names, or None for a topic that is not a model reply's.
draw
¶
draw(
candidates: Iterable[Candidate],
coverage_now: Mapping[str, Coverage],
*,
per_run: int,
seen: Iterable[str] = ()
) -> list[Candidate]
Return the replies to queue for a human next.
Per role, at most min(per_run, gap) replies, none in seen (already labelled or queued).
A role whose topic is absent from coverage_now is not drawn. Within a role the draw is
deterministic (a hash of role and subject) and round-robin across the judge's verdicts,
rarest first, so a queue holds both classes whenever the candidates do.
Parameters:
-
candidates(Iterable[Candidate]) –Unlabelled production replies.
-
coverage_now(Mapping[str, Coverage]) –learn.examples.coveragekeyed by topic, which carries each role's gap. -
per_run(int) –The most replies one run queues per role.
-
seen(Iterable[str], default:()) –Subjects to skip.
Returns:
-
list[Candidate]–The replies to queue, grouped by role in the order of
coverage_now. Never raises.
label_of
¶
label_of(
verdict: JudgeVerdict,
*,
role: str,
subject: str,
trace: str = "",
by: str = "",
source: Literal["model", "judge"] = "model",
at: datetime | None = None
) -> Example
Return a Judge's verdict on one reply as an example.
Parameters:
-
verdict(JudgeVerdict) –Any
Judge's verdict. -
role(str) –The labelled reply's role.
-
subject(str) –The reply's id in the label store.
-
trace(str, default:'') –The reply's trace id, kept in
ref["trace"]when given. -
by(str, default:'') –The judging model's name; the verdict's judge when "".
-
source(Literal['model', 'judge'], default:'model') –modelfor an offline labeller,judgefor the production judge. Neverhuman. -
at(datetime | None, default:None) –When it was given.
Returns:
-
Example–The example,
goodwhen the verdict passed,badotherwise.
Raises:
-
ValueError–sourceis notmodelorjudge: a model's verdict is never ground truth.
llm_ports.evals.calibration¶
Calibrate a judge in both directions: agreement with human labels, and replies it must fail.
A judge earns trust two ways, and each measures something the other cannot:
- AGREEMENT with a human labeller on the same items.
cohen_kappais the chance-corrected rate to read;gwet_ac1is its prevalence-robust companion, for an item whose labels are nearly all one answer (where kappa's chance correction degenerates). Raw agreement flatters a lopsided item. - SEEDED-BAD replies: corrupted copies of real replies that any honest judge must fail.
seed_placeholderblanks every string (the schema-answered-honestly law),seed_figureappends a figure no input carried (the number law). A judge that passes these passes everything.
trust joins the two into one verdict a judge's verdicts carry: trusted only when it agrees with
the human on enough items (kappa, or its interval's lower end, at a floor) AND its seeded-bad
replay is calibrated. A judge with no record is untrusted, so swapping the model never inherits
the old one's trust.
tally and verdict read the rows of a replay that judged good samples beside their seeded-bad
copies. A row is a mapping with:
role: the judged role's name.kind:"good"for a real reply, else the seed's name ("tbd","figure", ...).after: the replayed verdict (pass,failorcould_not_run, asjudge.verdict_label).before: the verdict previously stored for a good sample; None on a seeded row.held_out: whether the good sample was outside the rows read while building the judge.
Invariants
- Pure: no model, no database, no clock.
- A statistic that the data cannot define is None, never a number.
- A
could_not_runrow is never a catch: a judge that did not answer measured nothing.
Trust
dataclass
¶
Trust(
trusted: bool,
why: str,
agreement: Agreement,
false_fail: int = 0,
human_good: int = 0,
seeded: int | None = None,
)
Whether a judge's verdicts may be acted on, with the numbers that decided it.
Attributes:
-
trusted(bool) –True only when every condition of
trustheld. -
why(str) –One line naming the deciding condition and its numbers.
-
agreement(Agreement) –The judge against the human on the shared items (n, kappa and its interval, AC1).
-
false_fail(int) –Items the human labelled good that the judge failed.
-
human_good(int) –Items the human labelled good.
-
seeded(int | None) –The seeded-bad replay's
verdictcode; None when no replay was given.
cohen_kappa
¶
cohen_kappa(pairs: Pairs) -> float | None
Return Cohen's kappa over binary (a, b) label pairs.
Both labels are "yes" or anything else (read as no); abstentions belong outside pairs.
Returns:
-
float | None–Kappa, or None when it is undefined: under two pairs, or both labellers one-sided on the
-
float | None–same answer (chance agreement 1.0).
gwet_ac1
¶
gwet_ac1(pairs: Pairs) -> float | None
Return Gwet's AC1 over binary (a, b) label pairs.
Chance agreement is 2 * pi * (1 - pi), with pi the pooled share of "yes", so the statistic
stays defined and bounded where one answer dominates.
Returns:
-
float | None–AC1, or None under two pairs.
seed_placeholder
¶
seed_placeholder(
reply: str, placeholder: str = PLACEHOLDER
) -> str | None
Return a copy of the reply with every string replaced by placeholder.
A JSON object or list keeps its shape; any other reply becomes placeholder itself.
Returns:
-
str | None–The seeded reply, or None for a JSON reply with no string to replace.
seed_figure
¶
seed_figure(reply: str, figure: str = FIGURE) -> str | None
Return a copy of the reply with figure appended to its longest free-text string.
A free-text string holds a space; a non-JSON reply has figure appended to the whole text.
Returns:
-
str | None–The seeded reply, or None for a JSON reply with no free text to carry the figure.
tally
¶
tally(
results: Sequence[Mapping[str, Any]],
) -> dict[str, dict[str, Any]]
Count a replay's rows per role, sorted by role.
Per role: n good samples, before_pass and after_pass among them, after_could_not_run,
held_out, and per seed kind <kind>_n and <kind>_fail (a could_not_run is not a fail).
verdict
¶
verdict(
results: Sequence[Mapping[str, Any]],
*,
floor: float = 0.0
) -> tuple[int, str]
Return the replay's exit code and the reason.
BLIND(2), did not finish looking: nothing was replayed, or some role has no answered good sample or no answered seeded-bad one. Such a replay proves nothing in that direction.FINDING(1): a seeded-bad reply passed, or a role's pass rate over its answered good samples is underfloor.CALIBRATED(0): measured and clean in both directions.
trust
¶
trust(
pairs: Pairs,
*,
replay: Sequence[Mapping[str, Any]] | None = None,
min_labelled: int = MIN_LABELLED,
min_kappa: float = MIN_KAPPA,
bound: Literal["point", "low"] = "point",
require_seeded: bool = True
) -> Trust
Return whether a judge is trusted, from its agreement with a human and its seeded-bad replay.
Checked in order, the first failure deciding: at least min_labelled pairs; the seeded-bad
replay CALIBRATED (or absent with require_seeded=False); kappa at least min_kappa, read
as the point estimate (bound="point") or the 95% interval's lower end (bound="low").
Parameters:
-
pairs(Pairs) –(judge, human)labels of the same real replies,"yes"for pass / good (learn.examples.pairsbuilds them from a label store; abstentions left out). -
replay(Sequence[Mapping[str, Any]] | None, default:None) –Rows of a seeded-bad replay, as
verdictreads them; None when there is none. -
min_labelled(int, default:MIN_LABELLED) –The fewest pairs that can earn trust.
-
min_kappa(float, default:MIN_KAPPA) –The agreement floor.
-
bound(Literal['point', 'low'], default:'point') –Which kappa the floor reads.
-
require_seeded(bool, default:True) –Whether a missing replay leaves the judge untrusted.
Returns:
-
Trust–The
Trust. Never raises.
Invariants
- No pairs is untrusted, never an error: a new judge starts untrusted.
- An undefined kappa (one-sided labels) never clears the floor.
- A replay row
verdictcannot read (a missing key) is a BLIND replay: untrusted.