A person's label can be relayed by code, named by the person¶
Decided in 1.12.0.
Context and problem statement¶
LangfuseExampleStore.put refused every human label: a human label was given only in
Langfuse's annotation-queue UI. The labels a judge or a fine-tune needs come from the person who
reads the replies, and that person reads them on whatever surface they already use. A consumer
that wants a good / bad button beside the reply had two choices: keep a second label store of its
own, or send the person to a different tool for every label. A second store splits the counts
coverage and agreement read, so the gap and the judge's kappa would each be half a number.
Decision drivers¶
- One label store: a label given anywhere is read by the same
examples,coverageandagreement. - A human label must stay a person's judgement. Code may carry it; code may not invent it.
- The review queue must not hold a reply already labelled, or a nightly top-up stalls on it.
Considered options¶
- Keep the refusal; a consumer keeps its own table of button labels.
putaccepts ahumanexample only whenbynames the person, writes it under the samelabel-<role>score config the annotation UI uses, and completes the reply's queue item.
Decision outcome¶
Option 2. A relayed human label is scored against the role's categorical config (created when
missing, as enqueue does), so Langfuse shows it beside annotation-UI labels. Its score id is keyed
by the person as well as the role and reply: one person relabelling replaces their label, two
people's labels on one reply are two examples, and learn.examples.latest still counts one per
source. After the scores land, each relayed reply's pending queue item is marked COMPLETED, the
status the annotation UI sets. An empty by is a request_fault; an outcome label is still
never written by code.
Consequences¶
- Good: a consumer's button and the annotation UI feed one count and one agreement.
- Good: the person is on every human label (
metadata.by), whichever surface it came from. - Neutral: Langfuse records a relayed label's source as
API, notANNOTATION; the store reads both the same. - Bad: a queue-item update that fails after the scores landed leaves that item pending until the
next relabel;
putreportsunavailablewith every score counted, so the caller can see it.