A pack has an identity beside its content sha¶
Decided in the release after 1.22.0. Extends ADR-0008; changes none of it.
Context and problem statement¶
ADR-0008 names a pack by content_sha, a hash of its template sources. Three holes make that name
weaker than the thing it is used for, which is tying a grade to exactly what was graded:
- The manifest is excluded. A version bump, a re-pointed grader, a different golden set or a
renamed template key changes what the pack does under evaluation, and none of them changes
content_sha. - Re-split templates collide. Sources are joined by a bare newline, so
["a\nb", "c"]and["a", "b\nc"]hash alike. - Included files are missed. The loader was a filesystem loader over the pack directory, so a
template could
{% include %}any file there; the sha covered none of them, and the manifest did not name them.
The manifest's output_schema, graders and golden were validated for shape and read by
nothing: each consumer resolved the dotted paths itself, and a typo surfaced when an eval first
reached it.
content_sha cannot be repaired in place. Stamps beside stored output rows, mirrored prompts in a
managed store, and tests in consumers all hold today's values; a changed algorithm would read as a
configuration change that never happened.
Decision drivers¶
- Every existing
content_shastays valid, byte for byte. - A name that moves whenever behaviour can move, and does not move otherwise.
- No file a render can read is outside what the name covers.
- The name is the same on every checkout platform.
- A broken pointer in a manifest fails when the pack is resolved, not when an eval reaches it.
Considered options¶
- Change
content_shato cover the manifest and includes. - Hash
PackManifest.model_dump()beside the sources. - A new name,
identity, over raw bytes, with a closed template set.
Decision outcome¶
Chosen option 3.
byte_sha(parts): sha256 over each(name, data)part asname NUL len(data) NUL data, in order, cut to 16 hex characters. The length prefix makes a re-split, a rename or a reorder a different input. It is a public primitive so a consumer can hash a wider scope (a pack plus its generated config and dataset) on the same rule, takingPack.identityas its first part.Pack.identityisbyte_shaover the manifest file's raw bytes, then every file of the closed set by sorted relative path.Pack.stamp()gains the keyidentity;stamp_str()and theshakey are unchanged.- The template set is closed.
PackManifest.includesnames the files a template reaches by{% include %},{% import %}or{% extends %}: relative POSIX paths inside the pack, any extension, subdirectories allowed (partials/h.j2,rubric.md), never..or a path that resolves outside it. The loader is aDictLoaderover exactlytemplatesandincludes; at load,jinja2.meta.find_referenced_templateschecks every file, and a reference outside the set, or a dynamic one that cannot be read statically, fails withPromptPackInvalid. One environment is built per pack at load and reused by every render. - A carriage return is refused in any pack file.
content_sharead text through universal newlines, so a CRLF checkout hashed like an LF one; raw bytes would not, and the identity of one commit would differ by platform. Refusing is the only rule that keeps both names stable. Repositories keep packs LF with* text=auto eol=lfin.gitattributes. - The manifest is resolved by the library (
prompts.resolve):output_schemaimported only from module roots the caller names, graders looked up in the caller's registry,goldenkept inside the pack directory. Any miss isPromptPackInvalidnaming the field.promptssits belowcallin the layer contract, so the fallback to the built-in call graders isevals.golden.resolve_pack, one layer up. - The vectors are frozen the day
identityships: threebyte_shapart lists (including the re-split pair that collides undercontent_sha) and the identity of each fixture pack. call.graders.GRADERSbecomes a read-only mapping, so one consumer's grader cannot leak into another's table;{**GRADERS, ...}still copies, and every reader takes a table argument.
Consequences¶
- Good: a grade, a mirror and a stamp can name everything that could change the pack's behaviour;
unevaluatedworks on identities as well as shas. - Good: no file is rendered that the name does not cover.
- Good:
content_shaand every value derived from it are untouched. - Bad: a pack with an unlisted include, a dynamic include, or a CRLF file now fails at load. Each is fixed in the pack (list the include, make it static, convert to LF), and each failure names the file.
- Bad: two names per pack.
content_shaanswers "did the text change",identityanswers "can this pack behave differently"; a run is comparable to another only whenidentitymatches. - Neutral: a model setting does not belong in a pack. Model, temperature and retries are the role's and its tier's; one prompt carrying them would be two experiments.
Option 1, rejected¶
Every stored stamp and every mirrored prompt would silently stop matching. ADR-0008's frozen algorithm is the reason the name is worth storing.
Option 2, rejected¶
model_dump() changes when the library adds a field with a default (this change adds two), so the
name would move on a library upgrade with no edit to the pack.