ADR-322: A run records the test evidence of the skill versions it used
- Status
-
Accepted
- Date
-
2026-11-17
- Authors
-
Netresearch DTT GmbH
Context
A run names the skill versions each model call carried (ADR-321). Whether a version had been tested, and on which model, was not recorded anywhere. A completion evaluation identifies its benchmark by a fingerprint over its prompts, settings, grading and the effective requests (ADR-271). A changed skill changes that fingerprint, but the evaluation did not say which skill versions it had measured, so nothing tied an evaluation to a version or a version to the evaluations that covered it.
Two premises bound what can be built:
- Evaluations run one-shot completions through the default configuration
(
Evaluationcalls the completion service, which callsService:: run () Llm), never the agent loop. The versions they can measure are the ones that configuration composes into a prompt. A skill loaded on demand during a run is never measured.Service Manager:: chat () - Evaluation results and agent runs have separate retention
(
Purge). A run cannot point at an evaluation and expect it to be there later.Privacy Data Command
Decision
- An evaluation names what it measured.
Llmputs the skill versions its composition gave the request, and the configuration they came from, on the response's metadata (Service Manager:: chat () Composed), after the call and outside the provider pipeline. The evaluation stores them when every prompt's request carried the same versions; otherwise it names none and stands as evidence for none. So does an evaluation whose gradings do not share one yardstick, or none of whose decisions could be made: it is stored, but measured nothing a run could rely on. It also stores the configuration identifier and the version of its grading protocol (Skill Versions tx_, two columns ofnrllm_ eval_ skill_ version tx_). Identities and digests only.nrllm_ eval_ result - A run records the evidence that applied, when it applied. After the
answer of the model call that first carried a version, the loop records a
skill_step (evidence Run, additiveTrace:: record Skill Evidence () @api): for each version, the evaluations stored at that moment that measured exactly its digest, newest first, at most three (Skill). An empty list states that none applied. The step names composed versions only: a version loaded during the run is left out, since no evaluation measured it loaded, even at the same digest, and so is a version a resumed segment carries over, which the segment that composed it named if it was composed. The usage step's inclusion (ADR-321) says which a left-out version was. Nothing is looked up later to fill in or replace it. A lookup that fails (for example before the database analyzer created the table) is logged and records no evidence step for that call: the run goes on, and the missing step reads apart from an empty list.Evidence Lookup Interface:: LIMIT - Judged against the model that answered. Each evaluation is marked
matchwhen the model the provider named on the call's answer is the evaluated model,other_otherwise (for example after a fallback),model unknownwhen either side names no model. Providers are not compared: the answer names its adapter, the evaluation its provider record. - A snapshot, kept apart from the approval. The run stores the evaluation's facts — uid, run date, set, benchmark fingerprint, configuration, provider, model, reported model, grader, protocol, prompt and pass counts, pass rate, mean score, scope. The usage entry keeps the approval; the evidence step keeps the measurement.
- Gone, never replaced. The run timeline shows the recorded snapshot.
Where the evaluation it came from is no longer stored — the same uid with
the same run date, fingerprint and set — it adds
report=gone; where that check cannot be made,report=unknown.
Consequences
●● For a run, an administrator reads which evaluations had measured the skill versions it used, at that time, and whether they were measured on the model that answered.
● A fallback to another model on the call that first carried the composed
versions shows as other_ on their evidence. A fallback on a later
call records no evidence.
● A later evaluation of the same version, or of a newer version, does not change what an earlier run recorded.
◐ The tables hold identities, digests and figures; they follow the evaluation results' retention and are purged with them.
✕ Skills loaded on demand, and anything a run does beyond a one-shot
completion — tool choice, write proposals, multi-step processes — are never
covered, because no evaluation runs the agent loop. They are left out of the
evidence, and the scope of every evidence is completion.
◑ Evidence matches by skill and digest, not by configuration or role: an evaluation through the default configuration counts for a run under another configuration, whose system prompt or other skills may differ. The snapshot stores the evaluation's configuration.
◑ An evaluation whose prompts disagreed on the versions they carried, or ran with a pinned provider and therefore without the configuration's skills, covers no version.
◐ Whoever may read a run sees its evidence: evaluation uid, set, model, pass rate and fit, also where the evaluation module itself is for administrators only. These are figures about a model, not personal data, and the same reader already sees the run's skill identifiers and model.
◑ At most three evaluations per version are recorded, the newest stored at the time. Older ones are not.
◑ Decision-graded evaluations are recorded like the others; their comparable revision stays unknown (ADR-271). Identical versions, model and input still do not guarantee identical answers.