---
title: "ADR-322: A run records the test evidence of the skill versions it used"
manual: "LLM Foundation"
version: "main"
permalink: "https://docs.typo3.org/permalink/netresearch/nr-llm:adr-322@main"
source: "Adr/Adr322ARunRecordsTheTestEvidenceOfTheSkillVersionsItUsed.rst"
rendered: "2026-11-18T05:36:33+00:00"
---

# ADR-322: A run records the test evidence of the skill versions it used {#adr-322}

-   *Status:* Accepted
-   *Date:* 2026-11-17
-   *Authors:* Netresearch DTT GmbH

## Context {#context}

A run names the skill versions each model call carried ([ADR-321](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-321@main)). Whether a version had been tested, and on which model, was not
recorded anywhere. A completion evaluation identifies its benchmark by a
fingerprint over its prompts, settings, grading and the effective requests
([ADR-271](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-271@main)). A changed skill changes that fingerprint, but the
evaluation did not say which skill versions it had measured, so nothing tied
an evaluation to a version or a version to the evaluations that covered it.

Two premises bound what can be built:

-   Evaluations run one-shot completions through the default configuration
    (`EvaluationService::run()` calls the completion service, which calls
    `LlmServiceManager::chat()`), never the agent loop. The versions they
    can measure are the ones that configuration composes into a prompt. A skill
    loaded on demand during a run is never measured.
-   Evaluation results and agent runs have separate retention
    (`PurgePrivacyDataCommand`). A run cannot point at an evaluation and
    expect it to be there later.

## Decision {#decision}

1.  **An evaluation names what it measured.** `LlmServiceManager::chat()`
    puts the skill versions its composition gave the request, and the
    configuration they came from, on the response's metadata
    (`ComposedSkillVersions`), after the call and outside the provider
    pipeline. The evaluation stores them when every prompt's request carried
    the same versions; otherwise it names none and stands as evidence for none.
    So does an evaluation whose gradings do not share one yardstick, or none of
    whose decisions could be made: it is stored, but measured nothing a run
    could rely on. It also stores the configuration identifier and the version
    of its grading protocol (`tx_nrllm_eval_skill_version`, two columns of
    `tx_nrllm_eval_result`). Identities and digests only.
1.  **A run records the evidence that applied, when it applied.** After the
    answer of the model call that first carried a version, the loop records a
    `skill_evidence` step (`RunTrace::recordSkillEvidence()`, additive
    `@api`): for each version, the evaluations stored at that moment that
    measured exactly its digest, newest first, at most three
    (`SkillEvidenceLookupInterface::LIMIT`). An empty list states that
    none applied. The step names composed versions only: a version loaded
    during the run is left out, since no evaluation measured it loaded, even at
    the same digest, and so is a version a resumed segment carries over, which
    the segment that composed it named if it was composed. The usage step's
    inclusion ([ADR-321](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-321@main)) says which a left-out version was.
    Nothing is looked up later to fill in or replace it. A lookup that fails
    (for example before the database analyzer created the table) is logged and
    records no evidence step for that call: the run goes on, and the missing
    step reads apart from an empty list.
1.  **Judged against the model that answered.** Each evaluation is marked
    `match` when the model the provider named on the call's answer is the
    evaluated model, `other_model` otherwise (for example after a fallback),
    `unknown` when either side names no model. Providers are not compared:
    the answer names its adapter, the evaluation its provider record.
1.  **A snapshot, kept apart from the approval.** The run stores the
    evaluation's facts — uid, run date, set, benchmark fingerprint,
    configuration, provider, model, reported model, grader, protocol, prompt
    and pass counts, pass rate, mean score, scope. The usage entry keeps the
    approval; the evidence step keeps the measurement.
1.  **Gone, never replaced.** The run timeline shows the recorded snapshot.
    Where the evaluation it came from is no longer stored — the same uid with
    the same run date, fingerprint and set — it adds `report=gone`; where
    that check cannot be made, `report=unknown`.

## Consequences {#consequences}

●● For a run, an administrator reads which evaluations had measured the skill
versions it used, at that time, and whether they were measured on the model
that answered.

● A fallback to another model on the call that first carried the composed
versions shows as `other_model` on their evidence. A fallback on a later
call records no evidence.

● A later evaluation of the same version, or of a newer version, does not
change what an earlier run recorded.

◐ The tables hold identities, digests and figures; they follow the evaluation
results' retention and are purged with them.

✕ Skills loaded on demand, and anything a run does beyond a one-shot
completion — tool choice, write proposals, multi-step processes — are never
covered, because no evaluation runs the agent loop. They are left out of the
evidence, and the scope of every evidence is `completion`.

◑ Evidence matches by skill and digest, not by configuration or role: an
evaluation through the default configuration counts for a run under another
configuration, whose system prompt or other skills may differ. The snapshot
stores the evaluation's configuration.

◑ An evaluation whose prompts disagreed on the versions they carried, or ran
with a pinned provider and therefore without the configuration's skills,
covers no version.

◐ Whoever may read a run sees its evidence: evaluation uid, set, model,
pass rate and fit, also where the evaluation module itself is for
administrators only. These are figures about a model, not personal data, and
the same reader already sees the run's skill identifiers and model.

◑ At most three evaluations per version are recorded, the newest stored at the
time. Older ones are not.

◑ Decision-graded evaluations are recorded like the others; their comparable
revision stays unknown ([ADR-271](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-271@main)). Identical versions, model
and input still do not guarantee identical answers.
