Quality evaluation

nr_llm can measure the quality of the answers a model produces against golden prompt sets and detect regressions between runs. Evaluation is an explicitly triggered, out-of-request operation — it never runs in the request pipeline and, with the default grader, spends no tokens. See ADR-060 for the design rationale.

A golden set is a collection of prompts, each with the expectations it should satisfy. A run executes the set against a model, grades every response, aggregates the results to a pass rate and mean score, stores the run, and compares it against the previous run for the same set and model.

Declaring a golden set in your extension

Implement GoldenPromptSetProviderInterface. The nr_llm.golden_prompt_set DI tag is applied automatically when your extension's Services.yaml has autoconfigure: true (the TYPO3 default):

<?php

declare(strict_types=1);

namespace Vendor\NrAiSearch\Evaluation;

use Netresearch\NrLlm\Service\Evaluation\Assertion;
use Netresearch\NrLlm\Service\Evaluation\GoldenPrompt;
use Netresearch\NrLlm\Service\Evaluation\GoldenPromptSet;
use Netresearch\NrLlm\Service\Evaluation\GoldenPromptSetProviderInterface;

final class AiSearchGoldenSetProvider implements GoldenPromptSetProviderInterface
{
    public function getGoldenPromptSets(): array
    {
        return [
            new GoldenPromptSet(
                identifier: 'nr_ai_search.faq',
                name: 'AI Search FAQ answers',
                description: 'Checks the model answers common FAQ prompts correctly.',
                prompts: [
                    new GoldenPrompt(
                        id: 'opening-hours',
                        prompt: 'When is the office open? Answer with the days.',
                        assertions: [
                            Assertion::contains('Monday'),
                            Assertion::regex('/9(:00)?\s*(am|–|-)/i'),
                        ],
                        reference: 'Monday to Friday, 9am to 5pm.',
                    ),
                    new GoldenPrompt(
                        id: 'contact-json',
                        prompt: 'Return the contact as JSON with an "email" field.',
                        assertions: [
                            Assertion::jsonSchema('{"type":"object","required":["email"],"properties":{"email":{"type":"string"}}}'),
                        ],
                    ),
                ],
            ),
        ];
    }
}

The identifier is namespaced (vendor_extension.set) so sets from different extensions cannot collide. Each prompt needs at least one assertion or a reference answer.

Assertion types

The deterministic grader supports four assertion types; a prompt passes only when all of its assertions hold, and the score is the fraction satisfied.

Type	Factory	Passes when
Exact	`Assertion::exact($value)`	the trimmed response equals `$value`
Contains	`Assertion::contains($value)`	`$value` is a substring of the response
Regex	`Assertion::regex($pattern)`	the response matches the PCRE `$pattern`
JSON schema	`Assertion::jsonSchema($schemaJson)`	the response is valid JSON satisfying the structural schema

The json_schema matcher is a lightweight structural check — a top-level type, object required keys, and recursive properties types. Extra keys are allowed. It is intentionally not a full JSON Schema draft validator.

Graders

Two grading strategies sit behind GraderInterface:

deterministic (default) — evaluates the assertions with no LLM call and no tokens.
llm_judge (opt-in) — asks a judge model through CompletionService to score the response 0.0–1.0 with a justification. It uses the reference answer when one is declared. Because it spends tokens, it runs only when explicitly selected; an unknown grader name falls back to the deterministic grader.

Running an evaluation

Use the nrllm:eval:run command:

# Deterministic grading against the configured default model
vendor/bin/typo3 nrllm:eval:run nr_ai_search.faq

# Evaluate a specific model with the LLM judge
vendor/bin/typo3 nrllm:eval:run nr_ai_search.faq --model gpt-5.2 --grader llm_judge

# Fail (non-zero exit) if quality regressed against the previous run — for CI
vendor/bin/typo3 nrllm:eval:run nr_ai_search.faq --fail-on-regression

The command prints the per-prompt gradings and the aggregate (pass rate, mean score), stores the run in tx_nrllm_eval_result, and reports whether the run regressed against the previous run for the same set and model. The regression tolerance is configurable with --max-pass-rate-drop and --max-mean-score-drop (both default to 0.1).

nr_llm ships an example set, nr_llm.smoke, so the command is runnable out of the box.

Quality-aware routing (opt-in)

Stored evaluation results feed an opt-in routing hook. The existing cost/latency selection modes of ModelSelectionService are unchanged; nothing routes by quality unless you call the hook explicitly:

use Netresearch\NrLlm\Service\Evaluation\QualityAwareModelSelector;

// Inject QualityAwareModelSelector, then:
$model = $selector->selectByQuality(
    ['capabilities' => ['chat']],
    minQuality: 0.7,
);

selectByQuality() takes the candidates ModelSelectionService would return for the criteria and re-ranks them by measured quality score (latest run per set, averaged). Candidates without evaluation data keep their base order behind the scored ones; with minQuality set, candidates below it (or without data) are excluded. Making quality a first-class sort key inside ModelSelectionService is a planned follow-up (see ADR-060).