.. SPDX-License-Identifier: CC-BY-4.0 SPDX-FileCopyrightText: Netresearch DTT GmbH .. include:: /Includes.rst.txt .. _administration-data-retention: ====================== Data retention & purge ====================== The extension writes several kinds of row that can carry request content: conversation transcripts, agent-run event payloads, evaluation output, the skill audit trail and provider telemetry. All of them are governed by one central privacy policy (:ref:`ADR-064 `): what is stored at all, and for how long. .. _administration-data-retention-what-is-stored: What is stored ============== A provider and a task can tighten this level with their own :guilabel:`Privacy level` field; it cannot loosen it. Agent chat and tool attempts also apply the resolved provider's ceiling before dispatch, including criteria selection, failed attempts and fallback. The run keeps the strictest ceiling across resumes and refilters earlier event payloads before a stricter attempt. An explicit ``metadata`` ceiling suppresses kept answers even at the global ``metadata`` default. See :ref:`ADR-270 ` for the decision and limits. Apply the TYPO3 database schema update before starting workers with this change. Old rows acquire their ceilings on their next guarded write or provider attempt; inactive historical rows are not retroactively reconciled. Previous exports, backups and functional queue/suspension state need their own retention handling. The extension configuration setting :guilabel:`Content Privacy Level` (``privacy.level``) decides how much request content is persisted: ``none`` / ``metadata`` (default) Content payloads are dropped. Metadata — timings, token counts, cost, tool names, sizes, error class — is kept. Agent-run steps are stored in this reduced form too, so persistence does not quietly build a prompt archive. ``redacted`` Content is stored with obvious credentials and email addresses masked and the length capped. A heuristic, not a guaranteed PII scrubber. ``full`` Content is stored verbatim. Choose this deliberately — for an agent run it means the whole transcript, including tool arguments and results. Conversation messages are the exception: their content is the feature. A session replays its own history on every turn, so the transcript is always stored — and therefore governed by retention rather than by the level. The final answer of a run that completes where nobody watches it arrive is a second exception, for the same reason (:ref:`ADR-234 `): a queued run, and any run that completes after an approval, an answered question or a remote agent's answer, including a backend chat. It is kept encrypted at ``metadata`` too, redacted at ``redacted``, not at all at ``none``, and it goes with the run row under ``privacy.retention.agentRun``. The run's initiator, administrators and service accounts holding ``agent:read`` can read it, as they can the rest of the run. .. _administration-data-retention-content-digests: Content digests of agent runs ----------------------------- :guilabel:`Run content digests` (``privacy.inputDigests``, off by default) lets an agent run step at the ``metadata`` level keep a keyed digest of each part of the content it drops (:ref:`ADR-323 `): each message sent, with its role, the model's answer, a tool's arguments and its result, and the whole request a model call answered. Two runs then show which input differed without the text being stored. The run timeline shows each digest by its first twelve hex digits. The digests are HMAC-SHA-256 keyed with the installation's encryption key and a purpose of their own; the key is never stored with them. They compare within one installation only. Tool-call ids and the provider's own items, which a provider generates afresh for every run, are left out, so two runs that sent the same input show the same digests. A tool result that carries a random value of its own differs on every run: ``fetch_external_url`` fences each page with a fresh nonce. An installation without an encryption key stores none. .. warning:: A digest does not hide a short or predictable text. Anyone who can read the run and also start a run of their own can send a guessed text — a yes or no, a record uid, a prompt built from a template — and compare the two digests, and so confirm the guess without knowing the key. Keep the setting off where a confirmed guess matters. No other level keeps digests: ``none`` keeps nothing, ``redacted`` would otherwise confirm a credential the redactor masked, and ``full`` stores the text itself. A step first written at ``redacted`` and later held to ``metadata`` by a provider's ceiling gets none, because its text is already masked. A step first written at ``full`` and held to ``metadata`` later gets no request digest, because ``full`` does not keep the request identity. When a run's level is tightened later, the digests are removed with the rest. Switching the setting off removes no digest already stored; they go with the run's other events under ``privacy.retention.agentRun``. .. _administration-data-retention-how-long: How long it is kept =================== ``privacy.retentionDays`` (default 30) is the window for every category. Each category can override it under ``privacy.retention.*`` — set ``0`` to keep the default: .. list-table:: :header-rows: 1 * - Setting - Covers * - ``privacy.retention.conversation`` - Conversation sessions and their message transcripts * - ``privacy.retention.agentRun`` - Finished agent runs and their event payloads * - ``privacy.retention.approval`` - Agent runs that never reached a terminal status, above all runs suspended for a human approval * - ``privacy.retention.telemetry`` - Provider pipeline metadata (no prompts, no responses) * - ``privacy.retention.evaluation`` - Graded evaluation output * - ``privacy.retention.skillAudit`` - Skill scan findings * - ``privacy.retention.governance`` - Tool-gate denials and guardrail blocks (no prompts, no responses) * - ``privacy.retention.toolAudit`` - Who switched a tool or a tool group on or off in the :ref:`Tools module `, and when A zero, negative or non-numeric value never means "delete immediately" — it means "no override". When to set one at all, and how to read the window that is in force, is covered under :ref:`administration-governance-retention-overrides`. Runs awaiting a decision are deliberately separate. A run suspended for an approval carries the state needed to resume it, so it is only deleted on the ``approval`` window. Give it a longer value than ``agentRun`` if approvers may take days. .. _administration-data-retention-purging: Running the purge ================= Nothing is deleted until a purge runs. Schedule the central command — it covers every content-bearing table in one pass: .. code-block:: bash vendor/bin/typo3 nrllm:privacy:purge It reports the window applied and the rows deleted per category. ``--days=N`` overrides every category at once, which is useful for a one-off cleanup: .. code-block:: bash vendor/bin/typo3 nrllm:privacy:purge --days=7 Two single-table variants exist for operators who want separate schedules. They read the same policy, so they cannot drift from it: .. code-block:: bash vendor/bin/typo3 nrllm:session:purge vendor/bin/typo3 nrllm:telemetry:purge All three appear in the scheduler's :guilabel:`Execute console commands` task — no extra registration is needed. .. note:: Usage rows (``tx_nrllm_service_usage``) are **not** purged: they are the billing ledger the budget limits and the Analytics module report on. They hold no prompt content, only counts, cost and the acting backend user.