Incident response 

Three runbooks. Each starts with the step that preserves evidence, because several of the useful actions — rotation, restore, re-anchoring — destroy or overwrite the state an investigation needs.

Suspected secret exposure 

Trigger: a credential from the vault appears somewhere it should not — a log, a ticket, a screenshot, a public repository — or an account is compromised that held secret.reveal, or a host running the PHP process is compromised.

Step 1 — preserve, before you change anything 

# Chain state as it stands now, plus the anchor comparison.
vendor/bin/typo3 vault:audit-verify > incident-verify-before.txt

# Snapshot the external anchor file as it stands now.
cp <auditSinkAnchorPath> incident-anchor-before.ndjson
Copied!

Copy the audit rows for the affected identifiers and the suspected time window out of the system as well (see Evidence collection for the export). A rotation in step 3 changes hash_before / hash_after and adds rows; the pre-incident picture is not recoverable afterwards.

Step 2 — determine scope from the audit log 

# Everything that happened to one identifier.
vendor/bin/typo3 vault:audit --identifier=<identifier>
Copied!

Answer these, and write down the answers:

  • Which secrets? Only the one observed, or every secret the compromised actor could reach? Check the actor's group grants and the per-secret tiers.
  • ``use`` or ``reveal``? A read row written through a machine path (secret.use) means the plaintext went into an integration. A reveal means a human saw it. Both are exposure; they imply different blast radii.
  • How long? First and last relevant row, not just the one that triggered the incident.
  • Which actor type? backend, cli or technical. A technical actor means code, not a person — find the code.
  • Any ``access_denied`` rows? Denials around the same window often show the attacker's reconnaissance and widen the scope.
  • Any ``break_glass_activated`` rows? If so, treat Break-glass usage policy and review as part of this incident.

Step 3 — rotate at the origin, then in the vault 

Rotate the credential at its source first — the API provider, the SMTP relay, the OAuth client — so the exposed value stops working. Only then store the replacement:

vendor/bin/typo3 vault:rotate <identifier>
Copied!

Rotating only the vault copy changes nothing about the exposure: the old value still authenticates.

Rotate every secret in scope, not only the one that was observed. If the PHP process or the host was compromised, the master key was reachable, so every secret is in scope, and the master key itself has to be rotated as well — Key rotation.

Step 4 — close the path 

Whatever made the exposure possible: revoke the session and reset the account; remove over-broad grants (an editor holding secret.reveal who only needs secret.use is the common finding); withdraw the admin override and pin it (Disabling the admin override); switch off allowCliAccess if it was enabled for a one-off and never turned back off, and where it must stay on, narrow cliAllowedOperations back to the low-risk default — a pipeline that was granted secret.reveal or master_key.rotate for one task and kept it is the same class of finding as the over-broad editor grant; consider the hardened profile (Migrating from standard to hardened).

Step 5 — record and re-anchor 

vendor/bin/typo3 vault:audit-verify > incident-verify-after.txt
vendor/bin/typo3 vault:audit-anchor
Copied!

Keep the before and after artefacts together with the scope notes.

Suspected audit tampering 

Trigger: vault:audit-verify or the verify scheduler task reports HASH_MISMATCH, UID_GAP, TABLE_RESET or EPOCH_DOWNGRADE; or an AuditIntegrityAlertEvent with AuditIntegrityReason::isTamperEvidence() true reaches your alerting.

Step 1 — freeze the evidence 

# Full table dump, unaltered. Not a filtered export.
mysqldump <db> tx_nrvault_audit_log > incident-audit-table.sql

# The external anchor file, byte for byte.
cp <auditSinkAnchorPath> incident-anchor.ndjson

# The verification output, with the findings.
vendor/bin/typo3 vault:audit-verify > incident-verify.txt
Copied!

Also pull the same period from the sinks the database owner cannot reach — the syslog archive, the SIEM. That is the comparison that decides the case. If the SIEM holds rows the database no longer does, you have proof of deletion rather than a suspicion.

Step 2 — rule out the benign causes first 

Most findings are operational, and confusing the two wastes the window in which real evidence is still available.

Finding Benign explanation to exclude
TABLE_RESET A database restore to an earlier point in time produces exactly this. Check your restore records and Audit-chain consistency after a restore. Also: a database clone from production into staging, then anchoring in staging.
HASH_MISMATCH on every row from one uid onwards The audit table and the master key came from different points in time, or the master key was rotated without the chain rekey completing.
EPOCH_DOWNGRADE auditHmacEpoch was lowered in the configuration, or a restored configuration does not match the restored data. An increase between consecutive rows is a legitimate migration boundary and is reported as a warning, not an error.
UID_GAP Rows lost in a partial restore, or a manual cleanup someone performed and did not document.
SINK_FAILURE / NO_EXTERNAL_SINK Availability, not integrity: an unreachable collector, a full disk, an allowed_hosts entry removed. Fix the delivery — but note that a sink which has been failing since before the suspect window means you have no independent evidence for it.

Step 3 — establish what happened 

  • Compare the anchor to the chain. The anchored sequence and tip say what the chain looked like at anchoring time. A shorter chain, a missing row at that sequence, or a different hash there is a rebuild.
  • Bound the window. Between the last anchor that still matches and the first one that does not.
  • Diff against the external sink. Rows present in syslog or the SIEM but absent from the table are deleted rows, and they name the actor.
  • Ask who could. Chain rewriting from epoch 1 up requires the master key, not just database write access. If the hashes recompute correctly under the current key but contradict the anchor, consider that the key holder is in scope.

Step 4 — respond 

Treat a confirmed tamper as a full compromise of the database, and — if the hashes are internally consistent yet contradict the anchor — of the master key as well. That means Key rotation and the exposure runbook above for every secret.

Only when the investigation is closed: re-anchor, and record the incident alongside the frozen artefacts so the next verification run has a documented baseline.

Break-glass usage policy and review 

Break-glass restores full administrator power for the duration of the window — every operation permission, and read, write and delete on every secret. It prevents nothing; its value is a named actor, a typed justification, a hash-chained audit row, a PSR-14 event, a visible banner, and an expiry. See Break-glass mode for the mechanism.

Policy 

  • Only for incidents. Not for routine maintenance and not for convenience. If a task needs break-glass every week, the group grants are wrong — fix the grants.
  • Reference a real record. --reason is mandatory and rejects empty or whitespace-only values, but nothing forces it to be useful. Require an incident, ticket or change-record identifier; "testing" tells a later reviewer nothing.
  • Take the smallest window that works. The default is 15 minutes, the range is 1 to 60, and out-of-range values are clamped rather than rejected so a fat-fingered --minutes=600 yields the ceiling instead of an error to re-read under pressure.
  • Close it explicitly. Do not wait for the expiry:

    vendor/bin/typo3 vault:break-glass --deactivate --reason="INC-4711 closed"
    Copied!

    An expiring window writes no audit row — nothing runs at the moment it lapses. Only an explicit deactivation produces the closing evidence.

  • Alert on activation. Listen for \Netresearch\NrVault\Event\BreakGlassActivatedEvent and BreakGlassDeactivatedEvent. The audit log proves what happened; a listener is what makes someone look.

Post-incident review checklist 

Run this after every activation, without exception. An activation nobody reviews is the admin override with extra steps.

# The evidence: both rows live under the pseudo-identifier __break_glass__.
vendor/bin/typo3 vault:audit --identifier=__break_glass__
Copied!
  • [ ] An break_glass_activated row exists, with the actor, the reason,
    the ttlMinutes and the expiresAt in its context.
  • [ ] The reason names a real incident or change record.
  • [ ] The actor was authorised to declare an emergency.
  • [ ] A break_glass_deactivated row exists. If not, the window expired —
    reconstruct the closed interval from the activation row's expiresAt and note that nobody closed it.
  • [ ] The window length was proportionate to the work.
  • [ ] Every vault operation performed inside the window is accounted for.
    Filter the audit log to the interval between activation and closure and confirm each row was part of the incident. This is the actual point of the exercise.
  • [ ] Nothing was done that the actor's normal grants would have covered —
    if so, they used break-glass instead of asking for the right grant.
  • [ ] The chain still verifies, and both rows are inside it:

    vendor/bin/typo3 vault:audit-verify
    Copied!
  • [ ] adminOverrideDisabledEffective reads yes again:

    vendor/bin/typo3 vault:break-glass --status
    Copied!
  • [ ] Whatever made break-glass necessary has a follow-up action — a grant
    to add, a runbook to write, a permission model to fix.
  • [ ] If the window was opened by an actor you did not expect, treat it as
    Suspected secret exposure and escalate.