---
title: "ADR-104: Worker heartbeat, stale-run reaper, retry and dead-letter"
manual: "TYPO3 LLM Extension"
version: "0.35"
permalink: "https://docs.typo3.org/permalink/netresearch/nr-llm:adr-104@0.35"
source: "Adr/Adr104StaleRunReaperAndRetry.rst"
modified: "2026-09-16T22:09:16+00:00"
---

# ADR-104: Worker heartbeat, stale-run reaper, retry and dead-letter

-   *Status:* Accepted
-   *Date:* 2026-07-21
-   *Authors:* Netresearch DTT GmbH

## Context

Queued runs ([ADR-102](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-102@0.35)) claim a row with a **lease**
(`claimed_by` \+ `lease_expires`), but until now nothing renewed or acted on
it — the lease was diagnostic only. Two gaps remained:

-   **A dead worker strands its run forever.** If the PHP process is killed, the
    container recycled or the machine rebooted mid-loop, the run stays RUNNING
    with a lease nobody renews. No other worker can take it (the claim is
    optimistic on QUEUED), so it never finishes.
-   **A transient provider failure is terminal.** A queued run that hit a 5xx, a
    rate limit or an exhausted fallback chain settled FAILED immediately — no
    retry, even though a different worker moments later might have succeeded.

The P1 roadmap asks for a heartbeat, a stale-run reaper, retry-per-failure-class
and a dead-letter terminus.

## Decision

**Heartbeat.** The runtime's trace hook (the same step-boundary closure the
cancellation probe uses, [ADR-103](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-103@0.35)) renews the lease on every
step for a worker run — an ownership-guarded UPDATE
(`WHERE status='running' AND claimed_by = :me`). A renewal that affects **no
row** means the worker lost the run (reaped, re-claimed or terminated); it
throws the internal `RunLeaseLostException`, caught by a dedicated ladder arm
that stops **without settling** (the row belongs to its new owner now) and
returns `AgentRunOutcome::LEASE_LOST`. Interactive `run()`/`approve()`
segments hold no lease and never renew — the heartbeat is worker-only. To avoid
a zombie worker appending an event whose sequence collides with the new owner's
stream, the boundary step is persisted **after** the renewal check, not before.

**Reaper.** `nrllm:agent:reap` (schedulable) finds RUNNING runs whose lease
has expired (`lease_expires > 0 AND lease_expires < now` — the `> 0` excludes
interactive runs) and either requeues them (budget permitting) or dead-letters
them. Both mutations re-check staleness **inside** the UPDATE, so a heartbeat
renewal that lands between the reaper's SELECT and its write wins and the merely
slow (not dead) worker is left alone.

**Retry.** A queued run's failure runs through a recovery hook in the ladder,
before the default settle. The failure is classified through the existing
[FailureClassifier](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-095@0.35) (extended so a `FallbackChainExhausted`
wrapper classifies by its most recent attempt rather than as UNKNOWN):

-   **Not retryable** (auth, configuration, 4xx) → dead-letter now with reason
    `NOT_RETRYABLE`.
-   **Retryable but the requeue budget is spent** → dead-letter with
    `RETRIES_EXHAUSTED`.
-   **Retryable and under budget** → ownership-guarded requeue (bumping
    `requeue_count`) and a re-dispatch with an exponential `DelayStamp`
    backoff, returning `AgentRunOutcome::REQUEUED`.

Interactive runs pass no recovery hook and surface failures unchanged.

**Budget.** A new `requeue_count` column, shared by both requeue sources
(failure retry and stale reclaim), capped by `AgentRuntime::MAX_REQUEUES` (3),
so a deterministically crashing or hanging run cannot loop forever.

**Dead-letter = FAILED + reason axis.** No new status is introduced (the status
model stays stable, and purge/retention are untouched). Dead-lettering is
expressed on the [ADR-092](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-092@0.35) reason axis via the two new
non-retryable reasons above.

## Consequences

-   `AgentRunOutcome` gained `REQUEUED` and `LEASE_LOST`;
    `AgentRunTerminationReason` gained `RETRIES_EXHAUSTED` and
    `NOT_RETRYABLE` (both `isRetryable() === false`). All are the documented
    minor-release growth path — consumers match with a default arm, and the
    playground never sees the new outcomes (they arise only on the worker path).
-   `tx_nrllm_agentrun` gained `requeue_count`.
-   Real exponential backoff requires the doctrine transport (which honours the
    `DelayStamp`); the default SyncTransport ignores the delay and retries
    in-process, bounded by `MAX_REQUEUES` — the same requirement ADR-102 already
    places on async execution.
-   A worker that renews its lease adds one guarded UPDATE per step boundary
    (alongside the ADR-103 read); steps are provider-call-slow, so the cost is
    noise.
-   The reaper only reclaims abandoned **queue** workers. An interactive run
    abandoned by a dying client keeps no lease and is still reaped by the
    age-based retention path (`nrllm:privacy:purge`).