--- title: "ADR-063: Provider Resilience — Circuit Breaker, Health, Idempotency" manual: "TYPO3 LLM Extension" version: "0.35" permalink: "https://docs.typo3.org/permalink/netresearch/nr-llm:adr-063@0.35" source: "Adr/Adr063ProviderResilience.rst" modified: "2026-09-16T22:09:16+00:00" --- # ADR-063: Provider Resilience — Circuit Breaker, Health, Idempotency - *Status:* Accepted - *Date:* 2026-07 - *Authors:* Netresearch DTT GmbH ## Context A comparison of this extension against a sibling agent framework surfaced a set of resilience features neither side had: a **circuit breaker** (stop hammering a provider that is down), **provider health scoring** (know which provider is currently healthy), and **request idempotency** (a retried request must not double-charge or produce a second, different answer). The middleware pipeline ([ADR-026: Provider Middleware Pipeline](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-026@0.35)) and the telemetry log ([ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35)) already give the seams to add these without touching provider adapters. The same comparison listed **OpenTelemetry** as a gap. It is addressed here too — by deciding *not* to build it (see below). ## Decision ### Circuit breaker A new `CircuitBreakerMiddleware` tracks consecutive failing calls **per provider**. After `circuitBreaker.failureThreshold` consecutive *tripping* failures the circuit **opens**: for `circuitBreaker.cooldownSeconds` further calls to that provider fail fast with a `CircuitOpenException` instead of waiting on a connection timeout. After the cooldown a single **half-open** probe is allowed; success **closes** the circuit, failure re-opens it. *Tripping* failures are exactly the set `FallbackMiddleware` already treats as retryable — `ProviderConnectionException` (network / timeout / 5xx / retries exhausted) and a `429` `ProviderResponseException` (rate limit). Client errors (other 4xx), misconfiguration and unsupported-feature errors mean the provider *answered*; they are not a health signal and neither trip nor reset the circuit. **Pipeline placement — innermost, priority 20.** This is the load-bearing design choice, so it is spelled out: ```text TelemetryMiddleware 110 observes every run IdempotencyMiddleware 105 replays a stored result by key CacheMiddleware 100 payload cache; short-circuits on hit BudgetMiddleware 75 pre-flight budget gate FallbackMiddleware 50 swaps configuration on retryable failure UsageMiddleware 25 records the served call CircuitBreaker 20 guards the actual provider call (THIS) ``` - **Inside FallbackMiddleware.** An open circuit throws `CircuitOpenException`, which `FallbackMiddleware` now treats as retryable. Because the breaker sits *inside* the fallback loop, that exception is raised from within `$next($configuration)` on each attempt, so Fallback catches it and advances to the next configuration/provider. A naïve reading puts the breaker "between Budget (75) and Fallback (50)"; that would be **wrong** — outside Fallback, an open primary circuit would abort the whole call before any fallback ran, the opposite of the intent (skip the sick provider, use a healthy one). The breaker therefore lives *below* Fallback. - **Inside UsageMiddleware.** The breaker wraps only the terminal, so the health signal reflects the pure provider call — usage bookkeeping (success-only, with its own error handling) never contaminates it, and a genuine success closes the circuit before any post-processing. `CircuitOpenException` extends `ProviderException` (so it carries the [ADR-053: One marker interface for all thrown exceptions](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-053@0.35) marker interface); `ProviderConnectionException` is `final`, so extending *that* was not an option — instead `FallbackMiddleware::isRetryable()` gained one explicit `instanceof` arm. **Circuit state storage: cache, not a table.** State (`consecutiveFailures` \+ `openedAt`) lives in the `nrllm_circuit` cache via `CircuitBreakerStoreInterface` (no hardcoded backend — the instance's Redis/Valkey is shared across web workers, so one worker tripping a circuit protects them all). Rationale: the state is inherently transient, self-decaying (a forgotten entry reads as *closed*, the fail-safe default), and needs no schema, no purge command, and no write on every call. A DB table would add write load on the hot path and a maintenance surface for data that is meant to be ephemeral. The half-open single-probe gate is pragmatic (refresh the open window before probing) rather than an atomic compare-and-set, which the cache backend cannot portably offer — a few extra probes after a cooldown is acceptable. The failure counter shares this posture: it is a non-atomic get-modify-set, so under heavy concurrent failure some increments are lost and the breaker trips a little late (the count still climbs about one per cooldown window) rather than never — acceptable for the same reasons. ### Provider health scoring `ProviderHealthService` reads the **existing** telemetry log (`tx_nrllm_telemetry`, [ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35)) over a rolling window and computes a per-provider `ProviderHealthScore` (success rate + mean latency into one comparable `0.0–1.0` number; success rate weighted 4:1 over latency). No new write path and no second source of truth — telemetry already records success/latency per run, so health is a **read model** over it. Reads go through a dedicated `ProviderHealthRepository`; the telemetry recorder's own interface stays append/purge-only, as [ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35) intended. The one built-in consumer is `FallbackMiddleware`, which asks `ProviderHealthService::reorder()` to prefer healthier providers among the fallback candidates. It is a **hint, opt-in, and minimal-invasive**: - Gated by `health.reorderFallback` — **OFF by default**. When off, the chain is returned untouched with *no* telemetry query, so the configured fallback order stays the default. - When on, it is a **stable** sort by descending health: providers of equal (or unknown) health keep their configured order, and no candidate is ever dropped. A provider with no telemetry scores *neutral*, never unhealthy — an un-exercised provider must not sink just for lack of data. A pure config-order-primary "tie-break" would be a no-op on a totally-ordered chain (there are no ties to break), so health-primary reordering is offered as the opt-in instead; the default-off flag is what keeps existing behaviour identical. ### Idempotency keys An optional idempotency key on any options object (`AbstractOptions::withIdempotencyKey()`) makes a repeated request return the stored result instead of calling the provider again. A new `IdempotencyMiddleware` (priority 105 — just inside Telemetry, outside Cache/Budget, so a replay short-circuits the behavioural stack and is not re-charged, yet is still observed) stores the result under the key and replays it on the next call with the same key. Failed calls and streaming generators are never stored. **Cache, not a table** — and *not* an overload of the existing cache key. The task hypothesis ("make it a thin layer on CacheMiddleware's key") was investigated and rejected: `CacheMiddleware` is deliberately array-only (it persists `array`), so it cannot round-trip the **typed** responses (`CompletionResponse`, …) that the chat/completion paths return — which is the case idempotency actually matters for. The dedicated middleware stores over its own `nrllm_idempotency` `VariableFrontend`, which serialises any response value, so idempotency works for every operation while staying cache-backed: idempotency results are transient and TTL-bounded by nature, so a table (with its purge/TCA surface) would be the wrong store. ### OpenTelemetry — deliberately not built OTel exporters/collectors are **out of scope for a TYPO3 extension**. The extension should not own an OTLP exporter, a collector endpoint, or trace sampling configuration — that is the **host instance's** observability responsibility (the same instance that owns logging, APM, and metrics scraping). What the extension *can* own — durable per-request metrics — already exists as `tx_nrllm_telemetry` ([ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35)) with a correlation id per run; a host that runs OTel can scrape or forward that. Building an in-extension OTel pipeline would duplicate host infrastructure, couple the extension to an exporter SDK, and add configuration the operator already manages centrally. ## Consequences - **Faster failure, healthier routing.** A downed provider is skipped within one cooldown window instead of timing out on every call; with the opt-in reorder on, a flapping provider is de-prioritised among fallbacks. - **Circuit state is cluster-wide but forgettable.** On a shared cache backend every worker sees the same circuit; a cache flush resets all circuits to closed (a conservative retry), which is acceptable. - **Health is advisory only.** With `health.reorderFallback` off (default), provider selection is byte-for-byte unchanged. The reorder, when enabled, re-loads each candidate configuration once to resolve its provider — a cost paid only on the opt-in path, on the (already-failing) fallback route. - **Idempotency is opt-in and best-effort.** No key ⇒ no behavioural change. A cache miss/flush simply re-runs the call. It does not deduplicate concurrent in-flight requests — it replays a *completed* result. - **One new retryable exception.** `FallbackMiddleware::isRetryable()` now also matches `CircuitOpenException`; the pipeline order test and the fallback tests pin this. - **Three new caches** (`nrllm_circuit`, `nrllm_health`, `nrllm_idempotency`), all without a hardcoded backend, all in the `nrllm` cache group so a group flush clears them. ## Alternatives considered - **Circuit state in a DB table** (`tx_nrllm_provider_circuit`). Rejected: ephemeral, hot-path state does not want a table, a per-call write, or a purge command. Cache is the natural store and shares state across workers for free. - **Circuit breaker outside FallbackMiddleware** (priority 60). Rejected: an open primary circuit would abort the call before fallback, defeating the purpose. The breaker must be *inside* the fallback loop. - **Health scoring on its own rolling-window state.** Rejected: telemetry already records exactly the success/latency signal; a second store would duplicate it and drift. Read the log. - **Health as a config-order-primary tie-break** (never reorders). Rejected as a no-op: a totally-ordered chain has no ties, so this would never change anything. Opt-in health-primary reorder (default off) gives a real, operator-controlled behaviour without changing the default. - **Idempotency as a thin overload of CacheMiddleware's key.** Rejected: CacheMiddleware is array-only by design and cannot store the typed responses the valuable (chat) case returns. A dedicated middleware over its own VariableFrontend is the minimal store that works for every response type. - **Idempotency in a DB table.** Rejected: transient, TTL-bounded data belongs in the cache, not a table with a purge/TCA surface. - **Build OpenTelemetry into the extension.** Rejected: host-instance responsibility; the telemetry log already exposes per-request metrics a host OTel stack can consume. ## Deferred / follow-ups - A backend readout of circuit state and health scores (a diagnostics panel). The services expose the data; only a view is missing. > [!NOTE] > **Overtaken.** The view exists: the analytics module renders a > **Provider health and circuits** table > (`ServiceAnalyticsProviderHealthReport`, joining > `ProviderHealthServiceInterface` with > `CircuitBreakerStoreInterface`). It names each score's sample count > and the rolling window, shows a provider with no telemetry as "no data" > rather than a zero, and states whether `health.reorderFallback` and > `circuitBreaker.enabled` are on — a score that decides nothing is the > likeliest misreading of such a panel. No new module was added > ([ADR-119: Where the backend modules live — Administration, for now](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-119@0.35)); it is a section of the existing analytics view. - Per-operation idempotency TTL override via call metadata (the middleware uses a single window today). - Concurrent-double-submit dedup. The current get-then-run-then-set has no atomic reserve-on-miss — a portable one is not available through the cache `FrontendInterface` — so two genuinely simultaneous same-key requests are not deduplicated; only sequential retries are (see *Consequences*). A future revision could gate run+store with `TYPO3CMSCoreLockingLockFactory`. - Strict circuit-breaker failure counting under concurrency. The failure counter is a non-atomic get-modify-set (best-effort, the same posture as the half-open gate), so concurrent failures can lose increments and trip the breaker slightly late. A future revision could make it exact with the same `TYPO3CMSCoreLockingLockFactory`. ## References - [ADR-026: Provider Middleware Pipeline](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-026@0.35) — Provider Middleware Pipeline (the pipeline and ordering this extends). - [ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35) — Telemetry Middleware (the log health scoring reads; the per-request metrics that make in-extension OTel unnecessary). - [ADR-053: One marker interface for all thrown exceptions](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-053@0.35) — Exception marker interface (`CircuitOpenException` inherits it via `ProviderException`). - ADR-021 — Provider Fallback Chain (the chain the health reorder reprioritises). - ADR-028 — Public Services Policy (all new services stay private).