--- title: "ADR-062: Streaming Request Lifecycle" manual: "TYPO3 LLM Extension" version: "0.35" permalink: "https://docs.typo3.org/permalink/netresearch/nr-llm:adr-062@0.35" source: "Adr/Adr062StreamingLifecycle.rst" modified: "2026-09-16T22:09:16+00:00" --- # ADR-062: Streaming Request Lifecycle - *Status:* Accepted - *Date:* 2026-07 - *Authors:* Netresearch DTT GmbH ## Context Every non-streaming provider call goes through the middleware pipeline ([ADR-026: Provider Middleware Pipeline](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-026@0.35)) and so inherits budget pre-flight ([ADR-025: Per-User AI Budgets](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-025@0.35)), usage accounting, and telemetry ([ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35)). Streaming did not: both `LlmServiceManager::streamChat()` and `streamChatWithConfiguration()` called the provider's `streamChatCompletion()` generator directly and returned it to the caller. The consequence was a live budget hole. A streamed chat: - ran **no** budget pre-flight — an over-budget user could stream freely; - produced **no** usage row in `tx_nrllm_service_usage`, so streamed tokens were invisible to cost dashboards and to the very budget aggregate that is supposed to gate the next call; - produced **no** telemetry row in `tx_nrllm_telemetry`, so a streamed call had no latency, no outcome, and no correlation id on record; - had no fallback, not even in the window where one is still possible. The pipeline cannot simply be reused. A PHP generator is **lazy**: calling `streamChatCompletion()` returns a suspended generator without running a line of it. Wrapping that as the pipeline terminal makes every middleware run against a stream that has not started — Budget would gate nothing meaningful, `UsageMiddleware` (which records *after* the terminal returns) would record zero tokens, and `TelemetryMiddleware` would measure a near-zero latency. This laziness is exactly why [ADR-026: Provider Middleware Pipeline](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-026@0.35) sanctioned streaming as a documented pipeline bypass. ## Decision Introduce a dedicated streaming lifecycle rather than forcing streams through the response pipeline. A single private collaborator, `NetresearchNrLlmServiceStreamingStreamingDispatcher`, owns it; the manager's two streaming methods build an *opener* closure and hand off to it. The dispatcher is a **wrapping generator** — it never changes the public `Generator`\ contract the seven providers and their consumers rely on. The lifecycle has four stages: 1. **Budget pre-flight — eager.** `StreamingDispatcher::stream()` runs the same `BudgetService::check()` gate as `BudgetMiddleware` *before* it returns a generator, so an over-budget caller is rejected at call time with a typed `BudgetExceededException` — not lazily on first iteration. This is the one part that must be eager: a caller that never drains the stream is never charged, but a caller that is already over budget must never receive a stream to drain. 1. **Provider selection with fallback — before the first chunk only.** The dispatcher walks the primary configuration's fallback chain (shallow, exactly as `FallbackMiddleware` does) and *primes* each candidate generator (`rewind()`) inside a try/catch. Priming runs the provider up to its first yield — the HTTP request and first delta — which is the last moment a provider can still be swapped. A retryable failure here (`ProviderConnectionException`, or a `429` `ProviderResponseException`) moves to the next candidate; a non-retryable failure bubbles up unchanged. Once a chunk has been handed to the caller a swap is impossible, so **fallback stops at the first chunk**. That single asymmetry with the non-streaming pipeline is intrinsic to streaming, not a shortcut. 1. **Drain accounting — lazy.** As the caller drains, the wrapper re-yields each chunk, appends it to a completion buffer, and stamps the time-to-first-token on the first chunk delivered. 1. **Settlement — in a \`\`finally\`\`.** Usage and telemetry are written in the drain generator's `finally` block, so they land on **every** exit path: normal completion, a mid-stream exception, and an abandoned generator (client disconnect or a consumer `break`). PHP runs a suspended generator's `finally` when the generator is destroyed, which is what makes early-break accounting work without a caller callback. An abandoned stream therefore records the **partial** tokens actually produced, never zero and never the full amount. ## Token counts are estimated The seven streaming adapters yield only text deltas and return `void` — none emits a usage frame at stream end. Real per-token usage is therefore *not available* on this path today. The dispatcher estimates it with the ≈4 chars/token heuristic already used by `RenderedPrompt::estimateTokens()`: the caller passes the prompt character count on the context metadata (it holds the messages; the dispatcher never sees the payload, honouring the [ADR-026: Provider Middleware Pipeline](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-026@0.35) "context carries no payload" rule) and the dispatcher counts the drained completion text. Recording an estimate is a deliberate improvement over the previous state, which recorded nothing at all — for budget enforcement an approximate figure is far better than a silent zero. Exact stream usage would require enabling provider-level usage frames (OpenAI `stream_options.include_usage`, Anthropic `message_delta` usage, …) and threading them out of the generator without breaking its `string` yield type — a follow-up, tracked separately, that would replace the estimate with the reported figure where a provider supports it. ## Attribution Mirroring the non-streaming split ([ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35)): - **Telemetry** names the *requested* primary configuration; a pre-first-chunk swap shows as `fallback_attempts > 0`. `cache_hit` is always `false` — streaming never caches. A new nullable `time_to_first_token_ms` column carries the TTFT; it is `NULL` for every non-streaming row (there is no partial-response milestone to measure), deliberately distinct from a real `0 ms`. - **Usage** attributes to the configuration that actually *served* — after a fallback swap that is the fallback's provider/model/uid, not the primary's. For an ad-hoc stream (a pinned provider with no configuration entity) the served provider comes from the context metadata the manager sets. ## Scope - New `StreamingDispatcher` (private service, autowired) and the `LlmServiceManager` wiring for both streaming entry points. - `streamChatWithConfiguration()` gains a trailing `array $metadata = []` parameter so streamed calls can carry budget attribution; it is additive and the three-argument callers stay source-compatible. `LlmServiceManagerInterface` — a Category-1 public API — keeps its `Generator` return contract, so this is backward compatible. - `tx_nrllm_telemetry` gains the nullable `time_to_first_token_ms` column and `TelemetryRecord` a matching trailing nullable field. - [ADR-009: Streaming Implementation](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-009@0.35) and [ADR-026: Provider Middleware Pipeline](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-026@0.35) updated: streaming is no longer a documented bypass. ## Alternatives considered - **Route the generator through the existing pipeline.** Rejected: generator laziness makes every middleware fire against a not-yet-started stream (see Context). Budget, usage, and telemetry would all be wrong. - **Change the provider yield type to carry a final usage object** (`Generator`). Rejected: the `string` yield type is part of the public capability contract; every consumer would have to defend against a non-string chunk. Real usage belongs on the generator *return* value or a side channel, tracked as the follow-up above. - **Eager drain inside the manager, then hand back a plain array.** Rejected: it defeats the entire point of streaming (memory efficiency, real-time output) and would change the public `Generator` return type. ## References - [ADR-009: Streaming Implementation](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-009@0.35) — Streaming Implementation (the generator mechanism this lifecycle wraps). - [ADR-025: Per-User AI Budgets](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-025@0.35) — Per-User AI Budgets (the pre-flight gate). - [ADR-026: Provider Middleware Pipeline](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-026@0.35) — Provider Middleware Pipeline (why streaming could not use it, now no longer a sanctioned bypass). - [ADR-058: Telemetry Middleware](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-058@0.35) — Telemetry Middleware (the row shape and attribution model streaming reuses).