ADR-062: Streaming Request Lifecycle 

Status

Accepted

Date

2026-07

Authors

Netresearch DTT GmbH

Context 

Every non-streaming provider call goes through the middleware pipeline (ADR-026: Provider Middleware Pipeline) and so inherits budget pre-flight (ADR-025: Per-User AI Budgets), usage accounting, and telemetry (ADR-058: Telemetry Middleware). Streaming did not: both LlmServiceManager::streamChat() and streamChatWithConfiguration() called the provider's streamChatCompletion() generator directly and returned it to the caller.

The consequence was a live budget hole. A streamed chat:

  • ran no budget pre-flight — an over-budget user could stream freely;
  • produced no usage row in tx_nrllm_service_usage, so streamed tokens were invisible to cost dashboards and to the very budget aggregate that is supposed to gate the next call;
  • produced no telemetry row in tx_nrllm_telemetry, so a streamed call had no latency, no outcome, and no correlation id on record;
  • had no fallback, not even in the window where one is still possible.

The pipeline cannot simply be reused. A PHP generator is lazy: calling streamChatCompletion() returns a suspended generator without running a line of it. Wrapping that as the pipeline terminal makes every middleware run against a stream that has not started — Budget would gate nothing meaningful, UsageMiddleware (which records after the terminal returns) would record zero tokens, and TelemetryMiddleware would measure a near-zero latency. This laziness is exactly why ADR-026: Provider Middleware Pipeline sanctioned streaming as a documented pipeline bypass.

Decision 

Introduce a dedicated streaming lifecycle rather than forcing streams through the response pipeline. A single private collaborator, NetresearchNrLlmServiceStreamingStreamingDispatcher, owns it; the manager's two streaming methods build an opener closure and hand off to it. The dispatcher is a wrapping generator — it never changes the public Generator<int, string, mixed, void> contract the seven providers and their consumers rely on.

The lifecycle has four stages:

  1. Budget pre-flight — eager. StreamingDispatcher::stream() runs the same BudgetService::check() gate as BudgetMiddleware before it returns a generator, so an over-budget caller is rejected at call time with a typed BudgetExceededException — not lazily on first iteration. This is the one part that must be eager: a caller that never drains the stream is never charged, but a caller that is already over budget must never receive a stream to drain.
  2. Provider selection with fallback — before the first chunk only. The dispatcher walks the primary configuration's fallback chain (shallow, exactly as FallbackMiddleware does) and primes each candidate generator (rewind()) inside a try/catch. Priming runs the provider up to its first yield — the HTTP request and first delta — which is the last moment a provider can still be swapped. A retryable failure here (ProviderConnectionException, or a 429 ProviderResponseException) moves to the next candidate; a non-retryable failure bubbles up unchanged. Once a chunk has been handed to the caller a swap is impossible, so fallback stops at the first chunk. That single asymmetry with the non-streaming pipeline is intrinsic to streaming, not a shortcut.
  3. Drain accounting — lazy. As the caller drains, the wrapper re-yields each chunk, appends it to a completion buffer, and stamps the time-to-first-token on the first chunk delivered.
  4. Settlement — in a ``finally``. Usage and telemetry are written in the drain generator's finally block, so they land on every exit path: normal completion, a mid-stream exception, and an abandoned generator (client disconnect or a consumer break). PHP runs a suspended generator's finally when the generator is destroyed, which is what makes early-break accounting work without a caller callback. An abandoned stream therefore records the partial tokens actually produced, never zero and never the full amount.

Token counts are estimated 

The seven streaming adapters yield only text deltas and return void — none emits a usage frame at stream end. Real per-token usage is therefore not available on this path today. The dispatcher estimates it with the ≈4 chars/token heuristic already used by RenderedPrompt::estimateTokens(): the caller passes the prompt character count on the context metadata (it holds the messages; the dispatcher never sees the payload, honouring the ADR-026: Provider Middleware Pipeline "context carries no payload" rule) and the dispatcher counts the drained completion text.

Recording an estimate is a deliberate improvement over the previous state, which recorded nothing at all — for budget enforcement an approximate figure is far better than a silent zero. Exact stream usage would require enabling provider-level usage frames (OpenAI stream_options.include_usage, Anthropic message_delta usage, …) and threading them out of the generator without breaking its string yield type — a follow-up, tracked separately, that would replace the estimate with the reported figure where a provider supports it.

Attribution 

Mirroring the non-streaming split (ADR-058: Telemetry Middleware):

  • Telemetry names the requested primary configuration; a pre-first-chunk swap shows as fallback_attempts > 0. cache_hit is always false — streaming never caches. A new nullable time_to_first_token_ms column carries the TTFT; it is NULL for every non-streaming row (there is no partial-response milestone to measure), deliberately distinct from a real 0 ms.
  • Usage attributes to the configuration that actually served — after a fallback swap that is the fallback's provider/model/uid, not the primary's. For an ad-hoc stream (a pinned provider with no configuration entity) the served provider comes from the context metadata the manager sets.

Scope 

  • New StreamingDispatcher (private service, autowired) and the LlmServiceManager wiring for both streaming entry points.
  • streamChatWithConfiguration() gains a trailing array $metadata = [] parameter so streamed calls can carry budget attribution; it is additive and the three-argument callers stay source-compatible. LlmServiceManagerInterface — a Category-1 public API — keeps its Generator return contract, so this is backward compatible.
  • tx_nrllm_telemetry gains the nullable time_to_first_token_ms column and TelemetryRecord a matching trailing nullable field.
  • ADR-009: Streaming Implementation and ADR-026: Provider Middleware Pipeline updated: streaming is no longer a documented bypass.

Alternatives considered 

  • Route the generator through the existing pipeline. Rejected: generator laziness makes every middleware fire against a not-yet-started stream (see Context). Budget, usage, and telemetry would all be wrong.
  • Change the provider yield type to carry a final usage object (Generator<int, string|UsageStatistics, …>). Rejected: the string yield type is part of the public capability contract; every consumer would have to defend against a non-string chunk. Real usage belongs on the generator return value or a side channel, tracked as the follow-up above.
  • Eager drain inside the manager, then hand back a plain array. Rejected: it defeats the entire point of streaming (memory efficiency, real-time output) and would change the public Generator return type.

References