ADR-009: Streaming Implementation 

Status 

Accepted (2024-03)

Context 

Streaming responses provide:

  • Better UX for long responses.
  • Lower time-to-first-token.
  • Real-time feedback.

Decision 

Use PHP Generators for streaming:

Example: Streaming chat responses 
public function streamChat(array $messages, array $options = []): Generator
{
    $response = $this->sendStreamingRequest($messages, $options);

    foreach ($this->parseSSE($response) as $chunk) {
        yield $chunk;
    }
}

// Usage
foreach ($llmManager->streamChat($messages) as $chunk) {
    echo $chunk;
    flush();
}
Copied!

Implementation details:

  • Server-Sent Events (SSE) parsing.
  • Chunked transfer encoding.
  • Memory-efficient iteration.
  • Provider-specific adaptations.

Consequences 

Positive:

  • ●● Memory efficient.
  • ● Natural iteration syntax.
  • ●● Real-time output.
  • ◐ Works with output buffering.

Negative:

  • ✕ No response object until complete.
  • ◑ Error handling complexity.
  • ◑ Connection management.
  • ✕ No caching possible.

Net Score: +3.5 (Positive impact - streaming UX benefits outweigh implementation complexity)

Update (2026-07): streaming is in the request lifecycle 

The generator mechanism decided here is unchanged, but streamed calls no longer sidestep the cross-cutting concerns the non-streaming path enforces. Originally a streamed call ran no budget pre-flight and produced neither a usage nor a telemetry row — a live budget hole.

Streamed calls now run through a dedicated streaming lifecycle (ADR-062: Streaming Request Lifecycle): an eager budget pre-flight before the first chunk, pre-first-chunk provider fallback, time-to-first-token measurement, and a finally that records usage (tx_nrllm_service_usage) and telemetry (tx_nrllm_telemetry) on every exit — completion, exception, or an abandoned generator. The lifecycle wraps the generator; the public Generator<int, string, mixed, void> contract is unchanged.

Two of the "Negative" consequences above are now bounded rather than open: "error handling complexity" and "connection management" are handled once, in the lifecycle, instead of at each call site. "No response object until complete" and "no caching possible" remain intrinsic to streaming. Token usage on this path is estimated (providers expose no usage frame mid-stream); see ADR-062: Streaming Request Lifecycle for the rationale and the follow-up toward exact figures.