Streaming support

Streaming allows you to receive LLM responses incrementally as they are generated, rather than waiting for the complete response. This improves perceived performance for long responses.

Streaming: the prompt is screened, the model and adapter are resolved, the budget is checked, the dispatcher opens the provider stream with fallback, and each chunk passes a sliding redaction window before the Generator yields it to the caller. — Request path down the left, chunk path back up the right. Redaction happens per chunk through a sliding window, so a secret split across two chunks is still caught without buffering the whole response.

Usage

Example: Streaming chat responses

$stream = $this->llmManager->streamChat($messages);

foreach ($stream as $chunk) {
    echo $chunk;
    ob_flush();
    flush();
}

The streamChat method returns a Generator that yields string chunks as the provider generates them. Each chunk contains a portion of the response text.

Audit and accounting

After a successful stream, the dispatcher screens at most the first 50,000 raw response bytes for its guardrail audit. A large provider chunk cannot extend this audit buffer. The audit cannot retract text already delivered; live redaction uses its separate sliding window.

The audit limit does not truncate delivered chunks or consumption accounting. Streaming token estimates count the complete raw output, including bytes beyond that prefix and bytes masked by live redaction.

Providers that implement StreamingCapableInterface support streaming. Check provider capabilities before using:

Example: Checking streaming support

use Netresearch\NrLlm\Provider\Contract\StreamingCapableInterface;

$provider = $this->llmManager->getProvider('openai');
if ($provider instanceof StreamingCapableInterface) {
    // Provider supports streaming
}

Transport failures

The bundled adapters' streaming methods send each request once. They propagate an exception raised by the PSR-18 transport without wrapping it as an nr_llm exception. In particular, Psr\Http\Client\NetworkExceptionInterface does not implement nr_llm's marker merely because it passed through an adapter. This differs from the buffered adapters' transport-error normalization.

The manager's streaming dispatcher classifies a PSR-18 network exception as a retryable connection failure. It can try the configured fallback before the first chunk is yielded. After output has begun, it propagates the failure; switching providers would combine two different answers. The dispatcher also propagates the last retryable failure when no fallback can serve the call. Callers that handle streaming transport failures should catch the PSR-18 network interface in addition to their nr_llm exception cases.