---
title: "ADR-240: The text of a tool round is streamed"
manual: "LLM Foundation"
version: "0.43"
permalink: "https://docs.typo3.org/permalink/netresearch/nr-llm:adr-240@0.43"
source: "Adr/Adr240TheTextOfAToolRoundIsStreamed.rst"
rendered: "2026-11-04T13:45:29+00:00"
---

# ADR-240: The text of a tool round is streamed {#adr-240}

-   *Status:* Accepted
-   *Date:* 2026-11-01
-   *Amends:* [ADR-062](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-062@0.43) (a tool call can stream too), [ADR-088](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-088@0.43) (an unbroken run waits)
-   *Authors:* Netresearch DTT GmbH

## Context {#context}

A model that may call tools answers through
`LlmServiceManager::chatWithToolsForConfiguration()`, and that call is
blocking: a reader sees nothing until the round is over. Over 224 calls on the
demo the time to the first token was recorded for none. The streaming path of
[ADR-062](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-062@0.43) is no help. It runs a generator of text outside the
middleware pipeline, knows no tool calls, reports estimated tokens, and on OpenAI
it only reaches Chat Completions, where a GPT-6 model takes no tools with its
reasoning on ([ADR-203](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-203@0.43)).

## Decision {#decision}

**The call stays blocking; the text leaves earlier.** A new contract,
`ToolStreamingCapableInterface`, has one method beside
`chatCompletionWithTools()`: the same arguments and a closure for the text,
returning the same `CompletionResponse`. So the pipeline stays what it is.
Budget, fallback, usage with real token counts, telemetry and the output
guardrails all run on the complete response, and nothing downstream can tell the
two calls apart. On OpenAI's Responses transport, the event that ends the stream
carries the `response` object of the blocking call, and one parser reads both.

**A caller asks by passing an observer.** `ToolOptions::withTextSink()` lives
for one call and is not part of `toArray()`, so it is not stored with a
suspended run. `ToolLoopService` sets it when the trace has a receiver, and
`AgentRunExecutor` gives the trace one when the caller of
`AgentRuntimeInterface::run()`, `approve()` or `submitInput()`
passes `onStep`. The observer then also receives steps of the kinds
`RunStep::KIND_TEXT_DELTA` and `RunStep::KIND_TEXT_RESTART`. They are no
records: they are not collected and not persisted, and
`RunStep::kinds()` does not list them. The llm step keeps the round's text.

**Text passes the guardrails on the way.** `StreamingDispatcher::toolTextRelay()`
decides per call. With a guardrail that holds a stream back
([ADR-238](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-238@0.43), for example a term list on this configuration in a
loop outside a run; its amendment leaves a run that offers tools streaming) the
round is not streamed at all: the caller gets the text after the verdict. Without
one, the text streams through the live redaction of
[ADR-088](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-088@0.43), with a smaller rescan block than a plain stream, because
a reader is waiting for it. Its window holds a run of text without whitespace back
until the run ends, so a bearer token quoted by the model is masked whole. The
output guardrails then screen the complete
response as always; one that refuses it fails the call after the text was seen,
as the end-of-stream audit of [ADR-086](https://docs.typo3.org/permalink/netresearch/nr-llm:adr-086@0.43) does for a plain stream.

**A second attempt says so.** The fallback chain, or a guardrail's retry, can send
a call twice in one pipeline run, blocking or streaming. The relay tells the sink
to drop what it holds before the second attempt, and the requested configuration
decides about the whole run, because its guardrails screen the response whatever
configuration of the chain serves it.

**Failures stay on the known path.** A connection or a secret that fails before the
stream has opened goes blocking, with the retries of `sendRequest()`, as does
a provider that answers the request for a stream with an error, and so
does an overload the provider reports inside the stream before the first text
(`rate_limit_exceeded`, `server_error`, which arrive after HTTP 200 and would
otherwise be nothing the pipeline retries). A timeout is not sent again, as on the
blocking path. Anything after the stream opened stands, and the pipeline's fallback
restarts the round as above.

**The time to the first text is telemetry.** The relay measures it and it lands in
`time_to_first_token_ms`, which the Verlauf of a run shows as `ttftMs`.

## Consequences {#consequences}

-   ● The first text of a chat answer no longer waits for the end of the round.
-   ● Usage on this path is what the provider reports, unlike the estimates of a
    plain stream.
-   ◑ Only OpenAI's Responses transport implements the contract. Chat Completions
    and every other provider send blocking, as before; a provider joins by
    implementing the interface.
-   ◑ The redaction window holds back 128 bytes, so the text a reader sees trails
    the model by that much, and an answer shorter than that arrives with the end of
    the round. Text without spaces, such as Japanese, arrives in larger pieces, line
    by line or every 4 KB, because a run without whitespace waits.
-   ◑ A round whose answer a non-holding guardrail then refuses has been read. A
    guardrail that must prevent that holds the stream.
-   ◑ A cache hit delivers no pieces; the text comes with the response.
-   ◑ `openai-processing-ms` says nothing on a stream, because the header leaves
    before the model has written; a streamed round has no provider processing time.