ADR-202: Reading a public web page through an address guard 

Context 

In the demo chats (NEXT-165) users asked the backend assistant five times to read or analyse a web page on another site. No tool could. probe_url reaches only the instance's own site hosts (ADR-044), and site_fetch_source returns only hits of the site's own search (ADR-049).

Network egress is declared per tool group and fails closed (ADR-061 item 6). Two positive scopes existed: own_site for system and configured_endpoint for rag (ADR-093). ADR-061 states that no free-form "any host" scope exists, so a new or mis-declared group cannot egress to an arbitrary target.

A tool that fetches a URL the model chooses is the textbook server-side request forgery surface: the model is steerable by the pages and skills it reads, and the request originates inside the operator's network, next to databases, caches and the cloud metadata service.

nr-vault already guards its HTTP client against this (SecureHttpClientFactory: private ranges, IPv4-mapped and transition IPv6 forms, legacy numeric host forms, a CURLOPT_RESOLVE pin per request). Two properties of it do not fit a model-chosen URL, measured against the resolved dependency, nr-vault 0.16.0:

• Its public check, isHostAllowed(), treats a literal entry in $GLOBALS['TYPO3_CONF_VARS']['HTTP']['allowed_hosts'] as an opt-in past the private-range block. That list exists for credentialed service calls; an operator who put an internal vault host on it did not decide that a model may fetch it.
• A host name its dns_get_record() lookup does not answer is passed through without a pin, and curl then resolves it with getaddrinfo(), which also reads /etc/hosts. db or host.docker.internal exist only there, and point inside the network.

Decision 

fetch_external_url reads ONE http(s) page and returns its readable text.

1. Its own group, its own scope. The tool is the only member of a new group web, declared external_filtered — a new ToolEgressScope case. system keeps own_site; putting the tool there would have widened probe_url's egress, since the scope is per group. The own-site and configured-endpoint resolvers of EgressPolicyService refuse the new scope explicitly.

2. An address guard in front of every request. ExternalUrlGuard checks the first URL and every redirect target, in this order: the group's scope; http(s), a host, no userinfo; an ASCII host name or IP literal; not a host of this installation (every site base and base variant, through SiteFinder); the operator's denylist; the operator's allowlist; the default port unless the allowlist names host:port; no HTTP proxy (item 10); no legacy numeric host form; and every address the host resolves to through DNS must be public (IpAddressClassifier) and not on the denylist. A host that does not resolve through DNS is refused rather than handed to curl.

   The classifier restates nr-vault's ranges (plus the documentation ranges, site-local IPv6 and the local-use NAT64 prefix 64:ff9b:1::/48, which nr-vault 1.0.0 does not refuse, fixed in nr-vault#388) instead of calling isHostAllowed(), for the reason in the context. Both NAT64 prefixes are refused whole rather than by their embedded IPv4, because the embedding position of the local-use prefix depends on a prefix length the address does not carry. The lists narrow and never widen: an allow-listed host that resolves to a private address is refused.

   List entries are host names or addresses. A name entry matches the host name of a URL; an address entry (IPv4, bare or bracketed IPv6, optionally with a port) matches an IP literal in a URL and, compared as a packed address, every address a host name resolves to — so the denylist cannot be stepped around by naming the same server differently.

   The request comes from the installation's own address. Services that trust that address (a partner API allow-listing the server, an intranet reached through a public address) are not private ranges and the guard cannot see them; they belong on the denylist, or the tool on an allowlist.

3. The connection is pinned to checked addresses. The request carries a CURLOPT_RESOLVE entry with the addresses the guard resolved and checked, so curl never resolves the host itself and a DNS answer that changes between check and connect (DNS rebinding) has nothing to act on. The client is nr-vault's, so its own per-request check runs underneath as a second layer, with the instance's TLS settings. When nr-vault's lookup answers, its middleware appends a pin of its own, and curl uses the LAST entry for a host and port — so the effective pin is usually nr-vault's. Its addresses passed nr-vault's range check; the guard's addresses are what curl uses when nr-vault's lookup does not answer. Either way curl connects only to an address one of the two checks accepted and never resolves the name itself. Without ext-curl there is no pin (Guzzle's stream handler ignores CURLOPT_RESOLVE), and the tool refuses to fetch.
4. Redirects are followed by the tool, one hop at a time, at most three; allow_redirects is off on every request. Each hop passes the whole guard and gets its own pin.
5. Bounded in bytes, characters and time. A response sink accepts at most 2 MiB and then reports a short write, which makes curl abort the transfer — the limit bounds what is downloaded, not only what is kept. The HTML parser (masterminds/html5, quadratic in the nesting depth: 40,000 nested <div> take about 40 s) gets at most 256 KiB, and markup a linear pre-scan finds nested deeper than 200 levels is not parsed at all but reduced to its text; the renderer's recursion stops at the same depth. The text returned is cut at 20,000 characters, and further in bytes so the whole result stays under 48,000 bytes: the loop's ToolResultBounder cuts results over 50,000 bytes at the tail, and the tail is the END marker. The connect timeout is 5 s and the whole fetch, redirects included, 20 s; the budget is checked before every guard call (its DNS lookup takes time), after it, and after the transfer before parsing. dns_get_record() takes no timeout of its own and ignores RES_OPTIONS; it honours the options timeout:N attempts:N line of /etc/resolv.conf, which is where an operator bounds one lookup. A successful answer that is not HTML or plain text is refused at the headers, before its body is read.

6. Marked as untrusted input. The text sits between <<<BEGIN/END UNTRUSTED EXTERNAL WEB CONTENT …>>> markers that carry a random nonce per call, under a preamble saying it is reference material, not instructions — the shape ADR-061 item 4 gave skill bodies. A page cannot know the nonce of the marker that closes its fence, and anything in it that looks like a marker (any case, any spacing, with or without a nonce) is defused. The data class is publicContent (the web group default), for the reason rag has it: the page is world-readable. The data class measures sensitivity, not trust; trust is what the markers carry.

   Outside the fence — in error results and the status line — no server-chosen text is echoed: an HTTP error reports the status code only, a media type only when it is a well-formed type/subtype of at most 64 characters, and a redirect target cut to 500 characters and labelled as an untrusted URL the server chose.

7. Disabled by default, not admin-only. isEnabledByDefault() is false: this is the first built-in that reaches hosts nobody declared, and the model-chosen URL itself leaves the installation. That follows how the repository ships a new kind of capability — the writers (ADR-135) and repository skills arrive disabled. requiresAdmin() is false: the admin tier guards system, host and cross-user data, and a public page is none of these; the users who asked for the tool are editors.
8. Host lists are extension settings, as the generic creator's denied tables are (ADR-197): tools.fetchExternalUrl.allowedHosts and .deniedHosts. Settings that cannot be read refuse every fetch.
9. Every call waits for a human approval by default. The URL itself can carry data out — a run steered by injected text can put what it has seen into a query string, and no address check can tell that from a search. So the tool implements the new ConfigurableApprovalInterface and asks for approval unless the operator sets tools.fetchExternalUrl.skipApprovalWithAllowlist AND a non-empty allowlist; the allowlist is what bounds where the data could go. The approval card's preview shows the host and the full query string. The interface is not RequiresApprovalInterface, which cannot be conditional, and not RemoteApprovalInterface, which is reserved for remote tools (ADR-134). ToolApprovalRule asks the declared write effect first, so the interface cannot lift the approval of a write, and ADR-134's "a write-without-approval builtin is not expressible" holds.
10. No proxy, unless the operator permits it with an allowlist. When the TYPO3 HTTP proxy setting, or HTTPS_PROXY (and HTTP_PROXY on the CLI) with NO_PROXY as the exclusion list, applies to a request, the proxy resolves the host and the pin does not reach the connection. The fetch is refused then, unless tools.fetchExternalUrl.allowViaProxy is set together with a non-empty allowlist. The address check before the request still runs in that case, but it no longer decides where the proxy connects.

What this does, what it does not, what proves it 

Does: fetch one public page per call, text only, with the limits above.

Does not: render JavaScript, submit forms, send cookies or credentials, read PDFs or other binary formats, or fetch hosts of this installation (those stay with probe_url). Internationalised host names are refused rather than converted, because the pin is keyed by the host string curl sees; the xn-- form works.

Proven by: Tests/Unit/Service/Tool/Web/IpAddressClassifierTest (IPv4, IPv6, IPv4-mapped, 6to4, Teredo, both NAT64 prefixes, decimal/octal/hex host forms, and one-directional agreement with nr-vault), ExternalUrlGuardTest (schemes, credentials, ports, own site hosts, proxies, name and address list entries including IPv6, a list that cannot readmit a private address, unresolvable and privately resolving hosts), Tests/Unit/Service/Tool/Builtin/FetchExternalUrlToolTest against a scripted transport (approval, preview, the pin, no pin without curl, redirect to a private address and to a privately resolving host, the redirect limit, the time budget before lookups and before parsing, the byte and character caps, the nonce fence, unechoed server text), HtmlTextExtractorTest (parse cap, deep nesting), ToolApprovalRuleTest (a configurable approval cannot lift a write), and Tests/Functional/Service/Tool/FetchExternalUrlToolTest for the container wiring.

Consequences 

• ● Editors can have a page on another site read and analysed, which the demo chats asked for five times.
• ● Without a proxy, no private, loopback, link-local, CGNAT or metadata address is reachable through the tool — by literal, by DNS answer, by redirect or by a DNS answer that changes before the connect.
• ● Nothing leaves without a human seeing the URL, unless the operator chose an allowlist and switched the approval off.
• ● probe_url and every other group keep the egress they had.
• ◐ The ranges live in two places, nr-vault and IpAddressClassifier. A test pins one direction: every address of its sample that nr-vault refuses is refused here too, so a range nr-vault adds and this side lacks turns it red if the sample carries an address of it.
• ◐ Through a permitted proxy the address guarantees above do not hold: the proxy resolves and connects, and only the allowlist bounds where it goes.
• ◐ Hosts that exist only in /etc/hosts or other non-DNS sources are unreachable, on purpose.
• ◐ One DNS lookup is bounded only by the system resolver's settings.
• ○ An approver has to read the query string. With approval switched off the allowlist is the only bound on where data can go; the tool cannot tell an exfiltrating query string from a search.

Alternatives considered 

• The tool in ``system``, beside ``probe_url``. Rejected: egress is per group, so system would have needed the new scope and probe_url would have gained it.
• nr-vault's ``isHostAllowed()`` as the only check. Rejected for the two properties in the context: the allowed_hosts opt-in and the unpinned path for names DNS does not answer.
• Enabled by default. Rejected: an operator should decide once that the installation may fetch from the internet on a model's behalf, and whether to restrict it to a list.
• A readability library (Readability.php and its ports). Rejected: a sizeable dependency for a TYPO3 extension. The HTML5 parser TYPO3 already ships (masterminds/html5, now declared directly) plus dropping the page chrome gives the model the main text.
• ``RequiresApprovalInterface`` with no way out. Rejected: an operator who restricted the tool to a handful of documentation hosts has bounded where data can go, and an approval per page read would make the tool unusable for the editors who asked for it.
• ``Dom\HTMLDocument`` (lexbor) on PHP 8.4+. Not used: it is a different DOM API from the one PHP 8.2 and 8.3 offer, so it would mean a second renderer for half the support matrix. The size cap and the depth pre-scan bound masterminds/html5 instead (about 0.7 s for 256 KiB of the worst shape it still parses).