Crawlers
The extension uses crawlers to visit all URLs configured for cache warmup. While visiting a URL, the appropriate page cache gets warmed. Learn more about which crawlers are available by default and how to implement a custom crawler on this page.
- interface Crawler
-
- Fully qualified name
-
\Elias
Haeussler\ Cache Warmup\ Crawler\ Crawler
Interface for crawlers used to crawl and warm up URLs.
Default crawlers
The extension ships with two default crawlers:
\Elias
: Used for cache warmup triggered within the TYPO3 backendHaeussler\ Typo3Warming\ Crawler\ Concurrent User Agent Crawler \Elias
: Used for cache warmup executed from the command-lineHaeussler\ Typo3Warming\ Crawler\ Outputting User Agent Crawler
Both crawlers use a custom User-
header for all warmup
requests. By using this custom header, it is possible to exclude
warmup requests from the statistics of analysis tools, for example.
The header is generated by a HMAC hash of the string
TYPO3/
.
The generated header value can be copied form the cache warmup modal
in the TYPO3 backend. Alternatively, a command warming:
is available which can be used to read the current User-
header.
Implement a custom crawler
Available interfaces
The actual cache warmup is done via the library
eliashaeussler/cache-warmup.
It provides \Elias
,
which must be implemented when developing your own crawler. There is
also a \Elias
that redirects user-oriented output to an instance of
\Symfony\
.
Configurable crawlers
Custom crawlers can also implement
\Elias
,
allowing users to configure warmup requests themselves.
See also
Feature #59 - Introduce configurable crawlers
of eliashaeussler/
library
Logging crawlers
Crawling results can be logged using a dedicated PSR-3 logger. For this, crawlers
must implement \Elias
and inject an appropriate PSR-3 logger. In TYPO3 context, this is mostly done using
TYPO3's log manager. Read more about logging in the official documentation.
See also
Feature #271 - Introduce support for PSR-3 loggers
of eliashaeussler/
library
Stoppable crawlers
Crawlers implementing \Elias
may cancel a cache warmup prematurely if any crawling failure occurs. This can be
especially useful for validation purposes to check whether any page within an XML
sitemap is inaccessible or failing.
See also
Feature #302 - Introduce stoppable crawler and --stop-on-failure option
of eliashaeussler/
library
Streamable crawlers
When running cache warmup from the TYPO3 backend, the current crawling progress is
streamed to the cache warmup progress modal. However, this is only supported for
crawlers implementing \Elias
.
Those crawlers will then get a \Elias
injected. It can be used to send events to the current event stream. The following
events are currently available:
\Elias
Haeussler\ Typo3Warming\ Http\ Message\ Event\ Warmup Finished Event \Elias
Haeussler\ Typo3Warming\ Http\ Message\ Event\ Warmup Progress Event
By default, when implementing a streamable crawler, there's no need to trigger these
events by your own. Instead, it's better to use the provided
\Elias
which takes care of sending appropriate events.
Steps to implement a new crawler
-
Create a new crawler
The new crawler must implement at least one of the following interfaces:
\Elias
Haeussler\ Cache Warmup\ Crawler\ Crawler \Elias
Haeussler\ Cache Warmup\ Crawler\ Configurable Crawler \Elias
Haeussler\ Cache Warmup\ Crawler\ Logging Crawler \Elias
Haeussler\ Cache Warmup\ Crawler\ Stoppable Crawler \Elias
Haeussler\ Cache Warmup\ Crawler\ Verbose Crawler \Elias
Haeussler\ Typo3Warming\ Crawler\ Streamable Crawler
-
Configure the new crawler
Add the new crawler to the extension configuration. Note that you should configure either the
crawler
orverbose
option, depending on what interface you have implemented.Crawler -
Flush system caches
Finally, flush all system caches to ensure the correct crawler class is used for further cache warmup requests.