Crawlers

The extension uses crawlers to visit all URLs configured for cache warmup. While visiting a URL, the appropriate page cache gets warmed. Learn more about which crawlers are available by default and how to implement a custom crawler on this page.

interface EliasHaeussler\CacheWarmup\Crawler\CrawlerInterface

Interface for crawlers used to crawl and warm up URLs.

crawl($urls)

Crawl a given list of URLs.

Parameters

  • $urls (array) -- List of URLs to be crawled.

getSuccessfulUrls()

Return all successfully crawled URLs.

Returns

A list of EliasHaeussler\CacheWarmup\CrawlingState instances

getFailedUrls()

Return all failing crawled URLs.

Returns

A list of EliasHaeussler\CacheWarmup\CrawlingState instances

Default crawlers

The extension ships with two default crawlers:

  • EliasHaeussler\Typo3Warming\Crawler\ConcurrentUserAgentCrawler: Used for cache warmup triggered within the TYPO3 backend

  • EliasHaeussler\Typo3Warming\Crawler\OutputtingUserAgentCrawler: Used for cache warmup executed from the command-line

Both crawlers use a custom User-Agent header for all warmup requests. By using this custom header, it is possible to exclude warmup requests from the statistics of analysis tools, for example. The header is generated by a HMAC hash of the string TYPO3/tx_warming_crawler.

The generated header value can be copied in the dropdown of the toolbar item in the backend. Alternatively, a command warming:showuseragent is available which can be used to read the current User-Agent header.

Implement a custom crawler

Available interfaces

The actual cache warmup is done via the library eliashaeussler/cache-warmup. It provides the EliasHaeussler\CacheWarmup\Crawler\CrawlerInterface, which must be implemented when developing your own crawler. There is also a EliasHaeussler\CacheWarmup\Crawler\VerboseCrawlerInterface that redirects user-oriented output to an instance of Symfony\Component\Console\Output\OutputInterface.

Configurable crawlers

Since version 0.7.13 of eliashaeussler/cache-warmup, custom crawlers can also implement the EliasHaeussler\CacheWarmup\Crawler\ConfigurableCrawlerInterface, allowing users to configure warmup requests themselves.

See also

Feature #59 - Introduce configurable crawlers of eliashaeussler/cache-warmup library

Steps to implement a new crawler

  1. Create a new crawler

    The new crawler must implement one of the following interfaces:

    Tip

    This extension provides some additional traits which you can use for your new crawler:

    • EliasHaeussler\Typo3Warming\Crawler\RequestAwareTrait

    • EliasHaeussler\Typo3Warming\Crawler\UseAgentTrait

  2. Configure the new crawler

    Add the new crawler to the extension configuration. Note that you should configure either the crawler or verboseCrawler option, depending on what interface you have implemented.

  3. Flush system caches

    Finally, flush all system caches to ensure the correct crawler class is used for further cache warmup requests.