Cache warmup of pages located in XML sitemaps

Console commands
Backend toolbar
Using the API
Custom crawlers

Extension key: warming
Package name: eliashaeussler/typo3-warming
Version: main
Language: en
Author: Elias Häußler & contributors
License: This extension documentation is published under the CC BY-NC-SA 4.0 (Creative Commons) license.

An extension for TYPO3 CMS that warms up Frontend caches based on an XML sitemap. Cache warmup can be triggered via TYPO3 backend or using a console command. It supports multiple languages and custom crawler implementations.

Introduction

A quick overview about the main features provided by this extension.

Learn more about this extension

Installation

Instructions on how to install this extension and which TYPO3 and PHP versions are currently supported.

Getting started

Configuration

Learn how to configure the extension in various ways. This includes extension configuration, site configuration and TypoScript configuration.

View configuration options

Usage

This section describes all possible usages of this extension. Learn how to use it in the Backend, as console command or using the PHP API.

Learn how to use this extension

Developer corner

A quick overview about all relevant classes provided by this extension.

Deep dive into classes & concepts

Migration

Required migration steps when upgrading the extension to a new major version.

View upgrade guide

Introduction

What does it do?

The extension provides a service to warm up Frontend caches based on XML sitemaps. Cache warmup can be triggered in various ways:

From the TYPO3 backend
Using a console command
Directly with the PHP API

It supports multiple languages and custom crawler implementations. Under the hood, the extension makes use of the eliashaeussler/cache-warmup library which provides the core cache warmup implementation. In addition, EXT:sitemap_locator is used to locate XML sitemaps.

Features

Frontend cache warmup of pages located in XML sitemaps
Integration in TYPO3 backend toolbar and page tree
Support of various sitemap providers (e.g. robots.txt or custom location)
Multi-language support of configured sites
Support for custom crawlers and crawling strategies
Console commands
Compatible with TYPO3 13.4 LTS and 14.2 (see version matrix)

Support

There are several ways to get support for this extension:

Security Policy

Please read our security policy if you discover a security vulnerability in this extension.

License

This extension is licensed under GNU General Public License 2.0 (or later).

Installation

Requirements

PHP 8.2 - 8.5
TYPO3 13.4 LTS - 14.2

Installation

Require the extension via Composer (recommended):

composer require eliashaeussler/typo3-warming

Or download it from the TYPO3 extension repository.

Version matrix

Extension versions	TYPO3 versions	eliashaeussler/cache-warmup versions	PHP versions
5.x	13.4 LTS - 14.2	5.x	8.2 - 8.5
4.x	12.4 LTS - 13.4 LTS	4.x	8.2 - 8.4
3.x	12.4 LTS - 13.4 LTS	3.x	8.1 - 8.4
2.x	12.4 LTS - 13.3	2.x	8.1 - 8.3
1.x	12.4 LTS	2.x	8.1 - 8.3
0.5.x	10.4 LTS - 11.5 LTS	0.5.0 - 0.8.x	7.4 - 8.3

Configuration

The extension can be configured in many ways. Especially the behavior during the cache warmup can be configured extensively. For example, it is possible to limit the maximum number of crawled pages or even use your own crawler implementations.

Learn what configuration options are available on the following pages:

Extension configuration

The extension currently provides the following configuration options:

crawler
crawlerOptions
verboseCrawler
verboseCrawlerOptions
parserOptions
clientOptions
limit
exclude
strategy
enablePageTree
supportedDoktypes
enableToolbar

Crawler

crawler

Type: string (FQCN)
Default: \EliasHaeussler\Typo3Warming\Crawler\ConcurrentUserAgentCrawler

Default crawler to be used for crawling the requested pages.

Note

Custom crawlers must implement the \EliasHaeussler\CacheWarmup\Crawler\Crawler interface.

crawlerOptions

Type: string (JSON)

JSON-encoded string of custom crawler options for the default crawler. Applies only to crawlers implementing the \EliasHaeussler\CacheWarmup\Crawler\ConfigurableCrawler interface. For more information read Configurable crawlers.

Changed in version 4.0.0

Client-related configuration was moved to clientOptions in version 4.0.0.

verboseCrawler

Type: string (FQCN)
Default: \EliasHaeussler\Typo3Warming\Crawler\OutputtingUserAgentCrawler

Verbose crawler to be used for cache warmup from the command-line.

Note

Custom verbose crawlers must implement the \EliasHaeussler\CacheWarmup\Crawler\VerboseCrawler interface.

verboseCrawlerOptions

Type: string (JSON)

JSON-encoded string of custom crawler options for the verbose crawler. Applies only to crawlers implementing the \EliasHaeussler\CacheWarmup\Crawler\ConfigurableCrawler interface. For more information read Configurable crawlers.

Changed in version 4.0.0

Client-related configuration was moved to clientOptions in version 4.0.0.

Parser

parserOptions

Type: string (JSON)

New in version 1.2.0

Feature: #502 - Allow configuration of XML parser client options

Changed in version 4.0.0

This configuration was renamed from parserClientOptions to parserOptions in version 4.0.0. Migrate existing configuration to clientOptions.

JSON-encoded string of parser options used within the XML parser to parse XML sitemaps. For more information read Parser configuration.

Client

clientOptions

Type: string (JSON)

New in version 4.0.0

Feature: #844 - Introduce client options extension configuration

JSON-encoded string of options for the client used within the crawler and XML parser to parse and crawl XML sitemaps. All available Guzzle client options are accepted and merged with TYPO3's global client configuration stored in $GLOBALS['TYPO3_CONF_VARS']['HTTP'].

Tip

If the XML sitemap is protected by HTTP authentication (basic auth), you can set the credentials as follows: {"auth": ["<username>", "<password>"]}

In case the XML sitemap does not have a valid SSL certificate, it is possible to disable the SSL verification: {"verify": false}

You can also combine both settings: {"auth": ["<username>", "<password>"], "verify": false}

Options

limit

Type: integer
Default: 250

Allows to limit the number of crawled pages in one iteration.

Tip

Can be set to 0 to crawl all available pages in XML sitemaps.

exclude

Type: string (comma-separated list)

Comma-separated list of exclude patterns to exclude URLs from cache warmup. The following formats are currently supported:

Regular expressions with delimiter #, e.g. #(no_cache|no_warming)=1#
Any pattern processable by the native PHP function fnmatch, e.g. *no_cache=1*

strategy

Type: string

Name of an available crawling strategy to use for cache warmup. Crawling strategies are used to prepare URLs before actually crawling them. This can be helpful to prioritize crawling of important URLs.

See also

Page tree

enablePageTree

Type: boolean
Default: 1

Enable cache warmup in the page tree context menu. This setting affects all users, including administrators.

supportedDoktypes

Type: string (comma-separated list)
Default: 1

Comma-separated list of doktypes to be supported for cache warmup in the page tree context menu. Defaults to default pages with doktype 1 only. If your project implements custom doktypes, you can add them here to support cache warmup from the context menu.

Site configuration

Important

Caches can be warmed up only if the entry point of the site contains the full domain name.

Cache warmup is based on the configured sites in the TYPO3 installation. Therefore, in order to control the cache warmup behavior, the site configuration can be used.

Take a look at the site configuration of EXT:sitemap_locator which provides the XML sitemap location feature.

In addition, the following site configuration options are available:

warming_exclude (site)

Type: boolean
Path: warming_exclude
Default: 0

New in version 4.0.0

Feature: #793 - Allow to exclude sites and languages from warming

Enable or disable cache warmup for this site. If this is set to 1, the whole site will be excluded from cache warmup. By default, all sites are included.

Configuration of exclude checkbox within the Sites module

warming_exclude (site_language)

Type: boolean
Path: languages > (site language) > warming_exclude
Default: 0

New in version 4.0.0

Feature: #793 - Allow to exclude sites and languages from warming

Enable or disable cache warmup for this site language. If set to 1, only pages of this specific site language will be excluded from cache warmup.

See also

Take a look at Sitemap providers to learn how the extension internally evaluates the site configuration values to determine the path to the XML sitemap of a site.

TypoScript configuration

The following global TypoScript configuration is available:

Constants

view.templateRootPath

Type: string
Path: module.tx_warming

Additional path to template root used in Backend context. Within this path, Fluid templates can be overwritten.

See also

Read more in the official documentation.

view.partialRootPath

Type: string
Path: module.tx_warming

Additional path to template partials used in Backend context. Within this path, Fluid partials can be overwritten.

See also

Read more in the official documentation.

Crawler configuration

The following configuration applies only to the default crawlers shipped by this extension.

See also

If you want to learn more about configurable crawlers, read more in the developer corner.

Both default crawlers utilize the same crawler options that are described in the option reference of the underlying eliashaeussler/cache-warmup library.

In addition, the following crawler options are available:

perform_subrequests

Type: boolean
Default: 0

New in version 3.3.0

Feature: #730 - Introduce sub request handler

Enable sub request handler for cache warmup requests. This will significantly increase performance of the whole cache warmup progress, since no HTTP requests will be performed anymore. Instead, the bootstrapped frontend application is re-used for each cache warmup request.

Note

Only URLs matching the site's base URL are handled by the sub request handler. All other URLs (e.g. pages linking to external URLs) will be handled by Guzzle's default handler.

Parser configuration

The XML parser utilizes the same parser options that are described in the option reference of the underlying eliashaeussler/cache-warmup library.

Permissions

Note

Cache warmup can also be enabled or disabled globally. Check out the extension configuration for page tree and backend toolbar.

Administrators are able to run cache warmup for all available sites and pages. All other users are by default not allowed to run those tasks. Thus, they cannot see both the cache warmup toolbar item and context menu items.

However, you can use User TSconfig to allow cache warmup for specific users/usergroups and sites/pages. Add the following configuration to the options.cacheWarmup User TSconfig:

allowedSites

Type: string (comma-separated list)
Path: options.cacheWarmup.allowedSites

Provide a comma-separated list of site identifiers. Those sites can then be warmed by the backend user.

Example:

options.cacheWarmup.allowedSites = my-dummy-site,another-dummy-site

allowedPages

Type: string (comma-separated list)
Path: options.cacheWarmup.allowedPages

Provide a comma-separated list of pages. Those pages can then be warmed by the backend user. Pages can be suffixed by a + sign to recursively include all subpages.

Example:

options.cacheWarmup.allowedPages = 1,2,3+

See also

You might also want to check out WarmupPermissionGuard for the actual implementation of the permission check.

Logging

New in version 1.1.0

Feature: #374 - Introduce logging for default crawlers

When using the default crawlers provided by this extension, crawling results are logged using a configured log writer. By default, TYPO3 configures a file writer with minimum log level warning.

However, you can always override logging configuration within your config/system/settings.php or config/system/additional.php file:

config/system/settings.php

return [
    'LOG' => [
        'EliasHaeussler' => [
            'Typo3Warming' => [
                'Crawler' => [
                    // Default crawler
                    'ConcurrentUserAgentCrawler' => [
                        'writerConfiguration' => [
                            \Psr\Log\LogLevel::WARNING => [
                                \TYPO3\CMS\Core\Log\Writer\FileWriter::class => [
                                    // A. Disable logging
                                    'disabled' => true,

                                    // -or- B. Configure logger
                                    'logFileInfix' => 'warming',
                                ],
                            ],
                        ],
                    ],

                    // Verbose crawler
                    'OutputtingUserAgentCrawler' => [
                        'writerConfiguration' => [
                            \Psr\Log\LogLevel::WARNING => [
                                \TYPO3\CMS\Core\Log\Writer\FileWriter::class => [
                                    // A. Disable logging
                                    'disabled' => true,

                                    // -or- B. Configure logger
                                    'logFileInfix' => 'warming',
                                ],
                            ],
                        ],
                    ],
                ],
            ],
        ],
    ],
];

See also

Read more about logging in the official TYPO3 documentation.

Custom log table

The extension ships with a custom log table tx_warming_domain_model_log. In addition, a custom log writer \EliasHaeussler\Typo3Warming\Log\Writer\DatabaseWriter is provided that writes all logged crawling results to said table.

Note

All log entries are written to the root page (uid=0).

Note that this log writer is not enabled by default. You must explicitly enable it in your config/system/settings.php or config/system/additional.php file:

config/system/settings.php

return [
    'LOG' => [
        'EliasHaeussler' => [
            'Typo3Warming' => [
                'Crawler' => [
                    // Default crawler
                    'ConcurrentUserAgentCrawler' => [
                        'writerConfiguration' => [
                            \Psr\Log\LogLevel::WARNING => [
                                \EliasHaeussler\Typo3Warming\Log\Writer\DatabaseWriter::class => [],
                            ],
                        ],
                    ],

                    // Verbose crawler
                    'OutputtingUserAgentCrawler' => [
                        'writerConfiguration' => [
                            \Psr\Log\LogLevel::WARNING => [
                                \EliasHaeussler\Typo3Warming\Log\Writer\DatabaseWriter::class => [],
                            ],
                        ],
                    ],
                ],
            ],
        ],
    ],
];

See also

View the sources on GitHub:

DatabaseWriter

Usage

Note

You need at least one sitemap to use cache warming. A simple XML sitemap can be created with TYPO3's SEO extension.

This section describes how to properly use the extension in various ways. Basically the cache warming can be executed in two different modes: Either for whole sites based on their XML sitemap or for single pages based on their page ID.

Page tree

Note

The context menu items are only visible for admins and permitted users. Read how to give non-admin users access to the context menu items at Permissions.

Next to the item in the backend toolbar, one can also trigger cache warmup using the context menu of pages inside the page tree.

Cache warmup context meu items within the page tree

Note

The option Warmup cache for this page is available for all pages whereas the option Warmup all caches is only available for sites' root pages.

Console commands

The extension provides the following console commands:

`warming:cachewarmup`

A command to trigger cache warmup of single pages and/or whole sites using their XML sitemaps.

Important

You must pass at least the --pages or --sites command option.

Tip

If you use warming:cachewarmup in your deployment process and the sitemap is protected with HTTP authentication (basic auth) or the website does not have a valid SSL certificate, you can configure this in the extension configuration with verboseCrawlerOptions and parserOptions.

vendor/bin/typo3 warming:cachewarmup

typo3/sysext/core/bin/typo3 warming:cachewarmup

The following command options are available:

-p|--pages

Type: integer
Required: false
Default: none
Multiple allowed: true

Use this option to provide IDs of pages whose Frontend caches should be warmed up. You can pass this option multiple times, either as single integer values or as a comma-separated list of integer values.

Example:

vendor/bin/typo3 warming:cachewarmup -p 1 -p 2 -p 3
vendor/bin/typo3 warming:cachewarmup -p 1,2,3

typo3/sysext/core/bin/typo3 warming:cachewarmup -p 1 -p 2 -p 3
typo3/sysext/core/bin/typo3 warming:cachewarmup -p 1,2,3

-s|--sites

Type: integer or string (site identifier or all)
Required: false
Default: none
Multiple allowed: true

Use this option to provide a list of sites to be warmed up. You can either pass the appropriate site identifiers or the site's root page IDs.

New in version 3.1.0

Feature: #720 - Allow warming up all available sites at once

You can also pass the special keyword all to warm up all available sites at once.

Example:

vendor/bin/typo3 warming:cachewarmup -s my-cool-site
vendor/bin/typo3 warming:cachewarmup -s 1
vendor/bin/typo3 warming:cachewarmup -s all

typo3/sysext/core/bin/typo3 warming:cachewarmup -s my-cool-site
typo3/sysext/core/bin/typo3 warming:cachewarmup -s 1
typo3/sysext/core/bin/typo3 warming:cachewarmup -s all

-l|--languages

Type: integer
Required: false
Default: -1 (all languages)
Multiple allowed: true

You can optionally limit the languages of sites to be warmed up to a given list of language IDs. If this option is omitted, all site languages will be included in the cache warmup.

Example:

vendor/bin/typo3 warming:cachewarmup -l 0 -l 1
vendor/bin/typo3 warming:cachewarmup -l 0,1

typo3/sysext/core/bin/typo3 warming:cachewarmup -l 0 -l 1
typo3/sysext/core/bin/typo3 warming:cachewarmup -l 0,1

-c|--config

Type: string
Required: false
Default: none
Multiple allowed: false

An external configuration file can be used to provide a more fine-grained configuration for cache warmup. The file path may contain references to extensions (see example below).

Note

Config file options will be merged with command options, whereas command options receive higher priority. It's even possible to use environment variables for configuration. Read more in the official documentation.

At the moment, the following file formats are supported:

JSON
PHP
YAML

Example:

vendor/bin/typo3 warming:cachewarmup --config cache-warmup.yaml
vendor/bin/typo3 warming:cachewarmup --config EXT:sitepackage/Configuration/cache-warmup.yaml

typo3/sysext/core/bin/typo3 warming:cachewarmup --config cache-warmup.yaml
typo3/sysext/core/bin/typo3 warming:cachewarmup --config EXT:sitepackage/Configuration/cache-warmup.yaml

--limit

Type: integer
Required: false
Default: limit value from extension configuration
Multiple allowed: false

Maximum number of pages to be crawled. Set to 0 to disable the limit. If this option is omitted, the limit value from extension configuration is used.

Example:

vendor/bin/typo3 warming:cachewarmup --limit 100

typo3/sysext/core/bin/typo3 warming:cachewarmup --limit 100

--strategy

Type: string
Required: false
Default: strategy value from extension configuration
Multiple allowed: false

Name of an available crawling strategy to use for cache warmup. If this option is omitted, the strategy value from extension configuration is used.

Example:

vendor/bin/typo3 warming:cachewarmup --strategy sort-by-priority

typo3/sysext/core/bin/typo3 warming:cachewarmup --strategy sort-by-priority

-x|--strict

Type: boolean
Required: false
Default: false
Multiple allowed: false

Exit with a non-zero status code in case cache warmup fails or errors occur during cache warmup.

Example:

vendor/bin/typo3 warming:cachewarmup --strict

typo3/sysext/core/bin/typo3 warming:cachewarmup --strict

`warming:showuseragent`

A command that shows the custom User-Agent header that is used for cache warmup requests by default crawlers.

Note

This command is not schedulable.

vendor/bin/typo3 warming:showuseragent

typo3/sysext/core/bin/typo3 warming:showuseragent

Using the API

Besides the usage via the TYPO3 backend and the console commands, there is also a public PHP API. It can be used to execute the cache warmup directly in PHP code.

class CacheWarmupService

Fully qualified name: \EliasHaeussler\Typo3Warming\Service\CacheWarmupService

Service to run cache warmup for sites and pages.

warmup ( $sites, $pages, $limit, $strategy)

Run cache warmup for given sites and pages.

param array $sites: List of site warmup requests.
param array $pages: List of page warmup requests.
param int|null $limit: Optional cache warmup limit.
param \EliasHaeussler\CacheWarmup\Crawler\Strategy\CrawlingStrategy|null $strategy: Optional crawling strategy.
returntype: \EliasHaeussler\Typo3Warming\Result\CacheWarmupResult

Example

use EliasHaeussler\CacheWarmup;
use EliasHaeussler\Typo3Warming;
use TYPO3\CMS\Core;

$cacheWarmupService = Core\Utility\GeneralUtility::makeInstance(Typo3Warming\Service\CacheWarmupService::class);
$siteFinder = Core\Utility\GeneralUtility::makeInstance(Core\Site\SiteFinder::class);

$sites = [];
$pages = [];

// Create site warmup requests
foreach ($siteFinder->getAllSites() as $site) {
    $sites[] = new Typo3Warming\ValueObject\Request\SiteWarmupRequest($site);
}

// Create page warmup requests
foreach ([1, 2, 3] as $page) {
    $pages[] = new Typo3Warming\ValueObject\Request\PageWarmupRequest($page);
}

// Define optional cache warmup options
$limit = 100;
$strategy = new CacheWarmup\Crawler\Strategy\SortByPriorityStrategy();

// Run cache warmup for sites and pages
$result = $cacheWarmupService->warmup($sites, $pages, $limit, $strategy);

// Fetch crawling states
$failedUrls = $result->getResult()->getFailed();
$successfulUrls = $result->getResult()->getSuccessful();

// Fetch excluded URLs
$excludedUrls = $result->getExcludedUrls();

Developer corner

Many of the cache warmup processes are configurable or can be extended with custom implementations. This section provides an insight into which development options are available and how these can be integrated into the cache warmup process.

Crawlers

The extension uses crawlers to visit all URLs configured for cache warmup. While visiting a URL, the appropriate page cache gets warmed. Learn more about which crawlers are available by default and how to implement a custom crawler on this page.

interface Crawler

Fully qualified name: \EliasHaeussler\CacheWarmup\Crawler\Crawler

Interface for crawlers used to crawl and warm up URLs.

crawl ( $urls)

Crawl a given list of URLs.

param array $urls: List of URLs to be crawled.
returntype: \EliasHaeussler\CacheWarmup\Result\CacheWarmupResult

Default crawlers

The extension ships with two default crawlers:

\EliasHaeussler\Typo3Warming\Crawler\ConcurrentUserAgentCrawler: Used for cache warmup triggered within the TYPO3 backend
\EliasHaeussler\Typo3Warming\Crawler\OutputtingUserAgentCrawler: Used for cache warmup executed from the command-line

Both crawlers use a custom User-Agent header for all warmup requests. By using this custom header, it is possible to exclude warmup requests from the statistics of analysis tools, for example. The header is generated by a HMAC hash of the string TYPO3/tx_warming_crawler.

The generated header value can be copied form the cache warmup modal in the TYPO3 backend. Alternatively, a command warming:showuseragent is available which can be used to read the current User-Agent header.

Implement a custom crawler

Available interfaces

The actual cache warmup is done via the library eliashaeussler/cache-warmup . It provides the \EliasHaeussler\CacheWarmup\Crawler\Crawler interface, which must be implemented when developing your own crawler.

Verbose crawlers

There is also a \EliasHaeussler\CacheWarmup\Crawler\VerboseCrawler interface that redirects user-oriented output to an instance of \Symfony\Component\Console\Output\OutputInterface.

interface VerboseCrawler

Fully qualified name: \EliasHaeussler\CacheWarmup\Crawler\VerboseCrawler

Interface that redirects user-oriented output to a given output.

setOutput ( $output)

Set output where to redirect user-oriented output.

param \Symfony\Component\Console\Output\OutputInterface $output: Output where to redirect user-oriented output.

Configurable crawlers

Custom crawlers can also implement the \EliasHaeussler\CacheWarmup\Crawler\ConfigurableCrawler, interface allowing users to configure warmup requests themselves.

interface ConfigurableCrawler

Fully qualified name: \EliasHaeussler\CacheWarmup\Crawler\ConfigurableCrawler

Interface allowing users to configure warmup requests themselves.

setOptions ( $options)

Set custom crawler options.

param array $options: Associative array of custom crawler options.

Logging crawlers

Crawling results can be logged using a dedicated PSR-3 logger. For this, crawlers must implement the \EliasHaeussler\CacheWarmup\Crawler\LoggingCrawler interface and inject an appropriate PSR-3 logger. In TYPO3 context, this is mostly done using TYPO3's log manager. Read more about logging in the official documentation.

interface LoggingCrawler

Fully qualified name: \EliasHaeussler\CacheWarmup\Crawler\LoggingCrawler

Interface that allows crawling results to be logged using a dedicated PSR-3 logger.

setLogger ( $logger)

Inject PSR-3 compatible logger.

param \Psr\Log\LoggerInterface $logger: PSR-3 compatible logger.

setLogLevel ( $logLevel)

Set minimum log level.

param string $logLevel: The minimum log level.

Stoppable crawlers

Crawlers implementing the \EliasHaeussler\CacheWarmup\Crawler\StoppableCrawler interface may cancel a cache warmup prematurely if any crawling failure occurs. This can be especially useful for validation purposes to check whether any page within an XML sitemap is inaccessible or failing.

interface StoppableCrawler

Fully qualified name: \EliasHaeussler\CacheWarmup\Crawler\StoppableCrawler

Interface that may cancel a cache warmup prematurely if any crawling failure occurs.

stopOnFailure ( $stopOnFailure)

Configure crawler to cancel cache warmup on failure.

param bool $stopOnFailure: Cancel cache warmup on failure.

Streamable crawlers

When running cache warmup from the TYPO3 backend, the current crawling progress is streamed to the cache warmup progress modal. However, this is only supported for crawlers implementing the \EliasHaeussler\Typo3Warming\Crawler\StreamableCrawler interface.

Those crawlers will then get an \EliasHaeussler\SSE\Stream\EventStream injected. It can be used to send events to the current event stream. The following events are currently available:

\EliasHaeussler\Typo3Warming\Http\Message\Event\WarmupFinishedEvent
\EliasHaeussler\Typo3Warming\Http\Message\Event\WarmupProgressEvent

By default, when implementing a streamable crawler, there's no need to trigger these events by your own. Instead, it's better to use the provided \EliasHaeussler\Typo3Warming\Http\Message\Handler\StreamResponseHandler which takes care of sending appropriate events.

interface StreamableCrawler

Fully qualified name: \EliasHaeussler\Typo3Warming\Crawler\StreamableCrawler

Interface that allows streaming of cache warmup events using an EventStream.

setStream ( $stream)

Set event stream used to send cache warmup events.

param \EliasHaeussler\SSE\Stream\EventStream $stream: Event stream used to send cache warmup events.

Steps to implement a new crawler

Create a new crawler

The new crawler must implement at least one of the following interfaces:
Configure the new crawler

Add the new crawler to the extension configuration. Note that you should configure either the crawler or verboseCrawler option, depending on what interface you have implemented.
Flush system caches

Finally, flush all system caches to ensure the correct crawler class is used for further cache warmup requests.

See also

View the sources on GitHub:

Crawling strategies

Before URLs are crawled by a crawler, they can be prepared by a specific strategy. This, for example, allows to prioritize specific URLs or provide additional information to URLs.

interface CrawlingStrategy

Fully qualified name: \EliasHaeussler\CacheWarmup\Crawler\Strategy\CrawlingStrategy

Interface for crawling strategy to prepare URLs before crawling.

prepareUrls ( $urls)

Prepare given URLs for crawling.

param array $urls: List of URLs to be prepared for crawling.
returntype: list<\EliasHaeussler\CacheWarmup\Sitemap\Url>

static getName ( )

Get name of crawling strategy for use as identifier.

returntype: string

Shipped crawling strategies

The extension ships with the following crawling strategies:

sort-by-changefreq: Sorts given URLs by their changefreq node value.
sort-by-lastmod: Sorts given URLs by their lastmod node value.
sort-by-priority: Sorts given URLs by their priority node value.

Implement a custom strategy

Create a new crawling strategy

The new strategy must implement the \EliasHaeussler\CacheWarmup\Crawler\Strategy\CrawlingStrategy interface. Make sure to properly implement the \EliasHaeussler\CacheWarmup\Crawler\Strategy\CrawlingStrategy::getName method to identify the crawling strategy.
Configure the new crawling strategy

Add the new strategy to the extension configuration. Use the strategy's name as configuration value.
Flush system caches

Finally, flush all system caches to ensure the correct crawling strategy is used for further cache warmup requests.

See also

View the sources on GitHub:

Sitemap providers

To localize the sitemap of a site, so-called "sitemap providers" are available. These are executed one after the other (depending on priority) to localize XML sitemaps according to certain criteria.

For deeper insights, take a look at how EXT:sitemap_locator uses sitemap providers to locate XML sitemaps.

Configuration API

In order to access the extension configuration, a slim PHP API exists. Each configuration option is accessible by an appropriate class property.

Changed in version 4.2.0

Several class methods were converted to class properties and deprecated in version 4.2.0. Calling those methods will be possible until version 5.0 of the extension, but deprecation notices will be logged on every method call.

class Configuration

Fully qualified name: \EliasHaeussler\Typo3Warming\Configuration\Configuration

API to access all available extension configuration options.

crawlerClass: Get the configured crawler class. To access the instantiated crawler, call the getCrawler method instead.

crawlerOptions: Get the configured crawler options.

verboseCrawlerClass: Get the configured verbose crawler class. To access the instantiated crawler, call the getVerboseCrawler method instead.

verboseCrawlerOptions: Get the configured verbose crawler options.

parserOptions: Get the configured parser options.

clientOptions: Get the configured client options.

limit: Get the configured crawler limit.

excludePatterns: Get the configured exclude patterns.

crawlingStrategy: Get the configured crawling strategy.

enabledInPageTree: Check whether cache warmup from page tree is enabled.

supportedDoktypes: Get all doktypes that support cache warmup from page tree.

enabledInToolbar: Check whether cache warmup from toolbar is enabled.

getCrawler ( )

Get the configured crawler.

returntype: \EliasHaeussler\CacheWarmup\Crawler\Crawler

getVerboseCrawler ( )

Get the configured verbose crawler.

returntype: \EliasHaeussler\CacheWarmup\Crawler\VerboseCrawler

getUserAgent ( )

Get the calculated user-agent.

Deprecated since version 4.2.0

Call \EliasHaeussler\Typo3Warming\Http\Message\Request\RequestOptions::getUserAgent instead. See migration guide for more information.

returntype: string

See also

View the sources on GitHub:

Configuration

`WarmupPermissionGuard`

The WarmupPermissionsGuard can be used to determine permissions of a given backend user to warm up caches of specific sites or pages.

Note

Resolved permissions for a given combination of page/site, language id and backend user are cached during runtime.

class WarmupPermissionGuard

Fully qualified name: \EliasHaeussler\Typo3Warming\Security\WarmupPermissionGuard

Guard to determine cache warmup permissions.

canWarmupCacheOfPage ( $pageId, $context)

Check if caches of the given page can be warmed up while respecting given permission context.

param int $pageId: ID of the page to be checked.
param \EliasHaeussler\Typo3Warming\Security\Context\PermissionContext $context: Permission context which holds language ID and backend user.
returntype: bool

canWarmupCacheOfSite ( $site, $context)

Check if caches of the given site can be warmed up while respecting given permission context.

param \TYPO3\CMS\Core\Site\Entity\Site $site: The site to be checked.
param \EliasHaeussler\Typo3Warming\Security\Context\PermissionContext $context: Permission context which holds language ID and backend user.
returntype: bool

See also

View the sources on GitHub:

Events

New in version 1.0.0

Feature #338 – Introduce events

During cache warmup, some PSR-14 events are dispatched. Events can be used to step in the cache warmup lifecycle and affect its crawling behavior in various ways.

Note

The eliashaeussler/cache-warmup library also dispatches some events. Read more in the official documentation.

The following events are currently dispatched:

BeforeCacheWarmupEvent

This event is dispatched right before cache warmup is triggered via \EliasHaeussler\CacheWarmup\CacheWarmer::run. It allows to add additional sitemaps and URLs to the cache warmer or modify the crawling behavior in other ways.

AfterCacheWarmupEvent

Once cache warmup is finished, this event is dispatched. It provides the cache warmup result together with the original instance of \EliasHaeussler\CacheWarmup\CacheWarmer and the used crawler.

See also

View the sources on GitHub:

Migration

This page lists all notable changes and required migrations when upgrading to a new major version of this extension.

See also

Make sure to check out the migration guide of the eliashaeussler/cache-warmup library as well.

Version 5.0.0

Removal of deprecated functionality

Several getter methods in \EliasHaeussler\Typo3Warming\Configuration\Configuration are now finally removed (see deprecation notices).

Use of a different secret for `User-Agent` header value

The additional secret used to HMAC-hash the generated User-Agent header value has been changed.
Values of the User-Agent header are now different compared to their previous representation.
Make sure to adapt all places that rely on the User-Agent header (e.g. exclusions in stats software like Matomo).

Version 4.2.0

Use of properties in Configuration API

Several class methods in \EliasHaeussler\Typo3Warming\Configuration\Configuration are converted to class properties.
Make sure to access the appropriate properties directly instead of calling the class methods.
Calling the legacy class methods is deprecated and won't be possible anymore with version 5.0 of the extension.

Deprecation of `getUserAgent()` method in Configuration API

Access to the generated User-Agent header is moved from the Configuration API to a dedicated \EliasHaeussler\Typo3Warming\Http\Message\Request\RequestOptions class.
Migrate calls from \EliasHaeussler\Typo3Warming\Configuration\Configuration::getUserAgent to \EliasHaeussler\Typo3Warming\Http\Message\Request\RequestOptions::getUserAgent.
Calling the legacy method is deprecated and won't be possible anymore with version 5.0 of the extension.

Version 4.0.0

Upgrade of eliashaeussler/cache-warmup library

Crawler option client_config is removed and must be migrated to extension configuration clientOptions.
Extension configuration parserClientOptions is renamed to parserOptions. Existing configuration must be migrated to extension configuration clientOptions.
Crawling response body is no longer attached to response objects. Enable crawler option write_response_body to restore previous behavior.
Read more in the library's release notes.

Version 3.0.0

Upgrade of eliashaeussler/cache-warmup library

Custom crawlers must be rewritten to match the updated codebase.
Read more in the library's release notes.

`StreamResponseHandler` is now result-aware

\EliasHaeussler\Typo3Warming\Http\Message\Handler\StreamResponseHandler now depends on a given \EliasHaeussler\CacheWarmup\Result\CacheWarmupResult.
The result object is generated and updated by the \EliasHaeussler\CacheWarmup\Http\Message\Handler\ResultCollectorHandler.
Make sure to use both handlers together when using the stream response handler.
Pass the result object from result collection handler when instantiating the stream response handler.
See \EliasHaeussler\Typo3Warming\Crawler\ConcurrentUserAgentCrawler for a dedicated example.

Version 2.0.0

Integration of EXT:sitemap_locator

Sitemaps cache was extracted to EXT:sitemap_locator. Use \EliasHaeussler\Typo3SitemapLocator\Cache\SitemapsCache instead of \EliasHaeussler\Typo3Warming\Cache\SitemapsCache.
Sitemap providers were extracted to EXT:sitemap_locator. Use \EliasHaeussler\Typo3SitemapLocator\Sitemap\Provider\Provider instead of \EliasHaeussler\Typo3Warming\Sitemap\Provider\Provider for custom provider implementations.
Sitemap locator was extracted to EXT:sitemap_locator. Use \EliasHaeussler\Typo3SitemapLocator\Sitemap\SitemapLocator instead of \EliasHaeussler\Typo3Warming\Sitemap\SitemapLocator.

Relocated sitemap model

The default \EliasHaeussler\Typo3Warming\Sitemap\SiteAwareSitemap model was moved to \EliasHaeussler\Typo3Warming\Domain\Model\SiteAwareSitemap. Update references to this class in your code.

Version 1.0.0

Default crawlers

Default crawlers are now final. Custom crawlers can no longer extend default crawlers. Implement the \EliasHaeussler\CacheWarmup\Crawler\Crawler or \EliasHaeussler\CacheWarmup\Crawler\VerboseCrawler interface instead.
CrawlerFactory from eliashaeussler/cache-warmup library is now used to instantiate crawlers. Dependency injection is no longer possible.
\EliasHaeussler\Typo3Warming\Crawler\ConfigurableClientTrait was removed. Use \EliasHaeussler\Typo3Warming\Http\Client\ClientFactory::get instead.
\EliasHaeussler\Typo3Warming\Crawler\RequestAwareInterface and \EliasHaeussler\Typo3Warming\Crawler\RequestAwareTrait were removed. Use \EliasHaeussler\Typo3Warming\Crawler\StreamableCrawler in combination with \EliasHaeussler\Typo3Warming\Http\Message\Handler\StreamResponseHandler instead.
\EliasHaeussler\Typo3Warming\Crawler\UserAgentTrait was removed. Provide an own implementation that calls \EliasHaeussler\Typo3Warming\Configuration\Configuration::getUserAgent instead.

Warmup request handling

\EliasHaeussler\Typo3Warming\ValueObject\Request\WarmupRequest is now final.
\EliasHaeussler\Typo3Warming\ValueObject\Request\WarmupRequest::$updateCallback was removed. Streamed warmup requests must now be handled by using \EliasHaeussler\Typo3Warming\Http\Message\Handler\StreamResponseHandler in a custom crawler instead.
Crawling result handling within \EliasHaeussler\Typo3Warming\ValueObject\Request\WarmupRequest was removed. Use the returned \EliasHaeussler\Typo3Warming\Result\CacheWarmupResult from \EliasHaeussler\Typo3Warming\Service\CacheWarmupService::warmup instead.
\EliasHaeussler\Typo3Warming\Service\CacheWarmupService::warmupPages and \EliasHaeussler\Typo3Warming\Service\CacheWarmupService::warmupSites were combined to a new method \EliasHaeussler\Typo3Warming\Service\CacheWarmupService::warmup. Use this method with dedicated instances of \EliasHaeussler\Typo3Warming\ValueObject\Request\SiteWarmupRequest and \EliasHaeussler\Typo3Warming\ValueObject\Request\PageWarmupRequest.

Sitemap providers

\EliasHaeussler\Typo3Warming\Sitemap\Provider\ProviderInterface was renamed to \EliasHaeussler\Typo3Warming\Sitemap\Provider\Provider.
\EliasHaeussler\Typo3Warming\Sitemap\Provider\AbstractProvider was removed. Custom sitemap providers must now implement \EliasHaeussler\Typo3Warming\Sitemap\Provider\Provider directly. The previously available trait method is now available within \EliasHaeussler\Typo3Warming\Utility\HttpUtility::getSiteUrlWithPath.
\EliasHaeussler\Typo3Warming\Sitemap\Provider\Provider::get now returns an array of \EliasHaeussler\Typo3Warming\Sitemap\SiteAwareSitemap instances.
A new sitemap provider \EliasHaeussler\Typo3Warming\Sitemap\Provider\PageTypeProvider was added. It is configured with highest priority. Read more at Sitemap providers.

Language handling

\EliasHaeussler\Typo3Warming\Sitemap\SiteAwareSitemap now requires a site language to be set.
Page uri generation now respects configured language overlays and is moved to \EliasHaeussler\Typo3Warming\Utility\HttpUtility::generateUri.

Extension configuration

Extension configuration exclude was added. Read more at exclude.
Extension configuration strategy was added. Read more at strategy.

Command options

New command option --format was added. Read more at warming:cachewarmup.
New command option --strategy was added. Read more at warming:cachewarmup.

Template paths

Template paths were rewritten:
- CacheWarmupToolbarItem.html was rewritten to Toolbar/CacheWarmupToolbarItem.html
- CacheWarmupToolbarItemActions.html was rewritten to Modal/SitesModal.html
Partial paths were rewritten:
- ToolbarItem.html was inlined to template Toolbar/CacheWarmupToolbarItem.html
- ToolbarItemAction.html was split into Modal/Sites/SiteGroup.html and Modal/Sites/SiteGroupItem.html
- ToolbarItemMissing.html was rewritten to Modal/Alert/NoSites.html
- ToolbarItemPlaceholder.html was removed
- ToolbarItemUserAgent.html was removed

Contributing

Thanks for considering contributing to this extension! Since it is an open source product, its successful further development depends largely on improving and optimizing it together.

The development of this extension follows the official TYPO3 coding standards. To ensure the stability and cleanliness of the code, various code quality tools are used and most components are covered with test cases. In addition, we use DDEV for local development. Make sure to set it up as described below. For continuous integration, we use GitHub Actions.

Create an issue first

Before you start working on the extension, please create an issue on GitHub: https://github.com/eliashaeussler/typo3-warming/issues

Also, please check if there is already an issue on the topic you want to address.

Contribution workflow

Note

This extension follows Semantic Versioning.

Preparation

Clone the repository first:

git clone https://github.com/eliashaeussler/typo3-warming.git
cd typo3-warming

Now start DDEV:

ddev start

Next, install all dependencies:

ddev composer install
ddev frontend install

You can access the DDEV site at https://typo3-ext-warming.ddev.site/.

Analyze code

# All analyzers
ddev cgl analyze

# Specific analyzers
ddev cgl analyze:dependencies

Check code quality

TYPO3

# All linters
ddev cgl lint

# Specific linters
ddev cgl lint:composer
ddev cgl lint:editorconfig
ddev cgl lint:php
ddev cgl lint:typoscript

# Fix all CGL issues
ddev cgl fix

# Fix specific CGL issues
ddev cgl fix:composer
ddev cgl fix:editorconfig
ddev cgl fix:php
ddev cgl fix:typoscript

# All static code analyzers
ddev cgl sca

# Specific static code analyzers
ddev cgl sca:php

Frontend

# All linters
ddev frontend run lint

# Specific linters
ddev frontend run lint:scss
ddev frontend run lint:ts

# Fix all CGL issues
ddev frontend run fix

# Fix specific CGL issues
ddev frontend run fix:scss
ddev frontend run fix:ts

Run tests

# All tests
ddev test

# Specific tests
ddev test acceptance
ddev test functional
ddev test unit

# All tests with code coverage
ddev test coverage

# Specific tests with code coverage
ddev test coverage:acceptance
ddev test coverage:functional
ddev test coverage:unit

# Merge code coverage of all test suites
ddev test coverage:merge

Code coverage reports are written to .Build/coverage. You can open the last merged HTML report like follows:

open .Build/coverage/html/_merged/index.html

Reports of acceptance tests are written to .Build/log/acceptance-reports. You can open the last HTML report like follows:

open .Build/log/acceptance-reports/records.html

Build documentation

# Rebuild and open documentation
composer docs

# Build documentation (from cache)
composer docs:build

# Open rendered documentation
composer docs:open

The built docs will be stored in .Build/docs.

Pull Request

Once you have finished your work, please submit a pull request and describe what you've done: https://github.com/eliashaeussler/typo3-warming/pulls

Ideally, your PR references an issue describing the problem you're trying to solve. All described code quality tools are automatically executed on each pull request for all currently supported PHP versions and TYPO3 versions.

Cache warmup of pages located in XML sitemaps

Introduction

Installation

Configuration

Usage

Developer corner

Migration

Introduction

What does it do?

Features

Support

Security Policy

License

Installation

Requirements

Installation

Version matrix

Configuration

Extension configuration

Crawler

crawler

crawlerOptions

verboseCrawler

verboseCrawlerOptions

Parser

parserOptions

Client

clientOptions

Options

limit

exclude

strategy

Page tree

enablePageTree

supportedDoktypes

Toolbar

enableToolbar

Site configuration

warming_exclude (site)

warming_exclude (site_language)

TypoScript configuration

Constants

view.templateRootPath

view.partialRootPath

Crawler configuration

perform_subrequests

Parser configuration

Permissions

allowedSites

allowedPages

Logging

Custom log table

Usage

Backend toolbar

Toolbar item

Cache warmup modal

Page tree

Console commands

warming:cachewarmup

-p|--pages

-s|--sites

-l|--languages

-c|--config

--limit

--strategy

-x|--strict

warming:showuseragent

Using the API

Example

Developer corner

Crawlers

Default crawlers

Implement a custom crawler

Available interfaces

Verbose crawlers

Configurable crawlers

Logging crawlers

Stoppable crawlers

Streamable crawlers

Steps to implement a new crawler

`warming:cachewarmup`

`warming:showuseragent`

`WarmupPermissionGuard`

Use of a different secret for `User-Agent` header value

Deprecation of `getUserAgent()` method in Configuration API

`StreamResponseHandler` is now result-aware