.. include:: /Includes.rst.txt

.. _crawlers:

========
Crawlers
========

The extension uses crawlers to visit all URLs configured for cache
warmup. While visiting a URL, the appropriate page cache gets warmed.
Learn more about which crawlers are available by default and how to
implement a custom crawler on this page.

..  php:namespace:: EliasHaeussler\CacheWarmup\Crawler

..  php:interface:: CrawlerInterface

    Interface for crawlers used to crawl and warm up URLs.

    ..  php:method:: crawl($urls)

        Crawl a given list of URLs.

        :param array $urls: List of URLs to be crawled.

    ..  php:method:: getSuccessfulUrls()

        Return all successfully crawled URLs.

        :returns: A list of :php:class:`EliasHaeussler\\CacheWarmup\\CrawlingState` instances

    ..  php:method:: getFailedUrls()

        Return all failing crawled URLs.

        :returns: A list of :php:class:`EliasHaeussler\\CacheWarmup\\CrawlingState` instances

.. _default-crawlers:

Default crawlers
================

The extension ships with two default crawlers:

-   :php:class:`EliasHaeussler\\Typo3Warming\\Crawler\\ConcurrentUserAgentCrawler`:
    Used for cache warmup triggered within the **TYPO3 backend**
-   :php:class:`EliasHaeussler\\Typo3Warming\\Crawler\\OutputtingUserAgentCrawler`:
    Used for cache warmup executed from the **command-line**

Both crawlers use a custom `User-Agent` header for all warmup
requests. By using this custom header, it is possible to exclude
warmup requests from the statistics of analysis tools, for example.
The header is generated by a HMAC hash of the string
`TYPO3/tx_warming_crawler`.

The generated header value can be copied in the dropdown of the
toolbar item in the backend. Alternatively, a command
`warming:showuseragent` is available which can be used to read the
current `User-Agent` header.

.. _implement-a-custom-crawler:

Implement a custom crawler
==========================

.. _available-interfaces:

Available interfaces
--------------------

The actual cache warmup is done via the library
`eliashaeussler/cache-warmup <https://github.com/eliashaeussler/cache-warmup>`__.
It provides the
:php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\CrawlerInterface`,
which must be implemented when developing your own crawler. There is
also a
:php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\VerboseCrawlerInterface`
that redirects user-oriented output to an instance of
:php:interface:`Symfony\\Component\\Console\\Output\\OutputInterface`.

.. _configurable-crawlers:

Configurable crawlers
---------------------

Since version 0.7.13 of `eliashaeussler/cache-warmup`, custom
crawlers can also implement the
:php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\ConfigurableCrawlerInterface`,
allowing users to configure warmup requests themselves.

.. seealso::

    `Feature #59 - Introduce configurable crawlers <https://github.com/eliashaeussler/cache-warmup/pull/59>`__
    of `eliashaeussler/cache-warmup` library

.. _steps-to-implement-a-new-crawler:

Steps to implement a new crawler
--------------------------------

.. rst-class:: bignums

1.  Create a new crawler

    The new crawler must implement one of the following interfaces:

    - :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\CrawlerInterface`
    - :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\VerboseCrawlerInterface`
    - :php:interface:`EliasHaeussler\\CacheWarmup\\Crawler\\ConfigurableCrawlerInterface`

    .. tip::

        This extension provides some additional traits which you can
        use for your new crawler:

        - :php:trait:`EliasHaeussler\\Typo3Warming\\Crawler\\RequestAwareTrait`
        - :php:trait:`EliasHaeussler\\Typo3Warming\\Crawler\\UseAgentTrait`

2.  Configure the new crawler

    Add the new crawler to the :ref:`extension configuration <extension-configuration>`.
    Note that you should configure either the `crawler` or
    `verboseCrawler` option, depending on what interface you have
    implemented.

3.  Flush system caches

    Finally, flush all system caches to ensure the correct crawler
    class is used for further cache warmup requests.