---
title: "Site Crawler Extension"
manual: "Crawler"
version: "main"
permalink: "https://docs.typo3.org/permalink/tomasnorre/crawler:start@main"
source: "Index.rst"
start: true
rendered: "2026-10-16T17:00:27+00:00"
---

# Site Crawler Extension {#start}

-   *Extension key:* crawler
-   *Package name:* tomasnorre/crawler
-   *Version:* main
-   *Language:* en
-   *Author:* Tomas Norre Mikkelsen
-   *Copyright:* 2005-2021 AOE GmbH, since 2021 Tomas Norre Mikkelsen
-   *License:* This document is published under the \`Open Content License

-   *Rendered:* Fri, 16 Oct 2026 17:00:27 +0000

---

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

---

**Table of Contents:**

-   [Introduction](https://docs.typo3.org/permalink/tomasnorre/crawler:introduction-1@main)
-   [Configuration](https://docs.typo3.org/permalink/tomasnorre/crawler:configuration-1@main)
    -   [Extension Manager Configuration](https://docs.typo3.org/permalink/tomasnorre/crawler:extension-manager-configuration-1@main)
    -   [Configuration records](https://docs.typo3.org/permalink/tomasnorre/crawler:configuration-records@main)
    -   [Page TSconfig Reference (tx_crawler.crawlerCfg)](https://docs.typo3.org/permalink/tomasnorre/crawler:page-tsconfig-reference-tx-crawler-crawlercfg@main)
    -   [HTTP Authentication](https://docs.typo3.org/permalink/tomasnorre/crawler:http-authentication-1@main)
    -   [Examples](https://docs.typo3.org/permalink/tomasnorre/crawler:examples-1@main)
-   [Executing the queue](https://docs.typo3.org/permalink/tomasnorre/crawler:executing-the-queue@main)
    -   [Run via command controller](https://docs.typo3.org/permalink/tomasnorre/crawler:run-via-command-controller@main)
    -   [Executing queue with cron-job](https://docs.typo3.org/permalink/tomasnorre/crawler:executing-queue-with-cron-job@main)
    -   [Run via backend](https://docs.typo3.org/permalink/tomasnorre/crawler:run-via-backend@main)
    -   [Building and Executing queue right away (from cli)](https://docs.typo3.org/permalink/tomasnorre/crawler:building-and-executing-queue-right-away-from-cli@main)
-   [Scheduler](https://docs.typo3.org/permalink/tomasnorre/crawler:scheduler-1@main)
-   [Use cases](https://docs.typo3.org/permalink/tomasnorre/crawler:use-cases-1@main)
    -   [Cache warm up](https://docs.typo3.org/permalink/tomasnorre/crawler:cache-warm-up@main)
    -   [Indexed Search](https://docs.typo3.org/permalink/tomasnorre/crawler:indexed-search@main)
-   [Features](https://docs.typo3.org/permalink/tomasnorre/crawler:features-1@main)
    -   [Automatic add pages to Queue](https://docs.typo3.org/permalink/tomasnorre/crawler:automatic-add-pages-to-queue@main)
    -   [Pollable processing instructions](https://docs.typo3.org/permalink/tomasnorre/crawler:pollable-processing-instructions@main)
    -   [Multi process support](https://docs.typo3.org/permalink/tomasnorre/crawler:multi-process-support@main)
    -   [Hooks](https://docs.typo3.org/permalink/tomasnorre/crawler:hooks-1@main)
    -   [PSR-14 Events](https://docs.typo3.org/permalink/tomasnorre/crawler:psr-14-events@main)
    -   [Priority Crawling](https://docs.typo3.org/permalink/tomasnorre/crawler:priority-crawling-1@main)
-   [Troubleshooting](https://docs.typo3.org/permalink/tomasnorre/crawler:troubleshooting-1@main)
-   [Links](https://docs.typo3.org/permalink/tomasnorre/crawler:links-1@main)