Crawler

Extension key:crawler Package name:tomasnorre/crawler Version:main Language:en Author:Tomas Norre Mikkelsen Copyright:2005-2021 AOE GmbH, since 2021 Tomas Norre Mikkelsen License:This document is published under the Open Content License. Rendered:2022-04-29 09:25

---

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

---

Table of Contents:

• Introduction
  • What does it do?
  • Screenshots
• Configuration
  • Extension Manager Configuration
  • Configuration records
  • Page TSconfig Reference (tx_crawler.crawlerCfg)
  • HTTP Authentication
  • Examples
• Executing the queue
  • Run via command controller
  • Executing queue with cron-job
  • Run via backend
  • Building and Executing queue right away (from cli)
• Scheduler
• Use cases
  • Cache warm up
  • Indexed Search
• Features
  • Automatic add pages to Queue
  • Pollable processing instructions
  • Multi process support
  • Hooks
  • PSR-14 Events
  • Priority Crawling
• Troubleshooting
• Links
• AOE GmbH