Crawler

Extension key

crawler

Package name

tomasnorre/crawler

Version

main

Language

en

Author

Tomas Norre Mikkelsen

Copyright

2005-2021 AOE GmbH, since 2021 Tomas Norre Mikkelsen

License

This document is published under the Open Content License.

Rendered

2022-12-20 16:58

---

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

---

Table of Contents:

• Introduction
  • What does it do?
  • Screenshots
• Configuration
  • Extension Manager Configuration
  • Configuration records
  • Page TSconfig Reference (tx_crawler.crawlerCfg)
  • HTTP Authentication
  • Examples
• Executing the queue
  • Run via command controller
  • Executing queue with cron-job
  • Run via backend
  • Building and Executing queue right away (from cli)
• Scheduler
• Use cases
  • Cache warm up
  • Indexed Search
• Features
  • Automatic add pages to Queue
  • Pollable processing instructions
  • Multi process support
  • Hooks
  • PSR-14 Events
  • Priority Crawling
• Troubleshooting
• Links
• AOE GmbH