Table of Contents
With the crawler release 9.1.0 we have changed the data stores in crawler queue from serialized to json data. If you are experiencing problems with the old data still in your database, you can flush your complete crawler queue and the problem should be solved.
We have build in a JsonCompatibilityConverter to ensure that this should not happen, but in case of it run:
$ vendor/bin/typo3 crawler:flushQueue all
If you are using direct request, see Extension Manager Configuration, and it doesn’t give you any result, or that the scheduler tasks stalls.
It can be because of a faulty configured TrustedHostPattern, this can be
changed in the
$GLOBALS['TYPO3_CONF_VARS']['SYS']['trustedHostsPattern'] = '<your-pattern>';
The crawler won’t process all entries at command-line-way. This might happened because the php run into an time out, to avoid this you can call the crawler like:
php -d max_execution_time=512 vendor/bin/typo3 crawler:buildQueue
If you experiences that the crawler queue only adds one url to the queue, you are probably on a new setup, or an update from TYPO3 8LTS you might have some migration not executed yet.
Please check the Upgrade Wizard, and check if the Introduce URL parts (“slugs”) to all existing pages is marked as done, if not you should perform this step.
See related issue: [BUG] Crawling Depth not respected #464
If you update the extension from older versions you can run into following error:
SQL error: 'Field 'sys_domain_base_url' doesn't have a default value'
Make sure to delete all unnecessary fields from database tables. You can do
this in the backend via Analyze Database Structure tool or if you
have TYPO3 Console
installed via command line command
In some cases you get an error, if the PHP path is not set correctly. It occures if you select the Site Crawler in Info-module.
In this case you have to set the path to your PHP in the Extension configuration.
Please be sure to add the correct path to your PHP. The path in this screenshot might be different to your PHP path.