Register the following hooks in ext_localconf.php of your extension.

excludeDoktype Hook

By adding doktype ids to following array you can exclude them from being crawled:

$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['crawler']['excludeDoktype'][] = <dokTypeId>

pageVeto Hook

You can also decide whether a page should not be crawled in an individual userfunction. Register your function here:

$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['crawler']['pageVeto'][] = Vendor\YourExt\Hooks\Crawler\PageVeto::class . '->excludePage';



namespace Vendor\YourExt\Hooks\Crawler;

use AOE\Crawler\Controller\CrawlerController;

class PageVeto
   public function excludePage(array &$params, CrawlerController $controller)
      if ($params['pageRow']['uid'] === 42) {
         return 'Page with uid "42" is excluded by page veto hook');

      return false;