Local file
It is also possible to provide the contents of your robots. in a local
robots. file in the web root of your TYPO3 installation. If configured,
the extension tries to inject XML sitemaps during the request process the same way
as in the usage of static routes.
As with static routes, the injection of XML sitemaps must be explicitly enabled in the appropriate site configuration.
Important
Your web server must be properly configured to route requests to an existing
robots. to TYPO3's entrypoint at index.. Otherwise, the
underlying middleware will never be called, thus being unable to inject XML
sitemaps during the request process.
Read more in the web server configuration section below.
Web server configuration
In order to route requests to an existing robots. file to TYPO3's main
entrypoint, your web server configuration must be modified. This depends on which
server type you're actually using.
Apache
Add the following to the .htaccess in the web root of your TYPO3 installation:
RewriteEngine on
RewriteRule ^robots.txt /index.php [L]Nginx
Add the following to the configuration file of your Nginx server:
location = /robots.txt {
rewrite ^ /index.php;
}