DEPRECATION WARNING
This documentation is not using the current rendering mechanism and is probably outdated. The extension maintainer should switch to the new system. Details on how to use the rendering mechanism can be found here.
EXT: sa_robotstxt¶
| Author: | Kasper Skårhøj |
|---|---|
| Created: | 2002-11-01T00:32:00 |
| Changed by: | Stefan Alpers |
| Changed: | 2005-08-23T19:05:38 |
| Email: | typo3@sa-net.de |
EXT: sa_robotstxt¶
Extension Key: sa_robotstxt
Copyright 2000-2002, typo3@sa-net.de, <typo3@sa-net.de>
This document is published under the Open Content License
available from http://www.opencontent.org/opl.shtml
The content of this document is related to TYPO3
- a GNU/GPL CMS/Framework available from www.typo3.com
Table of Contents¶
EXT: sa_robotstxt 1
Introduction 1
What does it do? 1
Screenshots 1
Users manual 1
FAQ 2
Adminstration 2
FAQ 2
Configuration 2
FAQ 2
Known problems 2
To-Do list 2
Changelog 2
Introduction¶
What does it do?¶
This extension enables enables the administrator to set different robots.txt files for each sys_domain records.
The Apache module mod_rewrite must be installed for this extension!
Screenshots¶
Users manual¶
When the extension is installed just select the file you want to use as a robots.txt for the particular domain. The extension will serve this file instead of the default robots.txt.
FAQ¶
Q: Why do I need a customized robots.txt file?
A: This might be useful when you are hosting a public and a semi public (extranet) domain on the same server and you don't want you extranet login to be indexed by search engines.
Configuration¶
The Apache module mod_rewrite must be installed for this extension!
Install the extension through the Extension Manager. Then copy the file robots.php from the extension's res folder to your Typo3 base directory. This script will handle all future spider requests. At last you have to configure the URL rewriting so all requests to robots.txt actually go to robots.php. Open the .htaccess file in your Typo3 base directory and add the following line:
RewriteRule ^robots.txt$ robots.php [NC]
The file rewrite.txt in the res folder contains an example of working .htaccess file. To test the extension rename an existing robots.txt in your Typo3 base folder to robots.old and point your web browser to http://webserver/robots.txt . The web server will return a generated default robots.txt which allows indexing for all directories.
User-agent: *
Disallow:
# robots.txt for webserver
# Generated by TYPO3
# sa_robotstxt by Stefan Alpers
# http://www.sa-net.de
A global default can be set by placing a robots.txt in the base directory of the Typo3 installation. For some samples look in extension directory.
If a file is set for the domain it will be shown instead.
Known problems¶
None so far
To-Do list¶
Don't know
Changelog¶
initial version
bug fix for longer domain names
database defaults added
EXT: sa_robotstxt - 2