EXT: robots.txt for multidomain¶

What does Dev/Null Robots.txt do?¶

This extension generates domain specific robots.txt files. Configuration is done from in sys_domain records and typoscript.

Features¶

• Preconfigured TS-Templates for robots.txt
• Different robots.txt per domain
• Supports RealURL auto-configuration

Credits¶

Automate RealURL configuration where possible – Jacob Floyd < jacob@cognifire.com >

Usage¶

This extension works nearly out of the box, especially if you use RealURL with automatic configuration. If you don't have that, then only URL-rewriting has to be configured. (See the Administration section for help configuring URL-Rewriting)

Assuming your website starts with id=1, make sure your robots.txt is working with the following url:

http://www.yourdomain.tld/index.php?id=1&type=1964

Next, configure URL-rewriting for each domain, as necessary. (See Administration )

[Optional] Add additional robots.txt configuration in domain records.

(See Domain Record Configuration and Crawler-specific Configuration )

When you're done, make sure that it works by visiting: http://www.yourdomain.tld/robots.txt

[Optional] Add any page specific meta-data (See Page Meta Tags )

Cooluri¶

If you use the extension Cooluri for URL-rewriting you have to define the link for each domain that should serve a robots.txt

RealURL¶

If you use RealURL's automatic configuration, you're in luck; RealURL auto-configuration is supported out of the box. You may have to delete your realurl_autoconf.php file in typo3conf to get RealURL to regenerate it.

However, you must manually update your RealURL configuration if: (a) you have customized your RealURL re-writing configuration; or (b) you have changed the default typeNum setting. To manually make RealURL serve your robots.txt, add the following to your RealURL configuration (see the RealURL documentation for details on how to do this). The default configuration puts this block in the 'fileName' part of the configuration array.

'index' => array(
  'robots.txt' => array(
    'keyValues' => array(
    'type' => '1964',
  ),
),

Remember: If you change your typeNum you have to change it here too!

.htaccess¶

This method is not recommended. It does not work in conjunction with RealURL, and n it hasn't been tested in any other siituations. You may have to create a rewrite rule for each domain.

RewriteCond  %{HTTP_HOST}  www.yourdoman.tld
RewriteRule  ^(robots\.txt)$ index.php?id=1&type=1964 [L,NC]

RewriteRule .* index.php [L]

### End: Rewrite stuff ###

Crawler Config List¶

List of crawler specific configurations. This will be inserted before the default configuration.

Default Crawler Directives¶

Default Disallow and Allow statements without User-agent. To disallow all for the given domain just enter

Disallow: /

User-agent¶

Crawler config¶

((generated))¶

<meta name="robots" content="noindex,nofollow" />

Reference¶

0 index,follow
1 noindex,follow
2 index,nofollow
3 noindex,nofollow

Output Defaults for robots.txt (useDefaults)¶

User-agent: *
Disallow: /fileadmin
Disallow: /t3lib
Disallow: /typo3conf
Disallow: /typo3temp
Allow:    /fileadmin/media

User-agent: *
Disallow: /

Feature request & Bugs¶

If you find a bug or a missing feature, please add it to the bugtracker of the extension at forge.

The address is:

http://forge.typo3.org/projects/extension-dev_null_robots/issues

For posting a bug or feature request you need an account of typo3.org.

You can see this extension in operation on one of my websites

http://www.dev-null.at

http://www.letsdive.at

2.0.0¶

1.4.0¶

1.3.0¶

1.2.0¶

1.1.1¶

1.1.0¶

1.0.1¶

1.0.0¶

12