What is llms.txt? 

llms.txt is an emerging standard for providing AI and Large Language Model (LLM) crawlers with structured information about a website. It serves as a machine-readable index that helps AI systems understand your website's content and structure.

The llmstxt.org specification defines how websites can provide this information in a standardized format.

Concept 

This extension provides a two-tier approach for LLM content access:

1. llms.txt Index File - A single file containing:

   • Website metadata (title, description, domain)
   • Page structure with SEO descriptions and keywords
   • Instructions for accessing full page content

2. Markdown Content Access - Access any page content via:

   • .md suffix - Returns clean Markdown with YAML frontmatter
   • Example: /about.md returns the "About" page as Markdown

Multi-Language Support 

Instead of generating separate llms.txt files per language, this extension uses a simpler approach:

• Single llms.txt - Contains the site structure in the default language

• Language-specific content - Access any page in any language using the .md suffix with language URL prefix:

  • Default: https://example.com/about.md
  • English: https://example.com/en/about.md
  • German: https://example.com/de/ueber-uns.md

This approach follows how multi-language sites actually work in TYPO3.

Features 

Requirements 

• TYPO3 13.0 - 14.x
• PHP 8.2 or higher

Supported Content Elements 

The extension converts the following TYPO3 content elements to Markdown:

• Header (header)
• Text (text)
• Text with Image (textpic, textmedia)
• Image (image)
• Bullet List (bullets)
• Table (table)
• HTML (html)
• Menu elements (menu_*)
• All other elements via HTML-to-Markdown fallback

Installation via Composer (Recommended) 

Install the extension via Composer:

composer require rtfirst/llms-txt

Then activate the extension and clear the cache:

vendor/bin/typo3 extension:setup
vendor/bin/typo3 cache:flush

For DDEV environments:

ddev composer require rtfirst/llms-txt
ddev typo3 extension:setup
ddev typo3 cache:flush

See also Installing extensions in the TYPO3 Getting Started documentation.

Installation in Classic Mode 

1. Download the extension from the TYPO3 Extension Repository (TER) or from GitHub.
2. Install the extension via the Extension Manager in the TYPO3 Backend.
3. Clear all caches.

Activate the Site Set 

After installation, you need to add the Site Set to your site configuration:

1. Go to Site Management > Sites in the TYPO3 Backend.
2. Edit your site configuration.
3. Go to the Sets tab.
4. Add the set LLMs.txt Generator (rtfirst/llms-txt).
5. Save and clear all caches.

Verify Installation 

After installation, verify that everything works:

1. Access https://your-site.com/llms.txt - You should see the generated llms.txt content.
2. Access any page with .md suffix, e.g., https://your-site.com/about.md - You should see Markdown content with YAML frontmatter.
3. Check the page properties of any page - You should see a new LLM tab.

Site Settings 

After adding the Site Set to your site configuration (see Activate the Site Set), you can configure the extension in Site Management > Settings.

Page Properties 

Each page has an LLM tab in the page properties with the following fields:

robots.txt Configuration 

To allow AI crawlers to discover and access your llms.txt file, add these lines to your public/robots.txt:

# Allow AI crawlers to access llms.txt
User-agent: GPTBot
Allow: /llms.txt

User-agent: Claude-Web
Allow: /llms.txt

User-agent: Anthropic-AI
Allow: /llms.txt

User-agent: Google-Extended
Allow: /llms.txt

HTML Header Link 

The extension automatically adds a <link> tag to all HTML pages to help AI crawlers discover the llms.txt file:

<link rel="alternate" type="text/plain" href="/llms.txt" title="LLM Content Guide">

Accessing llms.txt 

The llms.txt index file is available at the root of your website:

https://example.com/llms.txt

This file contains:

• Website metadata (title, description, domain)
• Page structure with descriptions and keywords
• Instructions for accessing page content in Markdown format

# My Website

> Your expert for quality products and services.

**Specification:** <https://llmstxt.org/>
**Domain:** https://example.com
**Language:** de
**Generated:** 2026-01-31 12:00:00

## LLM-Optimized Content Access

This site provides LLM-friendly Markdown output for all pages:

### Markdown Format
Append `.md` to any page URL to get plain Markdown with YAML frontmatter.
- **Example:** `https://example.com/page-slug.md`

### Multi-Language Access
Use language-specific URL prefixes with the `.md` suffix:
- **Default language:** `https://example.com/page.md`
- **English:** `https://example.com/en/page.md`

## Page Structure

- **[Home](/)**
  Welcome to our website with all important information.
  [Markdown](/index.html.md)

  - **[About](/about/)**
    Learn about our company history and values.
    [Markdown](/about.md)

  - **[Services](/services/)**
    Professional services for your needs.
    *Keywords: services, consulting, support*
    [Markdown](/services.md)

- **[Contact](/contact/)**
  Get in touch with us via phone or email.
  [Markdown](/contact.md)

Accessing Markdown Content 

Append .md to any page URL to get the content as clean Markdown with YAML frontmatter.

https://example.com/about.md

---
title: "About Us"
description: "Learn about our company history and values."
language: en
date: 2024-06-15
lastmod: 2026-01-31
canonical: "/about"
format: markdown
generator: "TYPO3 LLMs.txt Extension"
---

# About Us

> Learn about our company history and values.

## Our History

Our company was founded in 1985...

## Our Values

- Quality and reliability
- Fair and transparent prices
- Personal consultation

https://example.com/index.html.md

https://example.com/.md

Multi-Language Access 

Access page content in different languages using the language URL prefix with the .md suffix:

# German (default language)
https://example.com/ueber-uns.md

# English
https://example.com/en/about.md

# French
https://example.com/fr/a-propos.md

The extension automatically:

• Detects the language from the URL prefix
• Loads the translated page content
• Sets the correct language in the YAML frontmatter

Caching 

The extension uses smart caching for optimal performance:

• llms.txt: Cached and regenerated when TYPO3 cache is cleared
• Markdown output: Cached for 24 hours per page/language combination

To force regeneration:

vendor/bin/typo3 cache:flush

Or in DDEV:

ddev typo3 cache:flush

Content Filtering 

The Markdown output is automatically cleaned for better LLM consumption:

Removed elements:

• Scripts and styles
• Navigation and footer elements
• Sidebar content
• Bootstrap accessibility spans (visually-hidden)
• Empty anchor tags (<a id="c1"></a>)

Preserved elements:

• Main content text
• Headings and structure
• Lists and tables
• Images (converted to Markdown syntax)
• Links (converted to absolute URLs)

Setting Up API Protection 

1. Go to Site Management > Settings in the TYPO3 Backend.
2. Find the LLMs-Text category.
3. Enter your API key in the API Key for Format Access field.
4. Save and clear all caches.

php -r "echo bin2hex(random_bytes(32)) . PHP_EOL;"

ddev exec php -r "echo bin2hex(random_bytes(32)) . PHP_EOL;"

Authenticating Requests 

Pass the API key via HTTP header (recommended):

# Access llms.txt
curl -H "X-LLM-API-Key: your-secret-key" https://example.com/llms.txt

# Access page as Markdown
curl -H "X-LLM-API-Key: your-secret-key" https://example.com/about.md

Or via query parameter:

https://example.com/llms.txt?api_key=your-secret-key
https://example.com/about.md?api_key=your-secret-key

Error Response 

Invalid or missing API key returns HTTP 401 Unauthorized with a JSON body:

{
  "error": "Unauthorized",
  "message": "Valid API key required. Provide via X-LLM-API-Key header or api_key query parameter."
}

Integration Examples 

import requests

headers = {
    "X-LLM-API-Key": "your-secret-key"
}

# Get llms.txt
response = requests.get("https://example.com/llms.txt", headers=headers)
print(response.text)

# Get page as Markdown
response = requests.get("https://example.com/about.md", headers=headers)
print(response.text)

const response = await fetch("https://example.com/llms.txt", {
  headers: {
    "X-LLM-API-Key": "your-secret-key"
  }
});

const content = await response.text();
console.log(content);

# Store API key in environment variable
export LLM_API_KEY="your-secret-key"

# Access llms.txt
curl -H "X-LLM-API-Key: $LLM_API_KEY" https://example.com/llms.txt

# Access multiple pages
for page in about services contact; do
  curl -H "X-LLM-API-Key: $LLM_API_KEY" "https://example.com/${page}.md" > "${page}.md"
done

Behavior When Enabled 

When API key protection is enabled:

1. llms.txt requires authentication
2. All .md endpoints require authentication
3. The HTML header link (<link rel="alternate">) is automatically hidden
4. The llms.txt file includes authentication instructions

Disabling API Protection 

To make endpoints publicly accessible again:

1. Go to Site Management > Settings
2. Clear the API Key for Format Access field
3. Save and clear all caches

The header link will automatically reappear and endpoints will be publicly accessible.

Architecture 

The extension uses three PSR-15 middlewares:

1. UrlSuffixMiddleware - Detects .md suffix and rewrites URLs
2. LlmsTxtMiddleware - Serves the /llms.txt endpoint
3. ContentFormatMiddleware - Transforms HTML to Markdown

Request: /about.md
   │
   ▼
UrlSuffixMiddleware (before site resolver)
   │  Strips .md suffix
   │  Sets request attribute 'llms_txt_format' = 'md'
   │  Rewrites URI to /about
   ▼
TYPO3 Site Resolver
   │
   ▼
LlmsTxtMiddleware (after site, before page resolver)
   │  Handles /llms.txt requests
   ▼
TYPO3 Page Resolver & Frontend
   │
   ▼
ContentFormatMiddleware (after content-length-headers)
   │  Checks for 'llms_txt_format' attribute
   │  Converts HTML response to Markdown
   ▼
Response: Markdown with YAML frontmatter

Services 

Content Converters 

The extension uses a converter pattern for content element to Markdown conversion. Each converter implements ContentConverterInterface:

interface ContentConverterInterface
{
    public function supports(string $cType): bool;
    public function convert(array $record, string $baseUrl): string;
}

<?php
declare(strict_types=1);

namespace Vendor\MyExtension\Converter;

use RTfirst\LlmsTxt\Converter\ContentConverterInterface;

class MyCustomConverter implements ContentConverterInterface
{
    public function supports(string $cType): bool
    {
        return $cType === 'my_custom_element';
    }

    public function convert(array $record, string $baseUrl): string
    {
        $header = $record['header'] ?? '';
        $bodytext = $record['bodytext'] ?? '';

        return "## {$header}\n\n{$bodytext}";
    }
}

services:
  Vendor\MyExtension\Converter\MyCustomConverter:
    tags:
      - name: 'llms_txt.content_converter'
        priority: 100

Event Listeners 

Caching 

The extension uses two cache layers:

1. llms.txt Index Cache (cache_pages)

   • Stores generated llms.txt content per site
   • Invalidated on cache flush

2. Format Output Cache (llms_txt_format)

   • Stores Markdown output per page/language
   • 24-hour default lifetime
   • Part of the pages cache group

Database Schema 

The extension adds fields to the pages table:

Code Quality 

The extension maintains high code quality standards:

• PHPStan Level 8 compliant
• PSR-12 code style (php-cs-fixer)
• Unit tests for converters and services

Run quality checks:

# Static analysis
vendor/bin/phpstan analyse packages/llms_txt --level=8

# Code style check
vendor/bin/php-cs-fixer fix packages/llms_txt --dry-run

# Fix code style
vendor/bin/php-cs-fixer fix packages/llms_txt

# Run tests
vendor/bin/phpunit -c packages/llms_txt/phpunit.xml

Dependencies 

• league/html-to-markdown (^5.1) - HTML to Markdown conversion
• typo3/cms-core (^13.0 || ^14.0)
• typo3/cms-frontend (^13.0 || ^14.0)

General Questions 

Installation Questions 

Configuration Questions 

Usage Questions 

https://example.com/index.html.md

# English
https://example.com/en/about.md

# German
https://example.com/de/ueber-uns.md

vendor/bin/typo3 cache:flush

ddev typo3 cache:flush

API Protection Questions 

curl -H "X-LLM-API-Key: your-key" https://example.com/llms.txt

https://example.com/llms.txt?api_key=your-key

Troubleshooting 

Community Support 

Contact the Author 

For direct support, you can contact the extension author:

• Author: Roland Tfirst
• Email: roland@tfirst.de

Documentation 

• This documentation: You're reading it
• llmstxt.org specification: https://llmstxt.org/
• TYPO3 Documentation: https://docs.typo3.org/

Report Issues 

Found a bug or have a feature request? Please report it on GitHub:

• GitHub Issues

When reporting issues, please include:

1. TYPO3 version (e.g., 13.4.1)
2. PHP version (e.g., 8.2.15)
3. Extension version (e.g., 1.0.5)
4. Steps to reproduce the issue
5. Expected behavior vs. actual behavior
6. Error messages or logs (if any)

Contribute 

Contributions are welcome! The source code is available on GitHub:

• GitHub Repository

To contribute:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run code quality checks (see Code Quality)
5. Submit a pull request

Changelog 

See the CHANGELOG.md for a detailed list of changes in each version.