..  include:: /Includes.rst.txt

..  _administrator:

=============
Administrator
=============

..  _installation:

Installation
============

Install the extension with Composer:

..  code-block:: bash

    composer require web-vision/ai-llms-txt

Then set up the extension, so that the TYPO3 caches are flushed:

..  code-block:: bash

    vendor/bin/typo3 extension:setup

The extension is also available in the TYPO3 Extension Repository (TER) under
the key ``ai_llms_txt``. Classic mode installations are not covered by the
automated tests.

..  _setup:

Setup
=====

The extension works without further configuration: it registers its page
types automatically and is enabled for every site.

1.  Import the route enhancer into each site configuration to get the
    ``/llms.txt`` and ``.md`` URLs, see :ref:`configuration-routes`.
2.  Optionally adjust the title, summary and other values in the site
    configuration, see :ref:`configuration-site`.
3.  To keep a site out of it entirely, disable
    :confval:`site-llmsTxtEnabled` for that site.

..  _urls:

Available URLs
==============

..  list-table::
    :header-rows: 1
    :widths: 55 45

    *   - URL
        - Output
    *   - ``https://example.com/llms.txt``
        - ``llms.txt`` (with route enhancer)
    *   - ``https://example.com/.well-known/llms.txt``
        - ``llms.txt`` (with route enhancer)
    *   - ``https://example.com/about.md``
        - Markdown version of ``/about`` (with route enhancer)
    *   - ``https://example.com/?type=1699``
        - ``llms.txt`` (always available)
    *   - ``https://example.com/about?type=1701``
        - Markdown version of ``/about`` (always available)

..  _testing-installation:

Testing the installation
========================

1.  Open ``https://example.com/llms.txt``. It starts with the site title as
    a ``#`` heading, followed by the summary and the list of pages.
2.  Append ``.md`` to the URL of any regular page, for example
    ``https://example.com/about.md``. The page content is shown as plain
    Markdown.

..  _cli-export:

Exporting pages as Markdown files
=================================

The command ``ai-llms:download-markdown`` saves the Markdown version of pages
as ``.md`` files, for example to feed them into an AI knowledge base. It
exports the root page of a site and every page below it that ``llms.txt`` can
list: pages hidden in menus or excluded from indexing are left out, as are the
page types of ``llmsTxtExcludeDoktypes``. The subpages of folders and of
excluded page types are included.

..  code-block:: bash

    # Export all sites to var/markdown-export/
    vendor/bin/typo3 ai-llms:download-markdown

    # Export one site, with YAML frontmatter, and preview first
    vendor/bin/typo3 ai-llms:download-markdown main --include-metadata --dry-run

..  list-table::
    :header-rows: 1
    :widths: 30 70

    *   - Argument / option
        - Description
    *   - ``siteIdentifier``
        - Site to export. Without it, all sites are exported.
    *   - ``-o``, ``--output-dir``
        - Target directory, relative to the project root.
          Default: ``var/markdown-export``.
    *   - ``-l``, ``--limit``
        - Maximum number of pages to export. Default: ``0`` (no limit).
    *   - ``-f``, ``--format``
        - File naming: ``slug`` (default; ``/about-us/team`` becomes
          ``about-us-team.md``, the root page ``index.md``), ``uid``
          (``page-5.md``) or ``title``.
    *   - ``-m``, ``--include-metadata``
        - Prepend YAML frontmatter with uid, title, description, URL, site
          and export time.
    *   - ``--flat``
        - Write all files into one directory instead of one per site.
    *   - ``--overwrite``
        - Overwrite existing files.
    *   - ``--dry-run``
        - List the files that would be written, without creating any file or
          directory.

The command fetches the Markdown version of each page over HTTP, so the site
must be reachable from the server under its configured base URL, without a
redirect. Redirects are not followed: a page that redirects, such as an
"External URL" page, is skipped, so that the server never requests the address
an editor entered. It ends with a non-zero exit code if a page or site could
not be processed.

..  _troubleshooting:

Troubleshooting
===============

``/llms.txt`` returns "page not found"
    The route enhancer is not imported into the site configuration, see
    :ref:`configuration-routes`. ``/?type=1699`` works without any routing
    and helps to tell a routing problem apart from other causes.

``/.well-known/llms.txt`` is not reachable
    Some web server configurations block paths starting with a dot. Allow
    ``/.well-known/`` to be passed to TYPO3, or use ``/llms.txt``.

A page's ``.md`` URL returns "page not found"
    This is intended when the extension is disabled for the site, or when
    the page has :guilabel:`Index this page` switched off (``no_index``).

A page is missing from ``llms.txt``
    Check the conditions in :ref:`configuration-pages`. If caching is
    enabled and the page was changed outside the backend, for example by its
    start time passing, it appears once the cache entry expires.

``llms.txt`` contains an "error: Failed to generate content" line
    Generation failed. The cause is logged through the TYPO3 logging
    framework (by default in :file:`var/log/`), not shown in the response.

..  _performance:

Performance
===========

``llms.txt`` is generated with batched database queries per tree level rather
than one query per page. On large sites, keep
:confval:`site-llmsTxtMaxDepth` low and set :confval:`site-llmsTxtCacheLifetime`
so the file is not rebuilt on every request.

The Markdown version of a page renders the full page through TYPO3 and
converts it, so it takes about as long as rendering the HTML page.