Breaking: #110775 - External document parsers of indexed_search replaced by text extractors 

See forge#110775

Description 

The system extension typo3/cms-indexed-search extracted the text of linked documents such as PDF or Word files with "external parsers". These duck-typed classes were registered in $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['indexed_search']['external_parsers'] and were bound to the indexer through a back-reference.

The external parsers are replaced by text extractors, which are registered as tagged services and used through the \TYPO3\CMS\Core\Resource\TextExtraction\TextExtractorRegistry , see Feature: #110775 - Structured text extraction in TextExtractorRegistry. This comes with the following breaking changes:

  • The array $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['indexed_search']['external_parsers'] is not evaluated anymore. The class \TYPO3\CMS\IndexedSearch\FileContentParser has been removed. Its work is done by text extractors in the namespace \TYPO3\CMS\IndexedSearch\TextExtraction .
  • Only files that are part of a File Abstraction Layer (FAL) storage are indexed. The indexer does not index external URLs anymore, and the extension setting indexExternalURLs has been removed. Files are resolved from links to public files of a storage, and from file dump URLs of non-public storages. Links using jumpurl , as well as links to files outside of a storage, including absolute file system paths, are not followed anymore. The indexer now handles every linked document as a FAL file: the extractor is selected by the MIME type stored in the FAL index, time and size are read from it, and access to files of non-public storages is checked by the token of the file dump URL. Documents fetched from a URL or read from an arbitrary path have no such record and would need a second indexing path next to FAL, with its own HTTP client and temporary files. JumpURLs are not generated by TYPO3 Core since TYPO3 v7.5.
  • The modification time, creation time and size of an indexed file are read from the FAL index instead of from the file system.
  • The text extractor of a file is selected by the MIME type stored in the FAL index, instead of by the file extension. A generic MIME type, such as application/zip for a Word document, is mapped to the concrete type by the file extension through $GLOBALS['TYPO3_CONF_VARS']['SYS']['FileInfo']['mimeTypeCompatibility'] . Office documents detected as ZIP archives or OLE compound files are covered by the default configuration. Files stored with a MIME type that is not mapped, such as application/octet-stream , are not indexed.
  • The following members of \TYPO3\CMS\IndexedSearch\Indexer have been removed: $external_parsers , $indexExternalUrl_content , initializeExternalParsers() , indexExternalUrl() , getUrlHeaders() , readFileContent() and fileContentParts() .
  • Indexer->indexRegularDocument() has been replaced by Indexer->indexFileDocument(FileInterface $file, string $fileUrl, bool $force = false) . The array returned by Indexer->extractHyperLinks() does not contain the key localPath anymore, links are resolved to files by Indexer->extractLinks() .
  • The constructors of Indexer , SearchController , IndexSearchRepository and AdministrationRepository changed. IndexSearchRepository->initialize() does not accept the external parsers anymore, and AdministrationRepository->external_parsers has been removed.
  • Text of HTML based documents, such as HTML files, is now always split into title, meta keywords, meta description and body. The setting config.index_metatags does not apply to the content of files. Title, keywords and description of a document are consistently taken from the extractor. A file name is only used as the title if the extractor does not return one.

Impact 

Custom external parsers registered in $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['indexed_search']['external_parsers'] are not called anymore. Documents of the file types they handled are not indexed anymore, until a text extractor is registered.

Extensions calling or extending the removed methods of the indexer, or relying on the removed classes and properties, will fail with a PHP error.

External URLs and files outside of a FAL storage are not added to the search index anymore. Entries of such documents that already exist in the index are not updated anymore.

Files whose stored MIME type is neither supported by an extractor nor mapped by mimeTypeCompatibility are skipped. The time tracker log of the indexed page lists them with the message "No text extractor found" and their MIME type. This also affects documents of OpenOffice.org 1.x (sxc, sxi, sxw) that older versions of libmagic detected as application/zip , which is only mapped for the current Office Open XML and OpenDocument formats.

An extractor whose external tool is not configured or not found does not support any MIME type, so its file types are neither indexed nor offered in the search form. This is logged once per request or console command with the log level "notice", naming the extractor, its MIME types and the missing tools.

Affected installations 

Installations with custom external parsers for the indexed search, with an extension that interacts with the Indexer to index files or URLs, or that enabled the extension setting indexExternalURLs . The extension scanner reports usages of the removed API.

Migration 

Remove the registration of the parser from ext_localconf.php and implement a text extractor instead, registered with the attribute #[AsTextExtractor] . The extractor receives the file as a system resource, so the content can be read with getContents() :

EXT:my_extension/ext_localconf.php
-$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['indexed_search']['external_parsers']['dwg'] = MyDwgParser::class;
Copied!
EXT:my_extension/Classes/TextExtraction/DwgTextExtractor.php
use TYPO3\CMS\Core\Attribute\AsTextExtractor;
use TYPO3\CMS\Core\Resource\TextExtraction\ExtractedText;
use TYPO3\CMS\Core\Resource\TextExtraction\StructuredTextExtractorInterface;
use TYPO3\CMS\Core\SystemResource\Type\SystemResourceInterface;

#[AsTextExtractor]
final class DwgTextExtractor implements StructuredTextExtractorInterface
{
    public function getSupportedMimeTypes(): array
    {
        return ['image/vnd.dwg'];
    }

    public function extractStructuredText(SystemResourceInterface $resource): ExtractedText
    {
        // Read $resource->getContents() and return the text
        return new ExtractedText($text, $title);
    }
}
Copied!

Files are now selected by their MIME type instead of their file extension. The file extensions of a MIME type are indexed as soon as the extractor declares the MIME type in getSupportedMimeTypes() . A link to a public file is only resolved through FAL if its file extension is one of these, so other linked files are not added to the FAL index. File extensions that must not be indexed can still be listed in the extension setting ignoreExtensions .

The label of a file type in the media type selector of the search form, which the method searchTypeMediaTitle() of a parser provided before, is the label mediaTypes.<file extension> of the plugin. A file type without a label is not offered in the selector. Add it with TypoScript, for example:

EXT:my_extension/Configuration/Sets/MySet/setup.typoscript
plugin.tx_indexedsearch._LOCAL_LANG.default.mediaTypes.dwg = AutoCAD drawings
Copied!

An exception thrown while a file is indexed, for example by an extractor or by the driver of a remote storage, is logged and the file is skipped, the page and other linked files are indexed nevertheless.

If documents are skipped because the file information of the server stores them with a generic MIME type, check the column mime_type of the table sys_file . Fix the MIME type detection of the server, for example by updating the libmagic database used by the PHP extension fileinfo , then correct the stored MIME type of the affected files in the FAL index and let the site be indexed again.

Do not map generic MIME types such as application/octet-stream through $GLOBALS['TYPO3_CONF_VARS']['SYS']['FileInfo']['mimeTypeCompatibility'] to make such files indexable. The setting is also applied when the MIME type of an uploaded file is detected and in the consistency check of file extension and MIME type of uploads, so such a mapping weakens the validation of uploaded files.

Documents that were indexed through a URL, or that are stored outside of a FAL storage, have to be moved into a storage to be indexed. Use the Indexer->indexFileDocument() method instead of Indexer->indexRegularDocument() when indexing a file from PHP.

Index entries of such documents that already exist stay searchable until they are removed. Delete them in the backend module Indexing, in the view Detailed statistics, which lists the documents indexed for each page. To remove all of them at once, empty the tables index_phash , index_fulltext , index_rel , index_section , index_grlist and index_words , and let the site be indexed again.