Breaking: #110775 - External document parsers of indexed_search replaced by text extractors
See forge#110775
Description
The system extension
typo3/cms-indexed-search
extracted the text
of linked documents such as PDF or Word files with "external parsers". These
duck-typed classes were registered in
$GLOBALS
and were bound to the indexer through a back-reference.
The external parsers are replaced by text extractors, which are registered
as tagged services and used through the
\TYPO3\, see
Feature: #110775 - Structured text extraction in TextExtractorRegistry. This comes with the following breaking
changes:
- The array
$GLOBALSis not evaluated anymore. The class['TYPO3_ CONF_ VARS'] ['EXTCONF'] ['indexed_ search'] ['external_ parsers'] \TYPO3\has been removed. Its work is done by text extractors in the namespaceCMS\ Indexed Search\ File Content Parser \TYPO3\.CMS\ Indexed Search\ Text Extraction - Only files that are part of a File Abstraction Layer (FAL) storage are
indexed. The indexer does not index external URLs anymore, and the
extension setting
indexhas been removed. Files are resolved from links to public files of a storage, and from file dump URLs of non-public storages. Links usingExternal URLs jumpurl, as well as links to files outside of a storage, including absolute file system paths, are not followed anymore. The indexer now handles every linked document as a FAL file: the extractor is selected by the MIME type stored in the FAL index, time and size are read from it, and access to files of non-public storages is checked by the token of the file dump URL. Documents fetched from a URL or read from an arbitrary path have no such record and would need a second indexing path next to FAL, with its own HTTP client and temporary files. JumpURLs are not generated by TYPO3 Core since TYPO3 v7.5. - The modification time, creation time and size of an indexed file are read from the FAL index instead of from the file system.
- The text extractor of a file is selected by the MIME type stored in the
FAL index, instead of by the file extension. A generic MIME type, such as
application/for a Word document, is mapped to the concrete type by the file extension throughzip $GLOBALS. Office documents detected as ZIP archives or OLE compound files are covered by the default configuration. Files stored with a MIME type that is not mapped, such as['TYPO3_ CONF_ VARS'] ['SYS'] ['File Info'] ['mime Type Compatibility'] application/, are not indexed.octet- stream - The following members of
\TYPO3\have been removed:CMS\ Indexed Search\ Indexer $external_,parsers $index,External Url_ content initialize,External Parsers () index,External Url () get,Url Headers () readandFile Content () file.Content Parts () Indexer->indexhas been replaced byRegular Document () Indexer->index. The array returned byFile Document (File Interface $file, string $file Url, bool $force = false) Indexer->extractdoes not contain the keyHyper Links () localanymore, links are resolved to files byPath Indexer->extract.Links () - The constructors of
Indexer,Search,Controller IndexandSearch Repository Administrationchanged.Repository Indexdoes not accept the external parsers anymore, andSearch Repository->initialize () Administrationhas been removed.Repository->external_ parsers - Text of HTML based documents, such as HTML files, is now always split
into title, meta keywords, meta description and body. The setting
config.does not apply to the content of files. Title, keywords and description of a document are consistently taken from the extractor. A file name is only used as the title if the extractor does not return one.index_ metatags
Impact
Custom external parsers registered in
$GLOBALS
are not called anymore. Documents of the file types they handled are not
indexed anymore, until a text extractor is registered.
Extensions calling or extending the removed methods of the indexer, or relying on the removed classes and properties, will fail with a PHP error.
External URLs and files outside of a FAL storage are not added to the search index anymore. Entries of such documents that already exist in the index are not updated anymore.
Files whose stored MIME type is neither supported by an extractor nor mapped
by
mime are skipped. The time tracker log of the
indexed page lists them with the message "No text extractor found" and their
MIME type. This also affects documents of OpenOffice.org 1.x (sxc,
sxi, sxw) that older versions of libmagic detected
as
application/, which is only mapped for the current Office Open
XML and OpenDocument formats.
An extractor whose external tool is not configured or not found does not support any MIME type, so its file types are neither indexed nor offered in the search form. This is logged once per request or console command with the log level "notice", naming the extractor, its MIME types and the missing tools.
Affected installations
Installations with custom external parsers for the indexed search, with an
extension that interacts with the
Indexer
to index files or URLs, or
that enabled the extension setting
index. The extension
scanner reports usages of the removed API.
Migration
Remove the registration of the parser from ext_localconf.php and
implement a text extractor instead, registered with the attribute
#. The extractor receives the file as a system resource,
so the content can be read with
get:
-$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['indexed_search']['external_parsers']['dwg'] = MyDwgParser::class;
use TYPO3\CMS\Core\Attribute\AsTextExtractor;
use TYPO3\CMS\Core\Resource\TextExtraction\ExtractedText;
use TYPO3\CMS\Core\Resource\TextExtraction\StructuredTextExtractorInterface;
use TYPO3\CMS\Core\SystemResource\Type\SystemResourceInterface;
#[AsTextExtractor]
final class DwgTextExtractor implements StructuredTextExtractorInterface
{
public function getSupportedMimeTypes(): array
{
return ['image/vnd.dwg'];
}
public function extractStructuredText(SystemResourceInterface $resource): ExtractedText
{
// Read $resource->getContents() and return the text
return new ExtractedText($text, $title);
}
}
Files are now selected by their MIME type instead of their file extension. The
file extensions of a MIME type are indexed as soon as the extractor declares
the MIME type in
get. A link to a public file is
only resolved through FAL if its file extension is one of these, so other
linked files are not added to the FAL index. File extensions that must not
be indexed can still be listed in the extension setting
ignore.
The label of a file type in the media type selector of the search form, which
the method
search of a parser provided before, is the
label
media of the plugin. A file type without a
label is not offered in the selector. Add it with TypoScript, for example:
plugin.tx_indexedsearch._LOCAL_LANG.default.mediaTypes.dwg = AutoCAD drawings
An exception thrown while a file is indexed, for example by an extractor or by the driver of a remote storage, is logged and the file is skipped, the page and other linked files are indexed nevertheless.
If documents are skipped because the file information of the server stores
them with a generic MIME type, check the column
mime_ of the table
sys_. Fix the MIME type detection of the server, for example by
updating the libmagic database used by the PHP extension
fileinfo
, then correct the stored MIME type of the affected files in
the FAL index and let the site be indexed again.
Do not map generic MIME types such as
application/ through
$GLOBALS
to make such files indexable. The setting is also applied when the MIME type
of an uploaded file is detected and in the consistency check of file
extension and MIME type of uploads, so such a mapping weakens the validation
of uploaded files.
Documents that were indexed through a URL, or that are stored outside of a FAL
storage, have to be moved into a storage to be indexed. Use the
Indexer->index method instead of
Indexer->index when indexing a file from PHP.
Index entries of such documents that already exist stay searchable until they
are removed. Delete them in the backend module Indexing, in the
view Detailed statistics, which lists the documents indexed for
each page. To remove all of them at once, empty the tables
index_,
index_,
index_,
index_,
index_ and
index_, and let the
site be indexed again.