Feature: #110775 - Structured text extraction in TextExtractorRegistry
See forge#110775
Description
Text extractors, used to extract the textual content from files, can now also provide the metadata they find in a file, declare the MIME types they support, and work on any system resource: files of a FAL storage as well as files of packages or public files outside of any storage.
A text extractor implementing the new interface
\TYPO3\
receives a
\TYPO3\
and returns an
\TYPO3\
object, which holds the text together with a title, a description and
keywords. FAL files implement
System, so the same
extractor serves both:
use TYPO3\CMS\Core\Attribute\AsTextExtractor;
use TYPO3\CMS\Core\Resource\TextExtraction\ExtractedText;
use TYPO3\CMS\Core\Resource\TextExtraction\StructuredTextExtractorInterface;
use TYPO3\CMS\Core\SystemResource\Type\SystemResourceInterface;
#[AsTextExtractor(priority: 10)]
final class PdfTextExtractor implements StructuredTextExtractorInterface
{
public function getSupportedMimeTypes(): array
{
return ['application/pdf'];
}
public function extractStructuredText(SystemResourceInterface $resource): ExtractedText
{
// Read $resource->getContents() ...
return new ExtractedText($text, $title, $description, $keywords);
}
}
get returns the lowercase MIME types which the
extractor can currently handle. The registry selects the extractor by the MIME
type of the resource, not by its file extension. An extractor that depends on
an external tool returns an empty list as long as the tool is not available.
This allows to list the supported file types without having a file at hand,
for example to offer them in a search form.
File types which are detected differently depending on the environment, for
example Office Open XML documents detected as ZIP archives, are resolved to one
MIME type by the file extension through the existing setting
$GLOBALS,
so an extractor only declares the official MIME type. FAL applies the setting
when it detects the MIME type of a file. The registry applies it as well, if
no extractor supports the stored MIME type, so files indexed before an entry
was added are found without updating the file index. The default entries for
Office documents are described in Important: #111287 - Office documents detected as container format are accepted.
The
\TYPO3\
provides three new methods:
findreturns the extractor responsible for the resource. All extractors are asked in the order of theirText Extractor (File Interface |System Resource Interface $resource) #priority, whichever interface they implement: a structured extractor matches if it supports the MIME type of the resource, any other extractor if its[As Text Extractor] canaccepts the FAL file. A MIME type mapped byExtract Text () mimeis only considered if no extractor matches.Type Compatibility extractextracts the text with the extractor returned by(File Interface |System Resource Interface $resource): ?Extracted Text find. The text of an extractor implementing onlyText Extractor () Textis wrapped into anExtractor Interface Extractedobject. A file reference is resolved to its original file.Text getreturns the file extensions known for the MIME types declared by all registered structured text extractors, for consumers that offer file types for selection.Supported File Extensions (): array
The
Plain shipped with TYPO3 Core implements both
interfaces and declares the MIME type
text/.
The interface
Text, the method
Text and the attribute
# are unchanged, so existing text extractors and their
consumers keep working. Both interfaces stay supported in TYPO3 v15, because
existing consumers call
get and expect a
Text, while
find and
extract serve extractors of both interfaces.
New text extractors should implement
Structured.
Implement
Text in addition only if the extractor
shall also be returned by
get, as the extractors shipped
with TYPO3 do.
Impact
Consumers of text extractors, such as the search indexing of
typo3/cms-indexed-search
, can use the title, description and
keywords of a file, for example to weight them higher than the body text, and
can extract the text of files that are not part of a FAL storage. Extension
authors can implement
Structured to provide these
values, and to make the file types they support visible to consumers.