Indexing architecture 

Indexing converts external or TYPO3-managed content into vector documents.

Core classes 

Madj2k\AiCore\Indexing\Indexer\IndexerInterface
Common interface for indexers.
Madj2k\AiAssistant\Indexing\Indexer\AbstractIndexer
TYPO3 adapter for configuration lookup and persisted source state.
Madj2k\AiCore\Indexing\DTO\IndexableDocument
Text plus metadata to be indexed.
Madj2k\AiCore\DTO\DocumentMetadata
Structured metadata for source identity and payload.
Madj2k\AiCore\Indexing\TextChunker
Splits text into chunks.
Madj2k\AiCore\Indexing\VectorDocumentIndexer
Creates embeddings and replaces vector documents safely.
Madj2k\AiAssistant\Indexing\Service\SourceStateService
Creates source hashes, content hashes and source state decisions.

Indexing stages 

  1. Resolve indexer configuration.
  2. Read source content.
  3. Build IndexableDocument with metadata.
  4. Detect source changes.
  5. Split text into chunks.
  6. Request embeddings.
  7. Write vector documents.
  8. Update source state.

Adapters and connectors 

File adapters extract text from files. Source connectors fetch external data. Indexers coordinate those services and decide how data is transformed into indexable documents.

JSON and JSONL handling 

JSON and JSONL indexers can separate searchable text from metadata. The configuration distinguishes json_text_fields and json_metadata_fields. Both use comma-separated dot notation and support nested paths, numeric array indices and wildcards.

Metadata mapping can write values to custom payload keys using the syntax payload_key=json.path. This allows a source field such as author.name to be stored as contact_name in the vector payload.

See JSON and JSONL indexing for details.