---
title: "Indexing architecture"
manual: "AI Assistant"
version: "main"
source: "Developers/Architecture/IndexingArchitecture.rst"
rendered: "2026-10-03T14:54:45+00:00"
---

# Indexing architecture {#indexing-architecture}

Indexing converts external or TYPO3-managed content into vector documents.

## Core classes {#core-classes}

-   **`Madj2k\AiCore\Indexing\Indexer\IndexerInterface`**

    Common interface for indexers.

-   **`Madj2k\AiAssistant\Indexing\Indexer\AbstractIndexer`**

    TYPO3 adapter for configuration lookup and persisted source state.

-   **`Madj2k\AiCore\Indexing\DTO\IndexableDocument`**

    Text plus metadata to be indexed.

-   **`Madj2k\AiCore\DTO\DocumentMetadata`**

    Structured metadata for source identity and payload.

-   **`Madj2k\AiCore\Indexing\TextChunker`**

    Splits text into chunks.

-   **`Madj2k\AiCore\Indexing\VectorDocumentIndexer`**

    Creates embeddings and replaces vector documents safely.

-   **`Madj2k\AiAssistant\Indexing\Service\SourceStateService`**

    Creates source hashes, content hashes and source state decisions.

## Indexing stages {#indexing-stages}

1.  Resolve indexer configuration.
1.  Read source content.
1.  Build `IndexableDocument` with metadata.
1.  Detect source changes.
1.  Split text into chunks.
1.  Request embeddings.
1.  Write vector documents.
1.  Update source state.

## Adapters and connectors {#adapters-and-connectors}

File adapters extract text from files. Source connectors fetch external data.
Indexers coordinate those services and decide how data is transformed into
indexable documents.

## JSON and JSONL handling {#json-and-jsonl-handling}

JSON and JSONL indexers can separate searchable text from metadata. The
configuration distinguishes `json_text_fields` and `json_metadata_fields`.
Both use comma-separated dot notation and support nested paths, numeric array
indices and wildcards.

Metadata mapping can write values to custom payload keys using the syntax
`payload_key=json.path`. This allows a source field such as `author.name` to
be stored as `contact_name` in the vector payload.

See [JSON and JSONL indexing](JsonJsonlIndexing.html#json-jsonl-indexing-architecture) for details.