How it works 

The analysis pipeline 

Everything below happens inside the scheduler task.

1. Scope. The site is resolved from startPageId, and the languages to process are taken from its site configuration (all of them, or the one given by languageId).
2. Page collection. The subtree below startPageId is walked to full depth, per language. Deleted and hidden pages are skipped, as are the pages listed in excludePages.
3. Text preparation. For every page, its language is resolved from the site configuration (see Language handling), then each field of analyzedFields is read. content falls back to the bodytext of the page's content elements in that language when the page field itself is empty. Stop words of that language are removed, then — if enableStemming is on — the text is stemmed with that language's stemmer.
4. Vectorisation. Pages are grouped by that same language code and each group is vectorised together, so all pages of one language share a single vocabulary and IDF corpus. Every page is vectorised exactly once, not once per comparison.

5. Comparison. Every page is compared with every other page of the same language using cosine similarity on those vectors, then combined with the recency term:

   score = cosine(vector1, vector2) * (1 - recencyWeight)
         + recencyBoost              * recencyWeight

   Copied!

   Pairs of different languages score 0.0 and are dropped.

6. Storage. Pairs scoring at or above the task's quality level are written to tx_semanticsuggestion_similarities, in one transaction, in batches of 100. The rows of the previous run of this task — same site, same scope, same language — are deleted first, so a run never disturbs another task's results.

The frontend does none of this: it reads the stored rows for the current page, filters them by the display quality level, excludePages and the language, and renders at most maxSuggestions of them.

Reading the scores 

The score is a cosine similarity between TF-IDF vectors, mixed with the recency term. Two consequences worth keeping in mind:

• Scores are lower than a naive word-count similarity would give. Real values on a normal site cluster well below 0.5; 0.3 is a reasonable threshold and 0.8 will match nothing. If you are migrating from a 2.x configuration, lower your thresholds.
• The recency term can carry a pair on its own. recencyBoost is the absolute difference between the two pages' normalised ages over a 30-day window, so with the default recencyWeight = 0.2 a pair with no textual similarity at all can still score 0.2. Set the weight to 0 for a purely textual score.

The relevance labels shown in the backend module are thresholds on that score: High above 0.8, Medium above 0.6, Low below.

Language handling 

The language is resolved per page, once, and the same value is then used for stop word removal, stemming and vectorisation. The order is:

1. The site configuration. The locale of the page's language, reduced to its two-letter code (de_DE.UTF-8 → de). This is the normal case and it always wins for a page belonging to a configured site.
2. Content analysis, by nlp_tools, only when the step above found nothing — a page outside any configured site.
3. defaultLanguage, when there is not even any text to analyse.

Stop words, stemming and detection profiles come from nlp_tools, which supports six languages:

Any other language still gets TF-IDF vectorisation, which is what does most of the work; it just loses stemming and gets an unhelpful stop word list. Note also that content-based detection can only ever return one of the six codes above.

German is the language this extension was tuned on: the Snowball stemmer is what makes Automobilindustrie and Automobil share a stem, and umlauts are handled. A lower quality level than for English (0.25 instead of 0.3) is a reasonable starting point on a German site.

Known limitations 

• Field weights are coarse, being implemented as text repetition — see analyzedFields.
• The comparison is quadratic in the number of pages of a scope. Split a very large site into several tasks on subtrees rather than raising the quality level to compensate.

Caching 

The extension registers a cache named semantic_suggestion (file backend, 24 h, member of the pages group). It holds analysis results and the intermediate nlp_tools results, and entries are tagged site_<rootPageId>.

Invalidation happens on two occasions:

• the scheduler task flushes its site's tag after saving,
• a DataHandler hook flushes the tag of the affected site when a page or a content element is changed, moved or deleted. Records being created carry a NEW… placeholder instead of a UID and cannot be resolved to a site at that point; they are picked up by the next analysis run anyway.

# flush everything, including this cache
vendor/bin/typo3 cache:flush