What does it do? 

Semantic Suggestion builds a "related pages" list from the text of the pages themselves. No category has to be maintained and no link has to be created by hand.

• A scheduler task walks a page tree, turns every page into a TF-IDF vector and compares the vectors pairwise.
• Every pair scoring above a configurable threshold is written to the table tx_semanticsuggestion_similarities.
• A frontend plugin reads that table for the page being rendered and displays the best matches with their title, media and a text excerpt.
• A backend module (Web > Semantic Suggestion) shows what was stored.

Because the frontend only reads pre-computed rows, displaying suggestions costs a single indexed query — all the work happens in the scheduler task.

What it does not do 

• It does not index anything on the fly. A page that was created after the last task run has no suggestions until the task runs again.
• It does not compare pages across sites, nor across languages. Both boundaries are enforced (see Multi-site and multilingual).
• It does not need Solr. A separate extension, semantic_suggestion_solr, can write into the same table from a Solr index; the two producers coexist without erasing each other's rows.

Performance 

The measured task duration depends far more on the average amount of text per page than on the page count alone. Use the backend module to see how many pairs a given quality level actually produces before lowering it.

Credits 

The text processing — language detection, stop words, stemming and TF-IDF vectorisation — is provided by the nlp_tools extension, which in turn uses wamania/php-stemmer (Snowball).

Requirements 

Installation with Composer 

composer require talan-hdf/semantic-suggestion
vendor/bin/typo3 extension:setup --extension=semantic_suggestion
vendor/bin/typo3 cache:flush

extension:setup creates the table tx_semanticsuggestion_similarities. You can also apply the schema from Admin Tools > Maintenance > Analyze Database Structure.

Installation from the TER 

1. Install and activate nlp_tools first — see the warning above.
2. Install semantic_suggestion from Admin Tools > Extensions, or upload the archive to typo3conf/ext/.
3. Activate the extension, then update the database schema.

Upgrading an existing installation 

Coming from 3.x or 4.0.0, two upgrade wizards must run. Skipping the first one makes suggestions disappear for every analysis whose task started on a subtree instead of a site root:

vendor/bin/typo3 upgrade:run semanticSuggestionMigrateRootPageId
vendor/bin/typo3 upgrade:run semanticSuggestionLabelAnalysisRows

They are also listed in Admin Tools > Upgrade > Upgrade Wizard. See Upgrade for what they change and how to verify the result.

First analysis run 

Nothing is displayed until the scheduler task has run once.

1. Create a Semantic Suggestion: Generate Similarities task in the Scheduler module and set its start page to a site root — see Scheduler task.

2. Run it with Execute now, or from the CLI:

   vendor/bin/typo3 scheduler:run --task=<uid>

   Copied!
3. Open Web > Semantic Suggestion to confirm that pairs were stored.
4. Add the plugin to your templates — see Integration.

If the module reports no analysis, run vendor/bin/typo3 semantic:diagnostic and read Troubleshooting.

Scheduler task 

Create a Semantic Suggestion: Generate Similarities task in the Scheduler module. Its fields:

Scheduling: daily or weekly, off-peak. Each run deletes and rewrites only its own rows, identified by root_page_id + scope_page_id + sys_language_uid, so several tasks can coexist — one per site, or one per subtree of a site.

TypoScript 

All settings live under plugin.tx_semanticsuggestion_suggestions.settings and are exposed to the Constant Editor under the category semantic_suggestion. The extension loads its own TypoScript instance-wide, so overrides belong in a site's root template.

Settings that are accepted but ignored 

These appear in older documentation and in configurations carried over from previous versions. No code reads them, so they can be dropped:

Extension configuration 

Admin Tools > Settings > Extension Configuration > semantic_suggestion offers parentPageId, proximityThreshold, maxSuggestions and excerptLength. They are a fallback only:

• the backend module uses them when the corresponding TypoScript setting is missing,
• the analysis uses this section when TypoScript is unreachable (the CLI case described above),
• the frontend plugin never reads them,
• parentPageId is read by no code at all.

Database columns 

tx_semanticsuggestion_similarities holds two different page references, and mixing them up is the usual cause of "my suggestions disappeared":

As a content element 

Add the Semantic Suggestions element (plugin semanticsuggestion_suggestions) wherever an editor should place the list. No TypoScript needed.

In a Fluid template 

The extension registers a reusable TypoScript object. Call it from any Fluid template:

<div class="section section-semantic-suggestion">
    <f:cObject typoscriptObjectPath="lib.semantic_suggestion" />
</div>

Or from TypoScript directly:

page.100 =< lib.semantic_suggestion

lib.semantic_suggestion references — rather than copies — the plugin's view, persistence and settings, so a template override applies to both integration paths at once.

Bootstrap Package 

The extension ships page templates for every Bootstrap Package layout, each placing the suggestions after the main content (colPos 0) and before the bottom content (colPos 9), inside <div class="section section-semantic-suggestion">.

Enable them in the Constant Editor, category semantic_suggestion > Template Integration, or in TypoScript constants:

plugin.tx_semanticsuggestion_suggestions.settings.overrideBootstrapTemplates = 1

Then flush the cache.

Covered layouts: Default, Simple, None, 2Columns, 2Columns2575, 2Columns5050, 2ColumnsOffsetRight, 3Columns, SpecialFeature, SpecialStart, SubnavigationLeft, SubnavigationRight, SubnavigationLeft2Columns, SubnavigationRight2Columns.

Overriding the template 

The plugin renders Templates/Suggestions/List.html. Point TypoScript at your own copy, using index 10 or above — 0 and 1 are taken by the extension:

plugin.tx_semanticsuggestion_suggestions.view {
    templateRootPaths.10 = EXT:my_sitepackage/Resources/Private/Templates/
    partialRootPaths.10  = EXT:my_sitepackage/Resources/Private/Partials/
    layoutRootPaths.10   = EXT:my_sitepackage/Resources/Private/Layouts/
}

Variables available in List.html:

Styling 

The shipped template reuses Bootstrap card markup, so it inherits the site's styles when Bootstrap is present. The classes it emits:

/* wrapper, only present when you add it yourself or use the shipped page templates */
.section-semantic-suggestion { }

/* emitted by the plugin template */
.card-menu { }              /* the list */
.card-menu-item { }         /* one suggestion, carries proximity="<score>" */
.card-menu-item .card-title { }
.card-menu-item .card-text { }   /* excerpt */
.card-menu-item .card-info { }   /* last modified, score */

Disabling suggestions on some pages 

Either remove the object for those pages:

[page["uid"] == 42]
    lib.semantic_suggestion >
[END]

or keep them out of the suggestion lists of other pages with excludePages. The two are not the same thing: the first hides the list on page 42, the second stops page 42 from being suggested.

What it shows 

Each section can be switched off in TypoScript — see TypoScript.

Access control 

• Administrators see every analysis of the instance.
• Other backend users see only the analyses of sites they hold a webmount on. A webmount on a subpage resolves to that page's site, so an editor mounted on one section still sees their whole site's analysis — and nothing from other sites.
• The rootPageId URL argument is validated against that same list, so it cannot be used to reach another site's data.

A user with no webmount inside any configured site sees an empty module.

Several sites in one instance 

Create one task per site, with startPageId on that site's root page:

Task "Similarities – Main site"      startPageId: 1    languageId: -1
Task "Similarities – Campaign site"  startPageId: 85   languageId: -1

The task resolves the site from startPageId and only ever walks that site's page tree, so tasks cannot contaminate each other. Extra tasks on subtrees of a site are allowed too — for instance a daily task on a fast-moving news section and a weekly one on the rest. Each rewrites only its own scope_page_id.

What is scoped per site automatically:

• Frontend display. The lookup filters on the current page's site, so a page of one site can never be suggested on another. No configuration needed.
• Cache invalidation. Analysis cache entries are tagged site_<rootPageId>, so editing content on one site does not invalidate another's analysis.
• Backend module. Permission-scoped per site — see Access control.

What is not scoped, and needs your attention:

• TypoScript. The extension's constants and setup are loaded instance-wide by ext_localconf.php, so every setting is shared until a site's root template overrides it. This is why the Bootstrap Package integration (Bootstrap Package) must be enabled per site and never globally.
• Extension configuration and the log file, which are instance-wide by nature.

A multilingual site 

Language handling is driven by the site configuration. Each language's locale gives the language code used for stop words, stemming and vectorisation:

# config/sites/main/config.yaml
languages:
  -
    languageId: 0
    title: 'English'
    locale: 'en_US.UTF-8'      # → analysed as "en"
  -
    languageId: 1
    title: 'Deutsch'
    locale: 'de_DE.UTF-8'      # → analysed as "de"

One task is enough. A multilingual site has a single root page and several languages entries; with the default languageId = -1 the task iterates over all of them in one run:

Task "Similarities – Main site"
  startPageId:  1     # the site root — one per SITE, not per language
  languageId:   -1    # all languages of this site
  qualityLevel: 0.3

Do not create one task per language pointing at different page UIDs: that pattern belongs to multi-site setups, where each site has its own root page.

Split by language only when a language needs a different quality level. Both tasks then target the same root page with an explicit language ID:

Task 1  startPageId: 1   languageId: 0   qualityLevel: 0.3
Task 2  startPageId: 1   languageId: 1   qualityLevel: 0.25

Rows are keyed by sys_language_uid on top of the site and the scope, so the two tasks never collide.

Languages never mix 

The language boundary is enforced three times over, so mixed-language suggestions are not something you have to configure away:

1. Pages are collected per language by the task.
2. Two pages whose detected language codes differ score 0.0 and are not stored.
3. The frontend query filters on sys_language_uid and re-checks the language of each candidate page record.

If suggestions really do cross a language, the cause is almost always a missing or malformed locale in the site configuration — see Troubleshooting.

Per-site templates 

Override the plugin template in the site's root template, using index 10 or above:

plugin.tx_semanticsuggestion_suggestions.view {
    templateRootPaths.10 = EXT:my_sitepackage/Resources/Private/Templates/
}

See Overriding the template.

The analysis pipeline 

Everything below happens inside the scheduler task.

1. Scope. The site is resolved from startPageId, and the languages to process are taken from its site configuration (all of them, or the one given by languageId).
2. Page collection. The subtree below startPageId is walked to full depth, per language. Deleted and hidden pages are skipped, as are the pages listed in excludePages.
3. Text preparation. For every page, its language is resolved from the site configuration (see Language handling), then each field of analyzedFields is read. content falls back to the bodytext of the page's content elements in that language when the page field itself is empty. Stop words of that language are removed, then — if enableStemming is on — the text is stemmed with that language's stemmer.
4. Vectorisation. Pages are grouped by that same language code and each group is vectorised together, so all pages of one language share a single vocabulary and IDF corpus. Every page is vectorised exactly once, not once per comparison.

5. Comparison. Every page is compared with every other page of the same language using cosine similarity on those vectors, then combined with the recency term:

   score = cosine(vector1, vector2) * (1 - recencyWeight)
         + recencyBoost              * recencyWeight

   Copied!

   Pairs of different languages score 0.0 and are dropped.

6. Storage. Pairs scoring at or above the task's quality level are written to tx_semanticsuggestion_similarities, in one transaction, in batches of 100. The rows of the previous run of this task — same site, same scope, same language — are deleted first, so a run never disturbs another task's results.

The frontend does none of this: it reads the stored rows for the current page, filters them by the display quality level, excludePages and the language, and renders at most maxSuggestions of them.

Reading the scores 

The score is a cosine similarity between TF-IDF vectors, mixed with the recency term. Two consequences worth keeping in mind:

• Scores are lower than a naive word-count similarity would give. Real values on a normal site cluster well below 0.5; 0.3 is a reasonable threshold and 0.8 will match nothing. If you are migrating from a 2.x configuration, lower your thresholds.
• The recency term can carry a pair on its own. recencyBoost is the absolute difference between the two pages' normalised ages over a 30-day window, so with the default recencyWeight = 0.2 a pair with no textual similarity at all can still score 0.2. Set the weight to 0 for a purely textual score.

The relevance labels shown in the backend module are thresholds on that score: High above 0.8, Medium above 0.6, Low below.

Language handling 

The language is resolved per page, once, and the same value is then used for stop word removal, stemming and vectorisation. The order is:

1. The site configuration. The locale of the page's language, reduced to its two-letter code (de_DE.UTF-8 → de). This is the normal case and it always wins for a page belonging to a configured site.
2. Content analysis, by nlp_tools, only when the step above found nothing — a page outside any configured site.
3. defaultLanguage, when there is not even any text to analyse.

Stop words, stemming and detection profiles come from nlp_tools, which supports six languages:

Any other language still gets TF-IDF vectorisation, which is what does most of the work; it just loses stemming and gets an unhelpful stop word list. Note also that content-based detection can only ever return one of the six codes above.

German is the language this extension was tuned on: the Snowball stemmer is what makes Automobilindustrie and Automobil share a stem, and umlauts are handled. A lower quality level than for English (0.25 instead of 0.3) is a reasonable starting point on a German site.

Known limitations 

• Field weights are coarse, being implemented as text repetition — see analyzedFields.
• The comparison is quadratic in the number of pages of a scope. Split a very large site into several tasks on subtrees rather than raising the quality level to compensate.

Caching 

The extension registers a cache named semantic_suggestion (file backend, 24 h, member of the pages group). It holds analysis results and the intermediate nlp_tools results, and entries are tagged site_<rootPageId>.

Invalidation happens on two occasions:

• the scheduler task flushes its site's tag after saving,
• a DataHandler hook flushes the tag of the affected site when a page or a content element is changed, moved or deleted. Records being created carry a NEW… placeholder instead of a UID and cannot be resolved to a site at that point; they are picked up by the next analysis run anyway.

# flush everything, including this cache
vendor/bin/typo3 cache:flush

The three tools 

Checks that the nlp_tools stack is reachable and working, on German samples:

vendor/bin/typo3 semantic:diagnostic

If it cannot instantiate the services, nothing else will work — reinstall nlp_tools before looking any further.

Everything the frontend can possibly show is in one table. This is the fastest way to tell a storage problem from a display problem:

SELECT root_page_id, scope_page_id, sys_language_uid, source,
       COUNT(*) AS pairs, MIN(similarity_score), MAX(similarity_score)
FROM tx_semanticsuggestion_similarities
GROUP BY root_page_id, scope_page_id, sys_language_uid, source;

Several scope_page_id values under one root_page_id is normal — it means several tasks cover different subtrees of the same site.

plugin.tx_semanticsuggestion_suggestions.settings.debugMode = 1

Writes to typo3temp/logs/semantic_suggestion.log and appends a debug block to the plugin output. Turn it off again afterwards — it is visible to visitors.

No suggestions anywhere 

Work down this list; each step rules out the ones above it.

The task runs but stores nothing 

Suggestions are irrelevant 

• Raise the display quality level first: it costs nothing and needs no re-run. Only raise the task's level once you know which value you want, since lowering it again requires a full re-analysis.
• Set recencyWeight to 0 if you see unrelated pages being suggested. The recency term is a difference of ages, not a freshness bonus, and it can carry a pair with no textual overlap.
• Check the language of the affected pages: an unsupported language loses stemming and gets the English stop word list, which makes scores noisier — see Language handling.
• Do not expect much from field weights. They are applied as text repetition, so only differences of half a point or more change anything.

Suggestions from another language 

The language boundary is enforced in three places, so this practically only happens when the site configuration is incomplete. Check that every language of the site has a full locale:

languages:
  -
    languageId: 1
    locale: 'de_DE.UTF-8'   # not just 'de'

Then re-run the task, since the stored rows were computed with the old configuration.

Suggestions from another site 

Not possible since 4.1.0 — unless the rows predate it and the migration wizard has not run. Verify:

-- must return 0
SELECT COUNT(*) FROM tx_semanticsuggestion_similarities WHERE scope_page_id = 0;

See Upgrade.

The task times out 

The comparison is quadratic in the number of pages of one scope, so the answer is to reduce the scope rather than to raise the threshold:

• split the site into several tasks on subtrees, each with its own frequency,

• or run it from the CLI, where max_execution_time is usually unlimited:

  vendor/bin/typo3 scheduler:run --task=<uid>

  Copied!

The backend module is empty or unreachable 

• "No module access" on TYPO3 14 with versions before 4.1.2: known bug, upgrade.
• An empty analysis list for an editor: they have no webmount inside a site that has an analysis — see Access control. Administrators always see everything.
• "No similarity analysis found" for an administrator: the table is empty, go back to The task runs but stores nothing.

A page renders empty after enabling the Bootstrap Package integration 

overrideBootstrapTemplates was enabled globally instead of in the Bootstrap Package site's own constants, and the shipped Default.html replaced another site's page template. Set the constant in the right root template only — see Bootstrap Package.

To 4.1 from 3.x or 4.0.0 

Version 4.1 changes the meaning of the root_page_id column and adds scope_page_id (see Database columns). Existing rows must be migrated, otherwise the frontend — which now filters on the site root — finds nothing for any analysis whose task started on a subtree.

vendor/bin/typo3 extension:setup --extension=semantic_suggestion

vendor/bin/typo3 upgrade:run semanticSuggestionMigrateRootPageId
vendor/bin/typo3 upgrade:run semanticSuggestionLabelAnalysisRows

Both are also listed in Admin Tools > Upgrade > Upgrade Wizard, as "Semantic Suggestion: split root_page_id into site root and analysis scope" and "Semantic Suggestion: label existing analysis rows with their source".

-- must return 0
SELECT COUNT(*) FROM tx_semanticsuggestion_similarities WHERE scope_page_id = 0;

Any remaining row is one the wizard could not resolve. Re-run the corresponding scheduler task to regenerate it, or delete it if that task is gone.

The storage threshold is now exactly the quality level you set on the task, with no hidden offset. Earlier versions silently stored from qualityLevel - 0.1, so a task configured at 0.3 was really storing from 0.2.

Nothing breaks, but each task now stores slightly fewer pairs. Those extra pairs were below the display threshold and therefore never shown — unless you had deliberately lowered the TypoScript value below the task's to exploit the buffer. If you did, lower the task's quality level to match and re-run it.

overrideBootstrapTemplates now defaults to 0. If you relied on the automatic Bootstrap Package integration, enable it explicitly in the constants of the site that uses Bootstrap Package — and read the warning in Bootstrap Package first if the instance hosts more than one site.

TYPO3 14 

Upgrade to at least 4.1.2 before moving an instance to TYPO3 14: on earlier versions the backend module could not be opened at all — every user, administrators included, was redirected with "No module access".

From 2.x 

2.x scored pages with a plain word-count cosine similarity, whose values are much higher than the TF-IDF scores used since 3.0. A configuration carried over unchanged will usually display nothing.

1. Make sure nlp_tools is installed (Composer does it for you).
2. Lower your thresholds: 0.25–0.35 instead of the 0.5–0.8 that made sense in 2.x.
3. Re-run the scheduler task — all scores are recomputed.
4. Drop proximityThreshold from your TypoScript in favour of qualityLevel, and stop setting minimumSimilarity on the task; it is now derived from the quality level and ignored as input.

Release highlights 

Upgrading between any of these is covered in Upgrade.