Features 

  • Support multiple languages.

Import 

Import of specific resources via static URLs. This allows import of specific tourist attractions and towns.

  • The following data types can be imported (not all properties are supported):

    • Organisations (responsible for content)
    • Towns
    • Tourist information
    • Tourist attraction

Backend module 

Allows to create new import configurations.

Allows to inspect:

  • Current existing organisations and their towns and tourist information
  • Existing import configuration
  • past imports and possible errors, as well as number of affected records.

Figure 1-1: Overview of currently available configurations and Organisations.

Figure 1-2: Overview of executed imports and their results.

Frontend 

A dedicated content element is provided to display tourist attraction.

A dedicated Page Type is provided to reference tourist attraction.

Installation 

The extension can be installed via composer, see t3gettingstarted:extensions-management. The composer package can be found at packagist.org.

Integrators can include and configure TypoScript once it is installed and enabled. See Include TypoScript from extensions. All available configurations can be found at Configuration.

At least one import configuration needs to be created and imported. See Import configuration.

Configuration 

API Key 

Some API requests are only possible by providing an API Key. This key can be configured via "Extension Configuration".

Storage Pids 

The configuration of imports is stored within TYPO3 records. Those records can be created via backend module. Each new record is stored on a default page. The default is 0. That would prevent editors from creating and editing records. This page uid can be configured via TypoScript:

module {
    tx_thuecat {
        settings {
            newRecordPid {
                tx_thuecat_import_configuration = 10
            }
        }
    }
}
Copied!

Import configuration 

Each import is defined via a special import configuration record. This record can be created via TYPO3 backend module.

There are different configurations available:

Static list of URLs
Allows to define a list of URLs which should be imported. These URLs should reference a single resource to import without any given parameters like a format.
Synchronization area
Allows to import a so called "Synchronisationsbereich". Find out more at https://cms.thuecat.org/developer. Add the given syncScopeId to the configuration to update the given resources for that specific sync scope. This requires an configured API Key.

All configurations also provide an input to define the page where records should be stored and updated. This page uid is also used to fetch accordingly site configuration. The related languages are used during the import.

Category based anchors 

Several imported properties are stored as sys_category records — the categories derived from an object's type, the keywords it carries, and further filter properties as they are added. Each such property is anchored by a pair of settings: the folder its categories are stored in, and the parent category they are created beneath.

Anchors are configured per site and per import target, not per import configuration. That is deliberate. They are structural data every imported object uses the same way, and a filter in the frontend has to be able to identify the parent category its selection rests on. With the anchors defined once per site, a plugin can.

The import target comes from the import configuration's Import target field and says which kind of records a run writes — thuecat objects or events. One site can hold an import configuration of each, so every anchor setting names its target: an import reads only its own, and the two category trees stay apart. A configuration that carries no target counts as thuecat; one carrying a value that matches no known target is rejected before the run fetches anything.

Every anchored property carries its own pair per target, resolved independently of the others. Properties may point at the same storage folder, but each pair stays separately configurable, because imported categories can legitimately live in different folders. The settings are grouped under Import in the site settings editor, one labelled pair per property and target.

Each anchor is resolved from the first level that supplies a positive page or category uid:

Site settings
Of the site owning the import configuration's storage page. Set them in Site Management > Sites. A site offers them once it lists the werkraummedia/thuecat-import set among its own dependencies — importing is a decision of the site, so no other set pulls it in.
Extension Configuration
Installation-wide, via Admin Tools > Settings > Extension Configuration > thuecat. Used only when the site supplies nothing — it is the fallback for installations that declare no site set. One value per target applies to every site, so it fits single-site installations and installations where all imports of that target share one category tree.
Unset
0, or nothing configured anywhere. That kind's mapping is switched off and the import runs without it. Reaching the end of the chain is the only way an anchor becomes unset — no error condition is ever downgraded to it.

Per kind, both anchors must be set or neither. Setting only one is rejected before the import fetches anything, as is an anchor outside the site that owns the storage page. The two kinds are validated independently: a broken keyword pair says nothing about the category pair. Failure messages name the settings to correct, including their target.

The values actually in effect are reported at the start of every run — see Effective settings of a run.

Keywords 

ThueCat objects carry schema:keywords: a controlled vocabulary of terms grouped into sets, alongside keywords editors typed by hand. The import stores them as sys_category records, so they can be selected in the backend and used to filter records in the frontend.

Keywords are a separate property from the categories derived from an object's type. They never share a tree, a record or an identifier, even where a keyword and a category happen to carry the same title.

Keyword import is switched on by the keyword pair of site settings — a storage folder and a parent category. See Category based anchors.

Both must be set, or neither. Leaving both empty switches keyword import off; setting only one is rejected before the import fetches anything. The anchor must lie inside the same site as the record storage page.

Because the anchors belong to the site, every import writing into that site shares one keyword tree. Places and events imported into the same site therefore share it too.

The upstream group structure is mirrored as intermediate categories, so a term appears beneath a category representing its set rather than in one flat list. Only the keyword a record actually carries becomes a relation; the groups above it exist to organise the tree.

Editors may rename an imported keyword category. Renames survive re-import: matching is on the stored remote identifier, never on the title. Titles are taken from upstream only when the category is first created.

Keywords no longer supplied by upstream lose their relation on the next import, while the category record itself remains for editors to keep using. A keyword that could not be fetched is left alone rather than removed, since a failed request cannot be distinguished from an upstream deletion.

Places consume their keywords in the frontend — filtering and meta tags, see Filtering by keyword. Events do not yet. Imported events receive keyword category relations, but nothing reads them: the event list cannot filter by keyword, and event keyword meta tags continue to come from the legacy keywords string column that the destination.one import writes. The relations accumulate unused until the event side is built; no migration will be needed then, because the data is already correct.

Import tuning 

Five settings bound what an import run may do. Each is available in two places:

Extension Configuration
Installation-wide, via Admin Tools > Settings > Extension Configuration > thuecat. The fields are pre-filled with the shipped defaults so the values in effect are visible.
Import configuration record
Per configuration, in the backend module. An empty field means "not set" and falls back to the installation-wide value.

Deliberately not site configuration: these bound a run, and a run belongs to an import configuration, not to a site.

The precedence for each setting is:

  1. the import configuration record, when set;
  2. the Extension Configuration value, when set;
  3. the shipped default.

0 and empty both count as "not set" at every level, so a field cleared in the backend returns to the fallback rather than meaning "unlimited".

Setting Default Meaning
readTimeout 120 s How long a single request may take to deliver its response. Without it one unresponsive host blocks the run indefinitely.
connectTimeout 30 s How long establishing the connection may take.
maxAttempts 3 How often a failing request is tried. Retries apply to transport failures and 5xx responses only — a 4xx is never retried. An exhausted retry is recorded in the import log naming the URL, the cause and the attempt count. Requests that recovered are summarised too; see Recovered retries.
runBudget 86400 s How long a whole run may take. When exceeded the run aborts deliberately at the next phase boundary, writes its import log and exits non-zero, rather than being killed with nothing on disk. A pass already under way is never interrupted.
fetchCacheLifetime 900 s How long a fetched API response stays reusable. See Fetch cache.

Effective settings of a run 

The values driving an import come from three places — the import configuration record, the site settings and the Extension Configuration — so each run reports what it actually used before it fetches anything.

The report is written to the import log as its first entry, at severity debug, and shown in the Summary column of the backend module. Command line runs print it as well, at normal verbosity; --quiet suppresses the console output while the log entry is still written.

It covers the storage page, the file folder, the API domain, the import target, that target's four category anchors and the five tuning settings. Only the run's own target appears; listing the other one's anchors would suggest they had a say in the run. An anchor nothing supplies is reported as unset rather than 0, so a switched-off mapping is visible as a decision rather than a number.

The API key is never part of the report — not its value, not a masked rendering, not its length.

Recovered retries 

A request that failed and then succeeded on a later attempt costs time but loses nothing, so it is easy to miss: the run completes, imports everything and reports success. A run against a struggling upstream then looks exactly like a healthy one.

Each run that had any such request therefore writes a single retriesRecovered entry to its import log, stating how many requests recovered and how many extra attempts they cost. It is a notice — below warning — so it never changes whether the run is considered successful, and the command still exits 0.

A run in which every request succeeded first time writes no entry, so the presence of one is the signal. Repeated appearances, or a rising attempt count, indicate the API is degrading before it starts failing outright.

Fetch cache 

Fetched API responses are cached in the database ( cache_thuecat_fetchdata ) for fetchCacheLifetime seconds, so a run that aborts part-way does not re-fetch everything when it is started again.

To run against fresh data, bypass the cache:

vendor/bin/typo3 thuecat:importviaconfiguration <uid> --fresh
Copied!

A bypassing run still writes what it fetches, so later runs benefit from it; only reading is skipped. Bypassing changes where responses come from, never what is imported.

An installation that configures this cache itself keeps its own backend and lifetime — the defaults apply only where nothing is configured:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['thuecat_fetchdata']
Copied!

Import without media 

Media dominates the cost of a run: every image is an API request for its metadata plus a download of the file, while the records carrying them are comparatively cheap. To import records without media:

vendor/bin/typo3 thuecat:importviaconfiguration <uid> --no_media
Copied!

No media is fetched, downloaded or related, in either shape the API uses — images referenced by URL, which each cost a request, and images inlined into the record's own response. The file folder is not touched either: neither the write-access probe nor the per-run staging folder is created, so the run succeeds even where the configured folder is missing or read-only.

Useful for restoring records after an aborted run, filling a fresh installation, or reproducing an import problem that has nothing to do with images.

Integration 

Frontend output 

EXT:thuecat registers Extbase controller actions and ships template stubs, but no content elements. You define the content elements in your own extension or sitepackage and point them at the registered plugins, so rendering stays under your control.

Plugins and templates 

The extension name is ThueCat. Five content elements are intended, built from four plugins (the filtered list is the list plugin with an editor preset). Each plugin renders one template:

Plugin name Content element Template
TouristAttractionList List List
TouristAttractionList Filtered list List
TouristAttractionListSelected Selected list SelectedList
TouristAttractionSearch Search-and-filter SearchForm
TouristAttractionShow Detail Show

The plain and filtered list are the same TouristAttractionList plugin and therefore share the List template; the filtered variant only adds an editor preset in its FlexForm (see Filtered list). The search element is a combined search and filter form.

Required pages 

Set up these pages once and configure their ids (see Site settings):

  • Storage folder -- a sysfolder holding the tourist attraction records.
  • Detail page -- carries the detail content element; target for links to a single attraction.
  • List / search page -- carries both the list content element and the search-and-filter content element.

On the list / search page neither plugin is optional: the search-and-filter form submits into the list, and the list renders the result. A page with only one of the two does not work.

Content elements 

Register one content element per plugin in your own extension or sitepackage, for example with Content Blocks. Each content element wires its CType to a plugin via the ThueCat extension name:

tt_content.myvendor_attractionlist =< lib.contentBlock
tt_content.myvendor_attractionlist {
    20 =< lib.contentBlock.20
    20 {
        pluginName = TouristAttractionList
    }
}
Copied!

The list content element points at the storage folder through its pages field.

A full, ready-to-use content element definition is shown in Content element example.

Templates 

The shipped stubs under EXT:thuecat/Resources/Private/Templates/ are registered at templateRootPaths.10. Override them by adding a higher index:

plugin.tx_thuecat.view {
    templateRootPaths.20 = EXT:my_extension/Resources/Private/Templates/
    partialRootPaths.20 = EXT:my_extension/Resources/Private/Partials/
    layoutRootPaths.20 = EXT:my_extension/Resources/Private/Layouts/
}
Copied!

The templates are

  • TouristAttraction/List
  • TouristAttraction/SelectedList
  • TouristAttraction/SearchForm
  • TouristAttraction/Show

Further topics 

Some fields carry richer data than a single value and have their own model accessors and partials; rendered output is cached. These are covered on their own pages:

Site settings 

Provide a site set that maps site settings onto the plugin configuration and include it in your site. Fill in the page ids under Settings in the site configuration.

Settings definition (Configuration/Sets/<YourSet>/settings.definitions.yaml):

settings:
  page.pid.thuecat_attraction_show:
    label: 'Detail Page for Tourist Attractions'
    description: 'The page providing the detail pages for tourist attractions'
    category: 'page.pids'
    type: 'int'
    default: 0
  page.pid.thuecat_attraction_search:
    label: 'Search Result Page for Tourist Attractions'
    description: 'The page providing the list of tourist attractions, used as target for search form submissions'
    category: 'page.pids'
    type: 'int'
    default: 0
  list.itemsPerPage:
    label: 'Tourist Attractions per Page'
    description: 'Number of tourist attractions shown per page in the list view'
    category: 'list'
    type: 'int'
    default: 20
Copied!

Mapping (Configuration/Sets/<YourSet>/setup.typoscript):

plugin.tx_thuecat.settings {
    page.pid {
        thuecat_attraction_show = {$page.pid.thuecat_attraction_show}
        thuecat_attraction_search = {$page.pid.thuecat_attraction_search}
    }
    itemsPerPage = {$page.settings.itemsPerPage}
}
Copied!

Search and list on one page 

The search-and-filter form adapts to what shares its page:

  • With a list (plain or filtered) on the page the form posts to the same page and the list re-renders with the result.
  • Without a list on the page the form targets the configured list / search page (page.pid.thuecat_attraction_search).
  • On a filtered list the preset fields are not shown in the search form, but rendered as hidden fields to preserve the pre-selection. The visitor refines the remaining fields but cannot widen past the preset.

After a search the form re-populates with the submitted values, so the visitor keeps their input.

Filtered list 

A filtered list carries an editor preset in its FlexForm (for example a fixed set of towns). The list re-applies the preset on every request: a visitor search refines within the preset but can never widen it, even with a tampered URL.

Registering a filter field 

Which filters the search mask offers is not a list in the controller. A filter field is a tagged service implementing \WerkraumMedia\ThueCat\Service\FilterField\FilterFieldDefinition , tagged search.filter.field. \WerkraumMedia\ThueCat\Service\SearchFilterOptionsService discovers every implementation through that tag and builds it. Another extension adds a filter without editing any class here.

Two storage shapes are supported, each with an abstract base to extend. Pick the one matching where the values live. Both live in Classes/Service/FilterField/; the examples are the shipped fields, in the same directory.

Values in a comma-separated uid column on the record itself, i.e. a TCA select without MM — base \WerkraumMedia\ThueCat\Service\FilterField\CommaColumnField :

<?php

namespace Vendor\Extension\Service\FilterField;

use WerkraumMedia\ThueCat\Service\FilterField\CommaColumnField;

final class TownFilterField extends CommaColumnField
{
    public function __construct()
    {
        parent::__construct(
            name: 'towns',
            recordColumn: 'town',
            optionTable: 'tx_thuecat_town',
        );
    }
}
Copied!

Values in an MM table over a parent/child option table, offered as the tree below a configured anchor — base \WerkraumMedia\ThueCat\Service\FilterField\HierarchicalMmField :

<?php

namespace Vendor\Extension\Service\FilterField;

use WerkraumMedia\ThueCat\Import\Settings\CategoryAnchorSetting;
use WerkraumMedia\ThueCat\Service\FilterField\HierarchicalMmField;

final class CategoryFilterField extends HierarchicalMmField
{
    public function __construct()
    {
        parent::__construct(
            name: 'categories',
            mmTable: 'sys_category_record_mm',
            mmFieldName: 'categories',
            optionTable: 'sys_category',
            parentColumn: 'parent',
            anchorSetting: CategoryAnchorSetting::CategoryParent,
        );
    }
}
Copied!

With autoconfigure enabled the tag is applied by the interface, so no entry in Services.yaml is needed.

What each argument means 

name
The demand property and the view key. The template binds the option set under this name, so a field named towns is rendered from {towns}. It is not the MM field name: the two happen to coincide today but answer to different owners.
recordColumn
The column holding the comma-separated uids. A record without a value carries the placeholder 0, which is never offered as an option.
mmFieldName
Which field of a shared MM table the relation belongs to. Categories and keywords share sys_category_record_mm and are told apart by this alone, so it is part of every query.
anchorSetting
A case of \WerkraumMedia\ThueCat\Import\Settings\CategoryAnchorSetting , naming the site setting that holds the category the offered tree starts below. The declaration carries the setting, never a resolved uid — anchors are per site and are resolved per request. An unconfigured anchor offers nothing.

The record table is deliberately absent. It belongs to the scope, not to the field, so one field definition serves every record kind that offers it.

What a field does not decide 

A field declares where its values are read from and nothing else. Scoping is resolved once per request and applied to every field alike:

  • Only values carried by records the sibling list can return are offered. With no list on the page, each field offers what its storage holds.
  • Values are bound to the site. A value living outside the list's storage pages is still offered; one belonging to another site is not.
  • Option sets never depend on the visitor's current selection. Choosing one town does not remove the others from the form.

Adding a further storage shape 

If neither base fits, the shape needs its own provider: implement \WerkraumMedia\ThueCat\Service\FilterField\OptionProvider\FilterOptionProvider , tagged search.filter.option.provider, returning true from supports() for the field class it reads. Providers are discovered by that tag, so the service picks one up without being edited.

Extending \WerkraumMedia\ThueCat\Service\FilterField\OptionProvider\AbstractOptionProvider supplies the restriction handling every option query needs — deleted, disabled, language, storage pages and the site bound — which is easy to get subtly wrong when writing a provider from scratch. The shipped providers, CommaColumnOptionProvider and HierarchicalOptionProvider , sit in the same namespace as worked examples.

Filtering by keyword 

Places can be filtered by the keywords the import stored on them. The filter behaves like the category filter beside it: selecting several keywords widens the result, while adding a town or a category narrows it.

Which keywords a plugin offers is chosen per plugin, in the FlexForm of the attraction list. A plugin with no keyword selected offers no keyword control and applies no keyword constraint, so existing plugins keep behaving as they did.

The selection is bounded by the site's keyword anchor: the field offers the terms below import.thuecat.keywords.parent and nothing else, so the type-category tree cannot be picked here by mistake. A site that configures no keyword anchor therefore offers nothing to choose.

Because the anchor belongs to the site, one site running several imports into the same keyword parent shares one vocabulary across its plugins. Pointing separate imports at separate keyword parents gives each list its own.

The chosen keywords survive pagination, and the filter mask shows them grouped by their parent set so long vocabularies stay navigable.

Keyword meta tags 

A place detail view emits a keywords meta tag built from the record's related keyword categories, joined by ", ". A place with no keyword relations emits no keyword meta tag.

How it works 

The search-and-filter form does not read its own settings to learn the preset. A resolver inspects the current page for a list content element, reads its FlexForm, and returns the active preset:

  • No list on the page -- the form targets the configured list / search page.
  • A list on the page -- the form stays on the page; a filtered list additionally contributes the preselected fields.

The lookup is language- and overlay-aware, because it reads the stored content element through the frontend's record retrieval.

The form is rendered by TouristAttractionSearch but submits under TouristAttractionList, so the demand travels in the list's namespace. On re-render the form adopts that demand to re-populate its fields, and the editor preset is forced onto the locked fields, so the hidden inputs always carry the editor's values.

Opening hours 

Import 

Opening hours are imported as inline database records (one row per weekday and time span, with an optional validity range). You do not work with those rows directly: the model computes a display-ready shape from them.

Computed model 

A tourist attraction exposes two computed accessors:

  • computedOpeningHours -- the regular hours.
  • computedSpecialOpeningHours -- deviating hours, for example public holidays.

Both return the same structure: a list of periods (validity ranges), each listing weekdays Monday-first, each weekday carrying all of its time spans. A day without hours is marked closed, so every weekday is present in the output. Past periods are dropped; the period covering today is flagged as current.

Rendering with the shipped partial 

The extension ships a partial that renders the computed shape as a per-day table (one row per weekday, all spans listed, closed days shown as closed). Render it from your Show template, passing the computed hours and a heading label:

<f:render partial="OpeningHours/PerDayTable" arguments="{
    openingHours: attraction.computedOpeningHours,
    heading: 'LLL:EXT:thuecat/Resources/Private/Language/locallang.xlf:content.openingHours'
}" />
<f:render partial="OpeningHours/PerDayTable" arguments="{
    openingHours: attraction.computedSpecialOpeningHours,
    heading: 'LLL:EXT:thuecat/Resources/Private/Language/locallang.xlf:content.specialOpeningHours'
}" />
Copied!

The partial takes two arguments:

Argument Meaning
openingHours A computed opening hours object (one of the accessors above).
heading A translation key for the section heading.

The partial renders nothing when there are no periods, so a missing heading or empty section never appears.

Media 

Import 

Images and other files are imported into FAL and related to the record. The target file directory is part of the import configuration.

An asset is downloaded once per run no matter how many records use it, and files are named after the URL they were downloaded from, so a re-import reuses what is already stored instead of fetching it again.

Redirects are followed, up to five hops; some hosts serve every asset that way. The file is named after the URL upstream supplied, not the redirect target, so an asset stays one file across runs. A redirect without a target, or a longer chain, counts as a failed download and leaves stored relations alone.

Which fields the import owns 

main_image and media_files mirror what upstream supplies. Each import rebuilds them: an image upstream no longer lists has its relation removed.

Images added to these fields by hand are removed by the next import. The relation carries no marker saying where it came from, so an editorial addition cannot be told apart from a leftover of an earlier import.

Use editorial_images for images maintained in the backend. The import never writes to that field and never removes anything from it, and its contents are available in the frontend as editorialImages.

A download that fails does not remove the image it failed to fetch. Only a 404 or 410 from the media server counts as the asset being gone; a server error, a refused or rate-limited request, or a run started with --no_media leave the stored relations alone.

FAL accessors 

A tourist attraction exposes its media as native Extbase FAL relations:

Accessor Meaning
mainImage The primary image (main_image), a single file reference. Import-owned.
mediaFiles Additional images and files (media_files). Import-owned.
editorialImages Editorially curated images (editorial_images), maintained in the backend. Never touched by the import.

mainImage returns a single file reference or none; mediaFiles and editorialImages return a (possibly empty) collection.

Rendering 

The FAL relations are rendered with the standard Fluid image view helper, so processing (cropping, scaling) and metadata (copyright, alternative text) are available:

<f:if condition="{attraction.mainImage}">
    <figure>
        <f:image image="{attraction.mainImage}" />
        <f:if condition="{attraction.mainImage.originalResource.properties.copyright}">
            <figcaption>{attraction.mainImage.originalResource.properties.copyright -> f:format.htmlspecialchars()}</figcaption>
        </f:if>
    </figure>
</f:if>

<f:for each="{attraction.mediaFiles}" as="image">
    <f:image image="{image}" />
</f:for>
Copied!

Caching 

Attraction lists are cached so a visitor rarely waits for one to be built twice. Three caches do the work, all in the pages cache group.

The three caches 

tx_thuecat_teaser
One rendered list item, keyed by the record, the detail page it links to and the language. An attraction shown by several lists — filtered, curated, on different pages — is rendered once and reused by all of them.
tx_thuecat_list
A whole rendered list, keyed by the plugin, the filter selection, the pagination page and the language.
tx_thuecat_searchmask
A rendered filter form, keyed by the plugin, its page, the selection and the language. Deliberately not by the pagination page, so paging through results reuses one form.

Invalidation 

Because the caches belong to the pages group, everything that already clears page caches clears these too: the Clear cache buttons in the backend, and any editor saving a record. Editors need do nothing differently — a changed attraction, town or category takes its cached output with it, and the next visitor sees the new content.

Entries are discarded by the records they show rather than by age, so nothing goes stale waiting for a lifetime to run out. A filter combination nobody has requested before is still built from scratch, but only its unseen items cost anything: the rest come from the teaser cache.

Lifetime 

The three caches configure a lifetime of one year. This is not a staleness setting — invalidation by tag is what keeps content current — it exists because TYPO3's default would otherwise expire entries after an hour.

Override it per cache in ext_localconf.php or config/system/additional.php:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']
    ['tx_thuecat_teaser']['options']['defaultLifetime'] = 3600;
Copied!

Shortening it only costs rebuilds without making anything fresher.

Content element example 

A complete, ready-to-use content element built with content_blocks:Index. This example is the selected list: an editor picks a fixed set of attractions, rendered in the chosen order. Replace myvendor with your own vendor name.

Definition (ContentBlocks/ContentElements/attraction-list-selected/config.yaml):

name: myvendor/attraction-list-selected
group: Thuecat Attraction
prefixFields: true
prefixType: vendor
fields:
  - identifier: TYPO3/Header
    type: Basic
  - identifier: 'pi_flexform'
    type: 'FlexForm'
    useExistingField: true
    fields:
      - identifier: 'settings.selectedRecords'
        type: 'Select'
        renderType: 'selectMultipleSideBySide'
        foreign_table: 'tx_thuecat_tourist_attraction'
        foreign_table_where: 'AND {#tx_thuecat_tourist_attraction}.{#sys_language_uid} IN (0, -1)'
Copied!

Wiring (ContentBlocks/ContentElements/attraction-list-selected/setup.typoscript):

tt_content.myvendor_attractionlistselected =< lib.contentBlock
tt_content.myvendor_attractionlistselected {
    20 =< lib.contentBlock.20
    20 {
        pluginName = TouristAttractionListSelected
    }
}
Copied!

The settings.selectedRecords FlexForm field lets the editor choose the attractions; the TouristAttractionListSelected plugin renders them via the SelectedList template in the picked order.

The other content elements follow the same shape: a FlexForm field for their settings (a town preset for the filtered list, none for the plain list, none for the search-and-filter form) wired to the matching plugin name from Frontend output.

Changelog 

5.0.0 

Breaking 

  • Support for TYPO3 v12 removed.
  • Category and keyword anchors moved from the import configuration to the target site, and are now scoped by import target. The flexform fields categoryStoragePid, categoryParent, keywordStoragePid and keywordParent are removed without replacement; values left in stored flexform XML are ignored. Configure the anchors in the site settings instead — eight settings, one set per target, because a site may hold an import configuration of each and their category trees have to stay apart:

    • import.thuecat.category.storagePid, import.thuecat.category.parent, import.thuecat.keywords.storagePid, import.thuecat.keywords.parent
    • import.events.category.storagePid, import.events.category.parent, import.events.keywords.storagePid, import.events.keywords.parent

    A site opts into the settings editor for them by listing the werkraummedia/thuecat-import set among its own dependencies. The anchors are no longer declared by thuecat_ces or events_ces. On installations without a site set they remain configurable instance-wide in the Extension Configuration, where the four importCategory* / importKeywords* keys are likewise replaced by eight importThuecat* / importEvents* ones.

    There is no upgrade wizard: note the existing values before updating, because an import whose anchors are not configured runs with category and keyword mapping switched off rather than failing.

    Check the importTarget of every import configuration while you are there. It decides which anchors a run resolves, and a configuration that writes event records without it set counts as the thuecat target — putting its categories in the ThueCat tree. A value matching no known target now aborts the run before it fetches anything, instead of importing with no categories. See Category based anchors.

  • An imported record is identified within the site that imports it. Previously a remote_id matched any record carrying it anywhere in the installation, so two sites importing the same upstream object shared one record: the second site's import updated the first site's record in place — the record kept its page, so one site silently edited another's content — and the second site never got a record of its own. Each site now holds its own record, and relation targets resolve within the importing site too. On an installation where two sites import overlapping data, the first run after the update creates the records the second site was missing, which looks like duplication but is each site gaining what it should always have had. There is no migration: records a foreign site's import wrote into keep whatever it last wrote, and editors reconcile them.
  • API responses are cached persistently: thuecat_fetchdata now defaults to Typo3DatabaseBackend with a 900 second lifetime instead of dying with the process. A run is no longer guaranteed to hit the API — see Fetch cache for the bypass options.
  • Imported images are now the original asset (schema:contentUrl over schema:url), stored under a filename derived from the download URL. Re-imported records point at a different file under a new name; the previously imported files stay on disk unreferenced. No migration pass. See Import.
  • The list item moved from Partials/TouristAttraction/ListItem.html to Templates/TouristAttraction/ListItem.html, and the list templates now receive rendered items in {items} rather than iterating attractions. An override left in Partials/ is silently ignored. See Frontend output.
  • The category and keyword fields on tourist attractions and tourist information are bounded by the site's anchors instead of offering every sys_category in the installation. A site that configures no anchor now offers nothing in these fields, where it previously offered everything; configure import.thuecat.category.parent and import.thuecat.keywords.parent to restore a selection. The same bound applies to the keyword field of the attraction list plugin. See Category based anchors.
  • Filter options are built by SearchFilterOptionsService . Removed without replacement: TouristAttractionRepository::findCategoryTreeForSearchForm() , findKeywordsTreeForSearchForm() , findTownsInStorageSortedByTitle() , TownRepository::findAllForSearchFormSortedByTitle() , and the frontend CategoryRepository . \Dto\CategoryNode is replaced by \Dto\FilterOption , which keeps its accessor names — template overrides need no change.

    The partial Partials/TouristAttraction/CategoryTree.html requires a property argument naming the demand property its checkboxes bind to; an override without it renders unbound checkboxes. Its element ids carry that property too, so a category and a keyword sharing a uid no longer collide.

  • Filter options are scoped by site, not by the list's storage pages. A value outside those pages but inside the site is now offered; a value from another site no longer is.
  • A record can be contained in more than one place. town is a multi-value relation instead of a single select, so TouristAttraction::getTown() is replaced by getTowns() , returning an ObjectStorage . Templates reading {attraction.town.title} must iterate {attraction.towns} instead. Existing rows keep the town they held; a row that held none reads as an empty relation. Applies to tourist attractions, tourist information and parking facilities alike. See One property, several target tables.
  • Every transient bucket reports a reference it cannot relate. Previously only containedInPlace logged referenceUnrelatable; managedBy and parkingFacilityNearBy dropped a reference whose record imported into another table without saying so. Expect new info entries in the import log from those buckets on the first run — they name relations that were already being lost, not new failures.

Features 

  • Support for TYPO3 v14 added.
  • Images and other files are imported via FAL; the target file directory is part of the import configuration. See Import.
  • One asset is downloaded once per run however many records use it, deduplicated on the download URL before any request. Measured across two production configurations: 614 requests fell to 398, and 144 to 71.
  • Media relations are cleaned up when upstream stops supplying an image, for inline and referenced media alike. Re-importing an unchanged record no longer accumulates sys_file_reference rows. See Which fields the import owns.
  • A failed download no longer removes the image it failed to fetch: only 404 and 410 count as the asset being gone. See Which fields the import owns.
  • Events are imported with their images, both schema:photo and schema:image landing in images; the previous cap of eight images per event is gone.
  • Media uploaded directly to a record is imported rather than rejected, for every record type, with its author and licence. The configured API key is sent when fetching from the API host.
  • Addresses are imported as inline database records ( tx_thuecat_address ) related through address_inline, one row per address per language. Each field has its own column, so an address is readable and queryable without decoding a blob, and editors see labelled fields. A record may carry more than one address: schema:address is read as a list where upstream sends one. Addresses no longer supplied upstream are removed on re-import.
  • Opening hours are imported as inline database records ( tx_thuecat_opening_hours ) and exposed display-ready via Place::getComputedOpeningHours() . See Opening hours.
  • Imports can be run without media via --no_media . See Import without media.
  • Import runs are bounded and report their progress: per-request connect and read timeouts, retries for transient failures, an overall run budget, and progress output at three verbosity levels. Aborted and failed runs write an import log instead of vanishing. See Import tuning and Recovered retries.
  • Every import run reports the settings that drove it — storage page, file folder, API domain, the category anchors and the tuning settings — as the first entry of its import log, in the backend module's summary column, and on the console. The API key is never included. See Effective settings of a run.
  • Attraction lists, list items and filter forms are cached in the pages cache group, so everything that already clears page caches clears these too. See Caching.
  • An event imported without a single date is named in the import log as eventWithoutDates at severity warning.
  • schema:keywords is imported as sys_category records in their own tree, for tourist attractions, tourist information and events. Term references, typed ontology literals and editor-typed free text all resolve to a localized title, and the upstream group structure is mirrored as intermediate categories. Keywords stay separate from the categories derived from an object's type: separate anchor, separate relation, separate identifier. Editor renames survive re-import, and a keyword upstream stops supplying loses its relation while the category record remains. See Keywords.
  • Imported keywords are usable in the frontend: the attraction list filters by keyword, the filter mask offers them grouped by their parent set, and the selection survives pagination. Selecting several keywords widens the result while other filters continue to narrow it. Which keywords a plugin offers is chosen per plugin and bounded by the site's keyword anchor, so a plugin without a keyword selected behaves exactly as before. See Filtering by keyword.
  • A place detail view emits a keywords meta tag built from its related keyword categories. A place with no keyword relations emits no such tag.
  • schema:containedInPlace is imported in full. Upstream uses the property for whatever contains a record — towns, organisations and other places — while only the first town was kept and the rest discarded. Each reference now lands on a relation chosen by the kind of record it imported into: town, contained_in_organisation, or one of contained_in_attraction, contained_in_tourist_information and contained_in_parking_facility. One production configuration was dropping 127 such relations per run. A record contained in a kind no relation accepts is still reported as referenceUnrelatable. The detail view renders the containing organisations and places. See One property, several target tables.
  • The town filter matches a record carrying several towns, and the search form offers every town its records carry. An attraction in two towns is returned once when both are selected.
  • thuecat:Trail is imported into its own record type ( tx_thuecat_trail ), with its descriptive texts, route and metrics each in their own column. Way-type segments, current conditions and the start and end locations become inline records; seasons a multi-value field. Trails carry keywords and images like any other record. An attraction contained in a trail resolves to it through contained_in_trail. No frontend output yet — the records exist in the backend.

Fixes 

  • Imported coordinates keep their full precision. Latitude and longitude are stored as delivered rather than in a decimal column, which TYPO3 rounds to two decimal places on write — enough to move a point by roughly two kilometres. Re-run the import to correct stored coordinates.
  • Translated records keep their own address. Address text was read with a helper that understood only a single {@language,@value} object, so a record whose upstream address was a list of language variants — the shape used as soon as an address is translated — imported with street, postal code, city, phone, fax and email all empty. The same defect silently affected media copyright years and licences, and accessibility certification values.
  • Paging through an attraction list no longer shows the wrong page. Every pagination page and filter combination of one plugin previously shared a single page-cache entry.
  • The search mask no longer hydrates every record in storage to build its filter options. Measured on 649 attractions: 31.8s to 0.04s, and the tree expansion from 375 queries to a fixed number.
  • The search form offers its category options again on a page whose plugin names no storage page. The option tree was built with the default storage restriction, which resolves to page 0 and matches nothing, so the form rendered an empty category list.
  • The search form shows the levels below a category's top level. Child categories were looked up by comparing an object relation against uids, which matched nothing, so only the topmost entry of each tree was offered.
  • Filtering by a category that groups others returns the records filed under those others. Selecting a group previously matched only records related to the group itself — none, where a vocabulary relates records to the terms it ends in — so every group selection came back empty. Applies to the type categories and to keywords alike.
  • A referenced resource that cannot be fetched no longer aborts the whole root URL. Only the affected reference is dropped, logged per owning record as referenceSkipped at warning.
  • A media entry the import cannot interpret no longer costs the whole root URL.
  • An image that cannot be downloaded no longer costs the run its media. Previously one failed download discarded every image downloaded under all other roots.
  • Media served behind a redirect is imported. A 301 was previously logged as Image could not be downloaded, which cost every asset on a host that redirects — trail images and logos come from one. Redirects are followed up to five hops. Re-run the import to fetch the images earlier runs skipped. See Import.
  • Staged media is promoted into the target folder even when the run recorded an error elsewhere.
  • A record no longer inherits relations from the record imported before it — town, organisation, parking, categories and images alike. Relations already stored from earlier runs are not corrected by this.
  • Hiding a storage folder no longer breaks the import that writes into it. Which pages count as belonging to a site was derived with the frontend's visibility rules, so a hidden folder — or one past its publication end time — dropped out of the site. Depending on which folder it was, the run aborted with … is outside the storagePid's site for a page plainly inside the site, or the import stopped recognising what it had already written and built a second copy beside it: a hidden keyword or category folder produced a whole duplicate category tree, splitting records' relations across both. Storage is now what decides, for records, categories and keywords alike; only deleted pages fall out of scope.
  • A schedule day that cannot seed a date series no longer aborts the event import; the value is logged as scheduleDaySkipped, and surplus usable days as scheduleDayDropped.
  • schema:exceptDate is honoured, for every frequency.
  • An event's stored dates are reconciled with its schedule on re-import, so occurrences that stopped being produced are removed. Only the import's own rows are affected; dates created in the backend survive.
  • A failed root URL is identifiable from the import log: mappingError and fetchingError now carry the root URL.
  • thuecat:importviaconfiguration prints a closing message stating whether the run completed, completed with warnings, or failed.

Tasks 

Nothing

Deprecation 

  • The json-based media storage is superseded by FAL handling. The json read accessors Media::getMainImage() , getImages() , getExtraImages() , getEditorialImages() , getAllImages() and Base::getMedia() trigger E_USER_DEPRECATED. Use the FAL properties on Base instead and re-run the import. See FAL accessors.
  • editorial_images is handled as the FAL field it always has been: replace {entity.media.editorialImages} with {entity.editorialImages} .
  • The json-based opening hours storage is superseded by inline database records. Place::getOpeningHours() , getMergedOpeningHours() , getSpecialOpeningHours() and getMergedSpecialOpeningHours() trigger E_USER_DEPRECATED, as do the legacy OpeningHours , MergedOpeningHours and MergedOpeningHourWeekDay models. Use getComputedOpeningHours() / getComputedSpecialOpeningHours() and re-run the import. See Computed model.
  • The json-based address storage is superseded by inline database records. Place::getAddress() triggers E_USER_DEPRECATED and returns the legacy LegacyAddress value object. Use getAddressInline() for the full set, or getFirstAddress() where one address is expected, and re-run the import. The address column is kept and still readable so records not yet re-imported keep rendering; it is no longer written and is removed in a later major. The misspelled Address::getLatitute() is getLatitude() on the new model; the legacy object keeps the old spelling.

4.2.0 

Breaking 

Nothing

Features 

  • Add PHP 8.4 support.
  • Add PHP 8.5 support.

Fixes 

  • Allow none admin users to import.

    There was a broken configuration of corresponding TCA tables. It forced to insert log entries on root, whiteout allowing users to insert on root.

Tasks 

Nothing

Deprecation 

Nothing

4.1.0 

Breaking 

Nothing

Features 

  • Add TYPO3 wizard to select pages in configuration. That should ease live of editors, compared to manually inserting proper page uid.
  • Add title to static import configuration. This title is auto filled on save with the titles of provided URLs.

Fixes 

  • Prevent Error: Call to undefined method TYPO3CMSCoreDatabaseSchemaSchemaInformation::introspectTable() with newer TYPO3 v13 versions. We now check for the old API, falling back to the new. All of this is still @internal TYPO3 API.

    The corresponding core change was commit 45a50e455955c78f6baa2aec3af3865101ee06b9. We also need to update codappix/typo3-php-datasets dev dependency for the same reason.

  • Handle 404 during import. Imports that would import a no longer existing resource resulted in a 503 Server Error. Those are now properly catched and logged. Those errors will no longer fail.

Tasks 

  • Prevent creation of composer.lock file. As this is an extension without a lock state. We have different TYPO3 versions as supported versions and it should be easy to switch between them for testing and development.
  • Streamline phpstan.neon formatting.

Deprecation 

Nothing

4.0.0 

Breaking 

  • Removed content element.

    No Content element is provided any longer. We recommend to build your own tailored content elements instead.

Features 

  • Add TYPO3 v13 LTS Support.

Fixes 

Nothing

Tasks 

Nothing

Deprecation 

Nothing

3.0.1 

Breaking 

Nothing

Features 

Nothing

Fixes 

  • Add missing dependency to typo3/cms-install. As this provides the upgrade wizard feature.
  • Handle broken opening hours. Those are skipped during import and written as error to TYPO3 logs. That way entries with broken opening hours can still be imported.

Tasks 

Nothing

Deprecation 

Nothing

3.0.0 

Breaking 

  • Drop support for TYPO3 10.4 and 11.5.
  • Drop support for PHP 7.4 and 8.0.

Features 

  • Add support for TYPO3 12.4.
  • Add support for PHP 8.3.
  • Add support for multiple slogans (array within slogan). The existing API will return the first slogan. A new method getSlogans is added which will return the array of slogans.

Fixes 

Nothing

Tasks 

Nothing

Deprecation 

Nothing

2.1.0 

Breaking 

Nothing

Features 

  • Add command to allow import of a single configuration. The command is also available as scheduler task. This finally allows to regularly execute imports. This also allows to import from CLI context with differently configured timeouts.
  • Add support for additional images added via TYPO3. Some installations might need to add further images to records imported from ThüCAT. The records are now extended to support adding images by editors. The images are not touched during import. The images are also ignored during clean ups, the editor is in full control.

    This feature for now is only added to tourist attractions by default. The feature is implemented in a way that all objects extending the WerkraumMedia\ThueCat\Domain\Model\Frontend\Base class are usable by adding an editorial_images field to their table.

  • Support translated offers. The translation for offers of tourist attractions got added.

Fixes 

  • Catch mapping exceptions during converting entities. Those are handled the same way, the entity is skipped and errors are logged. That way further entities can be imported while only none working entities are skipped.
  • Fix broken check of logged errors within functional tests. The logging is now adjusted to only log errors. The file will be checked for each test that does not expect errors to happen.
  • Handle incoming array instead of string for AccessibilityCertification. That prevents mapping exceptions for objects containing the corresponding certification with more info than a single value.

Tasks 

  • Converted log entry for none converted entity from error to notice. As this might hint at an issue but most probably is okay, e.g. due to none active language, missing name, etc.
  • Separate default templates from templates for testing. That way we no longer test the delivered templates, but they should not be used anyway. Also we can now use templates only for testing to ensure that frontend rendering works as expected, without worrying about sites using the templates. The templates were copied and extended for editorial images.

Deprecation 

Nothing

2.0.0 

Breaking 

  • Permissions of backend modules. The modules got new identifiers. User permissions need to be adjusted. An update wizard is provided that will migrate the permissions.
  • Drop support for PHP 7.2 + 7.3.

Features 

  • Add support for TYPO3 v11 and PHP 8.0 + 8.1 + 8.2.
  • Last import date is now shown within backend module beside each import configuration.
  • New import configuration type "Contains Place". This allows to provide a single entity, e.g. a Town that has multiple schema:containsPlace entries. Each of them will be imported.
  • Import will no longer break on mapping issues. Those will be logged and are available within the existing backend module. This allows to skip some objects which can not be handled yet. The log can be used to open issues. We then can improve the mapping.
  • Import author of media. This allows to either render the license author or the author.
  • Filter and sort opening hours. Filter out opening hours from the past, they are not available to the template anymore. Sort opening hours from early to later based on their end timing.

    This should improve the UX of website visitors. It is not possible yet to sort opening hours by hand within the thuecat backend.

  • Support special opening hours. It is possible to define special opening hours, e.g. for holidays. Those are now also imported and provided to the templates.
  • Allow to check if an opening hour is valid for a single day.
  • The URL of attractions is now imported and provided to the template.
  • Media has a new method getExtraImages() which will return everything from getImages() except the getMainImage(). We now also filter out the main image from other images, it will not exist twice anymore.
  • Allow to hide parking facilities.
  • Parking facilities can now be rendered sorted by alphabet. Use new method getParkingFacilitiesNearBySortedByAlphabet().
  • Support types of public transport when returning distance to public transport. A new array key types was added. This is an array of types, e.g. CityBus or Streetcar. These can be used with f:translate ViewHelper to provide proper none technical labels.
  • Configure EXT:scheduler table garbage collection task to clean up import records. It is now possible to select the tables within the TYPO3 scheduler task to be cleaned up.
  • Respect schema:givenName and schema:familyName of schema:author for media. We only respected schema:name until now.
  • Provide new key copyrightAuthor holding the actual author for convenience. It will pick the author value falling back to license.author.
  • Provide new method to retrieve merged opening hours and merged special opening hours. The merge happens on the valid time span of each. The data structure is a bit different as different hours will be merged.

Fixes 

  • Allow to import objects (e.g. Tourist Attractions) which are managed by generic organisations instead of specific Tourist Marketing Company. Those organisations will now also be imported, just like Tourist Marketing Company before. Both are organizations internally and only used for the TYPO3 backend module.
  • Handle multiple thuecat:offerType values within Offer. The API is none breaking, the models still return only a single offer.

    They will filter down to the first offer which contains Offer within the value. Examples:

    Given: Childcare and CourseOffer will result in CourseOffer.

    Given: Childcare will result in Childcare.

    Existing imported data is still handled.

  • Remove trailing : in German translation of content.distanceToPublicTransport. This was the only label with :.
  • Keep editorial sorting of tourist attractions within content element. This was not the case yet, the records were sorted by dbms, e.g. by uid.

Tasks 

  • Removed API Key from site configuration. The key was already moved to extension configuration as documented. Still we extended the site configuration, which is now cleaned up, see: https://github.com/werkraum-media/thuecat/issues/55
  • Remove seconds from opens and closes of opening hours as we don't expect them to be used.
  • Use new icons in streamlined color and UI. Provide new icon for storage folders. Provide new icon for content element.
  • Add first acceptance tests for backend modules.

Deprecation 

Nothing

1.2.2 

Breaking 

Nothing

Features 

Nothing

Fixes 

Nothing

Tasks 

  • Changed Fluid templates:

    • Improve rendering of opening hours

Deprecation 

Nothing

1.2.1 

Breaking 

Nothing

Features 

Nothing

Fixes 

  • Use proper extension name for translation within templates.

    This was sitepackage and was changed to Thuecat.

Tasks 

  • Changed Fluid templates:

  • Improved composer authors:

    • Added Carlos
    • Added homepages
    • Added roles

Deprecation 

Nothing

1.2.0 

Breaking 

Nothing

Features 

Nothing

Fixes 

Nothing

Tasks 

  • Changed Fluid templates to be more polished using current Bootstrap markup.

Deprecation 

Nothing

1.1.1 

Breaking 

Nothing

Features 

  • Added missing French translations.

Fixes 

Nothing

Tasks 

Nothing

Deprecation 

Nothing

Importer architecture 

A short tour of how Classes/Import/ is laid out and how a single import flows. Audience: TYPO3 extension developers picking up this codebase. The goal is to cover the moving parts and the boundaries between them, not every method.

Components 

  • Importer — orchestrates one run. Picks a UrlProvider, walks every URL, drives the DataHandler loop, returns the run's max log severity.
  • UrlProvider (interface) — produces the list of root URLs for one configuration. Implementations: StaticUrlProvider (hand-listed URLs), SyncScopeUrlProvider (queries upstream for everything updated in a sync scope), ContainsPlaceUrlProvider.
  • Parser + Parser\Entity\* — pure transformation from a JSON-LD @graph to a DataHandlerPayload. One Entity class per imported record type (Organisation, Town, TouristInformation, TouristAttraction, ParkingFacility); TransientEntity\* covers nested value objects (Address, OpeningHours, Offer, …). EntityInterface::handlesTypes() declares which @type strings a parser claims; priority breaks ties.
  • DataHandlerPayload — typed bag the parser fills and the resolver/importer drain. Holds four buckets: dataMap, cmdMap, transients (unresolved JSON-LD references), translations (per-language partial rows). Plus auxiliary maps for default-language vs. translation rows.
  • Resolver + ResolverContext — walks the parsed tree, resolves transients (looks up FKs by remote_id, fetches missing nodes, re-parses them), drains translations once parents have uids. Visit-once contract is enforced via ResolverContext::remoteIdToKey; the default-language and translation status maps short-circuit re-resolution across rounds.
  • Importer\FetchData — HTTP boundary. Caches JSON-LD responses by (url, apiKey) SHA. Throws ResourceNotFoundException (subclass of InvalidResponseException) for upstream 404 so callers can decide whether to drop, stub, or abort.
  • ImportLogger — single producer of tx_thuecat_import_log + tx_thuecat_import_log_entry rows. Accumulates savingEntity rows from the payload plus DataHandler errorLog and caught exceptions, then flushes everything in one DataHandler call. Tracks max severity for the run.

Flow of one importConfiguration() call 

  1. Resolve the UrlProvider and the site's defaultLanguage + translationLanguages map (from SiteFinder).
  2. Build a ResolverContext that owns the run's state (storagePid, language map, status maps, remoteIdToKey).
  3. URL loop — for each URL the provider returns:

    • Fetch JSON-LD. InvalidResponseException here → log as fetchingError and continue (run survives one broken root).
    • Run Parser::parse(), then Resolver::resolve(). Any Throwable from this branch → log as mappingError and continue.
    • Merge the resolved payload into a single accumulatedPayload.
  4. Snapshot the default-language datamap for the logger (translation rows are excluded so the savingEntity counts match what users see).
  5. Drain loop — while dataMap or cmdMap is non-empty:

    • Fresh DataHandler per pass (state doesn't survive start()).
    • process_datamap() then process_cmdmap().
    • Capture $dataHandler->errorLog into the import logger as dataHandlerError rows (severity error).
    • Merge substNEWwithIDs, promote NEW… placeholders in remoteIdToKey to real uids via ResolverContext::promoteNewKeys().
    • Re-run Resolver::resolve() against the now-empty payload — translations and post-localize fields land here.
    • Iteration cap: count($translationLanguages) * 2 + 2. Round 0 writes defaults; each translation language needs one round to stage localize and one to fill the new translation row.
  6. Flush the logger (savingEntity + recorded errors → one log row + N entries).
  7. Return the run's max severity (info for clean, error if anything raised).

Why the loop has multiple passes 

DataHandler's cmdMap collapses to [$table][$uid][$command] = $value — a second localize for the same parent uid silently overwrites the first. Each translation language therefore needs its own round: round N stages exactly one localize (which materializes the translation row), round N+1 picks up the new translation uid via promoteNewKeys and writes its translated fields via the translations bucket.

ResolverContext::defaultStatus and translationStatus keep re-resolution idempotent across rounds — already-drained payloads short-circuit instead of re-fetching or re-querying.

Logging contract 

  • Every run produces one tx_thuecat_import_log row with N tx_thuecat_import_log_entry children.
  • Entry types: savingEntity (one per default-language row inserted/updated), dataHandlerError (one per errorLog[] line DataHandler raised), mappingError / fetchingError (one per caught exception in the URL loop).
  • Severity vocabulary is PSR-3 (debugemergency). DataHandler errors and caught exceptions are recorded as error; savingEntity rows are info.
  • Editors filter the BE list view by severity. The Command::SUCCESS/FAILURE exit code is driven from ImportLogger::getMaxSeverity() — anything >= error fails the command.
  • DataHandler's enableLogging stays true. Setting it false would short-circuit errorLog too, so we accept the duplicate write to sys_log and let editors filter there.

Extending the importer 

  • New imported type: add an Entity class in Parser/Entity/, declare its TCA + DB columns, register the service-locator tag import.entity. The parser picks it up via handlesTypes(). If translatable, mirror an existing translatable TCA (e.g. tx_thuecat_organisation).
  • New URL source: implement UrlProvider, register tag import.url.provider, and add a matching type string to ImportConfiguration::getType() plus its FlexForm.
  • New transient field: extend the relevant Entity::parse() to push refs into the payload's transients bucket; teach Resolver to drain the new key. This covers a scalar reference — one FK on the owner row. A property that produces a set of relations follows a different shape; see Relation-set properties. A property whose references may land in different tables follows a third; see One property, several target tables.
  • New log severity / type: extend ImportLogger::SEVERITY_*, the TCA select-list on tx_thuecat_import_log_entry.type, and the xlf labels.

Relation-set properties 

Some imported properties are not one value but a set of relations — media files, keywords, @type categories. Two of them (media, keywords) are built to the same shape, and the next one should follow it rather than rediscover it. What follows is that shape and the reasons for each part; the parts are load-bearing, not stylistic.

Several upstream shapes, one relation set 

Upstream rarely expresses such a property one way. schema:keywords arrives as an @id reference to a vocabulary term, as a typed literal naming an ontology term by CURIE, or as free text an editor typed. All three resolve to the same internal entry — identity, title, parent — and land in one relation set. Detection belongs in a small reader class; the resolver should not branch on shape.

Identity must be derived so that repeated imports reuse rather than accumulate. A URI is already an identity; free text has none, so one is derived from the value (lowercased, mb_* throughout — strtolower() is byte-wise and splits Ölmühle from ölmühle into two records). Prefix the identifier by source and shape so two shapes can never collide on one stored row.

The property is collected run-scoped, never handed to the payload 

Resolution collects entries onto ResolverContext; it does not stage them into DataHandlerPayload. A single flush after the last root writes them.

This is the part most easily got wrong. Two independent reasons:

  • A relation field is submitted as the complete set, and the framework replaces what is stored with what is submitted. Anything missing from the submitted set is thereby removed. Staging during resolution submits an incomplete set, so entries resolved later in the run are wiped by the earlier write.
  • Targets are shared heavily across roots — one vocabulary term is referenced by hundreds of objects. Resolution runs once per root URL, so a per-root write means one root removing what another had just written.

A collector is therefore run-scoped, guarded by a first-claim key of table|ownerKey|field|identity. Owner and field belong in that key: the same target claimed by two records must yield two relations, and only a repeat by the same owner collapses.

Removal falls out of submitting the complete set 

Because submission replaces, a target upstream no longer supplies loses its relation with no deletion code — provided the submitted set is complete, which is what the deferred flush guarantees. Only relations are removed; the shared target record itself stays, since editors may still use it.

The corollary is the dangerous half: an entry that failed to resolve is also missing from the set, and would therefore be removed. A technical failure is indistinguishable from an upstream deletion, so the run records which owner/field had a failure and carries that owner's stored targets forward into the submitted set. Only upstream positively reporting a target absent (404, 410) may cost a relation — see FetchFailureVerdict. Every other failure keeps what is stored, because a credential, rate-limit or server fault arrives for every target on that host at once and would otherwise strip a whole run.

One hole is known and shared by every property built this way: an owner that collects nothing never enters the flush loop, so its relations survive even when upstream dropped all of them. It is recorded in the project backlog and wants a pattern-level fix, not a per-property one.

Each property gets its own everything 

Sharing an established path is the tempting shortcut and the wrong one. A new relation-set property gets its own transient bucket, its own configuration anchor, its own collector, its own run-scoped dedup map, and its own relation column. Where two properties both borrow sys_category, that shared table is an implementation detail with no semantic meaning: they must not share an anchor, a relation field, an identifier, or a dedup bucket.

Two properties sharing a dedup map hand each other staged keys, and the trees silently merge. Two sharing an anchor put one property's records under the other's root, where a rootline-scoped lookup then finds the wrong row.

Where a target's storage location varies by table, the owning entity declares it — EntityInterface::KEYWORD_FIELD and MEDIA_FIELDS are read from the entity, never assumed by the resolver. The resolver sees only the payload, so such a declaration travels in the bucket entry alongside the reference it belongs to.

Boundaries not to be modified 

  • The ``@type`` category pathapplyCategoryMapper(), the _categories bucket, wireCategories(). It looks like a general "category relations" mechanism and is not: it stamps the category anchor and dedups through the category map. A property placed there silently acquires both. A diff touching applyCategoryMapper() while adding a new property is the sign the boundary was crossed.
  • The payload's per-row transient harvest — routing a relation set through it splits one record's set across two places at flush, which the completeness requirement above cannot work with.
  • ResolverContext::promoteNewKeys() — it must learn every new run-scoped key map. A map left out is looked up by its NEW… placeholder in the next persistence round, misses, and stages a second row for a target that already exists. Silently: nothing errors.

Tripwires 

Each of these failures is invisible in ordinary testing, so each wants a test that fails when the boundary is crossed:

  • Two properties whose targets carry deliberately identical titles, asserting each tree holds exactly its own members and that same-titled rows are distinct records with distinct identifiers.
  • Two roots referencing the same target, asserting one stored record and two relations — this is what catches dedup state living in a local instead of on the context.
  • A re-import dropping one target of several, asserting the relation is gone and the target record remains.
  • A failed fetch alongside a surviving entry, asserting nothing is removed. Note this needs a surviving entry: an owner whose every entry fails never reaches the flush, so the guard is inert and the test proves nothing.
  • Ancestors or grouping records, if the property has them, asserting they are not related to the owner — only the target the record actually cites is a relation.

One property, several target tables 

schema:containedInPlace is the case where a single upstream property points at records of different kinds. Upstream uses it for whatever contains an object: the town it sits in, the organisation responsible for it, or another place — a POI inside a park, a car park inside a shopping centre.

A transient bucket normally names one target table and one relation field. This one cannot, so Resolver::BUCKET_MAP is keyed by table throughout: bucket => [table => field]. Most buckets hold a single entry; this one holds five.

Imported as Relation
Town town
Organisation contained_in_organisation
Tourist attraction contained_in_attraction
Tourist information contained_in_tourist_information
Parking facility contained_in_parking_facility

The field is chosen by the table the referenced record actually imported into (ResolverContext::remoteIdToTable), not by the reference's @type. The parser already decided the table; re-deriving the kind from the type URI would give a second classifier for the same question, free to drift from the first.

Why one field per table 

"Any place" cannot be one relation. Extbase resolves a relation through a single concrete, table-mapped class — its only polymorphism is a recordType column selecting a subclass within one table (DataMapper::getTargetType()). A property typed across several tables produces a query against a table named after the class and fails. The core's own multi-table group fixture (blog_example's tx_blogexample_domain_model_tag.items) shows the same conclusion from the other side: the owning record never maps that property, and every readable side is typed to one concrete class.

So each target table gets its own field, and TouristAttraction::getContainedInPlaces() merges them back into one list for templates. Adding a place table to the import means adding it to the map and adding its field to the TCA of every owner table — the map is also the allowlist.

Every relation here is multi-value: a record can belong to more than one town (an airport serving two cities) and to places of several kinds at once.

Probing and reporting 

Because a bucket may name several tables, the pre-fetch lookup probes them in map order and takes the first hit. remote_id is unique per record, so one hit settles the question; the order only decides how many queries run first. Commonest kind first.

A reference whose record imported into a table the bucket has no field for is logged as referenceUnrelatable at info: the record exists and only the relation was dropped, which is upstream data drift rather than a fault. A reference that produced no record at all — a type this extension does not model — is not reported, because there was never a relation to lose. That distinction is the point of the report; a change that makes it fire for everything, or for nothing, has broken it.

Importing a sys_category-backed field 

Several imported fields store their values as sys_category records: an attraction's categories (derived from @type), its keywords, an event's keywords_relation. They arrive from different places and mean different things, but the work is identical — find or create a record per value, nest it, translate it, relate the owner to some of them.

One service does that work. Reach for it when adding the next such field; implementing it again is how the trees drifted apart before.

Class Responsibility
\WerkraumMedia\ThueCat\Import\SysCategory\SysCategoryProvisioner Finds or creates one record and answers its datamap key, or null where the term cannot be created.
\WerkraumMedia\ThueCat\Import\SysCategory\SysCategoryAnchor Where a consumer's tree lives: parent uid, storage pid, and the identifier prefix (type:, keyword:) that keeps identifiers from colliding.
\WerkraumMedia\ThueCat\Import\SysCategory\SysCategoryProvisioningState One consumer's deduplication for the run. Never shared — sharing it merges the trees.
\WerkraumMedia\ThueCat\Import\SysCategory\SysCategoryTerm One term: source value, titles per language, and the source value of its parent.
\WerkraumMedia\ThueCat\Import\SysCategory\TitleResolver What a term is called in each language, and whether the fallback map was needed.

What the provisioner guarantees 

  • Reuse by identifier, so an editor's rename survives and no re-import duplicates a record. The match is guarded by the anchor's rootline, so a record belonging to another tree is never taken.
  • Movement, not replacement. A stored record whose parent changed is re-parented in place. sys_category uids appear in plugin flexforms; a replacement looks identical in the tree and is wrong everywhere it is referenced.
  • Translations for the languages the site configures, and no others.
  • Skipping. A term with no default-language title is not created — a record an editor cannot read is worse than none — and its children attach to the nearest ancestor that was created.

Adding a field 

  1. Add the relation column to the owner table's TCA and list it in that entity's RELATION_FIELDS.
  2. Give the consumer its own SysCategoryAnchor — own parent, storage pid, identifier prefix — and its own SysCategoryProvisioningState .
  3. Resolve titles through TitleResolver if the values come from a vocabulary; pass them directly if they do not.
  4. Call provision() per term, parents before children, and relate the owner only to the terms it actually names.

Bind the state to a map on \WerkraumMedia\ThueCat\Import\ResolverContext where keys must survive between DataHandler passes: promoteNewKeys() rewrites NEW… placeholders to real uids there, and a state still holding placeholders stages a second record on the next round.

Two decisions the service does not make 

What a title means. TitleResolver asks upstream per language, treats a label carrying no language as English, and falls back to the mapper's titleMap for the default language only. That reading suits the class vocabularies and not keyword terms, whose untagged labels are German — the same JSON-LD shape means different things in different vocabularies, so one consumer is always served wrongly. Consulting the map for the default language is also what puts a value in the import report.

Which parent a value hangs from, where its source offers several. See Building the @type hierarchy.

Building the @type hierarchy 

The categories field is the one consumer whose values carry a hierarchy of their own: @type values are classes, and upstream models schema:Museum as a CivicStructure, under Place, under Thing. The import mirrors that so editors get a tree rather than a few hundred flat names.

Where it comes from 

Two vocabularies, fetched whole and merged into one index by \WerkraumMedia\ThueCat\Import\Vocabulary\VocabularyProvider :

  • https://schema.org/version/latest/schemaorg-current-https.jsonld
  • https://thuecat.org/ontology/thuecat/1.0/?format=jsonld

Whole documents rather than per-type lookups: the per-type endpoints are rate limited, and one climb would need a request per ancestor. ThueCat extends schema.org, so chains cross between them and the index must hold both to resolve one.

\WerkraumMedia\ThueCat\Import\Vocabulary\VocabularyIndexCache keeps the distilled index for 14 days, measured from a fetchedAt timestamp it stores itself rather than from the cache backend — TYPO3 cannot read an entry past its lifetime, and an expired entry is exactly what a failed refresh falls back on. A refresh is all or nothing: pairing a fresh vocabulary with a stale one drops the failed one's classes and breaks every chain crossing between them.

Building one chain 

\WerkraumMedia\ThueCat\Import\SysCategory\ChainBuilder walks upward from the type and returns the classes to create, ancestors first.

  • Cut-off. No category for schema:Thing or schema:Place: every imported record belongs to them, so they distinguish nothing. A type left without ancestors becomes a root.
  • Redundant parents. A class naming both CivicStructure and Museum, where Museum is itself a CivicStructure, has named one chain and not a fork. The nearer parent wins; the restated ancestor keeps its own level further up.
  • Genuine branches. Where the remaining parents do not meet, one is chosen — a tree cannot have two. Which one depends on what the record is, so \WerkraumMedia\ThueCat\Import\SysCategory\ParentStrategies holds a \WerkraumMedia\ThueCat\Import\SysCategory\ParentStrategy per owner table: attractions prefer a branch reaching TouristAttraction, then Place; events prefer Event; anything else takes the deepest branch. A branch reaching no preferred root is logged at warning severity, because no rule fits it and a person has to look.

Preferred roots steer without appearing. TouristAttraction and Place sit in the mappers' ignoredValues() as structural supertypes an editor should never see: the strategy uses them to choose a branch, the cut-off declines to create them, and the chain lands on the configured anchor.

Only the types a record names become relations. Ancestors exist to give the tree its levels.

Testing 

  • Functional tests live in Tests/Functional/, base class AbstractImportTestCase. HTTP is staged via GuzzleClientFaker (file-keyed by URL); use expectFetch() / expectNotFound() / expectFetchForUrl() per scenario.
  • Fixtures: payload data sets in Tests/Functional/Fixtures/Import/*.php, JSON-LD response bodies under Tests/Functional/Fixtures/Import/Guzzle/<domain>/<path>/<id>.json.
  • Assertions: Tests/Functional/Assertions/Import/*.php define expected DB state after import.
  • Run via ddev phpunit thuecat functional. Single file: append a test path. Static analysis: ddev phpstan thuecat. Style: ddev php-cs-fixer thuecat.

Maintenance 

List of changes that need to be done for maintenance reasons. Those affect the extension itself, not users of the extension.

E.g. changes once we drop a certain TYPO3 version. We might have new code backported for compatibility in older TYPO3 versions. Those changes are documented so we know what to do once we drop an older version.

TCA searchFields 

Drop TCA ctrl/searchFields. Those are kept for TYPO3 v13 backwards compatibility and can be dropped once we drop v13 support.

PHPDoc Blocks with type hints mentioning Necessary for Extbase/Symfony. 

Those are necessary (at least with TYPO3 v12) because of Extbase and the underlying Symfony component.

Extbase uses the PHPDocExtractor first, before using the ReflectionExtractor, both part of Symfony property-info package. The ReflectionExtractor will check the mutator followed by accessors prior checking the property itself. Some of our properties have different return values by accessors than the stored value that is set to the property. We therefore need to keep the PHPDoc block as this is checked first.

Sitemap