Installation

Installing this extension does nothing in and of itself. You still need to extend the TCA definition of some tables with the appropriate syntax and create specific connectors for the application you want to connect to.

TYPO3 CMS 11 or 12 is required, as well as the "scheduler" system extension.

For TYPO3 CMS 12, extension "reactions" is also required. It needs to be required "manually". It is not declared in External Import's composer.json manifest in order to preserve compatibility with TYPO3 11.

Upgrading and what's new

Upgrade to 7.2.0

The HandleDataStep process now keeps null values found in the imported data. This is an important change, but is has a concrete effect only if the target field is nullable (i.e. it has an eval property including null or has property nullable set to true in its TCA configuration). In such cases, existing values will be set to null where they would have been left untouched before. It may also affect user functions in transformations where a null value was not expected to be found until now.

Upgrade to 7.1.0

External Import now supports PHP 8.2.

When running the preview mode from the backend module, some steps now provide a download button, to retrieve the data being handled in its current state.

When setting a fixed value, the new column configuration property should be preferred over the historical transformation property.

It is now possible to define explicitly the order in which columns are processed.

Upgrade to 7.0.0

Support for old-style Connector services was droppped (i.e. connectors registered as TYPO3 Core Services). If you use custom connector services, make sure to update them (see the update instructions provided by extension "svconnector").

When editing Scheduler tasks in the External Import backend module, it is no longer possible to define a start date (this tiny feature was a lot of hassle to maintain across TYPO3 versions).

All hooks were removed. If you were still using hooks, please refer to the archived page about hooks to find replacement instructions.

A new ReportStep has been introduced, which triggers a webhook reporting about the just finished import run. In order for this step to run (and do the reporting) even when the process is aborted, a new possibility has been added for steps to run despite the interruption. This actually fixes a bug with the ConnectorCallbackStep which was never called when the process was aborted. If you use such a post-processing, you can now report about failed imports if needed.

Upgrade to 6.3.0

External Import now supports Connector services registered with new system introduced with extension "svconnector" version 5.0.0, while staying compatible with the older versions.

Another small new feature is the possibility to define a storage pid for the imported data on the command line or when creating a Scheduler task, which overrides storage information that might be found in the TCA or in the extension configuration.

Upgrade to 6.2.0

The Substructure Preprocess event is now fired for both array-type and XML-type data (previously, only for array-type data). To know which type of data is being handled, a new getDataType() method is available. The type of structure that must be returned after modfication (by calling setStructure() must be either an array or a \DomNodeList, as opposed to just an array in older versions. Existing event listeners may need to be adapted.

Upgrade to 6.1.0

Records which have no external key set (the value referenced by the referenceUid property) are now skipped in the import. Indeed it makes no sense to import records without such keys, as they can never be updated and - if several are created in a single import run - they will override each other. Still it is a change of behaviour and should be noted.

Upgrade to 6.0.0

All properties that were deprecated in version 5.0.0 were removed and the backwards-compatibility layer was dropped. Please refer to the 5.0.0 upgrade instructions and check if you have applied all changes.

All hooks were marked as deprecated. They will be removed in version 7.0.0. You should migrate your code to use either custom process steps or the newly introduced PSR-14 events. See the hooks chapter for information about how to migrate each hook.

External Import is now configured for using the standard (Symfony) dependency injection mechanism. This means it is not necessary to instantiate the \Cobweb\ExternalImport\Importer class using Extbase's \TYPO3\CMS\Extbase\Object\ObjectManager anymore when using the Importer as an API.

The PHP code was cleaned up as much as possible and strict typing was declared in every class file. This may break your custom code if you were calling public methods without properly casting arguments.

New stuff

The arrayPath is now available as both a general configuration option and a column configuration option. It was also enriched with more capabilities.

A new exception \Cobweb\ExternalImport\Exception\InvalidRecordException was introduced which can be used inside user function to remove an entire record from the data to import if needed.

A new transformation property isEmpty is available for checking if a given data can be considered empty or not. For maximum flexibility, it relies on the Symfony Expression language.

It is also possible to set multiple mail recipients for the import report instead of a single one (see the extension configuration).

Upgrade to 5.1.0

There is a single change in version 5.1.0 that may affect existing imports: when a user function fails to handle the value it was supposed to transform (by throwing an exception), that value is now removed from the imported dataset. Before that it was left unchanged.

Upgrade to older version

In case you are upgrading from a very old version and proceeding step by step, you find all the old upgrade instructions in the Appendix.

Other requirements

As is mentioned in the introduction, this extension makes heavy use of an extended syntax for the TCA. If you are not familiar with the TCA, you are strongly advised to read up on it in the TCA Reference manual.