Installation

Installing this extension does nothing in and of itself. You still need to extend the TCA definition of some tables with the appropriate syntax and create specific connectors for the application you want to connect to.

TYPO3 CMS 12 or 13 is required, as well as the "scheduler" and "reactions" system extensions.

Upgrading and what's new

Upgrade to 8.1.0

\Cobweb\ExternalImport\Importer::getContext() and \Cobweb\ExternalImport\Importer::setContext() have been deprecated in favor of \Cobweb\ExternalImport\Importer::getCallType() and \Cobweb\ExternalImport\Importer::setCallType(). These methods rely on the \Cobweb\ExternalImport\Enum\CallType enumeration which is used more consistenty throughout External Import.

A new event ChangeConfigurationBeforeRunEvent makes it possible to modify the External Import configuration at run-time. This happens before any of the import steps is executed.

Upgrade to 8.0.0

Configurations can now be part of several groups. As such, the "group" property is deprecated and is replaced with the groups property (with an array value rather than string).

Note

A Rector rule is provided for migration. Use it in your rector.php file:

return RectorConfig::configure()
    ...
    ->withRules([
        ...
        \Cobweb\ExternalImport\Rector\ChangeGroupPropertyRector::class,
    ])
    ...
;

System extension "reactions" is now a requirement. The "Import external data" reaction can now target a group of configurations.

The logging mechanism has been changed to store the backend user's name rather than its id. This makes it much easier for the Log module and keeps working even if a user is removed. An update wizard is available for updating existing log records.

Warning

Don't drop the "cruser_id" field before running the update wizard, or it won't be able to do its job.

In version 7.2.0, a change was introduced to preserve null values from the imported data. It affected only fields with 'eval' => 'null' in their TCA. Since version 8.0.0, null are preserved also for relation-type fields ("group", "select", "inline" and "file") which have no minitems property or 'minitems' => 0. This makes it effectively possible to remove existing relations. This is an important change of behavior, which - although more correct - may have unexpected effects on your date.

A new disabled flag makes it possible to completely hide a configuration.

Upgrade to 7.3.0

This version introduces a new reaction dedicated to deleting already import data.

Upgrade to 7.2.0

The HandleDataStep process now keeps null values found in the imported data. This is an important change, but is has a concrete effect only if the target field is nullable (i.e. it has an eval property including null or has property nullable set to true in its TCA configuration). In such cases, existing values will be set to null where they would have been left untouched before. It may also affect user functions in transformations where a null value was not expected to be found until now.

Upgrade to 7.1.0

External Import now supports PHP 8.2.

When running the preview mode from the backend module, some steps now provide a download button, to retrieve the data being handled in its current state.

When setting a fixed value, the new column configuration property should be preferred over the historical transformation property.

It is now possible to define explicitly the order in which columns are processed.

Upgrade to 7.0.0

Support for old-style Connector services was droppped (i.e. connectors registered as TYPO3 Core Services). If you use custom connector services, make sure to update them (see the update instructions provided by extension "svconnector").

When editing Scheduler tasks in the External Import backend module, it is no longer possible to define a start date (this tiny feature was a lot of hassle to maintain across TYPO3 versions).

All hooks were removed. If you were still using hooks, please refer to the archived page about hooks to find replacement instructions.

A new ReportStep has been introduced, which triggers a webhook reporting about the just finished import run. In order for this step to run (and do the reporting) even when the process is aborted, a new possibility has been added for steps to run despite the interruption. This actually fixes a bug with the ConnectorCallbackStep which was never called when the process was aborted. If you use such a post-processing, you can now report about failed imports if needed.

New stuff

The arrayPath is now available as both a general configuration option and a column configuration option. It was also enriched with more capabilities.

A new exception \Cobweb\ExternalImport\Exception\InvalidRecordException was introduced which can be used inside user function to remove an entire record from the data to import if needed.

A new transformation property isEmpty is available for checking if a given data can be considered empty or not. For maximum flexibility, it relies on the Symfony Expression language.

It is also possible to set multiple mail recipients for the import report instead of a single one (see the extension configuration).

Upgrade to 5.1.0

There is a single change in version 5.1.0 that may affect existing imports: when a user function fails to handle the value it was supposed to transform (by throwing an exception), that value is now removed from the imported dataset. Before that it was left unchanged.

Upgrade to older version

In case you are upgrading from a very old version and proceeding step by step, you find all the old upgrade instructions in the Appendix.

Other requirements

As is mentioned in the introduction, this extension makes heavy use of an extended syntax for the TCA. If you are not familiar with the TCA, you are strongly advised to read up on it in the TCA Reference manual.