Installation

Installing this extension does nothing in and of itself. You still need to extend the TCA definition of some tables with the appropriate syntax and create specific connectors for the application you want to connect to.

TYPO3 CMS 13 or 14 is required, as well as the "scheduler" and "reactions" system extensions.

Upgrading and what's new

Upgrade to 9.1.0

CobwebExternalImportImporter::getContext() and CobwebExternalImportImporter::setContext(), deprecated since version 8.1.0, have been removed. Use CobwebExternalImportImporter::getCallType() and CobwebExternalImportImporter::setCallType() instead.

The CobwebExternalImportImporter::SYNCHRONYZE_DATA_STEPS constant has been renamed to CobwebExternalImportImporter::SYNCHRONIZE_DATA_STEPS (fixing a typo). The old name is deprecated and will be removed in version 10.0.0.

CobwebExternalImportEventDatamapPostprocessEvent::getData() is deprecated and will be removed in version 10.0.0. Use getStructuredData() instead.

Support for the old group property was not entirely removed in version 9.0.0. This is now done.

Upgrade to 9.0.0

Version 9.0.0 adds support for TYPO3 14, while dropping support for TYPO3 12.

Since TYPO3 14, Scheduler tasks are entirely defined using TCA and are thus strictly restricted to admin users. Rather than trying to work around this even more than what was done up to now, adding, editing and deleting Scheduler tasks from the External Import backend module has been restricted to admin users.

Another consequence of this restructuring is that a new Scheduler task \Cobweb\ExternalImport\Task\SynchronizationTask replacing \Cobweb\ExternalImport\Task\AutomatedSyncTask with a cleaner structure. See details below about the upgrade/migration process.

The deprecated group property was definitely removed. If you still used it, you need to switch to the groups property instead.

A new examples chapter (with just one example for now) provides detailed explanations for the most complex configurations.

Scheduler tasks upgrade/migration

Please follow these steps for upgrading properly:

If you are running TYPO3 13, ignore this whole process. The new Scheduler task exists only for TYPO3 14 and above.
Run the upgrade wizard "Migrate the contents of the tx_scheduler_task database table into a more structured form." provided by the TYPO3 Core.
Run the upgrade wizard "Migrate old AutomatedSyncTask to newer SynchronizationTask" provided by External Import.

Upgrade to 8.2.0

XPath functions can be used in the columns configuration to directly return a string value. Previously XPath expressions could only be used to select a node or list of nodes in the XML structure.

A new event ModifyReactionResponseEvent is available to modify the response of a reaction before it is sent back. Both the response body and the HTTP return code may be changed.

Loading of the TCA has been encapsulated into a repository class, making it easier to follow the evolutions of the TYPO3 Core and allowing developers who might need it to dynamically manipulate the full TCA before the External Import configurations are extracted from it.

Upgrade to 8.1.0

CobwebExternalImportImporter::getContext() and CobwebExternalImportImporter::setContext() have been deprecated in favor of CobwebExternalImportImporter::getCallType() and CobwebExternalImportImporter::setCallType(). These methods rely on the \Cobweb\ExternalImport\Enum\CallType enumeration which is used more consistenty throughout External Import.

A new event ChangeConfigurationBeforeRunEvent makes it possible to modify the External Import configuration at run-time. This happens before any of the import steps is executed.

Upgrade to 8.0.0

Configurations can now be part of several groups. As such, the "group" property is deprecated and is replaced with the groups property (with an array value rather than string).

Note

A Rector rule is provided for migration. Use it in your rector.php file:

return RectorConfig::configure()
    ...
    ->withRules([
        ...
        \Cobweb\ExternalImport\Rector\ChangeGroupPropertyRector::class,
    ])
    ...
;

System extension "reactions" is now a requirement. The "Import external data" reaction can now target a group of configurations.

The logging mechanism has been changed to store the backend user's name rather than its id. This makes it much easier for the Log module and keeps working even if a user is removed. An update wizard is available for updating existing log records.

Warning

Don't drop the "cruser_id" field before running the update wizard, or it won't be able to do its job.

In version 7.2.0, a change was introduced to preserve null values from the imported data. It affected only fields with 'eval' => 'null' in their TCA. Since version 8.0.0, null are preserved also for relation-type fields ("group", "select", "inline" and "file") which have no minitems property or 'minitems' => 0. This makes it effectively possible to remove existing relations. This is an important change of behavior, which - although more correct - may have unexpected effects on your date.

A new disabled flag makes it possible to completely hide a configuration.

New stuff

The arrayPath is now available as both a general configuration option and a column configuration option. It was also enriched with more capabilities.

A new exception \Cobweb\ExternalImport\Exception\InvalidRecordException was introduced which can be used inside user function to remove an entire record from the data to import if needed.

A new transformation property isEmpty is available for checking if a given data can be considered empty or not. For maximum flexibility, it relies on the Symfony Expression language.

It is also possible to set multiple mail recipients for the import report instead of a single one (see the extension configuration).

Upgrade to older version

In case you are upgrading from a very old version and proceeding step by step, you find all the old upgrade instructions in the Appendix.

Other requirements

As is mentioned in the introduction, this extension makes heavy use of an extended syntax for the TCA. If you are not familiar with the TCA, you are strongly advised to read up on it in the TCA Reference manual.