Appendix

Changes

The general configuration must now be placed in $GLOBALS['TCA'][table-name]['external']['general'] instead of $GLOBALS['TCA'][table-name]['ctrl']['external'].

The "additionalFields" property from the general configuration (and not from the "MM" property) has been moved to its own configuration space. Rather than $GLOBALS['TCA'][table-name]['ctrl']['external'][some-index]['additionalFields] it is now $GLOBALS['TCA'][table-name]['external']['additionalFields'][some-index]. Furthermore, it is no longer a simple comma-separated list of fields, but an array structure with all the same options as standard column configurations. For more details, see the relevant chapter.

The "MM" property is deprecated. It should not be used anymore. Instead the new multipleRows or children properties should be used according to your import scenario.

The "userFunc" property of the transformations configuration has been renamed to userFunction and its sub-property "params" has been renamed "parameters".

If both "insert" and "update" operations are disabled in the general configuration (using the disabledOperations property), External Import will now delete records that were not marked for update (even if the actual update does not take place). Previously, no records would have been deleted, because the entire matching of existing records was skipped.

Accessing the external configuration inside a custom step with $this->configuration or $this->getConfiguration() is deprecated. $this->getImporter()->getExternalConfiguration() instead.

The "scheduler" system extension is required instead of just being suggested.

New stuff

It is possible to import nested structures using the children property. For example, you can now import data into some table and its images all in one go by creating a nested structure for the "sys_file_reference" table.

The multipleRows and multipleSorting properties allow for a much clearer handling of denormalized external sources.

Check out the revamped Mapping data chapter which should hopefully help you get a better picture of what is possible with External Import and how different properties (especially the new ones) can be combined.

Custom steps can now receive an array of arbitrary parameters.

Breaking changes

The \Cobweb\ExternalImport\Step\StoreDataStep class puts the list of stored records into the "records" member variable of the \Cobweb\ExternalImport\Domain\Model\Data object. This used to be a simple list of records for the imported table. Since child tables are now supported, the structure has changed so that there's now a list of records for each table that was imported. The table name is the key in the first dimension of the array. If you were relying on this data in a custom step, you will need to update your code as no backward-compatibility was provided for this change.

Importer API changes

The External Import configuration is now fully centralized in a \Cobweb\ExternalImport\Domain\Model\Configuration object. Every time you need some aspect of the configuration, you should get it via the instance of this class rather than through any other mean. The most current use case was getting the name of the current table and index from the \Cobweb\ExternalImport\Importer class, using Importer::getTableName() and Importer::getIndex(). Such methods were deprecated and should not be used anymore. Use instead:

$table = $importer->getExternalConfiguration()->getTable();
$index = $importer->getExternalConfiguration()->getIndex();

The Importer::synchronizeData() method was renamed to Importer::synchronize() and the Importer::importData() method was renamed to Importer::import(). The old methods were kept, but are deprecated.

The Importer::synchronizeAllTables() method should not be used anymore as it does not allow for a satisfying reporting. Instead a loop should be done on all configurations and Importer::synchronize() called inside the loop. See for example \Cobweb\ExternalImport\Command\ImportCommand::execute().

Other deprecated methods are Importer::getColumnIndex() and Importer::getExternalConfig().

The Importer::getExistingUids() method was moved to a new class called \Cobweb\ExternalImport\Domain\Repository\UidRepository (which is a Singleton).

Transformation properties

All column properties that are related to the "Transform data" scope have been grouped into a new property called transformations. This is an ordered array, which makes it possible to use transformation properties several times on the same field (e.g. calling several user functions) and to do that in a precise order. As an example, usage of such properties should be changed from:

$GLOBALS['TCA']['fe_users']['columns']['starttime']['external'] = [
      0 => [
            'field' => 'start_date',
            'trim' => true
            'userFunction' => [
                  'class' => \Cobweb\ExternalImport\Task\DateTimeTransformation::class,
                  'method' => 'parseDate'
            ]
      ]
];

to:

$GLOBALS['TCA']['fe_users']['columns']['starttime']['external'] = [
      0 => [
            'field' => 'start_date',
            'transformations => [
                  10 => [
                        'trim' => true
                  ],
                  20 => [
                        'userFunc' => [
                              'class' => \Cobweb\ExternalImport\Task\DateTimeTransformation::class,
                              'method' => 'parseDate'
                        ]
                  ]
            ]
      ]
];

If you want to preserve "old-style" order, the transformation properties were called in the following order up to version 3.0.x: "trim", "mapping", "value", "rteEnabled" and "userFunc". Also note that "value" was ignored if "mapping" was also defined. Now both will be taken into account if both exist (although that sounds rather like a configuration mistake).

A compatibility layer ensures that old-style transformation properties are preserved, but this is a temporary convenience, which will be removed in the next version. So please upgrade your configurations.

Renamed properties

To continue the move to unified naming conventions for properties started in version 3.0, the mapping and MM properties which had underscores in their names were moved to lowerCamelCase name.

The old properties are interpreted for backwards-compatibility, but this will be dropped in the next major version. The backend module will show you the deprecated properties.

Breaking changes

While all hooks were preserved as is, in the sense that they still receive a back-reference to the \Cobweb\ExternalImport\Importer object, the processParameters hook was modified due to its particular usage (it is called in the backend module, so that processed parameters can be viewed when checking the configuration). It now receives a reference to the \Cobweb\ExternalImport\Domain\Model\Configuration object and not to the \Cobweb\ExternalImport\Importer object anymore. Please update your hooks accordingly.