The schema below provides an overview of the external import process:
The process is comprised of steps, each of which correspond to a
PHP class (found in
Classes/Step). The steps are not the same
when synchronizing (pulling) data or when using the API (pushing).
In the above schema, the steps with a gradient background belong to
both processes. The ones with a single color background are called
only by the corresponding process.
Each step may affect the raw data (the data provided by the external source) and the so-called “records” (the data as it is transformed by External Import along the various steps). A step can also set an “abort” flag, which will interrupt the import process after the step has completed.
This chapter gives an overview of what each step does:
- Check permissions
\Cobweb\ExternalImport\Step\CheckPermissionsStepThis step checks whether the current user has the rights to modify the table into which data is being imported. If not, the process will abort.
- Validate configuration
\Cobweb\ExternalImport\Step\ValidateConfigurationStepThis step checks that the main configuration as well as each column configuration are valid. If any of them is not, the process will abort. The process will also abort if there is no general configuration or not a single column configuration.
- Validate connector
\Cobweb\ExternalImport\Step\ValidateConnectorStepThis steps checks if a Connector has been defined for the synchronize process. In a sense, it is also a validation of the configuration, but restricted to a property used only when pulling data.
Up to that point, the
\Cobweb\ExternalImport\Domain\Model\Dataobject contains no data at all.
- Read data
\Cobweb\ExternalImport\Step\ReadDataStepThis step reads the data from the external source using the defined Connector. It stores the result as the “raw data” of the
- Handle data
\Cobweb\ExternalImport\Step\HandleDataStepThis step takes the raw data, which may be a XML structure or a PHP array, and makes it into an associative PHP array. The keys are the names of the columns being mapped and any additional fields declared with the additionalFields property. The values are those of the external data. The results are stored in the “records” of the
- Validate data
\Cobweb\ExternalImport\Step\ValidateDataStepThis steps checks that the external data passes whatever conditions have been defined. If this is not the case, the process is aborted.
- Transform data
\Cobweb\ExternalImport\Step\TransformDataStepThis step applies all the possible transformations to the external data, in particular mapping it to other database tables. The “records” in the
\Cobweb\ExternalImport\Domain\Model\Dataobject are updated with the transformed values.
- Store data
\Cobweb\ExternalImport\Step\StoreDataStepThis is where data is finally stored to the database. Some operations related to MM relations also happen during this step. The “records” in the
\Cobweb\ExternalImport\Domain\Model\Dataobject now contain the “uid” field.
- Clear cache
\Cobweb\ExternalImport\Step\ClearCacheStepThis step runs whatever cache clearing has been configured.
- Connector callback
\Cobweb\ExternalImport\Step\ConnectorCallbackStepIn this step the connector is called again in case one wishes to perform some clean up operations on the source from which the data was imported (for example, mark the source data as having been imported). The
postProcessOperations()method of the connector API is called.