Transformations configuration

A number of properties relate to transforming the data during the import process. All of these properties are used during the “Transform data” step. They are sub-properties of the transformations property.

Properties

Property Data type Step/Scope
isEmpty array Transform data
mapping Mapping configuration Transform data
rteEnabled boolean Transform data
trim boolean Transform data
userFunction array Transform data
value simple type (string, integer, boolean) Transform data

mapping

Type
Mapping configuration
Description
This property can be used to map values from the external data to values coming from some internal table. A typical example might be to match 2-letter country ISO codes to the uid of the “static_countries” table.
Scope
Transform data

value

Type
Simple type (string, integer, boolean)
Description
With this property, it is possible to set a fixed value for a given field. For example, this might be used to set a flag for all imported records.
Scope
Transform data

trim

Type
boolean
Description

If set to true, every value for this column will be trimmed during the transformation step.

Note

With newer versions of PHP, trying to trim a non-string causes an error. To account for that, since External Import 6.0.1, non-string data is left unchanged by this transformation. This may cause changes in your import, as previously the data used to be cast on the fly and trimmed.

If you are affected by this change, you should create a custom transformation with a userFunction to cast your data explicitly before calling trim.

Scope
Transform data

rteEnabled

Type
boolean
Description

If set to true when importing HTML data into a RTE-enable field, the imported data will go through the usual RTE transformation process on the way to the database.

Note

Since the data goes through the RTE transformation process, you should mind the settings of the RTE for the given field if the results are unexpected. This is particularly true for tags which are not inside other tags and need to be explicitly allowed using the allowTagsOutside option for example (see the RTE configuration reference).

Scope
Transform data

userFunction

Type
array
Description

This property can be used to define a function that will be called on each record to transform the data from the given field. See example below.

Example

Here is a sample setup referencing a user function:

       $GLOBALS['TCA']['fe_users']['columns']['starttime']['external'] = [
0 => [
   'field' => 'start_date',
   'transformations' => [
      10 => [
         'userFunction' => [
            'class' => \Cobweb\ExternalImport\Task\DateTimeTransformation::class,
            'method' => 'parseDate'
         ]
      ]
   ]
]
       ];

The definition of a user function takes three parameters:

class
(string) Required. Name of the class to be instantiated.
method
(string) Required. Name of the method that should be called.
parameters (formerly “params”)
(array) Optional. Can contain any number of data, which will be passed to the method. This used to be called “params”. Backwards-compatibility is ensured for now, but please update your configuration as soon as possible.

In the example above we are using a sample class provided by External Import that can be used to parse a date and either return it as a timestamp or format it using either of the PHP functions date() or strftime() .

Note

Since External Import 5.1.0, if the user function throws an exception while handling a value, that value will be unset and thus removed from the imported dataset. The rationale is that such a value is considered invalid and should not be further processed nor saved to the database.

The user function can also specifically throw the \Cobweb\ExternalImport\Exception\InvalidRecordException. The effect is to remove the entire record from the imported dataset.

For more details about creating a user function, please refer to the Developer’s Guide.

Scope
Transform data

isEmpty

Type
array
Description

This property is used to assess if a value in the given column can be considered empty or not and, if yes, act on it. The action can be either to set a default value or to remove the entire record from the imported dataset.

Deciding whether a given value is “empty” is a bit tricky, since null, false, 0 or an empty string - to name a few - could all be considered empty depending on the circumstances. By default, this property will rely on the PHP function empty(). However it is also possible to evaluate an expression based on the values in the record using the Symfony Expression Language.

expression

(string) A condition using the Symfony Expression Language syntax. If it evaluates to true, the action (see below) will be triggered. The values in the record can be used, by simply referencing them with the column name.

If no expression is defined, the PHP function empty() is used.

See the Symfony documentation for reference.

invalidate
(bool) Set this property to true to discard the entire record from the imported dataset if the expression (or empty()) evaluated to true. invalidate takes precedence over default.
default
(mixed) If the expression (or empty()) evaluate to true, this value will be set in the record instead of the empty value.

Example

'store_code' => [
    'exclude' => 0,
    'label' => 'Code',
    'config' => [
        'type' => 'input',
        'size' => 10
    ],
    'external' => [
        0 => [
            'field' => 'code',
            'transformations' => [
                10 => [
                    'trim' => true
                ],
                20 => [
                    'isEmpty' => [
                        'expression' => 'store_code === ""',
                        'invalidate' => true
                    ]
                ],
            ]
        ]
    ]
],

In this example, the store_code field is compared with an empty string. Any record with an empty string in that column will be removed from the dataset.

Note

Since you can write any expression as long as it evaluates to a boolean value, this property actually makes it possible to test another condition than just emptiness, although it may be confusing to use it in this way.

Warning

There’s a weird behavior in the Symfony Expression Language: if the value being evaluated is missing from the record, the parser throws an error as if the syntax were invalid. The workaround implemented in External Import is that an evaluation throwing an exception is equivalent to the evaluation returning true. This makes it possible to handle missing values, but has the drawback that a real syntax error will not be detected and all values will be considered empty.

Such events are logged (at notice-level).