DAM to FAL migration¶

In the install tool¶

nr_dam_falmigration registers two updates for the upgrade wizard in the install tool at the position, you should execute it: Before all updates but after the initial database update. If you see the latter, please execute it first as it probably will create tables and fields that are required by the migration updates.

If there are files or categories to import from tx_dam those should show up in the upgrade wizard and you can simply execute them:

The files update

On the command line¶

nr_dam_falmigration provides three commands to run the migrations from command line:

Migrate files:

php typo3/cli_dispatch.phpsh extbase dammigration:migratefiles [--storage=123] [--dryrun] [--count]

Migrate categories:

php typo3/cli_dispatch.phpsh extbase dammigration:migratecategories [--storage=123] [--dryrun] [--count]

Migrate files and categories (shortcut for above):

php typo3/cli_dispatch.phpsh extbase dammigration:migrate [--storage=123] [--dryrun] [--count]

Options are the same for all of the above commands:

--storage=123

Limit the import to a (local) storage uid

--dryrun

Don't run the sql statements but print them out

--count

Only count the files/categories/references etc. (see output in the install tool wizards)

Mode of operation¶

If you intend to extend this extension, some background information about the operations might be useful:

1. File migration

   1. It's checked, if the extension 'filemetadata' is installed and if not, it will be installed automatically.
   2. For each local storage the new tx_dam files inside the storages path are inserted into sys_file. Thereby the tx_dam.uid is written into sys_file._migrateddamuid - tx_dam files that were already imported (have a sys_file with that _migrateddamuid) will not be imported again on subsequent runs.
   3. The new sys_file_metadata is imported from tx_dam files present in sys_file. The identification happens analogical to the sys_file identification: Each sys_file_metadata record belongs to one sys_file record which belongs to one tx_dam record.
   4. The new sys_file_reference's are imported from tx_dam_mm_ref's of tx_dam files present in sys_file. The identification happens analogical to the sys_file identification: Each sys_file_reference record belongs to one sys_file record which belongs to one tx_dam record.
   5. All <media> tags found in the sys_refindex table will be replaced by their <link> tag pendants - thus it's important that you don't update the refindex until you migrated the media tags. As the tags will be replaced, this happens only once for each record containing fields with media tags.
   6. Related records are sanitized: With the new FAL fields, TYPO3 requires those fields to be present in the foreign table and contain the number of referenced files. The migration attempts to do this for all FAL fields on all tables.

2. Category migration

   1. All new sys_category's are inserted from tx_dam_cat. Thereby the tx_dam_cat.uid is written into sys_category._migrateddamcatuid - tx_dam categories that were already imported (have a sys_category with that _migrateddamcatuid) will not be imported again on subsequent runs.
   2. The new sys_category_record_mm's are imported from tx_dam_mm_cat's of tx_dam_cat's present in sys_category. The identification happens analogical to the sys_category identification: Each sys_category_record_mm record belongs to one sys_category record which belongs to one tx_dam_cat record.
   3. Related records are sanitized: With the new category fields, TYPO3 requires those fields to be present in the foreign table and contain 1. The migration attempts to do this for all category fields on all tables.

Difference to dam_falmigration¶

There is already another extension to migrate DAM data to FAL: dam_falmigration. It can handle the migration of files, metadata, references, categories, collections, tt_news-relations and selections.

Though we developed this extension because dam_falmigration desperately needs all files to be physically available during migration. This is quite cumbersome if you want to migrate huge datasets of remote installations (in our very case the files were also moved to a cloud storage in parallel, thus we only needed to get their metadata and relations into FAL).

Moreover we could get the migrations to work much faster than with dam_falmigration because the data isn't fetched, processed and written by PHP but handled in single SQL INSERT ... SELECT transactions (by that we could f.i. import about 100k of records in like 2 minutes). This also allowed us to completely handle the migrations as one transaction which is automatically rolled back whenever something goes wrong. As we the migrations are thus faster, more resource efficient and reliable we could wrap them and create an update step for the install tool upgrade wizard.

However dam_falmigration offers some migrations that we didn't implement as we didn't need them (yet) or because we solved them in a different way: For operations that utilize the deprecated DAM API we've developed a compatibility layer which also hooks into TCA and FlexForm configuration handling to rewrite DAM to FAL relations at runtime.

Override mapping¶

The default field mapping can be found in ext_localconf.php and you may override it from any other extensions. Those should depend on nr_dam_falmigration in order to be invoked in the correct order. Also they should register the mappings depending on the TYPO3 version so that they can be installed prior to a update to TYPO3 6.2 (see nr_dam_falmigration/ext_localconf.php for an example way).

In order to override mappings, call the method Netresearch\NrDamFalmigration\Service\FileMigrationService::appendMappings from your ext_localconf.php, which takes two arguments: The target table and and an array where the keys are the fields of the target table and the values are the source statement. Those can contain the correctly prefixed field names, SQL statements, variables (:variable) and references to the default mapping (:default(target_table.field)):

// ext_localconf.php
<?php
\Netresearch\NrDamFalmigration\Service\FileMigrationService::appendMappings(
    'sys_file_metadata',
    array(
        'width' =>
            "IF (tx_dam.tx_myext_width != '', tx_dam.tx_myext_width, :default(sys_file_metadata.width))",
        'height' =>
            "IF (tx_dam.tx_myext_height != '', tx_dam.tx_myext_height, :default(sys_file_metadata.height))",
        'tx_myext_importtime' => 'UNIX_TIMESTAMP()'
    )
);

Below you find a table listing the source tables and variables available for each target table:

// ext_localconf.php
<?php
$signalSlotDispatcher->connect(
    'Netresearch\\NrDamFalmigration\\Service\\AbstractMigrationService',
    Netresearch\\NrDamFalmigration\\Service\\AbstractMigrationService::CREATE_INSERT_QUERY_SIGNAL,
    'Vendor\\Package\\Service\\MyAspect',
    'createQuery'
);

// Classes/Service/MyAspect.php
<?php
namespace Vendor\Package\Service;
class MyAspect {
    public function createQuery(Service $service, $args) {
        if ($args['to'] == 'sys_file') {
            $args['vars']['myArg'] = 123;
        }
    }
}