A CLI and REST frontend to TYPO3's DataHandler 

This is an import extension that provides CLI and REST endpoints for creating, updating, and deleting records in TYPO3. It uses TYPO3's DataHandler API, so data is inserted as if you submitted a form in the Backend. This means changes are visible in the record history, and you can even revert to previous versions.

Backend user permissions 

REST calls authenticate as a backend user. That user's permissions are the permissions of the Interest extension. This means the REST calls can only create the records the backend user has permission to do. The extension uses TYPO3's own backend permission checking functionality.

Remote ID mapping 

During an import, you don't want to have to keep track of what UID each new record receives in the TYPO3 database, and it might be different between your development, staging, and live environments anyway. Sometimes one record in the import source can become multiple records in TYPO3.

The interest extension keeps track of the UIDs for you and maps them to whatever name (aka. a "remote ID") your data source gives it. The remote ID can be any string, not just a number. That makes debugging much easier!

Some remote ID examples:

• News-123
• NewsLink-123
• myCoolImage.jpg
• Hash-ab3235fdab0a

You can even manually create remote IDs for records in TYPO3, so the remote ID ContactPage always represents the UID of the Contact Us page. Another example is files that are uploaded to TYPO3 manually: You can use the file name and parts of the path as a remote ID and common reference point.

Send data in any order: Relation tracking and insert deferral 

Import data often out of order. You have to insert the parent page before you insert the children.

The Interest extension makes it possible to insert data in any order. Yes, you can send over the data of the child pages before the parent is present.

It does this with relation tracking and insert deferral:

• Relation tracking: Interest keeps track of a record's relations. Especially the records that have not yet been created. Once one of the missing related records are created, the extension will make sure that the relations are set up. It will even track the intended order of relations: If record 7 is created before record 4, it will insert the new record before the other.
• Insert deferral: Some records are not useful or create errors if they exist without one or more of their relations. Interest can store the data away and wait with inserting it until the relation is created. By default sys_file_reference records are deferred until the related sys_file is created. The extension provides an API for registering more insert deferrals.

Flexible file uploading 

Files can be created based on an URL, Base64-encoded data, or a URL to a MediaHandler-compatible website, such as YouTube and Vimeo.

You can even make relations to files e.g. on an Amazon S3 or Microsoft Azure mount within TYPO3: It is easy to give them remote IDs based on their ID in the external storage system.

The extension also supports putting files in any number of subfolders based on a hash of the filename. E.g.: interest_files/a/0/image.jpg

Data transformations 

All record data values can be modified using PHP and stdWrap. Each call to an endpoint can also include meta data that is not saved to the database, but can be used to make conditional decisions, such as deciding which PID a record should go into depending on some otherwise unused value in the source data.

Optimizations 

Interest includes a number of optimizations that will make it easier for you to maintain a data import with few changes:

• The same operation will not be repeated twice in a row. The extension hashes the data of the record operation and checks it.
• Files are not downloaded again, unless the file has changed. The extension tracks modification date and ETag headers.
• Disable reference indexing. Updating the reference index is time consuming when you are writing a lot of data. You can disable it during import and and update the index afterwards.
• For TYPO3 v9, the CLI module also disables registering of Extbase commands to greatly improve performance.

High Performance 

This extension has not been made to make imports faster, only better. Using TYPO3's backend APIs will always be slower than a list of SQL commands. On the other side, Interest tries to ensure that the date will always be as TYPO3 expects it, using core APIs whenever possible. And you can run it in multiple threads.

Read operations 

Read operations are currently not supported. It is a question how helpful it would be. In our experience, an import scenario where you need to use the Interest extension to read out data is not well designed. You should be able to trust that the data you are sending to Interest is correct: Check the data beforehand.

The only exception to this is the touch timestamps that are attached to each remote ID mapping. They can be used to see when a remote ID was touched last (i.e. when Interest was sent an operation directed towards it, even if no data was written in the end). It can be used to track when your data source last mentioned a particular remote ID to the Interest extension, for example to know which records to delete because they are no longer mentioned in your data source.

Frontend operations 

Just like the TYPO3 Backend, the Interest extension is not suited as your website. It is an admin tool.

REST 

Behavior 

Log 

Logging of REST calls, including request and response data and execution time.

REST API 

The REST API's write permissions are drawn from the authenticated backend user. You can configure the user in the TYPO3 backend as any other user. We recommend using a non-admin user with permissions limited to exactly the tables, page mounts, and file storage that are required.

Authenticating admin users is also possible and slightly more performant. However, it also comes with potential security risks.

CLI commands 

The write permissions of the CLI commands is drawn from the default _cli_ backend user.

Accessing metadata 

Both the CLI commands and REST endpoints support sending metadata alongside with the record data. The metadata is common to the command or request, and it can e.g. be used to set storage PID dependent on a criteria.

Here's an example using market metadata to configure the PID:

tx_interest.persistence {
  storagePid {
    cObject = CASE
    cObject {
      key.data = field : metaData | market

      Denmark = TEXT
      Denmark.value =

      Norway = TEXT
      Norway.value = 1814

      Sweden = TEXT
      Sweden.value =
    }
  }

UserTS properties 

UserTS configuration properties are set within tx_interest in the UserTS field of a backend user or backend user group, or within a .userts file.

interest:clearhash 

Interest stores a hash of the data in each operation together with the remote ID. This command clears this hash. The hash is used to prevent an operation from being executed repeatedly.

Sometimes, especially in connection with testing, clearing the hash and re-running the operation makes sense.

typo3 interest:clearhash REMOTE_ID [-c|--contains]

interest:create 

Create a record.

typo3 interest:create ENDPOINT REMOTE_ID [LANGUAGE [WORKSPACE]] [-u|--update] [-d|--data]  [-m|--metaData] [-b|--batch] [--disableReferenceIndex]

interest:delete 

Delete a record.

typo3 interest:delete REMOTE_ID [LANGUAGE [WORKSPACE]] [--disableReferenceIndex]

interest:pendingrelations 

View statistics and optionally try to resolve pending relations.

typo3 interest:pendingrelations [-r|--resolve]

interest:update 

Update a record.

typo3 interest:update ENDPOINT REMOTE_ID [LANGUAGE [WORKSPACE]] [-c|--create] [-d|--data]  [-m|--metaData] [-b|--batch] [--disableReferenceIndex]

Scheme: basic 

Initial authentication is done using the basic HTTP authentication schema, where a Base64-encoded string is submitted to the server. The string is a TYPO3 backend username and the corresponding password, separated by a colon: :.

Given the username "testuser" and password "test1234", the concatenated string will be "testuser:test1234" and the Base64-encoded version: "dGVzdHVzZXI6dGVzdDEyMzQ=".

curl -XPOST \
     -H 'Authorization: basic dGVzdHVzZXI6dGVzdDEyMzQ=' \
     -v 'https://example.org/rest/authenticate'

This request will return a JSON response body including a token that can be used on subsequent requests:

{"success":true,"token":"f3c0946fb05aae4ad50897e9060ab4e8"}

When using Apache there is a need to add the following to .htaccess

RewriteEngine On
RewriteCond %{HTTP:Authorization} ^(.*)
RewriteRule .* - [e=HTTP_AUTHORIZATION:%1]

Scheme: bearer (OAuth) 

For any other request, you must use bearer HTTP authentication scheme, supplying an authentication token, such as the one supplied by the basic authentication request mentioned above:

curl -XPOST \
     -H 'Authorization: bearer f3c0946fb05aae4ad50897e9060ab4e8' \
     -v 'https://example.org/rest/pages/testPage' \
     -d '{"data":{"title":"Test Name","pid":"siteRootPage"}}'

Multiple records in the same table 

You can insert or update multiple records within [table]. Your data array could look something like this:

{
   "Record-1": {
      "title": "My first record",
      "page": ["Page-916"]
   },
   "Record-2": {
      "title": "My second record",
      "page": ["Page-376"]
   }
}

Multiple records in multiple tables 

You can also leave out the table and insert or update multiple records within multiple tables:

"pages": {
   "Page-1": {
      "title": "My first page"
   },
   "Page-2": {
      "title": "My second page"
   }
},
"tt_content": {
   "Content-1": {
      "heading": "Welcome to the first page",
      "pid": "Page-1"
   },
   "Content-2": {
      "heading": "Welcome to the second page",
      "pid": "Page-2"
   }
}

Multilingual records 

It is even possible to insert records in multiple languages by adding a language layer to the data:

"pages": {
   "Page-1": {
      "en": {
         "title": "My first page"
      },
      "nb": {
         "title": "Min første side"
      }
   },
   "Page-2": {
      "en": {
         "title": "My second page"
      }
   },
},
"tt_content": {
   "Content-1": {
      "en": {
         "heading": "Welcome to the first page",
         "pid": "Page-1"
      },
      "nb" {
         "heading": "Velkommen til den første siden",
         "pid": "Page-1"
      }
   },
}

POST 

Create a record.

PUT 

Update a record.

PATCH 

Update a record if it exists, otherwise create it.

DELETE 

Delete a record.

curl -XDELETE \
     -H 'Authorization: bearer f3c0946fb05aae4ad50897e9060ab4e8' \
     -v 'https://example.org/rest/pages/testPage'

Specifying the HTTP request header 

Because webhook-based requests always use the POST method, it must be overridden in your data. This means

method refers to the HTTP request method that would otherwise have been used by the REST request. The available values are POST, PUT, PATCH, and DELETE. More information in the REST API section on HTTP Methods.

Example request data 

The request data can be formatted for both single and batch operations. A single-record request could look like this:

{
   "method": "PATCH",
   "table": "pages",
   "remoteId": "testPage",
   "data": {
      "title": "Test Name",
      "pid":"siteRootPage"
   }
}

A batch request could look like this:

{
   "method": "POST",
   "data": {
      "pages": {
         "Page-1": {
            "title": "My first page",
         },
         "Page-2": {
            "title": "My second page",
         }
      },
      "tt_content": {
         "Content-1": {
            "heading": "Welcome to the first page",
            "pid": "Page-1"
         },
         "Content-2": {
            "heading": "Welcome to the second page",
            "pid": "Page-2"
         }
      }
   }
}

You can read more about creating batch requests in the REST API section on Batch requests.

Base64-encoded file data 

With the extension's default configuration, this example will create the file at fileadmin/tx_interest/image.png.

{
  "sys_file": {
    "ExampleRemoteId": {
      "fileData": "iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8...",
      "name": "image.png"
    }
  }
}

The name field must be supplied and include a file extension.

Publicly accessible URL 

{
  "sys_file": {
    "ExampleRemoteId": {
      "url": "https://example.com/image.png",
      "name": "image.png"
    }
  }
}

{
  "sys_file": {
    "ExampleRemoteId": {
      "url": "https://youtu.be/zpOVYePk6mM",
    }
  }
}

Events 

The events are listed in order of execution.

class HttpRequestRouterHandleByEvent

Fully qualified name

\FriendsOfTYPO3\Interest\Router\Event\HttpRequestRouterHandleByEvent

Called in HttpRequestRouter::handleByMethod(). Can be used to modify the request and entry point parts before they are passed on to a RequestHandler.

EventHandlers for this event should implement \FriendsOfTYPO3\Interest\Router\Event\HttpRequestRouterHandleByEventHandlerInterface.

getEntryPointParts ( )

Returns an array of the entry point parts, i.e. the parts of the URL used to detect the correct entry point. Given the URL http://www.example.com/rest/tt_content/ContentRemoteId and the default entry point rest, the entry point parts will be ['tt_content', 'ContentRemoteId'].

returntype

array

setEntryPointParts ( $entryPointParts)

param array $entryPointParts

 

getRequest ( )

returntype

PsrHttpMessageServerRequestInterface

setRequest ( $request)

param Psr\Http\Message\ServerRequestInterface $request

 

class RecordOperationSetupEvent

Fully qualified name

\FriendsOfTYPO3\Interest\DataHandling\Operation\Event\RecordOperationSetupEvent

Called inside the AbstractRecordOperation::__construct() when a *RecordOperation object has been initialized, but before data validations.

EventHandlers for this event should implement \FriendsOfTYPO3\Interest\DataHandling\Operation\Event\RecordOperationEventHandlerInterface.

EventHandlers for this event can throw these exceptions:

\FriendsOfTYPO3\Interest\DataHandling\Operation\Event\Exception\StopRecordOperationException

To quietly stop the record operation. This exception is only logged as informational and the operation will be treated as successful. E.g. used when deferring an operation.

\FriendsOfTYPO3\Interest\DataHandling\Operation\Event\Exception\BeforeRecordOperationEventException

Will stop the record operation and log as an error. The operation will be treated as unsuccessful.

getRecordOperation ( )

returntype

FriendsOfTYPO3InterestDataHandlingOperationAbstractRecordOperation

class RecordOperationInvocationEvent

Fully qualified name

\FriendsOfTYPO3\Interest\DataHandling\Operation\Event\RecordOperationInvocationEvent

Called as the last thing inside the AbstractRecordOperation::__invoke() method, after all data persistence and pending relations have been resolved.

EventHandlers for this event should implement \FriendsOfTYPO3\Interest\DataHandling\Operation\Event\AfterRecordOperationEventHandlerInterface.

getRecordOperation ( )

returntype

FriendsOfTYPO3InterestDataHandlingOperationAbstractRecordOperation

class HttpResponseEvent

Fully qualified name

\FriendsOfTYPO3\Interest\Middleware\Event\HttpResponseEvent

Called in the middleware, just before control is handled back over to TYPO3 during an HTTP request. Allows modification of the response object.

EventHandlers for this event should implement \FriendsOfTYPO3\Interest\Middleware\Event\HttpResponseEventHandlerInterface.

getResponse ( )

returntype

PsrHttpMessageResponseInterface

setResponse ( $response)

param Psr\Http\Message\ResponseInterface $response

 

Internal representation and identity 

Inside the extension, a record's state and identity is maintained by two data transfer object classes:

• A record's unique identity from creation to deletion is represented by \FriendsOfTYPO3\Interest\Domain\Model\Dto\RecordInstanceIdentifier.
• A record's current state, including the data that should be written to the database is represented by \FriendsOfTYPO3\Interest\Domain\Model\Dto\RecordRepresentation.

When creating a RecordRepresentation, you must also supply a RecordInstanceIdentifier:

use FriendsOfTYPO3\Interest\Domain\Model\Dto\RecordInstanceIdentifier;
use FriendsOfTYPO3\Interest\Domain\Model\Dto\RecordRepresentation;

new RecordRepresentation(
    [
        'title' => 'My record title',
        'bodytext' => 'This is a story about ...',
    ],
    new RecordInstanceIdentifier(
        'tt_content',
        'ContentElementA',
        'en'
    )
);

Record operations 

Record operations are the core of the Interest extension. Each represents one operation requested from the outside. One record operation is not the same as one database operation. Some record operations will not be executed (if it is a duplicate of the previous operation on the same remote ID) or deferred (if the record operation requires a condition to be fulfilled before it can be executed).

The record operations are invokable, and are executed as such:

Mapping table 

The extension keeps track of the mapping between remote IDs and TYPO3 records in the table tx_interest_remote_id_mapping. In addition to mapping information, the table contains metadata about each record.

use FriendsOfTYPO3\Interest\Domain\Model\Dto\RecordInstanceIdentifier;
use FriendsOfTYPO3\Interest\Domain\Model\Dto\RecordRepresentation;
use FriendsOfTYPO3\Interest\Domain\Repository\RemoteIdMappingRepository;
use FriendsOfTYPO3\Interest\DataHandling\Operation\DeleteRecordOperation;
use TYPO3\CMS\Core\Utility\GeneralUtility;

$mappingRepository = GeneralUtility::makeInstance(RemoteIdMappingRepository::class);

foreach($mappingRepository->findAllUntouchedSince(time() - 86400) as $remoteId) {
    (new DeleteRecordOperation(
        new RecordRepresentation(
            [],
            new RecordInstanceIdentifier('table', $remoteId)
        );
    ))();
}

use GuzzleHttp\Client;
use FriendsOfTYPO3\Interest\Domain\Repository\RemoteIdMappingRepository;

$mappingRepository = GeneralUtility::makeInstance(RemoteIdMappingRepository::class);

$metaData = $mappingRepository->getMetaDataValue(
    $remoteId,
    self::class
) ?? [];

$headers = [
    'If-Modified-Since' => $metaData['date'],
    'If-None-Match'] => $metaData['etag'],
];

$response = GeneralUtility::makeInstance(Client::class)
    ->get($url, ['headers' => $headers]);

if ($response->getStatusCode() === 304) {
    return null;
}

$mappingRepository->setMetaDataValue(
    $remoteId,
    self::class,
    [
        'date' => $response->getHeader('Date'),
        'etag' => $response->getHeader('ETag'),
    ]
);

return $response->getBody()->getContents();