TYPO3 Core APIs

Previous Key:doc_core_api
Version:latest (9-dev)
Language:en
Description:Reference to the Core APIs of TYPO3, e.g. main classes, Extension API, RTE API.
Keywords:tsref, typoscript, reference, forDevelopers, forAdvanced
Copyright:2000-2017
Author:Documentation Team
Email:documentation@typo3.org
License:Open Publication License available from www.opencontent.org/openpub/
Rendered:2018-04-19 20:37

The content of this document is related to TYPO3 CMS, a GNU/GPL CMS/Framework available from www.typo3.org

Official Documentation

This document is included as part of the official TYPO3 documentation. It has been approved by the TYPO3 Documentation Team following a peer- review process. The reader should expect the information in this document to be accurate - please report discrepancies to the Documentation Team (documentation@typo3.org). Official documents are kept up-to-date to the best of the Documentation Team's abilities.

Core Manual

This document is a Core Manual. Core Manuals address the built in functionality of TYPO3 CMS and are designed to provide the reader with in- depth information. Each Core Manual addresses a particular process or function and how it is implemented within the TYPO3 source code. These may include information on available APIs, specific configuration options, etc.

Core Manuals are written as reference manuals. The reader should rely on the Table of Contents to identify what particular section will best address the task at hand.

Sitemap

Introduction

Overview

TYPO3 is known for its extensibility. To really benefit from this power, a complete documentation is needed: "Core APIs" and its companion, "Inside TYPO3", aim to provide such information to developers and administrators. Not all areas are covered with the same amount of detail, but at least some pointers are provided.

"Inside TYPO3" contains the overall introduction to the architecture of the TYPO3 core. It also contains API descriptions to a certain degree but mostly in the form of examples and short table listings. "Core APIs" goes into much more detail about such APIs and covers subjects more closely related to development.

These documents do not contain any significant information about the frontend of TYPO3. Creating templates, setting up TypoScript objects etc. is not the scope of these documents; they address the backend part of the core only.

The TYPO3 Documentation Team hopes that these two documents, "Inside TYPO3" and "TYPO3 Core APIs", will form a complete picture of the TYPO3 Core architecture and the backend. They will hopefully be the reference of choice in your work with TYPO3. It took Kasper more than a year to get the first version published and we've tried to maintain it as best as we could.

Code examples

Many of the code examples found in this document come from the TYPO3 Core itself.

Quite a few others come from the "examples" extension which is available in the TER. You can install it if you want to try out these examples yourself and use them as a basis for your own stuff.

Yet some other examples just belong to this manual. Some may be moved to the "examples" extension at some later stage.

Feedback

For general questions about the documentation get in touch by writing to documentation@typo3.org .

If you find a bug in this manual, please be so kind as to check the online version on https://docs.typo3.org/typo3cms/CoreApiReference/. From there you can hit the "Edit me on GitHub" button in the top right corner and submit a pull request via GitHub. Alternatively you can just file an issue using the bug tracker: https://github.com/TYPO3-Documentation/TYPO3CMS-Reference-CoreApi/issues.

Maintaining high quality documentation requires time and effort and the TYPO3 Documentation Team always appreciates support. If you want to support us, please join the documentation mailing list/forum (http://forum.typo3.org/index.php/f/44/).

Credits

This manual was originally written by Kasper Skårhøj. It was further maintained, refreshed and expanded by François Suter.

Dedication

I want to dedicate this document to the people in the TYPO3 community who have the discipline to do the boring job of writing documentation for their extensions or contribute to the TYPO3 documentation in general. It's great to have good coders, but it's even more important to have coders with character to carry their work through till the end - even when it means spending days writing good documents. Go for completeness!

- kasper

Extension Architecture

Introduction

TYPO3 can be extended in nearly any direction without loosing backwards compatibility. The Extension API provides a powerful framework for easily adding, removing, installing and developing such extensions to TYPO3. This is in particular powered by the Extension Manager (EM) inside TYPO3 and the online TYPO3 Extension Repository (TER) found at typo3.org for easy sharing of extensions.

"Extensions" is a general term in TYPO3 which covers many kinds of additions to TYPO3. The main types are:

  • Plugins which play a role on the website itself, e.g. a discussion board, guestbook, shop, etc. Therefore plugins are content elements, that can be placed on a page like a text element or an image.
  • Modules are backend applications which have their own entry in the main menu. They require a backend login and work inside the framework of the backend. We might also call something a module if it exploits any connectivity of an existing module, that is if it simply adds itself to the function menu of existing modules. A module is an extension in the backend.
  • Services are libraries that provide a given service through a clearly defined API. A service may exist both in the frontend and the backend. Please refer to the Services Section for more information about this type of extension.
  • Distributions are fully packaged TYPO3 CMS web installations, complete with files, templates, extensions, etc. Distributions are covered in their own chapter.

Extensions and the Core

Extensions are designed in a way so that extensions can supplement the core seamlessly. This means that a TYPO3 system will appear as "a whole" while actually being composed of the core application and a set of extensions providing various features. This philosophy allows TYPO3 to be developed by many individuals without loosing fine control since each developer will have a special area (typically a system extension) of responsibility which is effectively encapsulated.

So, at one end of the spectrum system extensions make up what is known as "TYPO3" to the outside world. At the other end, extensions can be entirely specific to a given project and contain only files and functionality related to a single implementation.

Files and locations

Files

An extension consists of:

  1. A directory named by the extension key (which is a worldwide unique identification string for the extension), usually located in typo3conf/ext for local extensions, or typo3/sysext for system extensions.
  2. Standard files with reserved names for configuration related to TYPO3 (of which most are optional, see list below)
  3. Any number of additional files for the extension functionality itself.

Reserved file names

This lists special files within an extension that have a special meaning by convention. If put at the according places, TYPO3 will find them and use for specific functionality. For example, if a svg logo of your extension is placed at Resources/Public/Icons/Extension.svg, the extension manager will show that image.

Nearly none of these are required, but for example you can not have a TYPO3 extension recognized by TYPO3 without the ext_emconf.php file etc. You can read more details like that in the table below.

In general, do not introduce your own files in the root directory of extensions with the name prefix ext_.

Filename Description
ext_emconf.php

Definition of extension properties. This is the only mandatory file in the extension. It describes the extension for the rest of TYPO3.

Name, category, status etc. Used by the extension manager. The content of this file is described in more details below. Note that it is auto-written by extension manager when extensions are imported from the repository.

Note

If this file is not present, the extension manager will not find the extension.

ext_localconf.php

Addition to LocalConfiguration.php which is included if found. Should contain additional configuration of $GLOBALS['TYPO3_CONF_VARS'].

This file contains hook definitions and plugin configuration. It must not contain a PHP encoding declaration.

All ext_localconf.php files of loaded extensions are included right after the files typo3conf/LocalConfiguration.php and typo3conf/AdditionalConfiguration.php during TYPO3 bootstrap.

Pay attention to the rules for the contents of these files. For more details, see the section below.

ext_tables.php

Included if found. Contains extensions of existing tables, declaration of backend modules, etc. All code in such files is included after all the default definitions provided by the Core and loaded after ext_localconf.php files during TYPO3 bootstrap.

Pay attention to the rules for the contents of these files. For more details, see the section below.

Note

In old TYPO3 core versions, this file contained additions to the global $GLOBALS['TCA'] array. This changed since core version 6.2 to allow effective caching:

TCA definition of new database tables must be done entirely in Configuration/TCA/<table name>.php. These files are expected to contain the full TCA of the given table (as an array) and simply return it (with a return statement).

Customizations of existing tables must be done entirely in Configuration/TCA/Overrides/<table name>.php.

ext_tables.sql

SQL definition of database tables.

This file should contain a table-structure dump of the tables used by the extension. It is used for evaluation of the database structure and is therefore important to check and update the database when an extension is enabled.

If you add additional fields (or depend on certain fields) to existing tables you can also put them here. In that case insert a CREATE TABLE structure for that table, but remove all lines except the ones defining the fields you need, here is an example adding a column to the pages table:

CREATE TABLE pages (
    tx_myext_field int(11) DEFAULT '0' NOT NULL,
);

TYPO3 will merge this table definition to the existing table definition when comparing expected and actual table definitions. Partial definitions can also contain indexes and other directives. They can also change existing table fields though that is not recommended, because it may create problems with the TYPO3 core and/or other extensions.

The ext_tables.sql file may not necessarily be "dumpable" directly to MySQL (because of the semi-complete table definitions allowed defining only required fields). But the extension manager or install tool can handle this. The only very important thing is that the syntax of the content is exactly like MySQL made it so that the parsing and analysis of the file is done correctly by the extension manager.

TYPO3 parses ext_tables.sql files. TYPO3 expects that all table definitions in this file look like the ones produced by the mysqldump utility. Incorrect definitions may not be recognized by the TYPO3 SQL parser or may lead to MySQL errors, when TYPO3 tries to apply them. If TYPO3 is not running on MySQL or directly compatible other DBMS like MariaDB, the system will parse the file towards the target DBMS like PostgreSQL.

ext_tables_static+adt.sql

Static SQL tables and their data.

If the extension requires static data you can dump it into a sql-file by this name. Example for dumping mysql data from bash (being in the extension directory):

mysqldump --add-drop-table \
          --password=[password] [database name] \
          [tablename]  > ./ext_tables_static.sql

--add-drop-table will make sure to include a DROP TABLE statement so any data is inserted in a fresh table.

You can also drop the table content using the extension manager in the backend.

Note

The table structure of static tables needs to be in the ext_tables.sql file as well - otherwise an installed static table will be reported as being in excess in the install tool.

Warning

Static data is not meant to be extended by other extensions. On re-import all extended fields and data is lost due to DROP TABLE statements.

ext_typoscript_constants.typoscript

Preset TypoScript constants. Will be included in the constants section of all TypoScript templates.

Warning

Use such a file if you absolutely need to load some TS (because you would get serious errors without it). Otherwise static templates or usage of the Extension Management API of class TYPO3\CMS\Core\Utility\ExtensionManagementUtility are preferred.

ext_typoscript_setup.typoscript

Preset TypoScript setup. Will be included in the setup section of all TypoScript templates.

Warning

Use such a file if you absolutely need to load some TS (because you would get serious errors without it). Otherwise static templates or usage of the Extension Management API of class TYPO3\CMS\Core\Utility\ExtensionManagementUtility are preferred.

ext_conf_template.txt

Extension Configuration template.

Configuration code in TypoScript syntax setting up a series of values which can be configured for the extension in the install tool. Read more about the file format here.

If this file is present 'Settings' of the install tool provides you with an interface for editing the configuration values defined in the file. The result is written as an array to LocalConfiguration.php in the variable $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS'][*extension_key* ]

Configuration/Backend/Routes.php and Configuration/Backend/AjaxRoutes.php Registry of backend routes. Extension that add backend modules must register their routes here to be correctly linkable in the backend. The file must return an array with routing details. See core extensions like backend for examples.
Resources/Public/Icons/Extension.svg

Extension icon. If exists, this icon is displayed in the extension manager. Preferred is using an SVG file, Extension icon will look nicer when provided as vector graphics (SVG) rather than bitmaps (GIF or PNG).

18x16 GIF, PNG or SVG icon for the extension.

class.ext_update.php

Local Update tool class

If this file is found it will install a new menu item, "UPDATE", in the extension manager when looking at details for the extension. When this menu item is selected the class inside of this file (named ext_update) will be instantiated and the method "main()" will be called and expected to return HTML content.

Also you must add the function "access()" and make it return a boolean value whether or not the menu item should be shown. This feature is meant to let you disable the update tool if you can somehow detect that it has already been run and doesn't need to run again. The point of this file is to give extension developers the possibility to provide an update tool if their extensions in newer versions require some updates to be done.

Reserved folders

In the early days, every extension baked it own bread when it came to file locations of PHP classes, public web resources and templates.

With the rise of extbase, a generally accepted structure for file locations inside extensions has been established. If extension authors stick to this, the system helps in various ways. For instance, if putting PHP classes into the Classes/ folder and naming classes accordingly, the system will be able to autoload these without further action from the developer.

Extension kickstarters like the Extension Builder extension will create the correct structure for you.

It is described below:

Classes
Contains all PHP classes. One class per file. Should have sub folders like Controller/, Domain/, Service/ or View/. For more details on class file namings an PHP namespaces, see chapter namespaces.
Classes/Controller
Contains MVC Controller classes.
Classes/Domain/Model
Contains MVC Domain model classes.
Classes/Domain/Repository
Contains data repository classes.
Classes/ViewHelpers
Helper classes used in (Fluid) views.
Configuration
General configuration folder. Some of the sub directories in here like TCA and Backend have special meaning and files in there are automatically included during TYPO3 bootstrap.
Configuration/Backend/
Contains backend routing configurations. See files description of Routes.php and AjaxRoutes.php above.
Configuration/TCA
One file per database table, using the name of the table for the file, plus ".php". Only for new tables.
Configuration/TCA/Overrides
For extending existing tables, one file per database table, using the name of the table for the file, plus ".php".
Configuration/TsConfig/Page
Page TSconfig, see chapter 'Page TSconfig' in the TSconfig Reference. Files should have the file extension .tsconfig.
Configuration/TsConfig/User
User TSconfig, see chapter 'User TSconfig' in the TSconfig Reference. Files should have the file extension .tsconfig.
Configuration/TypoScript
TypoScript static setup (setup.typoscript) and constants (constants.typoscript). Use subfolders if your have several static templates.
Documentation
Contains the extension documentation in ReStructuredText (ReST, .rst) format. Read more on the topic in chapter extension documentation. Documentation/ and its subfolders may contain several ReST files, images and other resources.
Documentation/Index.rst
This file contains the cover page of the extension manual in ReST format. The name or format of the file may not be changed. You may include other ReST files as you like. See the "Extension Template" on docs.typo3.org for more information about structure and syntax of extension manuals.
Resources
Contains the subfolders Public/ and Private/, which contain resources, possibly in further subfolders, e.g. Templates/, CSS/, Language/, Images/ or JavaScript/. This is also the directory for non–TYPO3 files supplied with the extension. TYPO3 is licensed under GPL version 2 or any later version. Any non–TYPO3 code must be compatible with GPL version 2 or any later version.
Resources/Private/Language
XLIFF files for localized labels.
Resources/Private/Layouts
Main layouts for (Fluid) views.
Resources/Private/Partials
Partial templates for repetitive use.
Resources/Private/Templates
One template per action, stored in a folder named after each Controller.
Resources/Public/Css
Any CSS file used by the extension.
Resources/Public/Images
Any images used by the extension.
Resources/Public/JavaScript
Any JS file used by the extension.
Tests/Unit
Contains unit tests and fixtures.
Tests/Functional
Contains functional tests and fixtures.

System, Global and Local extensions

The files for an extension are located in a folder named by the extension key . The location of this folder can be either inside typo3/sysext/, typo3/ext/ or typo3conf/ext/.

The extension must be programmed so that it does automatically detect where it is located and can work from all three locations. If it is not possible to make the extension that flexible, it is possible to lock its installation requirement to one of these locations in the ext_emconf.php file (see "lockType").

Local extensions

Local extensions are located in the typo3conf/ext/ directory.

This is where to put extensions which are local for a particular TYPO3 installation. The typo3conf directory is always local, containing local configuration (e.g. LocalConfiguration.php), local modules etc. If you put an extension here it will be available for a single TYPO3 installation only. This is a "per-database" way to install an extension.

Note

Local extension can successfully be symlinked to other local extensions on a server as long as they are running under the same TYPO3 source version (which would typically also be symlinked). This method is useful for maintenance of the same local extension running under several sites on a server.

Global extensions

Global extensions are located in the typo3/ext/ directory.

This is a "per-server" way to install an extension; they are global for the TYPO3 source code on the web server. These extensions will be available for any TYPO3 installation sharing the source code.

Note

These features have not been consistently supported in recent versions of TYPO3, so you may encounter problems when using it.

System extensions

System extensions are located in the typo3/sysext/ directory.

This is system default extensions which cannot and should not be updated by the EM. They are distributed with TYPO3 core source code and generally understood to be a part of the core system.

Loading precedence

Local extensions take precedence which means that if an extension exists both in typo3conf/ext/ and typo3/ext/ the one in typo3conf/ext/ is loaded. Likewise global extension takes precedence over system extensions. This means that extensions are loaded in the order of priority local-global-system.

In effect you can therefore have - say - a "stable" version of an extension installed in the global dir (typo3/ext/) which is used by all your projects on a server sharing source code, but on a single experimental project you can import the same extension in a newer "experimental" version and for that particular project the locally available extension will be used instead.

Choosing an extension key

The "extension key" is a string uniquely identifying the extension. The folder where the extension resides is named by this string. The string can contain characters a-z0-9 and underscore. No uppercase characters should be used (keeps folder-,file- and table/field-names in lowercase). Furthermore the name must not start with an "tx" or "u" (this is prefixes used for modules) and because backend modules related to the extension should be named by the extension name without underscores, the extension name must still be unique even if underscores are removed (underscores are allowed to make the extension key easily readable).

The naming conventions of extension keys are automatically validated by the registration at the repository, so you have nothing to worry about here.

There are two ways to name an extension:

  • Project specific extensions (not generally usable or shareable): Select any name you like and prepend it "user_" (which is the only allowed use of a key starting with "u"). This prefix denotes that this extension is a local one which does not come from the central TYPO3 Extension Repository or is ever intended to be shared. Probably this is an "adhoc" extension you have made for some special occasion.
  • General extensions: Register an extension name online at the TYPO3 Extension Repository. Your extension name will automatically be validated and you are sure to have a unique name returned which nobody else in the world uses. This makes it very easy to share your extension later on with every one else, because it ensures that no conflicts with other extension will happen. But by default a new extension you make is defined "private" which means nobody else but you have access to it until you permit it to be public. It's free of charge to register an extension name. By definition all code in the TYPO3 Extension Repository is covered by the GPL license because it interfaces with TYPO3. You should really consider making general extensions!

Tip

It is far easier to settle for the right extension key from the beginning. Changing it later involves a cascade of name changes to tables, modules, configuration files etc. Think carefully.

About GPL and extensions

Remember that TYPO3 is GPL software and at the same moment you extend TYPO3 your extensions are legally covered by GPL. This does not force you to share your extension, but it should inspire you to do so and legally you cannot prevent anyone who gets hold of your extension code from using it and further develop it. The TYPO3 Extension API is designed to make sharing of your work easy as well as using others work easy. Remember TYPO3 is Open Source Software and we rely on each other in the community to develop it further.

Important

It's also your responsibility to make sure that all content of your extensions is legally covered by GPL. The webmaster of TYPO3.org reserves the right to kick out any extension without notice that is reported to contain non-GPL material.

Security

You are responsible for security issues in your extensions. People may report security issues either directly to you or to the TYPO3 Security Team. Whatever the case you should get in touch with the Security Team which will validate the security fixes. They will also include information about your (fixed) extension in their next Security bulletin. If you don't respond to requests from the Security Team, your extension will be forcibly removed from the TYPO3 Extension Repository.

More details on the security team's policy on handling security issues can be found at http://typo3.org/teams/security/extension-security-policy/.

Registering an extension key

Before starting a new extension you should register an extension key on typo3.org (unless you plan to make an implementation-specific extension – of course – which it does not make sense to share).

Go to typo3.org, log in with your (pre-created) username / password and go to Extensions > Extension Keys and click on the "Register keys" tab. On that page you can enter the key name you want to register.

The extension registration form

The extension registration form on typo3.org.

Naming conventions

Based on the extension key of an extension these naming conventions should be followed:

Attention

((The following table is unreadable and has been translated to the following normal text. The table will be dropped soon.))

  General Example User-specific Example

Extension key

(Lowercase "alnum" + underscores. )

Assigned by the TYPO3 Extension Repository. cool_shop Determined by yourself, but prefixed "user_" user_my_shop
Database tables and fields Prefix with "tx_[ key ]_" where key is without underscores!

Prefix: tx_coolshop_

Examples:

tx_coolshop_products

tx_coolshop_categories

Prefix with "[ key ]_"

Prefix: user_my_shop_

Examples:

user_my_shop_products

user_my_shop_categories

Backend module

(Names are always without underscores!)

Name: The extension key name without underscores, prefixed "tx" txcoolshop Name: No underscores, prefixed "u" uMyShop or umyshop or ...
Abbreviations
TER = TYPO3 extension repository
extkey = extension key
modkey = backend module key
Public extensions
  1. Public extensions are available from the TER or via Packagist. Private extensions are not published to the TER or Packagist.

  2. The extkey is made up of alphanumeric characters and underscores only and should start with a letter.

    Example: cool_shop

  3. The extkey is valid if the TER accepts it. This makes sure that the name follows the rules and is unique.

  4. Database tablenames look like tx_ + extkey (without underscores) + _specification.

    Examples: tx_coolshop_products, tx_coolshop_categories, tx_coolshop_more_categories, tx_coolshop_domain_model_tag.

Backend modules
  1. The modkey is made up of alphanumeric characters only. It does not contain underscores and starts with a letter.

    Example: coolshop

Frontend PHP classes
For frontend PHP classes, follow the same conventions as for database tables and fields.

You may also want to refer to the TYPO3 Core Coding Guidelines for more on general naming conventions in TYPO3.

Tip

If you study the naming conventions above closely you will find that they are complicated due to varying rules for underscores in key names. Sometimes the underscores are stripped off, sometimes not.

The best practice you can follow is to avoid using underscores in your extensions keys at all! That will make the rules simpler. This is highly encouraged.

Note on "old" extensions:

Some the "classic" extensions from before the extension structure came about do not comply with these naming conventions. That is an exception made for backwards compatibility. The assignment of new keys from the TYPO3 Extension Repository will make sure that any of these old names are not accidentally reassigned to new extensions.

Further, some of the classic plugins (tt_board, tt_guest etc) use the "user_" prefix for their classes as well.

Extending "extensions classes"

As a standard procedure you should include the "class extension code" even in your own extensions. This is placed at the bottom of every class file:

if (defined('TYPO3_MODE') && isset($GLOBALS['TYPO3_CONF_VARS'][TYPO3_MODE]['XCLASS']['ext/myext/pi1/class.tx_myext_pi1.php'])) {
        include_once($GLOBALS['TYPO3_CONF_VARS'][TYPO3_MODE]['XCLASS']['ext/myext/pi1/class.tx_myext_pi1.php']);
}

Normally the key used as example here ("ext/myext/pi1/class.tx_myext_pi1.php") would be the full path to the script relative to the PATH_site constant. However because modules are required to work from both typo3/sysext/, typo3/ext/ and typo3conf/ext/ it is a policy that any path before "ext/" is omitted.

Installing extensions

There are only two (possibly three) steps involved in using extensions with TYPO3:

  1. You must import it.

    This simply means to copy the extensions files into the correct directory in either typo3/ext/ (global) or typo3conf/ext/ (local). More commonly you import an extension directly from the online TYPO3 Extension Repository (TER). When an extension is found located in one of the extension locations, it is available to the system.

    The Extension Manager (EM) should take care of this process, including updates to newer versions if needed. Notice that backend modules will have their "conf.php" file modified in the install process depending on whether they are installed locally or globally!

    Another convenient way to install extensions is offered by using composer (https://getcomposer.org/). Besides TYPO3 CMS itself the TYPO3 composer repository includes all TYPO3 Extensions that are uploaded to TER. Read more on https://composer.typo3.org/ .

  2. You must install it.

    An extension is loaded only if its state is set to active in the PackageStates.php file. Extensions are loaded in the order they appear in this list.

    An enabled extension is always global to the TYPO3 Installation - you cannot disable an extension from being loaded in a particular branch of the page tree. The EM takes care enabling extensions. It's highly recommended that the EM is doing this, because the EM will make sure the priorities, dependencies and conflicts are managed according to the extension characteristics, including clearing of the cache-files if any.

  3. You might need to configure it.

    Certain extensions may allow you to configure some settings. Again the EM is able to handle the configuration of the extensions based on a certain API for this. Any settings - if present - configured for an extension are available as an array in the variable $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS'][extensionKey].

Loaded extensions are registered in a global variable, $TYPO3_LOADED_EXT, available in both frontend and backend of TYPO3.

This is how the data structure for an extension in this array looks:

$TYPO3_LOADED_EXT[extension key] = array(
        "type" =>                S, G, L for system, global or local type of availability.
        "siteRelPath" => Path of extension dir relative to the PATH_site constant
                                e.g. "typo3/ext/my_ext/" or "typo3conf/ext/my_ext/"
        "typo3RelPath" => Path of extension dir relative to the "typo3/" admin folder
                                e.g. "ext/my_ext/" or "../typo3conf/ext/my_ext/"
        "ext_localconf" => Contains absolute path to 'ext_localconf.php' file if present
        "ext_tables" => [same]
        "ext_tables_sql" => [same]
        "ext_tables_static+adt.sql" => [same]
        "ext_typoscript_constants.txt" => [same]
        "ext_typoscript_setup.txt" => [same]
        "ext_typoscript_editorcfg.txt" => [same]
)

The order of the registered extensions in this array corresponds to the order they were listed in PackageStates.php.

The inclusion of ext_tables.php or ext_localconf.php files (see next chapter) is done by traversing (a copy of) the $TYPO3_LOADED_EXT array.

Declaration file

The ext_emconf.php is the single most important file in an extension. Without it, the Extension Manager (EM) will not detect the extension, much less be able to install it. This file contains a declaration of what the extension is or does for the EM. The only thing included is an associative array, $EM_CONF[extension key]. The keys are described in the table below.

When extensions are imported from the online repository this file is written anew! So don't put any custom stuff in there - only change values in the $EM_CONF array if needed.

Key Data type Description
title string, required The name of the extension in English.
description string, required Short and precise description in English of what the extension does and for whom it might be useful.
version string Version of the extension. Automatically managed by EM / TER. Format is [int].[int].[int]
category string

Which category the extension belongs to:

  • be

    Backend (Generally backend-oriented, but not a module)

  • module

    Backend modules (When something is a module or connects with one)

  • fe

    Frontend (Generally frontend oriented, but not a "true" plugin)

  • plugin

    Frontend plugins (Plugins inserted as a "Insert Plugin" content element)

  • misc

    Miscellaneous stuff (Where not easily placed elsewhere)

  • services

    Contains TYPO3 services

  • templates

    Contains website templates

  • example

    Example extension (Which serves as examples etc.)

  • doc

    Documentation (e.g. tutorials, FAQ's etc.)

  • distribution

    Distribution, an extension kickstarting a full site

constraints array

List of requirements, suggestions or conflicts with other extensions or TYPO3 or PHP version. Here's how a typical setup might look:

'constraints' => array(
    'depends' => array(
        'typo3' => '4.5.0-6.1.99',
        'php' => '5.3.0-5.5.99'
    ),
    'conflicts' => array(
        'dam' => ''
    ),
    'suggests' => array(
        'tt_news' => '2.5.0-0.0.0'
    )
)
depends
List of extensions that this extension depends on. Extensions defined here will be loaded before the current extension.
conflicts
List of extensions which will not work with this extension.
suggests

List of suggestions of extensions that work together or enhance this extension. Extensions defined here will be loaded before the current extension. Dependencies take precedence over suggestions.

Note: If a "suggested" extension depends on the current extension (directly or indirectly), the suggestion is not taken into account for loading order calculation. Read more at Forge #57825.

The above example indicated that the extension depends on a version of TYPO3 between 4.5 and 6.1 (as only bug and security fixes are integrated into TYPO3 when the last digit of the version changes, it is safe to assume it will be compatible with any upcoming version of the corresponding branch, thus .99). Also the extension has been tested and is known to work properly with PHP 5.3, 5.4 and 5.5. It will conflict with the DAM (any version) and it is suggested that it might be worth installing "tt_news" (version at least 2.5.0).

state string

Which state is the extension in

  • alpha

    Alpha state is used for very initial work, basically the state is has during the very process of creating its foundation.

  • beta

    Under current development. Beta extensions are functional but not complete in functionality. Most likely beta-extensions will not be reviewed.

  • stable

    Stable extensions are complete, mature and ready for production environment. You will be approached for a review. Authors of stable extensions carry a responsibility to be maintain and improve them.

  • experimental

    Experimental state is useful for anything experimental - of course. Nobody knows if this is going anywhere yet... Maybe still just an idea.

  • test

    Test extension, demonstrates concepts etc.

  • obsolete

    The extension is obsolete or deprecated. This can be due to other extensions solving the same problem but in a better way or if the extension is not being maintained anymore.

  • excludeFromUpdates

    This state makes it impossible to update the extension through the extension manager (neither by the Update mechanism, nor by uploading a newer version to the installation). This is very useful if you made local changes to an extension for a specific installation and don't want any admin to overwrite them.

    New since TYPO3 4.3.

uploadfolder boolean If set, then the folder named "uploads/tx_[extKey-with-no- underscore]" should be present!
createDirs list of strings Comma list of directories to create upon extension installation.
clearCacheOnLoad boolean If set, the EM will request the cache to be cleared when this extension is loaded.
author string Author name
author_email email address Author email address
author_company string Author company
autoload array

To get better class loading support for websites in non-composer mode+ the following information can be provided.

Extensions having one folder with classes or single files

Considering you have an Extbase extension (or an extension where all classes and interfaces reside in a Classes folder) or single classes you can simply add the following to your ext_emconf.php file:

'autoload' => [
   'classmap' => [
      'Classes',
      'a-class.php',
   ]
],

Extensions using namespaces

If the extension has namespaced classes following the PSR-4 standard, then you can add the following to your ext_emconf.php file:

  'autoload' => [
     'psr-4' => [
        'Vendor\\ExtName\\' => 'Classes'
     ]
  ],

Important: The prefix **must** end with a backslash.
autoload-dev array Same as the configuration "autoload" but it is only used if the ApplicationContext is set to Testing.

Deprecated configuration

The following fields are deprecated and should not be used anymore:

  • dependencies
  • conflicts
  • suggests
  • docPath
  • CGLcompliance
  • CGLcompliance_note
  • private
  • download_password
  • shy
  • loadOrder
  • priority
  • internal
  • modify_tables
  • module
  • lockType
  • TYPO3_version
  • PHP_version

Configuration files

Files ext_tables.php and ext_localconf.php are the two most important files for the execution of extensions within TYPO3. They contain configuration used by the system on almost every request. They should therefore be optimized for speed.

  • ext_localconf.php is always included in global scope of the script, either frontend or backend.

    While you can put functions and classes into the script, it is a really bad practice because such classes and functions would always be loaded. It is better to have them included only as needed.

    These are the typical functions that extension authors should place within ext_localconf.php

    • Registering hooks or any simple array assignments to $TYPO3_CONF_VARS options
    • Registering additional Request Handlers within the Bootstrap
    • Adding any PageTSconfig or Default TypoScript via ExtensionManagementUtility APIs
    • Registering Extbase Command Controllers
    • Registering Scheduler Tasks
    • Adding reports to the reports module
    • Adding slots to signals via Extbase's SignalSlotDispatcher
    • Registering Icons to the IconRegistry
    • Registering Services via the Service API
  • ext_tables.php is not always included in the global scope of the frontend context.

    This file is only included when * a TYPO3 Backend or CLI request is happening * or the TYPO3 Frontend is called and a valid Backend User is authenticated

    This file gets usually included later within the request and after TCA information is loaded, and a Backend User is authenticated as well.

    These are the typical functions that should be placed inside ext_tables.php

    • Registering of Backend modules or Backend module functions
    • Adding Context-Sensitive-Help docs via ExtensionManagementUtility API
    • Adding TCA descriptions (via ExtensionManagementUtility::addLLrefForTCAdescr())
    • Adding table options via ExtensionManagementUtility::allowTableOnStandardPages
    • Assignments to the global configuration arrays $TBE_STYLES and $PAGES_TYPES
    • Adding new fields to User Settings ("Setup" Extension)

    Additionally, it is possible to extend TYPO3 in a lot of different ways (adding TCA, Backend Routes, Symfony Console Commands etc) which do not need to touch these files.

    It is heavily recommended to AVOID any checks on TYPO3_MODE or TYPO3_REQUESTTYPE constants (e.g. if(TYPO3_MODE === 'BE')) within these files as it limits the functionality to cache the whole systems' configuration. Any extension author should remove the checks if not explicitly necessary, and re-evaluate if these context-depending checks could go inside the hooks / caller function directly.

    Additionally, it is recommend to use the extension name (e.g. "tt_address") instead of $_EXTKEY within the two configuration files as this variable will be removed in the future. This also applies to $_EXTCONF.

    However, due to limitations to TER, the $_EXTKEY option should be kept within an extensions ext_emconf.php.

    See any system extension for best practice on this behaviour.

  • $TYPO3_LOADED_EXT[extensionKey] contains information about whether the module is loaded as local, global or system type, including the proper paths you might use, absolute and relative.

  • Your ext_tables.php and ext_localconf.php file must be designed so that they can safely be read and subsequently imploded into one single file with all the other configuration scripts!

  • You must never use a "return" statement in the files global scope - that would make the cached script concept break.

  • You must never use a "use" statement in the files global scope - that would make the cached script concept break and could conflict with other extensions.

  • You should not rely on the PHP constant __FILE__ for detection of include path of the script - the configuration might be executed from a cached script and therefore such information should be derived from e.g. \TYPO3\CMS\Core\Utility\GeneralUtility::getFileAbsFileName() or ExtensionManagementUtility::extPath().

Best practice for ext_tables.php and ext_localconf.php

It is a good practice to use call_user_func with an closure function. The following example contains the complete code:

<?php
defined('TYPO3_MODE') or die();

call_user_func(function () {
    // Add your code here
});

In most cases, the file ext_tables.php is no longer needed, since most of the code can be placed in ConfigurationTCA*.php files.

Configuration options

In the ext_conf_template.txt file configuration options for an extension can be defined. They will be accessible in the TYPO3 backend from the Extension Manager.

There's a specific syntax to declare these options properly, which is similar to the one used for TypoScript constants (see "Declaring constants for the Constant editor" in "TypoScript Syntax and In-depth Study"). This syntax applies to the comment line that should be placed just before the constant. Consider the following example (taken from system extension "rsaauth"):

# cat=basic/enable; type=string; label=Path to the temporary directory:This directory will contain...
temporaryDirectory =

First a category (cat) is defined ("basic") with the subcategory "enable". Then a type is given ("string") and finally a label, which is itself split (on the colon ":") into a title and a description (this should actually be a localized string). The above example will be rendered like this in the EM:

Configuration screen for the rsaauth extension

The configuration tab displays all options from a single category. A selector is available to switch between categories. Inside an option screen, options are grouped by subcategory. At the bottom of the screenshot, the label – split between header and description – is visible. Then comes the field itself, in this case an input, because the option's type is "string".

Available option types:

Option type Description
boolean checkbox
color colorpicker
int integer value
int+ positive integer value
integer integer value
offset offset
options option select
small small text field
string text field
user user function
wrap wrap field

Once you saved the configuration in the ExtensionManager, it will be stored in $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS']['your_extension_key'] as an array.

To retrieve the configuration use the API:

$backendConfiguration = (bool)\TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(
   \TYPO3\CMS\Core\Configuration\ExtensionConfiguration::class)
   ->get('your_extension_key');

To fetch the value of temporaryDirectory from the example above, you could simply use:

$backendConfiguration = (bool)\TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(
   \TYPO3\CMS\Core\Configuration\ExtensionConfiguration::class)
   ->get('your_extension_key', 'temporaryDirectory');

You can also define nested options using the TypoScript notation:

directories {
   # cat=basic/enable; type=string; label=Path to the temporary directory
   tmp =
   # cat=basic/enable; type=string; label=Path to the cache directory
   cache =
}

This will result in a multidimensional array:

$extensionConfiguration = $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS']['your_extension_key'];
$extensionConfiguration['directories.']['tmp']
$extensionConfiguration['directories.']['cache']

Important

Notice the dot at the end of the directories key. This notation must be used for every grouping key and is a convention of the TypoScript parser.

Extending the $TCA array

Being a PHP array, the Table Configuration Array can be easily extended. It can be accessed as the global variable $GLOBALS['TCA']. TYPO3 also provides APIs for making this simpler.

Storing the changes

There are various ways to store changes to $GLOBALS['TCA']. They depend - partly - on what you are trying to achieve and - a lot - on the version of TYPO3 CMS which you are targeting.

There are two main ways to store your changes to the TCA: inside an extension or straight in the typo3conf folder. Both are described below in more details.

Storing in extensions

The advantage of putting your changes inside an extension is that they are nicely packaged in a self-contained entity which can be easily deployed on multiple servers.

The drawback is that the extension loading order must be finely controlled. Indeed if your extension modifies another extension, your extension must be loaded after the extension you are modifying. This can be achieved by registering that other extension as a dependency of yours. See the description of constraints in Core APIs.

For more information about an extension's structure, please refer to the extension architecture chapter in Core APIs.

Storing in the Overrides folder

Since TYPO3 CMS 6.2 (6.2.1 to be precise) changes to $GLOBALS['TCA'] must be stored inside a folder called Configuration/TCA/Overrides with one file per modified table. These files are named along the pattern <tablename>.php.

Thus if you want to customize the TCA of tx_foo_domain_model_bar, you'd create the file Configuration/TCA/Overrides/tx_foo_domain_model_bar.php.

The advantage of this method is that all such changes are incorporated into $GLOBALS['TCA'] before it is cached. This is thus far more efficient.

Important

Be aware that you cannot extend the TCA of extensions if it was configured within its ext_tables.php file, usually containing the "ctrl" section referencing a "dynamicConfigFile". Please ask the extension author to switch to the Configuration/TCA/<tablename>.php setup.

Important

Only TCA-related changes should go into Configuration/TCA/Overrides files. Some API calls may be okay as long as they also manipulate only $GLOBALS['TCA']. For example, it is fine to register a plugin with \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addPlugin() in Configuration/TCA/Overrides/tt_content.php because that API call only modifies $GLOBALS['TCA'] for table "tt_content".

Storing in ext_tables.php files

Until TYPO3 CMS 6.1 (still supported for 6.2) changes to $GLOBALS['TCA'] are packaged into an extension's ext_tables.php file. This is strongly discouraged in more recent versions of TYPO3 CMS.

Nowadays the only usecase for TCA changes in ext_tables.php is to override TCA definitions done in the ext_tables.php of a legacy extension. TCA overrides cannot be used in this case until the author of the legacy extension migrates his code.

Changing the TCA "on the fly"

It is also possible to perform some special manipulations on $GLOBALS['TCA'] right before it is stored into cache, thanks to the tcaIsBeingBuilt signal. This signal was introduced in TYPO3 CMS 6.2.1.

Customization examples

Many extracts can be found throughout the manual, but this section provides more complete examples.

Example 1: extending the fe_users table

The "examples" extension adds two fields to the "fe_users" table. Here's the complete code, taken from file Configuration/TCA/Overrides/fe_users.php:

<?php
if (!defined('TYPO3_MODE')) {
        die ('Access denied.');
}


// Add some fields to FE Users table to show TCA fields definitions
// USAGE: TCA Reference > $GLOBALS['TCA'] array reference > ['columns'][fieldname]['config'] / TYPE: "select"
$temporaryColumns = array (
        'tx_examples_options' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options',
                'config' => array (
                        'type' => 'select',
                        'items' => array (
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.0', '1'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.1', '2'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.2', '--div--'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.3', '3'),
                        ),
                        'size' => 1,
                        'maxitems' => 1,
                )
        ),
        'tx_examples_special' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_special',
                'config' => array (
                        'type' => 'user',
                        'size' => '30',
                        'userFunc' => 'Documentation\\Examples\\Userfuncs\\Tca->specialField',
                        'parameters' => array(
                                'color' => 'blue'
                        )
                )
        ),
);

\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns(
        'fe_users',
        $temporaryColumns
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addToAllTCAtypes(
        'fe_users',
        'tx_examples_options, tx_examples_special'
);

First of all, the fields that we want to add are detailed according to the $GLOBALS['TCA'] syntax for columns. This configuration is stored in the $temporaryColumns array.

After that come two additional steps:

  • first the columns are actually added to the table by using \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns().
  • then the fields are added to the "types" definition of the "fe_users" table by using \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addToAllTCAtypes(). It is possible to be more fine-grained.

This does not create the corresponding fields in the database. The new fields must also be defined in the ext_tables.sql file of the extension:

CREATE TABLE fe_users (
        tx_examples_options int(11) DEFAULT '0' NOT NULL,
        tx_examples_special varchar(255) DEFAULT '' NOT NULL
);

Warning

The above statement uses the SQL CREATE TABLE statement. This is the way TYPO3 expects it to be. The Extension Manager will automatically transform this into a ALTER TABLE statement when it detects that the table already exists.

By default new fields are added at the bottom of the form when editing a record from that table. If the table uses tabs, new fields are added at the bottom of the "Extended" tab (this tab is created if it does not exist). The following screenshot shows the placement of the two new fields when editing a "fe_users" record:

New fields for fe\_users table

The new fields added at the bottom of the "Extended" tab

The next example shows how to place a field more precisely.

Example 2: extending the tt_content table

In this second example, we will add a "No print" field to all content element types. First of all, we add its SQL definition in ext_tables.sql:

CREATE TABLE tt_content (
        tx_examples_noprint tinyint(4) DEFAULT '0' NOT NULL
);

Then we add it to the $GLOBALS['TCA'] in Configuration/TCA/Overrides/tt_content.php:

$temporaryColumn = array(
        'tx_examples_noprint' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:tt_content.tx_examples_noprint',
                'config' => array (
                        'type' => 'check',
                )
        )
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns(
        'tt_content',
        $temporaryColumn
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addFieldsToPalette(
        'tt_content',
        'visibility',
        'tx_examples_noprint',
        'after:linkToTop'
);

The code is mostly the same as in the first example, but the last line is very different and requires an explanation. The "pages" and "tt_content" use palettes extensively for all fields and not just for secondary options, for increased flexibility. So in this case we use addFieldsToPalette() instead of addToAllTCAtypes(). We need to specify the palette's key as the second argument (visibility). Precise placement of the new field is achieved with the fourth parameter (after:linkToTop). This will place the "no print" field right after the "link to top" field, instead of putting it in the "Extended" tab.

The result is the following:

New fields for tt\_content table

The new field added next to an existing one

Note

Obviously this new field will now magically exclude a content element from being printed. For it to have any effect, it must be used during the rendering by modifying the TypoScript used to render the "tt_content" table. Although this is outside the scope of this manual, here is an example of what you could do, for the sake of showing a complete process.

Assuming you are using "css_styled_content" (which is installed by default), you could add the following TypoScript to your template:

tt_content.stdWrap.outerWrap = <div class="noprint">|</div>
tt_content.stdWrap.outerWrap.if.isTrue.field = tx_examples_noprint

This will wrap a "div" tag with a "noprint" class around any content element that has its "No print" checkbox checked. The final step would be to declare the appropriate selector in the print-media CSS file so that "noprint" elements don't get displayed.

This is just an example of how the effect of the "No print" checkbox can be ultimately implemented. It is meant to show that just adding the field to the $GLOBALS['TCA'] is not enough.

Verifying the $TCA

You may find it necessary – at some point – to verify the full structure of the $GLOBALS['TCA'] in your TYPO3 installation. The System > Configuration module makes it possible to have an overview of the complete $GLOBALS['TCA'], with all customizations taken into account.

The Configuration module

Checking the existence of the new field via the Configuration module

If you cannot find your new field, it probably means that you have made some mistake.

This view is also useful when trying to find out where to insert a new field, to explore the combination of types and palettes that may be used for the table that we want to extend.

The Extension Manager (EM)

Extensions are managed from the Extension Manager inside TYPO3 by "admin" users. The module is located at "Admin tools > Ext Manager" and offers a menu with options to see loaded extensions (those that are installed or activated), available extensions on the server and the possibility to import extensions from online resources, typically the TER (TYPO3 Extension Repository) located at typo3.org.

The Extension Manager (from TYPO3 7.6 to TYPO3 8.7)

Interface of the Extension Manager (from TYPO3 7.6 to TYPO3 8.6) showing all available extensions.

The interface is really easy to use. You just click the +/- icon to the left of an extension in order to install it and follow the instructions.

Creating a new extension

This chapter is not a tutorial about how to create an Extension. It only aims to be a list of steps to perform and key information to remember.

First you have to register an extension key. This is the unique identifier for your extension.

Kickstarting the extension

Although it is possible to write every single line of an extension from scratch, there is tool which makes it easier to start. It is called "Extension builder" (key: "extension_builder") and can be installed from TER.

The Extension Builder comes with its own BE module:

A view from the Extension Builder

The Domain Modeller screen of the Extension Builder. The comfort of building your model with drag and drop.

Note that this tool is not a complete editor. It helps you creating the scaffolding of your extension, generating the necessary files. It's then up to you to fill these with the relevant code.

Warning

The Extension Builder has some possibility to preserve code, but it should still be used with care.

After the extension is written to your computer's disk you will be able to install it locally and start using it.

Please refer to the Extension Builder's manual for more information.

Creating a new distribution

This chapter describes the main steps in creating a new distribution. It should not be considered as a full fledge tutorial.

Concept of distributions

Distributions are full TYPO3 CMS websites ready to be unpacked. They provide an easy quick start for using TYPO3 CMS. The most well known distribution is "The official Introduction Package". Distributions can most easily installed in the backend extension manager in "Get preconfigured distribution", it lists all available distributions for the given core version.

A distribution is just an extension enriched with some further data that is loaded or executed upon installing that extension. A distribution takes care of the following parts:

  • Deliver initial database data
  • Deliver fileadmin files
  • Deliver configuration for a package
  • Hook into the process after saving configuration to trigger actions dependent on configuration values
  • Deliver dependent extensions if needed (e.g., customized versions or extensions not available through TER)

Kickstarting the distribution

A distribution is a special kind of extension. The first step is thus to create a new extension. Start by registering an extension key, which will be the unique identifier of your distribution.

Next create the Extension declaration file as usual, except for the "category" property which must be set to distribution.

Configuring the distribution display in the EM

You should provide two preview images for your distribution. Provide a small 220x150 pixels for the list in the extension manager as Resources/Public/Images/Distribution.png and a larger 300x400 pixels welcome image as Resources/Public/Images/DistributionWelcome.png. The welcome image is displayed in the distribution detail view inside the extension manager.

Fileadmin files

Create the following folder structure inside your extension:

  • Initialisation
  • Initialisation/Files

All the files inside that second folder will be copied to fileadmin/<extkey> during installation, where "extkey" is the extension key of your distribution.

A good strategy on files (as followed by ext:introduction) is to construct the distribution in a way that it can be unloaded after initial import and removed from the file system.

To achieve that, when creating content for your distribution, all your content related files (assets) should be located within fileadmin/<extkey> in the first place, and content elements or other records should reference these files via FAL. A good export preset will then contain the content related assets within your dump.

If there are files not directly referenced in tables selected for export (for example ext:form .yml form configurations), you can locate them within fileadmin/<extkey>, too. Only those need to be copied to Initialization/Files - all other files referenced in database rows will be within your export dump.

Note you should not end up with having all your site configuration (TypoScript files, logos, css and so on) within fileadmin. This is considered bad practice. The main site setup should be an extension, keep in mind that fileadmin is for editors. In case of the introduction distribution, the main site setup (templates, content elements, ...) is included in the extension bootstrap_package, and ext:introduction has a dependency to this. This way, ext:introduction only provides the database dump and the asset files, while ext:bootstrap_package is the real site setup. This ends up with only content related stuff being located in fileadmin, delivered by ext:introduction.

Database data

The database data is delivered as TYPO3 CMS export data.xml. Generate this file by exporting your whole installation from the tree root with the import/export module.

Warning

Do NOT include backend users in the dump! If you do, you end up having your user on other systems who loaded your distribution. Give the export a special check in this area. Having your backend user in the dump is most likely a security vulnerability of your distribution if that distribution is uploaded to the public.

The file has to be named data.xml (or data.t3d, where the .t3d format is harder to maintain). The dump file must be located in the Initialisation folder.

It is also possible to have referenced files (images / media) in an own folder called Initialisation/data.xml.files/ - a good export preset should prepare that.

Note

Due to core bugs, importing extracted files from standalone file folder only works since core version 8.7.10 and 9.1.0. For older target core versions, files must not be extracted (tab Advanced options), but directly included in data.xml.

Another core issue prevents loading data.xml if it is bigger than 10MB. In this case the only option left is going with data.t3d

Exporting the correct data can be a bit tricky to get right. It is a good idea to create an "Export preset" within the Export module for that and deliver an sql dump of that preset within the distribution. The introduction distribution comes with a maintained sql dump that could be useful as kick start. Just load that row into table tx_impexp_presets and adapt to the needs of your distribution. The ext:introduction preset is configured as:

  • Export db data as data.xml
  • Export only referenced FAL file relations into data.xml.files directory, do not just export all files from fileadmin
  • Do not export be_users (!)
  • Do not export some other tables like sys_log and friends
Distribution configuration

A distribution is technically handled as an extension. Therefore your can make use of all configuration options as needed.

After installing the extension, the signal hasInstalledExtensions is dispatched. You may use this to alter your website configuration (e.g. color scheme) on the fly.

Delivering custom dependencies

Normally extension dependencies are setup in the Extension declaration file.

However sometimes, extensions are not available in the TYPO3 Extension Repository (TER), or you need to deliver a modified version. Therefore, a distribution can act as its own extension repository. Add unpacked extensions to Initialisation/Extensions/ to provide dependencies. Your main extension has to be dependent on these extension as normal dependencies in ext_emconf.php.

Extensions delivered inside an extension have the highest priority when extensions need to be fetched.

Caution

This will not overwrite extensions already present in the system.

Test your distribution

To test your distribution, simply copy your extension to an empty TYPO3 CMS installation and try to install it from the Extension Manager.

To test a distribution locally without uploading to TER, just install a blank TYPO3 (last step in installer "Just get me to the Backend"), then go to extension manager, select "Get extensions" once to let the extension manager initialize the extension list (this is needed if your distribution has dependencies to other extensions, for instance ext:introduction depends on ext:bootstrap_package). Next, copy or move the distribution extension to typo3conf/ext, it will then show up in extension manager default tab "Installed Extensions".

Install the distribution extension from there. The extension manager will then resolve TER dependencies, loads the database dump and will handle the file operations. Under the hood, this does the same as later installing the distribution via "Get preconfigured distribution", when it has been uploaded or updated in TER, with the only difference that you can provide and test the distribution locally without uploading to TER first.

Warning

It is not enough to clean all files and the page tree if you want to try again to install your distribution. Indeed, TYPO3 CMS remembers that it previously imported your distribution and will skip any known files and the database import. Make sure to clean the table "sys_registry" if you want to work around that, or, even better, install a new blank TYPO3 to test again. Tip: Optimize creating the empty TYPO3 instance with a script, you probably end up testing the import a couple of times until you are satisfied with the result.

More information

The introduction extension is a good starting point to see how distributions are handled in practice. It also comes with an impexp preset to easily export database data with correct settings and dependencies.

Some additional backgrounds can be retrieved from the blueprint for this feature.

Adding documentation

If you plan to upload your extension to the TYPO3 Extension Repository (TER), you should first consider adding a documentation to your extension. A documentation will help users and administrators to quickly install and configure your extension and give it more weight.

The documentation platform https://docs.typo3.org centralizes documentation for every project. It supports two different kind of documentation:

  1. (recommended) A Sphinx project, stored within EXT:extkey/Documentation/
  2. A simple README file stored as EXT:extkey/README.rst as seen on Github

Sphinx project

Sphinx is the official format for official TYPO3 documentation. A Sphinx-based documentation is a set of plain text files making up the chapters or sections of the documentation. It uses a markup language called "reStructuredText" (reST).

Advantages of this new documentation format are numerous:

  • Output formats: Sphinx projects may be automatically rendered as HTML or TYPO3-branded PDF.
  • Cross-references: It is easy to cross-reference other chapters and sections of other manuals (either TYPO3 references or extension manuals).
  • Multilingual: Unlike OpenOffice, Sphinx projects may be easily localized and automatically presented in the most appropriate language to TYPO3 users.
  • Collaboration: As the documentation is plain text, it is easy to work as a team on the same manual or quickly review changes using any versioning system.

Although it is possible to write every single line of a Sphinx-based documentation from scratch, the TYPO3 community provides tools that help write and manage Sphinx projects:

  • The extension "Sphinx" (Sphinx Python Documentation Generator and Viewer) installs a local Sphinx environment to view, edit and compile documentation in the backend of your TYPO3 website. It can be installed from the TYPO3 Extension Repository (TER) like any other extension.
  • The Sphinx extension is able to convert existing OpenOffice manuals (manual.sxw) into Sphinx projects with just one click.
  • An example manual is available on the TYPO3 Documentation Github repository.
  • The Extension Builder provides a skeleton documentation based on the above-mentioned Git repository.
  • A good primer to get started using the reStructuredText markup.

README.rst

A "README.rst" is a simple text file stored at the root of your extension directory and briefly describing the purpose of your extension. It is best suited when installing or using your extension is straightforward. The format of this file is reStructuredText, as for chapters of a Sphinx project.

Tip

In TYPO3 6.2, the system extension "documentation" is using such a simple manual.

Other resources

Beyond the general overview given in this chapter, other sections in this manual will be of particular interest to extension developers:

TYPO3 explained

The source is the documentation! (General wisdom)

The TYPO3 APIs are first and foremost documented inside of the source scripts. It would be impossible to maintain documentation at more than one location given the fact that things change and sometimes fast. This chapter describes the most important elements of the API.

Directory structure

By default a TYPO3 installation consists of a structure of main directories within the web server document root. You will find this structure to be almost always like that. Depending on the installation variant you choose however, this may be slightly different. For instance, it is possible to have all PHP files except the entry points index.php within the composer managed vendor/ directory, outside of the document root. This setup however did not fully settle yet, and is not documented here in detail. So, if you look at "casual" TYPO3 installations, you will almost always find the directory structure as outlined below.

Directory Description
fileadmin/

This is a directory in which editors store files. Typically images, PDFs or video files appear in this directory and/or its subdirectories.

Note this is only the default editor's file storage. This directory is handled via the FAL API internally, there may be further storage locations configured outside of fileadmin/, even pointing to different servers or using 3rd party digital asset management systems.

Note

Note this directory is meant for editors! Integrators should not locate frontend website layout related files in here: Storing HTML templates, logos, Css and similar files used to build the website layout in here is considered bad practice. Integrators should locate and ship these files within a project specific extension.

typo3/ TYPO3 Backend directory. This directory contains most of the files coming with the TYPO3 Core. The files are arranged logically in the different system extensions in the sysext/ directory, according to the application area of the particular file. For example, the "frontend" extension amongst other things contains the "TypoScript library", the code for generating the Frontend website. In each system extension the PHP files are located in the folder Classes/. See extension files locations for more information on how single extensions are structured.
typo3conf/

TYPO3 configuration directory. This directory contains local extensions in typo3conf/ext folder. It may also contain a typo3conf/l10n directory that holds localisation files for frontend or backend languages other than english.

The most important file within typo3conf/ however is LocalConfiguration.php. This one contains local settings of the main global PHP array $GLOBALS['TYPO3_CONF_VARS], crucial settings like database connect credentials are in here. The file is managed by the install tool and the extension manager and the content should not be managed manually since extension manager or install tool may override manually changed settings again.

The file LocalConfiguration.php can be enriched by AdditionalConfiguration.php which is never touched by TYPO3 internal management tools. Be aware that having settings within AdditionalConfiguration.php may prevent the system from doing automatic upgrades and should be used with care and only if you know what you are doing.

typo3conf/ext/ Directory for local TYPO3 extensions. Each subdirectory contains one extension.
typo3conf/l10n Directory for extension localisations. The "Language" module of the TYPO3 Backend manages this directory.
typo3temp/ Directory for temporary files. It contains subdirectories for temporary files of extensions and TYPO3 components.

PHP Namespaces

Since version 6.0, TYPO3 CMS uses PHP namespaces for all classes in the Core.

The general structure of namespaces is the following:

\{VendorName}\{PackageName}\({CategoryName}\)*{ClassName}

For the Core, the vendor name is TYPO3\CMS and the package name corresponds to a system extension.

All classes must be located inside the Classes folder at the root of the (system) extension. The category name may contain several segments that correspond to the path inside the Classes folder.

Finally the class name is the same as the corresponding file name, without the .php extension.

"UpperCamelCase" is used for all segments.

Tip

See the chapter about 'ClassAliasMap.php' in the 6.2 documentation.. It may help you with migrating code from old to new conventions.

Core example

The good old t3lib_div class has been renamed to:

\TYPO3\CMS\Core\Utility\GeneralUtility

This means that the class is now found in the "core" system extension, in folder Classes/Utility, in a file named GeneralUtility.php.

Usage in extensions

Extension developers are free to use their own vendor name. Important: It may consist of one segment only. Vendor names must start with an uppercase character and are usually written in UpperCamelCase style. In order to avoid problems with different filesystems, only the characters a-z, A-Z, 0-9 and the dash sign "-" are allowed for package names – don't use special characters:

// good vendor name:
\Webcompany

// wrong vendor name:
\Web\Company

Attention

The vendor name TYPO3\CMS is reserved and may not be used by extensions!

The package name corresponds to the extension key. Underscores in the extension key are removed in the namespace and replaced by upper camel-case. So extension key:

weird-name_examples

would become:

Weird-nameExamples

in the namespace.

As mentioned above, all classes must be located in the Classes folder inside your extension. All sub-folders translate to a segment of the category name and the class name is the file name without the .php extension.

Looking at the "examples" extension, class:

examples/Classes/Controller/DefaultController.php

corresponds to namespace:

\Documentation\Examples\Controller\DefaultController

Inside the class, the namespace is declared as:

<?php
namespace Documentation\Examples\Controller;

Namespaces in Extbase

When registering components in Extbase, the vendor name must be used on top of the extension key.

For a backend module:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::registerModule(
    '<vendorName>.<ExtensionName>',
    // ...
);

For a frontend module:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::configurePlugin(
    '<vendorName>.<ExtensionName>',
    // ...
);

Important

  • Do not forget the dot after the vendor name.
  • Do not use dots inside the vendor name.

Namespaces for test classes

As for ordinary classes, namespaces for test classes start with a vendor name followed by the extension key.

All test classes reside in a Tests folder and thus the third segment of the namespace must be "Tests". Unit tests are located in a Unit folder which is the fourth segment of the namespace. Any further subfolders will be subsequent segments.

So a test class in EXT:foo_bar_baz/Tests/Unit/Bla/ will have as namespace \Vendor\FooBarBaz\Tests\Unit\Bla.

Creating instances

The following example shows how you can create instances by means of GeneralUtility::makeInstance():

$contentObject = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(
   \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::class);

Or, use use to make the code more readable:

use TYPO3\CMS\Core\Utility\GeneralUtility;
use TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer;

$contentObject = GeneralUtility::makeInstance(ContentObjectRenderer::class);

include and required

There is no need for require() or include() statements. All classes adhering to namespace conventions will automatically be located and included by the autoloader.

References

For more information about PHP namespaces in general, you may want to refer to the PHP documentation and in particular the Namespaces FAQ.

Autoloading

The autoloader takes care of finding classes in TYPO3. It is closely related to \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance() which takes care of singleton and XCLASS handling.

As a developer you should always instantiate classes either through \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance() or with the Extbase \TYPO3\CMS\Extbase\Object\ObjectManager> (which internally uses makeInstance() again).

Autoloading classes since TYPO3 7.x

TYPO3 6.2 was still delivered with a couple of different autoloaders, that all had different approaches and rules to find a class. This led to the naming conventions in and outside extbase and the optional ext_autoload.php file to load classes that didn't follow the conventions. Since TYPO3 7.0 all this is gone and there is only a single autoloader left, the one of composer. No matter if you run TYPO3 in composer mode or not, TYPO3 uses the composer autoloader to resolve all class file locations. However, the autoloader is little bit more sophisticated in composer mode as it then supports PSR-4 autoloading.

Loading classes without composer mode

This means, you did not install TYPO3 via a require-statement inside your composer.json. It's a regular old-school install where the TYPO3 source and the symlinks (typo3/index.php) are setup manually. In this case, every time you install an extension, the autoloader scans the whole extension directory for classes. No matter if they follow any convention at all. There is just one rule. Put each class into its own file. The generated classmap is a huge array with a mapping of classnames to their location on the disk.

Example:

<?php

// autoload_classmap.php @generated by TYPO3

$typo3InstallDir = PATH_site;

return array(
   'Schnitzler\\Templavoila\\Clipboard\\Clipboard' => $typo3InstallDir . 'typo3conf/ext/templavoila/Classes/Clipboard/Clipboard.php',
   'tx_templavoila_pi1' => $typo3InstallDir . 'typo3conf/ext/templavoila/Compatibility/class.tx_templavoila_pi1.php',
   ...
);

This method is failsafe unless the autoload information cannot be written. In this case, check the install tool for warnings and make sure that typo3temp is writable.

Troubleshooting:

If your classes cannot be found, try the following approaches.

  • Dump the class loading information manually with the following command: php typo3/cli_dispatch.phpsh extbase extension:dumpclassloadinginformation
  • If that command itself fails, please (manually) uninstall the extension and simply try reinstalling it (via the extension manager).
  • If you are still not lucky, the issue is definitely on your side and you should double check the write permissions on typo3temp.

Loading classes with composer mode

In composer mode, the autoloader checks for (classmap and PSR-4) autoloading information inside your extensions' composer.json. If you do not provide any information, the autoloader falls back to the classmap autoloading like in non composer mode.

Troubleshooting:

  • Dump the class loading information manually via composer dumpautoload and check that the autoload information is updated. Typically you would check vendor/composer to hold files like autoload_classmap.php and autoload_psr4.php etc.

Example:

$ tree vendor/composer
.
├── ClassLoader.php
├── LICENSE
├── autoload_classmap.php
├── autoload_files.php
├── autoload_namespaces.php
├── autoload_psr4.php
├── autoload_real.php
├── autoload_static.php
├── include_paths.php
└── installed.json

Best practices

  • If you didn't do so before, have a look at the PSR-4 standard. It defines very good rules for naming classes and the files they reside in. Really, read the specs and start using PSR-4 in your projects. It's unlikely that there will be any other more advanced standard in the near future in the PHP world. PSR-4 is the way to go and you should embrace it.
  • Even if you do not use composer mode and the class mapping of the autoloader allows you to use whatever you want, stick to PSR-4. It's not only a very good standard to find classes, but it will also help organizing your code.
  • PSR-4 is all about namespaces. No matter if you like namespaces or not, use them. Namespaces exist since PHP 5.3, so you will be able to use them in any modern TYPO3 project due to the minimum PHP requirements of TYPO3 itself.

Tip

PSR-4 is a standard that has been develop by the PHP Framework Interop Group (FIG). PSR-4 is an advanced standard for autoloading php classes and replaces PSR-0. If you want to know more about the PHP FIG in general and PSR-4 in specific, please visit http://www.php-fig.org/psr/psr-4/.

Bootstrapping

TYPO3 CMS has a clean bootstrapping process driven mostly by class \TYPO3\CMS\Core\Core\Bootstrap. This class contains a host of methods each responsible for a little step along the initialization of a full TYPO3 process, be it the backend or other contexts.

Some contexts add their own bootstrap class (like the command line, which additionally requires \TYPO3\CMS\Core\Core\CliBootstrap.

Note

The frontend's bootstrapping process is not yet fully encapsulated in a bootstrap class.

Warning

This bootstrapping API is internal and may change at any time in the near future even in minor updates. It is thus discouraged to use it in third party code. Use this class only if other extensibility possibilities such as Hooks, Signals or XCLASS are not enough to reach your goals.

One can see the bootstrapping process in action in file typo3/sysext/backend/Classes/Http/Application.php:

use TYPO3\CMS\Core\Core\Bootstrap;

###

$this->bootstrap = Bootstrap::getInstance()
   ->initializeClassLoader($classLoader)
   ->setRequestType(TYPO3_REQUESTTYPE_BE | (!empty($_GET['ajaxID']) ? TYPO3_REQUESTTYPE_AJAX : 0))
   ->baseSetup($this->entryPointLevel);

// Redirect to install tool if base configuration is not found
if (!$this->bootstrap->checkIfEssentialConfigurationExists()) {
   $this->bootstrap->redirectToInstallTool($this->entryPointLevel);
}

foreach ($this->availableRequestHandlers as $requestHandler) {
   $this->bootstrap->registerRequestHandlerImplementation($requestHandler);
}

$this->bootstrap->configure();

###

Note that most methods of the Bootstrap class must be called in a precise order. It is perfectly possible to define one's own bootstrapping process, but care should be taken about the call order.

Also note that all bootstrapping methods return the instance of the Bootstrap class itself, allowing calls to be chained.

Application Context

Each request, no matter if it runs from the command line or through HTTP, runs in a specific application context. TYPO3 CMS provides exactly three built-in contexts:

  • Production (default) - should be used for a live site
  • Development - used for development
  • Testing - is only used internally when executing TYPO3 core tests. It must not be used otherwise.

The context TYPO3 runs in is specified through the environment variable TYPO3_CONTEXT. It can be set on the command line:

# run the TYPO3 CMS CLI commands in development context
TYPO3_CONTEXT=Development ./typo3/cli_dispatch.phpsh

or be part of the web server configuration:

# In your Apache configuration, you usually use:
SetEnv TYPO3_CONTEXT Development

# Set context with mod_rewrite
# Rules to set ApplicationContext based on hostname
RewriteCond %{HTTP_HOST} ^dev\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Development]

RewriteCond %{HTTP_HOST} ^staging\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Production/Staging]

RewriteCond %{HTTP_HOST} ^www\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Production]
# In your Nginx configuration, you can pass the context as a fastcgi parameter
location ~ \.php$ {
   include         fastcgi_params;
   fastcgi_index   index.php;
   fastcgi_param   TYPO3_CONTEXT  Development/Dev;
   fastcgi_param   SCRIPT_FILENAME  $document_root$fastcgi_script_name;
}
Custom Contexts

In certain situations, more specific contexts are desirable:

  • a staging system may run in a Production context, but requires a different set of credentials than the production server.
  • developers working on a project may need different application specific settings but prefer to maintain all configuration files in a common Git repository.

By defining custom contexts which inherit from one of the three base contexts, more specific configuration sets can be realized.

While it is not possible to add new "top-level" contexts at the same level like Production and Testing, you can create arbitrary sub-contexts, just by specifying them like <MainContext>/<SubContext>.

For a staging environment a custom context Production/Staging may provide the necessary settings while the Production/Live context is used on the live instance.

Note

This even works recursively, so if you have a multiple-server staging setup, you could use the context Production/Staging/Server1 and Production/Staging/Server2 if both staging servers needed different configuration.

Attention

Testing Is reserved for internal use when executing TYPO3 core functional and unit tests It must not be used otherwise. Instead sub-contexts must be used: Production/Testing or Development/Testing

Usage Example

The current Application Context is set very early in the bootstrap process can be accessed through public API for example in the AdditionalConfiguration.php file to automatically set different configuration for different contexts.

In file typo3conf/AdditionalConfiguration.php:

switch (\TYPO3\CMS\Core\Utility\GeneralUtility::getApplicationContext()) {
   case 'Development':
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 1;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '*';
      break;
   case 'Production/Staging':
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 0;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '192.168.1.*';
      break;
   default:
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 0;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '127.0.0.1';
}

Variables and Constants

After TYPO3's bootstrap sequence has completed, a number of global variables, constants and classes available to any script.

The column "Avail. in FE" is an indicator that tells you if the constant, variable or class mentioned is also available to scripts running under the frontend of the "cms" extension.

Constants

Constants normally define paths and database information. These values are global and cannot be changed when they are first defined. This is why constants are used for such vital information.

These constants are defined at various points during the bootstrap sequence.

Note

To make the table below a bit more compact, namespaces were left out. Here are the fully qualified class names referred to below:

  • "SystemEnvironmentBuilder" = \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder
  • "Bootstrap" = \TYPO3\CMS\Core\Core\Bootstrap
Table 1: Traditional List
Constant Defined in Description Avail. in FE
TYPO3_MODE \TYPO3\CMS\Backend\Http\Application::defineLegacyConstants() \TYPO3\CMS\Core\Console\CommandApplication::defineLegacyConstants() \TYPO3\CMS\Frontend\Http\Application::defineLegacyConstants() \TYPO3\CMS\Install\Http\Application::defineLegacyConstants() Mode of TYPO3: Set to either "FE" or "BE" depending on frontend or backend execution and context.

Yes

value = "FE"

TYPO3_OS SystemEnvironmentBuilder::defineBaseConstants() Operating system; Windows = "WIN", other = "" (presumed to be some sort of Unix) Yes
PATH_thisScript SystemEnvironmentBuilder::definePaths() Abs. path to current script. Yes
TYPO3_mainDir SystemEnvironmentBuilder::definePaths() This is the directory of the backend administration for the sites of this TYPO3 installation. Hardcoded to typo3/. Must be a subdirectory to the website. See elsewhere for descriptions on how to change the default admin directory, typo3/, to something else. Yes
PATH_typo3 SystemEnvironmentBuilder::definePaths() Abs. path of the TYPO3 admin dir (PATH_site + TYPO3_mainDir). No
PATH_site SystemEnvironmentBuilder::definePaths() Absolute path to directory with the frontend (one directory above PATH_typo3) Yes
PATH_typo3conf SystemEnvironmentBuilder::definePaths() Absolute TYPO3 configuration path (local, not part of source). Yes
TYPO3_version SystemEnvironmentBuilder::defineBaseConstants() The TYPO3 version, as a "x.y.z" number. Development versions will be either "x.y.z-dev" for stable versions or "x.y-dev" for the current master. Yes
TYPO3_branch SystemEnvironmentBuilder::defineBaseConstants() The TYPO3 version Branch, as a "x.y" number. Without the patch level. Yes
Table 2: Base Constants

Check \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::defineBaseConstants() for updates.

String constants
Constant Value Description
NUL chr(0) A null
TAB chr(9) A tabulator
LF chr(10) A linefeed
CR chr(13) A carriage return
SUB chr(26) A sub (substitute) character
CRLF CR + LF Carriage return + linefeed pair
Operating system identifier
Constant Value Description
TYPO3_OS self::getTypo3Os()) Either "WIN" or empty string
Service error constants
Constant Value Description
T3_ERR_SV_GENERAL -1 General error - something went wrong
T3_ERR_SV_NOT_AVAIL -2 During execution it showed that the service is not available and should be ignored. The service itself should call $this->setNonAvailable()
T3_ERR_SV_WRONG_SUBTYPE -3 Passed subtype is not possible with this service
T3_ERR_SV_NO_INPUT -4 Passed subtype is not possible with this service
T3_ERR_SV_FILE_NOT_FOUND -20 File not found which the service should process
T3_ERR_SV_FILE_READ -21 File not readable
T3_ERR_SV_FILE_WRITE -22 File not writable
T3_ERR_SV_PROG_NOT_FOUND -40 Passed subtype is not possible with this service
T3_ERR_SV_PROG_FAILED -41 Passed subtype is not possible with this service

Global variables

Note

Variables in italics may be set in a script prior to the bootstrap process so they are optional.

Note

To make the table below a bit more compact, namespaces were left out. Here are the fully qualified class names referred to below:

  • "SystemEnvironmentBuilder" = \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder
  • "Bootstrap" = \TYPO3\CMS\Core\Core\Bootstrap
  • "PackageManager" = \TYPO3\CMS\Core\Package\PackageManager
Global variable Defined in Description Avail. in FE
$GLOBALS['TYPO3_CONF_VARS'] typo3/sysext/core/Configuration/DefaultConfiguration.php TYPO3 configuration array. Please refer to file typo3/sysext/core/Configuration/DefaultConfigurationDescription.php where each option is described in detail in the comments. The same comments are also available in the Install Tool when you choose "All Configuration". Yes
$TYPO3_LOADED_EXT PackageManager::loadPackageManagerStatesFromCache() PackageManager::initializeCompatibilityLoadedExtArray() Array with all loaded extensions listed with a set of paths. You can check if an extension is loaded by the function \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::isLoaded($key) where $key is the extension key. Yes
$EXEC_TIME SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables() Is set to time() so that the rest of the script has a common value for the script execution time. YES
$SIM_EXEC_TIME SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables() Is set to $EXEC_TIME but can be altered later in the script if we want to simulate another execution-time when selecting from e.g. a database (used in the frontend for preview of future and past dates) Yes
$PAGES_TYPES typo3/sysext/core/ext_tables.php See Page types (occasionally)
$TCA Bootstrap::loadExtensionTables() See TCA Reference Yes, partly
$TBE_MODULES typo3/sysext/core/ext_tables.php The backend main/sub-module structure. See section elsewhere plus source code of class \TYPO3\CMS\Backend\Module\ModuleLoader which also includes some examples. (occasionally)
$TBE_STYLES typo3/sysext/core/ext_tables.php Contains information related to BE skinning. (will be removed on CMS 9) (occasionally)
$T3_SERVICES SystemEnvironmentBuilder::initializeGlobalVariables() Global registration of services. Yes
$T3_VAR SystemEnvironmentBuilder::initializeGlobalVariables()

Space for various internal global data storage in TYPO3. Each key in this array is a data space for an application. Keys currently defined for use is:

['callUserFunction'] + ['callUserFunction_classPool']: Used by \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction to store singleton objects.

['RTEobj'] : Used to hold the current RTE object if any. See \TYPO3\CMS\Backend\Utility\BackendUtility.

['ext'][ extension-key ] : Free space for extensions.

Yes
$BE_USER Bootstrap::initializeBackendUser() Backend user object. See Backend User Object. (depends)
$TBE_MODULES_EXT [In ext_tables.php files of extensions] Used to store information about modules from extensions that should be included in "function menus" of real modules. See the Extension API for details. (occasionally)
$TCA_DESCR [tables.php files] Can be set to contain file references to local lang files containing TCA_DESCR labels. See section about Context Sensitive Help. No
Exploring global variables

Many of the global variables described above can be inspected using the Admin Tools > Configuration module.

Warning

This module is always viewed in the BE context. Variables defined only in the FE context will not be visible there.

The Configuration module in Admin Tools

Viewing the $TCA array using the Admin Tools > Configuration module

Backend User Object

The backend user of a session is always available to the backend scripts as the global variable $BE_USER. The object is created in \TYPO3\CMS\Core\Core\Bootstrap::initializeBackendUser() and is an instance of the class \TYPO3\CMS\Core\Authentication\BackendUserAuthentication (which extends \TYPO3\CMS\Core\Authentication\AbstractUserAuthentication).

In addition to $BE_USER one other global variables is of interest - $FILEMOUNTS, holding an array with the File mounts of the $BE_USER.

Checking user access

The $BE_USER object is mostly used to check user access right, but contains other helpful information. This is presented here by way of a few examples:

Checking access to current backend module

$MCONF is module configuration and the key $MCONF['access'] determines the access scope for the module. This function call will check if the $BE_USER is allowed to access the module and if not, the function will exit with an error message.

$BE_USER->modAccess($MCONF, 1);
Checking access to any backend module

If you know the module key you can check if the module is included in the access list by this function call:

$BE_USER->check('modules', 'web_list');

Here access to the module "Web > List" is checked.

Access to tables and fields?

The same function ->check() can actually check all the ->groupLists inside $BE_USER. For instance:

Checking modify access to the table "pages":

$BE_USER->check('tables_modify', 'pages');

Checking read access to the table "tt_content":

$BE_USER->check('tables_select', 'tt_content');

Checking if a table/field pair is allowed explicitly through the "Allowed Excludefields":

$BE_USER->check('non_exclude_fields', $table . ':' . $field);
Is "admin"?

If you want to know if a user is an "admin" user (has complete access), just call this method:

$BE_USER->isAdmin();
Read access to a page?

This function call will return true if the user has read access to a page (represented by its database record, $pageRec):

$BE_USER->doesUserHaveAccess($pageRec, 1);

Changing the "1" for other values will check other permissions:

  • use "2" for checking if the user may edit the page
  • use "4" for checking if the user may delete the page.
Is a page inside a DB mount?

Access to a page should not be checked only based on page permissions but also if a page is found within a DB mount for ther user. This can be checked by this function call ($id is the page uid):

$BE_USER->isInWebMount($id)
Selecting readable pages from database?

If you wish to make a SQL statement which selects pages from the database and you want it to be only pages that the user has read access to, you can have a proper WHERE clause returned by this function call:

$BE_USER->getPagePermsClause(1);

Again the number "1" represents the "read" permission; "2" is "edit" and "4" is delete permission. The result from the above query could be this string:

((pages.perms_everybody & 1 = 1)OR(pages.perms_userid = 2 AND pages.perms_user & 1 = 1)OR(pages.perms_groupid in (1) AND pages.perms_group & 1 = 1))
Saving module data

This stores the input variable $compareFlags (an array!) with the key "tools_beuser/index.php/compare"

$compareFlags = \TYPO3\CMS\Core\Utility\GeneralUtility::_GP('compareFlags');
$BE_USER->pushModuleData('tools_beuser/index.php/compare', $compareFlags);
Getting module data

This gets the module data with the key "tools_beuser/index.php/compare" (lasting only for the session)

$compareFlags = $BE_USER->getModuleData('tools_beuser/index.php/compare', 'ses');
Getting TSconfig

This function can return a value from the "User TSconfig" structure of the user. In this case the value for "options.clipboardNumberPads":

$BE_USER->getTSConfigVal('options.clipboardNumberPads');
Getting the username

The full "be_users" record of a authenticated user is available in $BE_USER->user as an array. This will return the "username":

$BE_USER->user['username']
Get User Configuration value

The internal ->uc array contains options which are managed by the User Tools > User Settings module (extension "setup"). These values are accessible in the $BE_USER->uc array. This will return the current state of "Notify me by email, when somebody logs in from my account" for the user:

$BE_USER->uc['emailMeAtLogin']

TYPO3 Core Engine (TCE)

Introduction

Database

The TYPO3 Core Engine is the class that handles all *data* writing to database tables configured in $TCA. In addition the class handles commands such as copy, move, delete. It will handle undo/history and versioning of records and everything will be logged to the sys_log. And it will make sure that write permissions are evaluated correctly for the user trying to write to the database. Generally, any processing specific option in the $TCA array is handled by TCE.

Using TCE for manipulation of the database content in the $TCA-configured tables guarantees that the data integrity of TYPO3 is respected. This cannot be safely guaranteed if you write to $TCA-configured database tables directly. It will also manage the relations to files and other records.

TCE requires a backend login to work. This is due to the fact that permissions are observed (of course) and thus TCE needs a backend user to evaluate against. This means you cannot use DataHandler from the frontend scope. Thus writing to tables (such as a guestbook) will have to be done from the frontend without DataHandler.

The features of the $TCA are described in the TCA Reference.

Files

TCE also has a part for handling files. The file operations are normally performed in the File > List module where you can manage a directory on the server by copying, moving, deleting and editing files and directories. The file operations are managed by two core classes, \TYPO3\CMS\Core\Utility\File\BasicFileUtility and \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility.

Database: DataHandler basics (formerly known as TCEmain)

When you are using TCE from your backend applications you need to prepare two arrays of information which contain the instructions to DataHandler (\TYPO3\CMS\Core\DataHandling\DataHandler) of what actions to perform. They fall into two categories: data and commands.

"Data" is when you want to write information to a database table or create a new record.

"Commands" is when you want to move, copy or delete a record in the system.

The data and commands are created as multidimensional arrays and to understand the API of DataHandler you simply need to understand the hierarchy of these two arrays.

Caution

The DataHandler needs a properly configured TCA. If your field is not configured in the TCA the DataHandler is not able to interact with it. This also is the case if you configured "type"="none" (which is in fact a valid type) or if an invalid type is specified. In that case the DataHandler is not able to determine the correct value of the field.

Commands Array

Syntax:

$cmd[ tablename ][ uid ][ command ] = value

Description of keywords in syntax:

Key Data type Description
tablename string Name of the database table. Must be configured in $TCA array, otherwise it cannot be processed.
uid integer The UID of the record that is manipulated. This is always an integer.
command string (command keyword)

The command type you want to execute.

Note

Only one command can be executed at a time for each record! The first command in the array will be taken.

See table below for :ref:`command keywords and values <tce-command-keywords>`

value mixed

The value for the command

See table below for :ref:`command keywords and values <tce-command-keywords>`

Command keywords and values
Command Data type Value
copy integer

The significance of the value depends on whether it is positive or negative:

  • Positive value: The value points to a page UID. A copy of the record (and possibly child elements/tree below) will be inserted inside that page as the first element.

  • Negative value: The (absolute) value points to another record from the same table as the record being copied. The new record will be inserted on the same page as that record and if $TCA[...]['ctrl']['sortby'] is set, then it will be positioned after.

  • Zero value: Record is inserted on tree root level.

  • array: The array has to contain the integer value as in examples above and may contain field => value pairs for updates. The array is structured like:

    [
       'action' => 'copy', // Defines where this is a move or copy command
       'target' => $pUid, // Defines the page to insert the record.
       'update' => $update, // Array with field => value to be updated.
    ]
    
move integer Works like "copy" but moves the record instead of making a copy.
delete 1

Value should always be "1"

This action will delete the record (or mark the record "deleted" if configured in $TCA).

undelete 1

Value should always be "1".

This action will set the deleted-flag back to 0.

localize integer

Pointer to a sys_language uid to localize the record into. Basically a localization of a record is making a copy of the record (possibly excluding certain fields defined with l10n_mode) but changing relevant fields to point to the right sys language / original language record.

Requirements for a successful localization is this:

  • [ctrl] options "languageField" and "transOrigPointerField" must be defined for the table
  • A sys_language record with the given sys_language_uid must exist.
  • The record to be localized by currently be set to "Default" language and not have any value set for the transOrigPointerField either.
  • There cannot exist another localization to the given language for the record (looking in the original record PID).

Apart from this, ordinary permissions apply as if the user wants to make a copy of the record on the same page.

version array

Versioning action.

Keys:

  • [action] : Keyword determining the versioning action. Options are:
    • "new": Indicates that a new version of the record should be created.Additional keys, specific for "new" action:
      • [treeLevels]: (Only pages) Integer, -1 to 4, indicating the number of levels of the page tree to version together with a page. This is also referred to as the versioning type:-1 ("element") means only the page record gets versioned (default)0 ("page") means the page + content tables (defined by ctrl-flag versioning_followPages )>0 ("branch") means the the whole branch is versioned ( full copy of all tables), down to the level indicated by the value (1= 1 level down, 2= 2 levels down, etc.)The treeLevel is recorded in the field t3ver_swapmode and will be observed when the record is swapped during publishing.
      • [label]: Indicates the version label to apply. If not given, a standard label including version number and date is added.
    • "swap": Indicates that the current online version should be swapped with another.Additional keys, specific for "swap" action:
      • [swapWith]: Indicates the uid of the record to swap current version with!
      • [swapIntoWS]: Boolean, indicates that when a version is published it should be swapped into the workspace of the offline record.
    • "clearWSID": Indicates that the workspace of the record should be set to zero (0). This removes versions out of workspaces without publishing them.
    • "flush": Completely deletes a version without publishing it.
    • "setStage": Sets the stage of an element. Special feature: The id- key in the array can be a comma list of ids in order to perform the stageChange over a number of records. Also, the internal variable ->generalComment (also available through :file:`tce_db.php` as "&generalComment") can be used to set a default comment for all stage changes of an instance of tcemain. Additional keys for this action is:
      • [stageId]: Values are: -1 (rejected), 0 (editing, default), 1 (review), 10 (publish)
      • [comment]: Comment string that goes into the log.
Examples of commands:
$cmd['tt_content'][54]['delete'] = 1;    // Deletes tt_content record with uid=54
$cmd['pages'][1203]['copy'] = -303;   //Copies page id=1203 to the position after page 303
$cmd['pages'][1203]['move'] = 303;  // Moves page id=1203 to the first position in page 303
Data Array

Syntax:

$data[tablename][uid][fieldname] = value

Description of keywords in syntax:

Key Data type Description
tablename string Name of the database table. Must be configured in $TCA array, otherwise it cannot be processed.
uid mixed The UID of the record that is modified. If the record already exists, this is an integer. If you're creating new records, use a random string prefixed with "NEW", e.g. "NEW7342abc5e6d".
fieldname string Name of the database field you want to set a value for. Must be configure in $TCA[ tablename ]['columns']
value string

Value for "fieldname".

Important

Always make sure $this->stripslashes_values is false before using DataHandler.)

Note

For FlexForms the data array of the FlexForm field is deeper than three levels. The number of possible levels for FlexForms is infinite and defined by the data structure of the FlexForm. But FlexForm fields always end with a "regular value" of course.

Examples of Data submission

This creates a new page titled "The page title" as the first page inside page id 45:

$data['pages']['NEW9823be87'] = array(
   'title' => 'The page title',
   'subtitle' => 'Other title stuff',
   'pid' => '45'
);

This creates a new page titled "The page title" right after page id 45 in the tree:

$data['pages']['NEW9823be87'] = array(
   'title' => 'The page title',
   'subtitle' => 'Other title stuff',
   'pid' => '-45'
);

This creates two new pages right after each other, located right after the page id 45:

$data['pages']['NEW9823be87'] = array(
   'title' => 'Page 1',
   'pid' => '-45'
);
$data['pages']['NEWbe68s587'] = array(
   'title' => 'Page 2',
   'pid' => '-NEW9823be87'
);

Notice how the second "pid" value points to the "NEW..." id placeholder of the first record. This works because the new id of the first record can be accessed by the second record. However it works only when the order in the array is as above since the processing happens in that order!

This creates a new content record with references to existing and one new system category:

$data['sys_category']['NEW9823be87'] = array(
    'title' => 'New category',
    'pid' => 1,
);
$data['tt_content']['NEWbe68s587'] = array(
    'header' => 'Look ma, categories!',
    'pid' => 45,
    'categories' => array(
        1,
        2,
        'NEW9823be87', // You can also use placeholders here
    ),
);

This updates the page with uid=9834 to a new title, "New title for this page", and no_cache checked:

$data['pages'][9834] = array(
    'title' => 'New title for this page',
    'no_cache' => '1'
);
Clear cache

TCE also has an API for clearing the cache tables of TYPO3:

Syntax:

$tce->clear_cacheCmd($cacheCmd);
$cacheCmd values Description
[integer] Clear the cache for the page id given.
"all"

Clears all cache tables (cache_pages, cache_pagesection, cache_hash).

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.all".

"pages"

Clears all pages from cache_pages.

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.pages".

"temp_cached" or "system"

Clears all cache entries cache group system.

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.system".

Hook for cache post-processing

You can configure cache post-processing with a user defined PHP function. Configuration of the hook can be done from ext_localconf.php. An example might look like:

$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['clearCachePostProc'][] = 'myext_cacheProc->proc';
require_once(\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::extPath('myext') . 'class.myext_cacheProc.php');
Flags in DataHandler

There are a few internal variables you can set prior to executing commands or data submission. These are the most significant:

Internal variable Data type Description
->deleteTree Boolean

Sets whether a page tree branch can be recursively deleted.

If this is set, then a page is deleted by deleting the whole branch under it (user must have delete permissions to it all). If not set, then the page is deleted only if it has no branch.

Default is false.

->copyTree Integer

Sets the number of branches on a page tree to copy.

If 0 then branch is not copied. If 1 then pages on the 1st level is copied. If 2 then pages on the second level is copied, and so on.

Default is zero.

->reverseOrder Boolean

If set, the data array is reversed in the order, which is a nice thing if you're creating a whole bunch of new records.

Default is zero.

->copyWhichTables list of strings (tables)

This list of tables decides which tables will be copied. If empty then none will. If "*" then all will (that the user has permission to of course).

Default is "*".

Using DataHandler in scripts

It's really easy to use the class \TYPO3\CMS\Core\DataHandling\DataHandler in your own scripts. All you need to do is include the class, build a $data/$cmd array you want to pass to the class and call a few methods.

Important

Mind that these scripts have to be run in the backend scope! There must be a global $BE_USER object.

In your script you simply insert this line to include the class:

What follows are a few code listings with comments which will provide you with enough knowledge to get started. It is assumed that you have populated the $data and $cmd arrays correctly prior to these chunks of code. The syntax for these two arrays is explained in the previous chapter.

DataHandler examples
Submitting data

This is the most basic example of how to submit data into the database. It is four lines. Line 1 instantiates the class, line 2 defines that values will be provided without escaped characters (recommended!), line 3 registers the $data array inside the class and initializes the class internally! Finally line 4 will execute the data submission.

1
2
3
4
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->start($data, array());
$tce->process_datamap();
Executing commands

The most basic way of executing commands. Line 1 creates the object, line 2 defines that values will be provided without escaped characters (recommended), line 3 registers the $cmd array inside the class and initializes the class internally! Finally line 4 will execute the commands.

1
2
3
4
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->start(array(), $cmd);
$tce->process_cmdmap();
Clearing cache

In this example the cache clearing API is used. No data is submitted, no commands executed. Still you will have to initialize the class by calling the start() method (which will initialize internal variables).

Note

Clearing a given cache is possible only for users that are "admin" or have specific permissions to do so.

1
2
3
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->start(array(), array());
$tce->clear_cacheCmd('all');

Since TYPO3 CMS 6.2, caches are organized in groups. Clearing "all" caches will actually clear caches from the "all" group and not really all caches. Check the caching framework architecture section for more details about available caches and groups.

Complex data submission

Imagine the $data array something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$data = array(
    'pages' => array(
        'NEW_1' => array(
            'pid' => 456,
            'title' => 'Title for page 1',
        ),
        'NEW_2' => array(
            'pid' => 456,
            'title' => 'Title for page 2',
        ),
    )
);

This aims to create two new pages in the page with uid "456". In the follow code this is submitted to the database. Notice how line 3 reverses the order of the array. This is done because otherwise "page 1" is created first, then "page 2" in the same PID meaning that "page 2" will end up above "page 1" in the order. Reversing the array will create "page 2" first and then "page 1" so the "expected order" is preserved.

To insert a record after a given record, set the other record's negative uid as pid in the new record you're setting as data.

Apart from this line 6 will send a "signal" that the page tree should be updated at the earliest occasion possible. Finally, the cache for all pages is cleared in line 7.

1
2
3
4
5
6
7
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->reverseOrder = 1;
$tce->start($data, array());
$tce->process_datamap();
\TYPO3\CMS\Backend\Utility\BackendUtility::setUpdateSignal('updatePageTree');
$tce->clear_cacheCmd('pages');
Both data and commands executed with alternative user object

In this case it is shown how you can use the same object instance to submit both data and execute commands if you like. The order will depend on the order of line 4 and 5.

In line 3 the start() method is called, but this time with the third possible argument which is an alternative $BE_USER object. This allows you to force another backend user account to create stuff in the database. This may be useful in certain special cases. Normally you should not set this argument since you want TCE to use the global $BE_USER.

1
2
3
4
5
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->start($data, $cmd, $alternative_BE_USER);
$tce->process_datamap();
$tce->process_cmdmap();

The "tce_db.php" API

This script is a gateway for POST forms to class \TYPO3\CMS\Core\DataHandling\DataHandler. It has historically been the script to which data was posted when you wanted to update something in the database.

Today it is used for editing by only a few scripts, actually only the "Quick Edit" module in "Web>Page" (frontend). The standard forms you find in TYPO3 are normally rendered and handled by alt_doc.php which includes \TYPO3\CMS\Core\DataHandling\DataHandler on its own.

For commands it is still used from various locations.

You can send data to this file either as GET or POST vars where POST takes precedence. The variable names you can use are:

GP var name Data type Description
data array

Data array on the form [tablename][uid][fieldname] = value.

Typically it comes from a POST form which submits a form field like <input name="data[tt_content][123][header]" value="This is the headline" />.

cmd array

Command array on the form [tablename][uid][command] = value. This array may get additional data set internally based on clipboard commands send in CB var!

Typically this comes from GET vars passed to the script like &cmd[tt\_content][123][delete]=1 which will delete Content Element with UID 123.

cacheCmd string Cache command sent to ->clear_cacheCmd
redirect string Redirect URL. Script will redirect to this location after performing operations (unless errors has occurred)
flags array Accepts options to be set in TCE object. Currently it supports "reverseOrder" (boolean).
mirror array Example: [mirror][table][11] = '22,33' will look for content in [data][table][11] and copy it to [data][table][22] and [data][table][33].
CB array Clipboard command array. May trigger changes in "cmd".
vC string Verification code

File functions basics

File operations in the TCE are handled by the class \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility which extends \TYPO3\CMS\Core\Utility\File\BasicFileUtility. The instructions for file manipulation are passed to this class as a multidimensional array.

Files Array

Syntax:

$file[ command ][ index ][ key ] = value

Description of keywords in syntax:

Key Data type Description
command string (command keyword)

The command type you want to execute.

See table below for :ref:`command keywords, keys and values<tce-file-keywords>`

index integer Integer index in the array which separates multiple commands of the same type.
key string

Depending on the command type. The keys will carry the information needed to perform the action. Typically a "target" key is used to point to the target directory or file while a "data" key carries the data.

See table below for :ref:`command keywords, keys and values<tce-file-keywords>`

value string

The value for the command

See table below for :ref:`command keywords, keys and values<tce-file-keywords>`

Command keywords and values
Command Keys Value
delete "data" "data" = Absolute path to the file/folder to delete
copy

"data"

"target"

"altName"

"data" = Absolute path to the file/folder to copy

"target" = Absolute path to the folder to copy to (destination)

"altName" = (boolean): If set, a new filename is made by appending numbers/unique-string in case the target already exists.

move

"data"

"target"

"altName"

(Exactly like copy, just replace the word "copy" with "move")
rename

"data"

"target"

"data" = New name, max 30 characters alphanumeric

"target" = Absolute path to the target file/folder

newfolder

"data"

"target"

"data" = Folder name, max 30 characters alphanumeric

"target" = Absolute path to the folder where to create it

newfile

"data"

"target"

"data" = New filename

"target" = Absolute path to the folder where to create it

editfile

"data"

"target"

"data" = The new content

"target" = Absolute path to the target file

upload

"data"

"target"

upload_$id

"data" = ID-number (points to the global var that holds the filename- ref ($_FILES["upload_" . $id]["name"]).

"target" = Absolute path to the target folder (destination)

upload_$id = File reference. $id must equal value of file[upload][...][data]!

See \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility::func_upload().

unzip

"data"

"target"

"data" = Absolute path to the zip-file. (file extension must be "zip")

"target" = The absolute path to the target folder (destination) (if not set, default is the same as the zip-file)

It is unlikely that you will need to use this internally in your scripts like you will need \TYPO3\CMS\Core\DataHandling\DataHandler. It is fairly uncommon to need the file manipulations in own scripts unless you make a special application. Therefore the most typical usage of this API is from TYPO3CMSBackendControllerFileFileController and the core scripts that are activated by the "File > List" module.

However, if needed, this is an example of how to initialize usage. It is taken from ImportExportController.php:

1
2
3
4
5
6
   // Initializing:
$this->fileProcessor = GeneralUtility::makeInstance(ExtendedFileUtility::class);
$this->fileProcessor->setActionPermissions();

$this->fileProcessor->start($this->file);
$this->fileProcessor->processData();

Explanation: Line 2 creates an instance of the class. Then the file operation permissions are loaded from the user object in line 3. Finally, the file command array is loaded in line 5 and internally additional configuration takes place according to $GLOBALS['TYPO3_CONF_VARS']!. In line 6 the command map is executed.

The "tce_file.php" API

This script serves as the file administration part of the TYPO3 Core Engine. It's a gateway for TCE (TYPO3 Core Engine) file-handling through POST forms. It uses \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility for the manipulation of the files.

This script is used from the File > List module where you can rename, create, delete etc. files and directories on the server.

You can send data to this file either as GET or POST vars where POST takes precedence. The variable names you can use are:

GP var name Data type Description
file array

Array of file operations. See previous information about basic file functions.

This could typically be a GET var like &file[delete][0][data]=[absolute file path] or a POST form field like:

"<input type="text" name="file[newfolder][0][data]" value=""/>
<input type="hidden" name="file[newfolder][0][target]"
value="[absolute path to folder to create in]"/>"
redirect string Redirect URL. Script will redirect to this location after performing operations.
CB array Clipboard command array. May trigger changes in "file"
vC string Verification code
overwriteExistingFiles boolean If existing files should be overridden.

FormEngine

Introduction

Looking at TYPO3's main constructs from an abstract position, the system splits into three most important pillars:

DataHandler
TYPO3\CMS\Core\DataHandling\...: Construct taking care of persisting data into the database. The DataHandler takes an array representing one or more records, inserts, deletes or updates them in the database and takes care of relations between multiple records. If editing content in the backend, this construct does all main database munging. DataHandler is fed by some controller that most often gets GET or POST data from FormEngine.
FormEngine
TYPO3\CMS\Backend\Form\...: FormEngine renders records, usually in the backend. It creates all the HTML needed to edit complex data and data relations. Its GET or POST data is then fed to the DataHandler by some controller.
Frontend rendering
TYPO3\CMS\Frontend\...: Render the website frontend. The frontend rendering, usually based on TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController uses TypoScript and / or Fluid to process and render database content into the frontend.

The glue between these three pillars is TCA (Table Configuration Array): It defines how database tables are constructed, which localization or workspace facilities exist, how it should be displayed in the backend, how it should be written to the database, and - next to TypoScript - which behaviour it has in the frontend.

This chapter is about FormEngine. It is important to understand this construct is based on TCA and is usually used in combination with the DataHandler. However, FormEngine is constructed in a way that it can work without DataHandler: A controller could use the FormEngine result and process it differently. Furthermore, all dependencies of FormEngine are abstracted and may come from "elsewhere", still leading to the form output known for casual records.

This makes FormEngine an incredible flexible construct. The basic idea is "feed something that looks like TCA and render forms that have the full power of TCA but look like all other parts of the backend".

The FormEngine code base has been significantly refactored in TYPO3 CMS version 7 and version 8 to be much more flexible, more easy to use and extend, and much more powerful than before. This is an ongoing process and some areas still need a major overhaul. The current state of the documentation aims to explain the main constructs of FormEngine and gives an insight on how to re-use, adapt and extend it with extensions. The core team expects to see more usages of FormEngine within core itself and within extensions in the future, and encourages developers to solve feature needs based on FormEngine. With the ongoing changes, those areas that may need code adaptions in the foreseeable future have notes within the documentation and developers should be available to adapt with younger cores. Watch out for breaking changes if using FormEngine and updating core.

Main rendering workflow

This is done by example. The details to steer and how to use only sub-parts of the rendering chain are explained in more detail in the following sections.

Editing a record in the backend - often from within the Page or List module - triggers the EditDocumentController by routing definitions using getModuleUrl() and handing over which record of which table should be edited. This can be an existing record, or it could be a command to create the form for a new record. The EditDocumentController is the main logic triggered whenever an editor changes a record!

The EditDocumentController has two main jobs: Trigger rendering of one or multiple records via FormEngine, and hand over any given data by a FormEngine POST result over to the DataHandler to persist stuff in the database.

The rendering part of the EditDocumentController job splits into these parts:

  • Initialize main FormEngine data array using POST or GET data to specify which specific record(s) should be edited.
  • Select which group of DataProviders should be used.
  • Trigger FormEngine DataCompiler to enrich the initialized data array with further data by calling all data providers specified by selected data provider group.
  • Hand over DataCompiler result to an entry "render container" of FormEngine and receive a result array.
  • Take result array containing HTML, CSS and JavaScript details and put them into FormResultCompiler which hands them over to the PageRenderer.
  • Let the PageRenderer output its compiled result.
Main FormEngine workflow

The controller does two distinct things here: First, it initializes a data array and lets it get enriched by data providers of FormEngine which add all information needed for the rendering part. Then feed this data array to the rendering part of FormEngine to end up with a result array containing all HTML, CSS and JavaScript.

In code, this basic workflow looks like this:

$formDataGroup = GeneralUtility::makeInstance(TcaDatabaseRecord::class);
$formDataCompiler = GeneralUtility::makeInstance(FormDataCompiler::class, $formDataGroup);
$nodeFactory = GeneralUtility::makeInstance(NodeFactory::class);
$formResultCompiler = GeneralUtility::makeInstance(FormResultCompiler::class);
$formDataCompilerInput = [
    'tableName' => $table,
    'vanillaUid' => (int)$theUid,
    'command' => $command,
];
$formData = $formDataCompiler->compile($formDataCompilerInput);
$formData['renderType'] = 'outerWrapContainer';
$formResult = $nodeFactory->create($formData)->render();
$formResultCompiler->mergeResult($formResult);

This basically means the main FormEngine concept is a two-fold process: First create an array to gather all render-relevant information, then call the render engine using this array to come up with output.

This two-fold process has a number of advantages:

  • The data compiler step can be regulated by a controller to only enrich with stuff that is needed in any given context. This part is supported by encapsulating single data providers in data groups, single data providers can be omitted if not relevant in given scope.
  • Data providing and rendering is split: Controllers could re-use the rendering part of FormEngine while all or parts of the data providers are omitted, or their data comes from "elsewhere". Furthermore, controllers can re-use the data providing part of FormEngine and output the result in an entirely different way than HTML. The latter is for instance used when FormEngine is triggered for a TCA tree by an ajax call and thus outputs a JSON array.
  • The code constructs behind "data providing" and "rendering" can be different to allow higher re-use and more flexibility with having the "data array" as main communication base in between. This will become more obvious in the next sections where it is shown that data providers are a linked list, while rendering is a tree.

Data compiling

This is the first step of FormEngine. The data compiling creates an array containing all data the rendering needs to come up with a result.

A basic call looks like this:

$formDataGroup = GeneralUtility::makeInstance(TcaDatabaseRecord::class);
$formDataCompiler = GeneralUtility::makeInstance(FormDataCompiler::class, $formDataGroup);
$formDataCompilerInput = [
   'tableName' => $table,
   'vanillaUid' => (int)$theUid,
   'command' => $command,
];
$formData = $formDataCompiler->compile($formDataCompilerInput);

The above code is a simplified version of the relevant part of the EditDocumentController. This controller knows by its GET or POST parameters which record ("vanillaUid") of which specific table ("tableName") should be edited (command="edit") or created (command="new"), and sets this as init data to the DataCompiler. The controller also knows that it should render a full database record and not only parts of it, so it uses the TcaDatabaseRecord data provider group to trigger all data providers relevant for this case. By calling ->compile() on this data group, all providers configured for this group are called after each other, and formData ends up with a huge array of data record details.

So, what happens here in detail?

  • Variable $formDataCompilerInput maps input values to keys specified by FormDataCompiler as "init" data.
  • FormDataCompiler returns a unified array of data. This array is enriched by single data providers.
  • A data provider group is a list of single data providers for a specific scope and enriches the array with information.
  • Each data provider is called by the DataGroup to add or change data in the array.

The variable $formData roughly consists of this data after calling $formDataCompiler->compile():

  • A validated and initialized list of current database row field variables.
  • A processed version of $TCA['givenTable'] containing only those column fields a current user has access to.
  • A processed list of items for single fields like select and group types.
  • A list of relevant localizations.
  • Information of expanded inline record details if needed.
  • Resolved flex form data structures and data.
  • A lot more

Basic goal of this step is to create an array in a specified format with all data needed by the render-part of FormEngine. A controller initializes this with init data, and then lets single data providers fetch additional data and write it to the main array. The deal is here that the data within that array is not structured in an arbitrary way, and each single data provider only adds data the render part of FormEngine understands and needs later. This is why the main array keys are restricted: The main array is initialized by FormDataCompiler, and each DataProvider can only add data to sub-parts of that array.

Note

The main data array is prepared by FormDataCompiler, each key is well documented in this class. To find out which data is expected to reside in this array, those comments are worth a look.

Note

It may happen in future versions of FormEngine (core version 9+) that the responsibility for the main structure and integrity of the data array will be moved away from FormDataCompiler into the single FormDataGroup class. This may even make the FormDataCompiler obsolete in total.

Data groups and providers

So we have this empty data array, pre-set with data by a controller and then initialized by FormDataCompiler, which in turn hands over the data array to a specific FormDataGroup. What are these data providers now? Data providers are single classes that add or change data within the data array. They are called in a chain after each other. A FormDataGroup has the responsibility to find out, which specific single data providers should be used, and calls them in a specific order.

Data compiling by multiple providers

Why do we need this?

  • Which data providers are relevant depends on the specific scope: For instance, if editing a full database based record, one provider fetches the according row from the database and initializes $data['databaseRow']. But if flex form data is calculated, the flex form values are fetched from table fields directly. So, while the DatabaseEditRow data provider is needed in the first case, it's not needed or even counter productive in the second case. The :php:`FormDataGroup`s are used to manage providers for specific scopes.
  • FormDataGroups know which providers should be used in a specific scope. They usually fetch a list of providers from some global configuration array. Extensions can add own providers to this configuration array for further data munging.
  • Single data providers have dependencies to each other and must be executed in a specific order. For Instance, the PageTsConfig of a record can only be determined, if the rootline of a record has been determined, which can only happen after the pid of a given record has been consolidated, which relies on the record being fetched from the database. This makes data providers a linked list and it is the task of a FormDataGroup to manage the correct order.

Main data groups:

TcaDatabaseRecord
List of providers used if rendering a database based record.
FlexFormSegment
List of data providers used to prepare flex form data and flex form section container data.
TcaInputPlaceholderRecord
List of data providers used to prepare placeholder values for type=input and type=text fields.
InlineParentRecord
List of data providers used to prepare data needed if an inline record is opened from within an ajax call.
OnTheFly
A special data group that can be initialized with a list of to-execute data providers directly. In contrast to the others, it does not resort the data provider list by its dependencies and does not fetch the list of data providers from a global config. Used in the core at a couple of places, where a small number of data providers should be called right away without being extensible.

Note

It is a good idea to set a breakpoint at the form data result returned by the DataCompiler and to have a look at the data array to get an idea of what this array contains after compiling.

Let's have a closer look at the data providers. The main TcaDatabaseRecord group consists mostly of three parts:

Main record data and dependencies
  • Fetch record from DB or initialize a new row depending on $data['command'] being "new" or "edit", set row as $data['databaseRow']
  • Add userTs and pageTsConfig to data array
  • Add table TCA as $data['processedTca']
  • Determine record type value
  • Fetch record translations and other details and add to data array
Single field processing
  • Process values and items of simple types like type=input, type=radio, type=check and so on. Validate their databaseRow values and validate and sanitize their processedTca settings.
  • Process more complex types that may have relations to other tables like type=group and type=select, set possible selectable items in $data['processedTca'] of the according fields, sanitize their TCA settings.
  • Process type=inline and type=flex fields and prepare their child fields by using new instances of FormDataCompiler and adding their results to $data['processedTca'].
Post process after single field values are prepared
  • Execute display conditions and remove fields from $data['processedTca'] that shouldn't be shown.
  • Determine main record title and set as $data['recordTitle']
Extending data groups with own providers

The base set of DataProviders for all DataGroups is defined within typo3/sysext/core/Configuration/DefaultConfiguration.php in section ['SYS']['formEngine']['formDataGroup'], and ends up in variable $GLOBALS['TYPO3_CONF_VARS'] after core bootstrap. The provider list can be read top-down, so the DependencyOrderingService typically does not resort this list to a different order.

Adding an own provider to this list means adding an array key to that array having a specification where the new data provider should be added in the list. This is done by the arrays depends and before.

As an example, the extension "news" uses an own data provider to do additional flex form data structure preparation. The core internal flex preparation is already split into two providers: TcaFlexPrepare determines the data structure and parses it, TcaFlexProcess uses the prepared data structure, processes values and applies defaults if needed. The data provider from the extension "news" hooks in between these two to add some own preparation stuff. The registration happens with this code in ext_localconf.php:

// Modify flexform fields since core 8.5 via formEngine: Inject a data provider
// between TcaFlexPrepare and TcaFlexProcess
if (\TYPO3\CMS\Core\Utility\VersionNumberUtility::convertVersionNumberToInteger(TYPO3_version) >= 8005000) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['formDataGroup']['tcaDatabaseRecord']
    [\GeorgRinger\News\Backend\FormDataProvider\NewsFlexFormManipulation::class] = [
        'depends' => [
            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class,
        ],
        'before' => [
            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexProcess::class,
        ],
    ];
}

This is pretty powerful since it allows extensions to hook in additional stuff at any point of the processing chain, and it does not depend on the load order of extensions.

Limitations:
  • It is not easily possible to "kick out" an existing provider if other providers have dependencies to them - which is usually the case.
  • It is not easily possible to substitute an existing provider with an own one.

Note

It may happen that the core splits or deletes the one or the other DataProvider in the future. If then an extension has a dependency to a removed provider, the DependencyOrderingService, which takes care of the sorting, throws an exception. There is currently no good solution in the core on how to mitigate this issue.

Note

Data providers in general should not know about renderType, but only about type. Their goal is to prepare and sanitize data independent of a specific renderType. At the moment, the core data provider just has one or two places, where specific :php:`renderType`s are taken into account to process data, and those show that these areas are a technical dept that should be changed.

Adding data to data array

Most custom data providers change or add existing data within the main data array. A typical use case is an additional record initialization for specific fields in $data['databaseRow'] or additional items somewhere within $data['processedTca']. The main data array is documented in FormDataCompiler->initializeResultArray().

Sometimes, own DataProviders need to add additional data that does not fit into existing places. In those cases they can add stuff to $data['customData']. This key is not filled with data by core DataProviders and serves as a place for extensions to add things. Those data components can be used in own code parts of the rendering later. It is advisable to prefix own data in $data['customData'] with some unique key (for instance the extension name) to not collide with other data a different extension may add.

Rendering

This is the second step of the processing chain: The rendering part gets the data array prepared by FormDataCompiler and creates a result array containing HTML, CSS and JavaScript. This is then post-processed by a controller to feed it to the PageRenderer or to create an ajax response.

The rendering is a tree: The controller initializes this by setting one container as renderType entry point within the data array, then hands over the full data array to the NodeFactory which looks up a class responsible for this renderType, and calls render() on it. A container class creates only a fraction of the full result, and delegates details to another container. The second one does another detail and calls a third one. This continues to happen until a single field should be rendered, at which point an element class is called taking care of one element.

Render tree example

Each container creates some "outer" part of the result, calls some sub-container or element, merges the sub-result with its own content and returns the merged array up again. The data array is given to each sub class along the way, and containers can add further render relevant data to it before giving it "down". The data array can not be given "up" in a changed way again. Inheritance of a data array is always top-bottom. Only HTML, CSS or JavaScript created by a sub-class is returned by the sub-class "up" again in a "result" array of a specified format.

class SomeContainer extends AbstractContainer
{
    public function render()
    {
        $result = $this->initializeResultArray();
        $data = $this->data;
        $data['renderType'] = 'subContainer';
        $childArray = $this->nodeFactory->create($data)->render();
        $resultArray = $this->mergeChildReturnIntoExistingResult($result, $childArray, false);
        $result['html'] = '<h1>A headline</h1>' . $childArray['html'];
        return $result;
    }
}

Above example lets NodeFactory find and compile some data from "subContainer", and merges the child result with its own. The helper methods initializeResultArray() and mergeChildReturnIntoExistingResult() help with combining CSS and JavaScript.

An upper container does not directly create an instance of a sub node (element or container) and never calls it directly. Instead, a node that wants to call a sub node only refers to it by a name, sets this name into the data array as $data['renderType'] and then gives the data array to the NodeFactory which determines an appropriate class name, instantiates and initializes the class, gives it the data array, and calls render() on it.

Note

The SingleFieldContainer and FlexFormElementContainer will probably vanish with core version 9.

Note

Data set by containers and given down to children will likely change in core version 9: All fields not registered in the main data array of FormDataCompiler and only added within containers will move into section renderData. Furthermore, it is planned to remove parameterArray and substitute it with something better. This will affect most elements and will probably break a lot of these elements.

Class inheritance
Main render class inheritance

All classes must implement NodeInterface to be routed through the NodeFactory. The AbstractNode implements some basic helpers for nodes, the two classes AbstractContainer and AbstractFormElement implement helpers for containers and elements respectively.

The call concept is simple: A first container is called, which either calls a container below or a single element. A single element never calls a container again.

NodeFactory

The NodeFactory plays an important abstraction role within the render chain: Creation of child nodes is always routed through it, and the NodeFactory takes care of finding and validating the according class that should be called for a specific renderType. This is supported by an API that allows registering new renderTypes and overriding existing renderTypes with own implementations. This is true for all classes, including containers, elements, fieldInformation, fieldWizards and fieldControls. This means the child routing can be fully adapted and extended if needed. It is possible to transparently "kick-out" a core container and to substitute it with an own implementation.

As example, the TemplaVoila implementation needs to add additional render capabilities of the flex form rendering to add for instance an own multi-language rendering of flex fields. It does that by overriding the default flex container with own implementation:

// Default registration of "flex" in NodeFactory:
// 'flex' => \TYPO3\CMS\Backend\Form\Container\FlexFormEntryContainer::class,

// Register language aware flex form handling in FormEngine
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1443361297] = [
    'nodeName' => 'flex',
    'priority' => 40,
    'class' => \TYPO3\CMS\Compatibility6\Form\Container\FlexFormEntryContainer::class,
];

This re-routes the renderType "flex" to an own class. If multiple registrations for a single renderType exist, the one with highest priority wins.

Note

The NodeFactory uses $data['renderType']. This has been introduced with core version 7 in TCA, and a couple of TCA fields actively use this renderType. However, it is important to understand the renderType is only used within the FormEngine and type is still a must-have setting for columns fields in TCA. Additionally, type can not be overridden in columnsOverrides. Basically, type specifies how the DataHandler should put data into the database, while renderType specifies how a single field is rendered. This additionally means there can exist multiple different renderTypes for a single type, and it means it is possible to invent a new renderType to render a single field differently, but still let the DataHandler persist it the usual way.

Adding a new renderType in ext_localconf.php

// Add new field type to NodeFactory
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1487112284] = [
    'nodeName' => 'selectTagCloud',
    'priority' => '70',
    'class' => \MyVendor\CoolTagCloud\Form\Element\SelectTagCloudElement::class,
];

And use it in TCA for a specific field, keeping the full database functionality in DataHandler together with the data preparation of FormDataCompiler, but just routing the rendering of that field to the new element:

$GLOBALS['TCA']['myTable']['columns']['myField'] = [
    'label' => 'Cool Tag cloud',
    'config' => [
        'type' => 'select',
        'renderType' => 'selectTagCloud',
        'foreign_table' => 'tx_cooltagcloud_availableTags',
    ],
];

The above examples are a static list of nodes that can be changed by settings in ext_localconf.php. If that is not enough, the NodeFactory can be extended with a resolver that is called dynamically for specific renderTypes. This resolver gets the full current data array at runtime and can either return NULL saying "not my job", or return the name of a class that should handle this node.

An example of this are the core internal rich text editors. Both "ckeditor" and "rtehtmlarea" register a resolver class that are called for node name "text", and if the TCA config enables the editor, and if the user has enabled rich text editing in his user settings, then the resolvers return their own RichTextElement class names to render a given text field:

// Register FormEngine node type resolver hook to render RTE in FormEngine if enabled
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeResolver'][1480314091] = [
    'nodeName' => 'text',
    'priority' => 50,
    'class' => \TYPO3\CMS\RteCKEditor\Form\Resolver\RichTextNodeResolver::class,
];

The trick is here that "ckeditor" registers his resolver with ah higher priority (50) than "rtehtmlarea" (40), so the "ckeditor" resolver is called first and wins if both extensions are loaded and if both return a valid class name.

Result array

Each node, no matter if it is a container, an element, or a node expansion, must return an array with specific data keys it wants to add. It is the job of the parent node that calls the sub node to merge child node results into its own result. This typically happens by merging $childResult['html'] into an appropriate position of own HTML, and then calling $this->mergeChildReturnIntoExistingResult() to add other array child demands like stylesheetFiles into its own result.

Container and element nodes should use the helper method $this->initializeResultArray() to have a result array initialized that is understood by a parent node.

Only if extending existing element via node expansion, the result array of a child can be slightly different. For instance, a FieldControl "wizards" must have a iconIdentifier result key key. Using $this->initializeResultArray() is not appropriate in these cases but depends on the specific expansion type. See below for more details on node expansion.

The result array for container and element nodes looks like this. $resultArray = $this->initializeResultArray() takes care of basic keys:

[
    'html' => '',
    'additionalInlineLanguageLabelFiles' => [],
    'stylesheetFiles' => [],
    'requireJsModules' => [],
]

CSS and language labels (which can be used in JS) are added with their file names in format EXT:extName/path/to/file. JavaScript is added only via RequireJS modules, the registration allows an init method to be called if the module is loaded by the browser.

Note

The result array handled by $this->mergeChildReturnIntoExistingResult() contains a couple of more keys, those will vanish with further FormEngine refactoring steps. If using them, be prepared to adapt extensions later.

Note

Nodes must never add JavaScript or CSS or similar stuff using the PageRenderer. This fails as soon as this container / element / wizard is called via AJAX, for instance within inline. Instead, those resources must be registered via the result array only, using stylesheetFiles and requireJsModules.

Node expansion

The "node expansion" classes FieldControl, FieldInformation and FieldWizard are called by containers and elements and allow "enriching" containers and elements. Which enrichments are called can be configured via TCA.

This API is the substitution of the old "TCA wizards array" and has been introduced with core version 8.

FieldInformation
Additional information. In elements, their output is shown between the field label and the element itself. They can not add functionality, but only simple and restricted HTML strings. No buttons, no images. An example usage could be an extension that auto-translates a field content and outputs an information like "Hey, this field was auto-filled for you by an automatic translation wizard. Maybe you want to check the content".
FieldWizard
Wizards shown below the element. "enrich" an element with additional functionality. The localization wizard and the file upload wizard of type=group fields are examples of that.
FieldControl
"Buttons", usually shown next to the element. For type=group the "list" button and the "element browser" button are examples. A field control must return an icon identifier.

Currently, all elements usually implement all three of these, except in cases where it does not make sense. This API allows adding functionality to single nodes, without overriding the whole node. Containers and elements can come with default expansions (and usually do). TCA configuration can be used to add own stuff. On container side the implementation is still basic, only OuterWrapContainer and InlineControlContainer currently implement FieldInformation and FieldWizard.

See the TCA reference ctrl section for more information on how to configure these for containers in TCA.

Example. The InputTextElement (standard input element) defines a couple of default wizards and embeds them in its main result HTML:

class InputTextElement extends AbstractFormElement
{
    protected $defaultFieldWizard = [
        'localizationStateSelector' => [
            'renderType' => 'localizationStateSelector',
        ],
        'otherLanguageContent' => [
            'renderType' => 'otherLanguageContent',
            'after' => [
                'localizationStateSelector'
            ],
        ],
        'defaultLanguageDifferences' => [
            'renderType' => 'defaultLanguageDifferences',
            'after' => [
                'otherLanguageContent',
            ],
        ],
    ];

    public function render()
        $resultArray = $this->initializeResultArray();

        $fieldWizardResult = $this->renderFieldWizard();
        $fieldWizardHtml = $fieldWizardResult['html'];
        $resultArray = $this->mergeChildReturnIntoExistingResult($resultArray, $fieldWizardResult, false);

        $mainFieldHtml = [];
        $mainFieldHtml[] = '<div class="form-control-wrap">';
        $mainFieldHtml[] =  '<div class="form-wizards-wrap">';
        $mainFieldHtml[] =      '<div class="form-wizards-element">';
        // Main HTML of element done here ...
        $mainFieldHtml[] =      '</div>';
        $mainFieldHtml[] =      '<div class="form-wizards-items-bottom">';
        $mainFieldHtml[] =          $fieldWizardHtml;
        $mainFieldHtml[] =      '</div>';
        $mainFieldHtml[] =  '</div>';
        $mainFieldHtml[] = '</div>';

        $resultArray['html'] = implode(LF, $mainFieldHtml);
        return $resultArray;
    }
}

This element defines three wizards to be called by default. The renderType concept is re-used, the values localizationStateSelector are registered within the NodeFactory and resolve to class names. They can be overridden and extended like all other nodes. The $defaultFieldWizards are merged with TCA settings by the helper method renderFieldWizards(), which uses the DependencyOrderingService again.

It is possible to:

  • Override existing expansion nodes with own ones from extensions, even using the resolver mechanics is possible.
  • It is possible to disable single wizards via TCA
  • It is possible to add own expansion nodes at any position relative to the other nodes by specifying "before" and "after" in TCA.
Add fieldControl example

To illustrate the principals discussed in this chapter see the following example which registers a fieldControl (button) next to a field in the pages table to trigger a data import via ajax.

Add a new renderType in ext_localconf.php:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1485351217] = [
   'nodeName' => 'importDataControl',
   'priority' => 30,
   'class' => \T3G\Something\FormEngine\FieldControl\ImportDataControl::class
];

Register the control in TCA/Overrides/pages.php:

'somefield' => [
   'label'   => $langFile . ':pages.somefield',
   'config'  => [
      'type' => 'input',
      'eval' => 'int, unique',
      'fieldControl' => [
         'importControl' => [
            'renderType' => 'importDataControl'
         ]
      ]
   ]
],

Add the php class for rendering the control in FormEngine/FieldControl/ImportDataControl.php:

declare(strict_types=1);

namespace T3G\Something\FormEngine\FieldControl;

use TYPO3\CMS\Backend\Form\AbstractNode;

class ImportDataControl extends AbstractNode
{
   public function render()
   {
      $result = [
         'iconIdentifier' => 'import-data',
         'title' => $GLOBALS['LANG']->sL('LLL:EXT:something/Resources/Private/Language/locallang_db.xlf:pages.importData'),
         'linkAttributes' => [
            'class' => 'importData ',
            'data-id' => $this->data['databaseRow']['somefield']
         ],
         'requireJsModules' => 'TYPO3/CMS/Something/ImportData',
      ];
      return $result;
   }
}

Add the JavaScript for defining the behavior of the control in Resources/Public/JavaScript/ImportData.js:

/**
* Module: TYPO3/CMS/Something/ImportData
*
* JavaScript to handle data import
* @exports TYPO3/CMS/Something/ImportData
*/
define(function () {
   'use strict';

   /**
   * @exports TYPO3/CMS/Something/ImportData
   */
   var ImportData = {};

   /**
   * @param {int} id
   */
   ImportData.import = function (id) {
      $.ajax({
         type: 'POST',
         url: TYPO3.settings.ajaxUrls['something-import-data'],
         data: {
            'id': id
         }
      }).done(function (response) {
         if (response.success) {
            top.TYPO3.Notification.success('Import Done', response.output);
         } else {
            top.TYPO3.Notification.error('Import Error!');
         }
      });
   };

   /**
   * initializes events using deferred bound to document
   * so AJAX reloads are no problem
   */
   ImportData.initializeEvents = function () {

      $('.importData').on('click', function (evt) {
         evt.preventDefault();
         ImportData.import($(this).attr('data-id'));
      });
   };

   $(ImportData.initializeEvents);

   return ImportData;
});

Add an ajax route for the request in Configuration/Backend/AjaxRoutes.php:

<?php
return [
   'something-import-data' => [
      'path' => '/something/import-data',
      'target' => \T3G\Something\Controller\Ajax\ImportDataController::class . '::importDataAction'
   ],
];

Add the ajax controller class in Classes/Controller/Ajax/ImportDataController.php:

declare(strict_types=1);

namespace T3G\Something\Controller\Ajax;

use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;

class ImportDataController
{
   /**
   * @param ServerRequestInterface $request
   * @param ResponseInterface $response
   * @return ResponseInterface
   */
   public function importDataAction(ServerRequestInterface $request, ResponseInterface $response)
   {
      $queryParameters = $request->getParsedBody();
      $id = (int)$queryParameters['id'];

      if (empty($id)) {
         $response->getBody()->write(json_encode(['success' => false]));
         return $response;
      }
      $param = ' -id=' . $id;

      // trigger data import (simplified as example)
      $output = shell_exec('.' . DIRECTORY_SEPARATOR . 'import.sh' . $param);

      $response->getBody()->write(json_encode(['success' => true, 'output' => $output]));
      return $response;
   }
}

Database Access

Contents:

Introduction

Database queries in TYPO3 are done with an API based on doctrine-dbal. The API is provided by the system extension core which is always loaded and thus always available.

Extension authors can use this low-level API to manage query operations directly on the configured DBMS.

Doctrine-dbal is feature rich. Drivers for various target systems enable TYPO3 to run on a long list of ANSI SQL compatible DBMS. If used properly, queries created with this API are translated to the specific database engine by doctrine without an extension developer taking care of that specifically.

The API provided by the core is basically a pretty small and lightweight facade in front of doctrine-dbal that adds some convenient methods as well as some TYPO3 CMS specific sugar. The facade additionally provides methods to retrieve specific connection objects per configured database connection based on the table that is queried. This enables instance administrators to configure different database engines for different tables while this is transparent for extension developers.

doctrine-dbal has been introduced with TYPO3 CMS version 8 and substitutes the old API based on $GLOBALS['TYPO3_DB']. Extension authors are encouraged to switch away from TYPO3_DB to the new API. A dedicated chapter helps with typical migration questions. With database abstraction being built in doctrine-dbal the old and optional extensions dbal and adodb are obsolete.

This document does not outline each and every single method the API provides. It sticks to those that are commonly used in extensions and some parts like the rewritten schema migrator are left out since they are usually of little to no interest for extensions.

Understanding Doctrine-Dbal and Doctrine-Orm

Doctrine is a two-fold project with doctrine-dbal being the low-level database abstraction and query building interface to specific database engines, while doctrine-orm is a high-level object relational mapping on top of doctrine-dbal.

The TYPO3 CMS core - only - implements the dbal part. doctrine-orm is neither required nor implemented nor used at the time of this writing.

Low-level and high-level database calls

This documentation is about low-level database calls. In many cases it is better to use higher level API's like the DataHandler or extbase repositories and to let the framework handle persistence details internally.

Tip

Always remember the high-level database calls and use them when appropriate!

Credits

Implementing the doctrine-dbal API into TYPO3 has been a huge project in 2016. Special thanks goes to awesome Mr. Morton Jonuschat for the initial design, integration and support and to more than 40 different people who actively contributed to migrate more than 1700 calls from TYPO3_DB-style to Doctrine within half a year. This was a huge community achievement, thanks everyone involved!

Configuration

Configuring doctrine-dbal for TYPO3 CMS is all about specifying the single database endpoints and handing over connection credentials. The frameworks supports the parallel usage of multiple database connections, a specific connection is mapped depending on its table name. The table space can be seen as a transparent layer that determines which specific connection is chosen for a query to a single or a group of tables: It allows "swapping-out" single tables from the Default connection to point them to a different database endpoint.

As with other central configuration options, the database endpoint and mapping configuration happens within typo3conf/LocalConfiguration.php and ends up in $GLOBALS['TYPO3_CONF_VARS'] after core bootstrap. The specific sub-array is $GLOBALS['TYPO3_CONF_VARS']['DB'].

A typical, basic example using only the Default connection with a single database endpoint:

// LocalConfiguration.php
// [...]
'DB' => [
   'Connections' => [
      'Default' => [
         'charset' => 'utf8',
         'dbname' => 'theDatabaseName',
         'driver' => 'mysqli',
         'host' => 'theHost',
         'password' => 'theConnectionPassword',
         'port' => 3306,
         'user' => 'theUser',
      ],
   ],
],
// [...]

Remarks:

  • The Default connection must be configured, this can not be left out or renamed.
  • For mysqli, if the host is set to localhost and if the default PHP options in this area are not changed, the connection will be socket based. This saves a little overhead. To force a TCP/IP based connection even for localhost, the IPv4 or IPv6 address 127.0.0.1 and ::1/128 respectively must be used as host value.
  • The connect options are hand over to doctrine-dbal without much manipulation from TYPO3 CMS side. Please refer to the doctrine connection docs for a full overview of settings.
  • If charset option is not specified it defaults to utf8.
  • The option wrapperClass is used by the TYPO3 CMS framework to "hang in" the extended Connection class TYPO3\CMS\Database\Connection as main facade around doctrine-dbal.

A slightly more complex example with two connections, mapping the sys_log table to a different endpoint:

// LocalConfiguration.php
// [...]
'DB' => [
   'Connections' => [
      'Default' => [
         'charset' => 'utf8',
         'dbname' => 'default_dbname',
         'driver' => 'mysqli',
         'host' => 'default_host',
         'password' => '***',
         'port' => 3306,
         'user' => 'default_user',
      ],
      'Syslog' => [
         'charset' => 'utf8',
         'dbname' => 'syslog_dbname',
         'driver' => 'mysqli',
         'host' => 'syslog_host',
         'password' => '***',
         'port' => 3306,
         'user' => 'syslog_user',
      ],
   ],
   'TableMapping' => [
      'sys_log' => 'Syslog'
   ]
],
// [...]

Remarks:

  • The array key Syslog is just a name, it can be different but it's good practice to give it a useful speaking name.
  • It is possible to map multiple tables to a different endpoint by adding further table name / connection name pairs to TableMapping.
  • Mind this "connection per table" approach is limited: If in the above example a join query that spans over different connections is fired, an exception is raised. It is up to the administrator to group affected tables to the same connection in those cases, or a developer should implement some fallback logic to suppress the join().

Attention

Connections to databases postgres, maria and mysql are actively tested. However, mssql is currently not actively tested.

Furthermore, the TYPO3 CMS installer supports only a single mysql or mariadb connection at the moment and the connection details can not be properly edited within the All configuration section of the install tool.

Basic CRUD

A list of basic usage examples of the query API. This is just a kickstart. Details on the single methods are found in the following chapters, especially QueryBuilder and Connection.

Note

The examples use the shorthand syntax for class names. Please refer to Class overview for the full namespace.

INSERT a row

A straight insert to a table:

GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->insert(
        'tt_content',
        [
            'pid' => (int)42,
            'bodytext' => 'bernd',
        ]
    );
INSERT INTO `tt_content` (`pid`, `bodytext`) VALUES ('42', 'bernd')
SELECT a single row

Straight fetch of a single row from tt_content table:

$uid = 4;
$row = GeneralUtility::makeInstance(ConnectionPool::class)
    ->getConnectionForTable('tt_content')
    ->select(
        ['uid', 'pid', 'bodytext'], // fields to select
        'tt_content', // from
        [ 'uid' => (int)$uid ] // where
    )
    ->fetch();

Result in $row:

array(3 items)
   uid => 4 (integer)
   pid => 35 (integer)
   bodytext => 'some content' (12 chars)

The engine quotes field names, adds default TCA restrictions like "deleted=0", and prepares a query executed with this final statement:

SELECT `uid`, `pid`, `bodytext`
    FROM `tt_content`
    WHERE (`uid` = '4')
        AND ((`tt_content`.`deleted` = 0)
        AND (`tt_content`.`hidden` = 0)
        AND (`tt_content`.`starttime` <= 1473447660)
        AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1473447660)))

Note

Default restrictions deleted, hidden, startime and endtime based on TCA setting of a table are only applied to select() calls, they are not added for delete() or other query types.

SELECT multiple rows with some WHERE magic

Advanced query using the QueryBuilder and manipulating the default restrictions:

$uid = 4;
// Get a query builder for a query on table "tt_content"
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
// Remove all default restrictions (delete, hidden, starttime, stoptime), but add DeletedRestriction again
$queryBuilder->getRestrictions()
    ->removeAll()
    ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
// Execute a query with "bodytext=klaus OR uid=4" and proper quoting
$rows = $queryBuilder
    ->select('uid', 'pid', 'bodytext')
    ->from('tt_content')
    ->where(
        $queryBuilder->expr()->orX(
            $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
            $queryBuilder->expr()->eq('uid', $queryBuilder->createNamedParameter($uid, \PDO::PARAM_INT))
        )
    )
    ->execute()
    ->fetchAll();

Result in $rows:

array(2 items)
   0 => array(3 items)
      uid => 4 (integer)
      pid => 35 (integer)
      bodytext => 'bernd' (5 chars)
   1 => array(3 items)
      uid => 366 (integer)
      pid => 13 (integer)
      bodytext => 'klaus' (5 chars)

The executed query looks like:

SELECT `uid`, `pid`, `bodytext`
    FROM `tt_content`
    WHERE ((`bodytext` = 'klaus') OR (`uid` = 4))
        AND (`tt_content`.`deleted` = 0)
UPDATE multiple rows
GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->update(
        'tt_content',
        [ 'bodytext' => 'bernd' ], // set
        [ 'bodytext' => 'klaus' ] // where
    );
UPDATE `tt_content` SET `bodytext` = 'bernd' WHERE `bodytext` = 'klaus'
DELETE a row
GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->delete(
        'tt_content', // from
        [ 'uid' => (int)4711 ] // where
    );
DELETE FROM `tt_content` WHERE `uid` = '4711'

Class overview

Doctrine provides a set of php objects to represent, create and handle SQL queries and their results. The basic class structure was slightly enriched by TYPO3 to add CMS specific features. Extension authors will typically interact with these classes and objects:

Connection
TYPO3\CMS\Core\Database\Connection: Object representing a specific connection to one connected database. Provides "shortcut" methods for simple standard queries like SELECT or UPDATE. An instance of the QueryBuilder can be retrieved to build more complex queries.
ConnectionPool
TYPO3\CMS\Core\Database\ConnectionPool: Main entry point for extensions to retrieve a specific connection a query should be executed on. Typically used to return a Connection or a QueryBuilder object.
ExpressionBuilder
TYPO3\CMS\Core\Database\Query\Expression\ExpressionBuilder: Object to model complex expressions. Mainly used for WHERE and JOIN conditions.
QueryBuilder
TYPO3\CMS\Core\Database\Query\QueryBuilder: Object to create all sort of complex queries executed on a specific connection. Provides the main CRUD methods for select(), delete() and friends.
QueryHelper
TYPO3\CMS\Core\Database\Query\QueryHelper: Set of static helper methods that can simplify the transition from old TYPO3_DB based code to the doctrine base API.
Restriction ...
TYPO3\CMS\Core\Database\Query\Restriction\...: Set of classes that add expressions like "deleted=0" to a query based on TCA settings of a table. This automatically adds TYPO3 specific restrictions like starttime and endtime, as well as deleted and hidden flags. Further restrictions for language overlays and workspaces are available. This documentation refers to these classes as the RestrictionBuilder.
Statement
Doctrine\DBAL\Driver\Statement: Result object retrieved if a SELECT or COUNT query has been executed. Single rows are returned as array by calling ->fetch() until the method returns false.

ConnectionPool

TYPO3's interface to execute queries via doctrine-dbal typically starts by asking the ConnectionPool for a QueryBuilder or a Connection object, handing over the table name to be queried:

// Get a query builder for a table
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tx_myext_comments');
// or
// Get a connection for a table
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tx_myext_comments');

The QueryBuilder is the default workhorse object used by extension authors to express complex queries, while a Connection instance can be used as shortcut to deal with some simple query cases and little written down code.

Pooling

TYPO3 can handle multiple connections to different database endpoints at the same time. This can be configured on a per-table basis in $TYPO3_CONF_VARS. It allows running tables on different databases, without an extension developer taking care of that.

The ConnectionPool implements this feature: It looks up a configured table-to-database mapping and can return a Connection or a QueryBuilder instance for that specific connection. Those objects internally know which target connection they are dealing with and will for instance quote field names accordingly.

The transparency of tables to different database endpoints is limited, though:

Executing a table JOIN between two tables that point to different connections will throw an exception. This restriction may in practice create implicit "groups" of tables that need to point to one connection at once if an extension or the TYPO3 core joins those tables.

This can turn out as a headache if multiple different extensions use for instance the core category or collection API with their mm table joins between core internal tables and their extension's counterparts.

That situation is not easy to deal with. At the time of this writing the core development will eventually implement some non-join fallbacks for typical cases that would be good to decouple, though.

Tip

In case joins cannot be decoupled but still affected tables must run on different databases, and if the code can not be easily adapted, some DBMS like PostgreSQL allow executing those queries by having own connection handlers to different other endpoints on its own.

QueryBuilder

The QueryBuilder is a rather huge class that takes care of the main query dealing.

An instance can get hold of by calling the ConnectionPool->getQueryBuilderForTable() and handing over the table. Never instantiate and initialize the QueryBuilder directly via makeInstance()!

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('aTable');

This documentation does not mention every single available method but sticks to those used in casual queries and normal code flow. There are a couple of not mentioned methods, most of them are either very seldom used or marked as internal. Extension authors typically don't have to deal with anything not mentioned here.

Warning

From security point of view, the documentation of ->createNamedParameter() and ->quoteIdentifier() are an absolute must read and follow section. Make very sure this is understood and use this for each and every query to prevent SQL injections!

The QueryBuilder comes with a happy little list of small methods:

  • Set type of query: ->select(), ->count(), ->update(), ->insert() and delete()
  • Prepare WHERE conditions
  • Manipulate default WHERE restrictions added by TYPO3 for ->select()
  • Add LIMIT, GROUP BY and other SQL stuff
  • ->execute() a query and retrieve a Statement (a query result) object

Most methods of the QueryBuilder return $this and can be chained:

$queryBuilder->select('uid')->from('pages');

Note

The QueryBuilder holds internal state and should not be re-used for different queries: Use one query builder per query. Get a fresh one by calling $connection->createQueryBuilder() if the same table is affected, or use $connectionPool->getQueryBuilderForTable() for a query on to a different table. Don't worry, creating those object instances is rather quick.

select() and addSelect()

Create a SELECT query.

Select all fields:

// SELECT *
$queryBuilder->select('*')

->select() and a number of other methods of the QueryBuilder are variadic and can handle any number of arguments. For ->select(), every argument is interpreted as a single field name to select:

// SELECT `uid`, `pid`, `aField`
$queryBuilder->select('uid', 'pid', 'aField');

Argument unpacking can be used if the list of fields is available as array already:

$fields = ['uid', 'pid', 'aField', 'anotherField'];
$queryBuilder->select(...$fields);

->select() supports AS and quotes identifiers automatically. This can become especially handy in join() operations:

// SELECT `tt_content`.`bodytext` AS `t1`.`text`
$queryBuilder->select('tt_content.bodytext AS t1.text')

->select() sets the list of fields that should be selected and ->addSelect() can add further items to an existing list.

Mind that ->select() replaces any formerly registered list instead of appending. Thus, it usually doesn't make much sense to call select() twice in a code flow, or to call it after an ->addSelect(). The methods ->where() and ->andWhere() share the same behavior: ->where() replaces all formerly registered constraints, ->andWhere() appends additional constraints.

A useful combination of ->select() and ->addSelect() can be:

$queryBuilder->select(...$defaultList);
if ($needAdditionalFields) {
   $queryBuilder->addSelect(...$additionalFields);
}

Calling ->execute() on a ->select() query returns a Statement object. To receive single rows a ->fetch() loop on that object is used, or ->fetchAll() to return a single array with all rows. A typical code flow of a SELECT query looks like:

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->execute();
while ($row = $statement->fetch()) {
   // Do something with that single row
   debug($row);
}

Note

->select() and ->count() queries trigger TYPO3 CMS magic that adds further default where clauses if the queried table is also registered via $GLOBALS['TCA']. See the RestrictionBuilder section for details on that topic.

count()

Create a COUNT query, a typical usage:

// SELECT COUNT(`uid`) FROM `tt_content` WHERE (`bodytext` = 'klaus')
//     AND ((`tt_content`.`deleted` = 0) AND (`tt_content`.`hidden` = 0)
//     AND (`tt_content`.`starttime` <= 1475580240)
//     AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1475580240)))
$count = $queryBuilder
   ->count('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
    )
   ->execute()
   ->fetchColumn(0);

Remarks:

  • Similar to the ->select() query type, ->count() automatically triggers RestrictionBuilder magic that adds default deleted, hidden, starttime and endtime restrictions if that is defined in TCA.
  • Similar to ->select() query types, ->execute() with ->count() returns a Statement object. To fetch the number of rows directly, use ->fetchColumn(0).
  • First argument to ->count() is required, typically ->count(*) or ->count('uid') is used, the field name is automatically quoted.
  • There is no support for DISTINCT, a ->groupBy() has to be used instead.
  • If combining ->count() with a ->groupBy(), the result may return multiple rows. The order of those rows depends on the used DBMS. To ensure same order of result rows on multiple different databases, a ->groupBy() should thus always be combined with a ->orderBy().
delete()

Create a DELETE FROM query. The method requires the table name to drop data from. Classic usage:

// DELETE FROM `tt_content` WHERE `bodytext` = 'klaus'
$affectedRows = $queryBuilder
   ->delete('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->execute();

Remarks:

  • For simple cases, it is often easier to write and read if using the ->delete() method of the Connection object.
  • In contrast to ->select(), ->delete() does not add WHERE restrictions like AND `deleted` = 0 automatically.
  • ->delete() does not magically transform a DELETE FROM `tt_content` WHERE `uid` = 4711 to something like UPDATE `tt_content` SET `deleted` = 1 WHERE `uid` = 4711 internally. A soft-delete must be handled on application level code with a dedicated lookup in $GLOBALS['TCA']['theTable']['ctrl']['deleted'] to check if a specific table can handle the soft-delete, together with an ->update() instead.
  • Multi-table delete is not supported: DELETE FROM `table1`, `table2` can not be created.
  • ->delete() ignores ->join()
  • ->delete() ignores setMaxResults(): DELETE with LIMIT does not work.
update() and set()

Create an UPDATE query. Typical usage:

// UPDATE `tt_content` SET `bodytext` = 'peter' WHERE `bodytext` = 'klaus'
$queryBuilder
   ->update('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')
   )
   ->set('bodytext', 'peter')
   ->execute();

->update() requires the table to update as first argument and a table alias as optional second argument. The table alias can then be used in ->set() and ->where() expressions:

// UPDATE `tt_content` `t` SET `t`.`bodytext` = 'peter' WHERE `u`.`bodytext` = 'klaus'
$queryBuilder
   ->update('tt_content', 'u')
   ->where(
      $queryBuilder->expr()->eq('u.bodytext', $queryBuilder->createNamedParameter('klaus')
   )
   ->set('u.bodytext', 'peter')
   ->execute();

->set() requires a field name as first argument and automatically quotes it internally. The second mandatory argument is the value a field should be set to, the value is automatically transformed to a named parameter of a prepared statement. This way, ->set() key/value pairs are automatically SQL injection save by default.

If a field should be set to the value of another field from the row, the quoting needs to be turned off and ->quoteIdentifier() has to be used:

// UPDATE `tt_content` SET `bodytext` = `header` WHERE `bodytext` = 'klaus'
$queryBuilder
   ->update('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->set('bodytext', $queryBuilder->quoteIdentifier('header'), false)
   ->execute();

Remarks:

  • For simple cases, it is often easier to use the ->update() method of the Connection object.
  • ->set() can be called multiple times if multiple fields should be updated.
  • ->set() requires a field name as first argument and automatically quotes it internally.
  • ->set() requires the value a field should be set to as second parameter.
  • ->update() ignores ->join() and ->setMaxResults().
  • The API does not magically add delete = 0 or other restrictions magically.
insert() and values()

Create an INSERT query. Typical usage:

$affectedRows = $queryBuilder
   ->insert('tt_content')
   ->values([
      'bodytext' => 'klaus',
      'header' => 'peter',
   ])
   ->execute();

Remarks:

  • It is often easier to use ->insert() or ->bulkInsert() of the Connection object.
  • ->values() expects an array of key/value pairs. Both keys (field names / identifiers) and values are automatically quoted. In rare cases, quoting of values can be turned off by setting the second argument to false. In those cases the quoting has to be done manually, typically by using ->createNamedParameter() on the values, use with care ...
  • ->execute() after ->insert() returns the number of inserted rows, which is typically 1.
  • QueryBuilder does not contain a method to insert multiple rows at once, use ->bulkInsert() of Connection object instead to achieve that.
from()

->from() is a must have call for ->select() and ->count() query types. ->from() needs a table name and an optional alias name. The method is typically called once per query build and the table name is typically the same as what was given to ->getQueryBuilderForTable(). If the query joins multiple tables, the argument should be the name of the first table within the ->join() chain:

// FROM `myTable`
$queryBuilder->from('myTable');

// FROM `myTable` AS `anAlias`
$queryBuilder->from('myTable', 'anAlias');

->from() can be called multiple times and will create the cartesian product of tables if not restricted by an according ->where() or ->andWhere() expression. In general, it is a good idea to use ->from() only once per query and model multi-table selection with an explicit ->join() instead.

where(), andWhere() and orWhere()

The three methods are used to create WHERE restrictions for SELECT, COUNT, UPDATE and DELETE query types. Each argument is typically an ExpressionBuilder object that will be cast to a string on ->execute():

// SELECT `uid`, `header`, `bodytext`
// FROM `tt_content`
// WHERE
//    (
//       ((`bodytext` = 'klaus') AND (`header` = 'a name'))
//       OR (`bodytext` = 'peter') OR (`bodytext` = 'hans')
//    )
//    AND (`pid` = 42)
//    AND ... RestrictionBuilder TCA restrictions ...
$statement = $queryBuilder
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
      $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('a name'))
   )
   ->orWhere(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('peter')),
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('hans'))
   )
   ->andWhere(
      $queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Note the parenthesis of the above example: ->andWhere() encapsulates both ->where() and ->orWhere() with an additional restriction.

Argument unpacking can become handy with these methods:

$whereExpressions = [
   $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
   $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('a name'))
];
if ($needsAdditionalExpression) {
   $whereExpressions[] = $someAdditionalExpression;
}
$queryBuilder->where(...$whereExpressions);

Remarks:

  • The three methods are variadic. They can handle any number of arguments. If for instance ->where() receives four arguments, they are handled as single expressions, all of them combined with AND.
  • ->where() should be called only once per query and it resets any previously set ->where(), ->andWhere() and ->orWhere() expression. Having a ->where() call after a previous ->where(), ->andWhere() or ->orWhere() typically indicates a bug or a rather weird code flow. Doing so is discouraged.
  • While creating complex WHERE restrictions, ->getSQL() and ->getParameters() are helpful debugging friends to verify parenthesis and single query parts.
  • If using only ->eq() expressions, it is often easier to switch to the according Connection object method to simplify quoting and increase readability.
  • It is possible to feed the methods with strings directly, but that is discouraged and typically only used in rare cases where expression strings are created at a different place that can not be resolved easily. In the core, those places are usually combined with QueryHelper::stripLogicalOperatorPrefix() to remove leading AND or OR parts. Using this gives an additional risk of missing or wrong quoting and is a potential security issue. Use with care if ever.
join(), innerJoin(), rightJoin() and leftJoin()

Joining multiple tables in a ->select() or ->count() query is done with one of these methods. Multiple joins are supported by calling the methods more than once. All methods require four arguments: The name of the left side table (or its alias), the name of the right side table, an alias for the right side table name and the join restriction as fourth argument:

// SELECT `sys_language`.`uid`, `sys_language`.`title`
// FROM `sys_language`
// INNER JOIN `pages_language_overlay` `overlay`
//     ON `overlay`.`sys_language_uid` = `sys_language`.`uid`
// WHERE
//     (`overlay`.`pid` = 42)
//     AND (
//          (`overlay`.`deleted` = 0)
//          AND (
//              (`sys_language`.`hidden` = 0) AND (`overlay`.`hidden` = 0)
//          )
//          AND (`overlay`.`starttime` <= 1475591280)
//          AND ((`overlay`.`endtime` = 0) OR (`overlay`.`endtime` > 1475591280))
//     )
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$statement = $queryBuilder
   ->select('sys_language.uid', 'sys_language.title')
   ->from('sys_language')
   ->join(
      'sys_language',
      'pages_language_overlay',
      'overlay',
      $queryBuilder->expr()->eq('overlay.sys_language_uid', $queryBuilder->quoteIdentifier('sys_language.uid'))
   )
   ->where(
      $queryBuilder->expr()->eq('overlay.pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Notes to the above example:

  • The query operates on table sys_language as main table, this table name is given to getQueryBuilderForTable().
  • The query joins table pages_language_overlay as INNER JOIN, giving it the alias overlay.
  • The join condition is `overlay`.`sys_language_uid` = `sys_language`.`uid`. It would have been identical to swap the expression arguments of the fourth ->join() argument ->eq('sys_language.uid', $queryBuilder->quoteIdentifier('overlay.sys_language_uid')).
  • The second argument of the join expression instructs the ExpressionBuilder to quote the value as a field identifier (a field name, here a table/field name combination). Using createNamedParameter() would lead to a quoting as value (' instead of ` in mysql) and the query would fail.
  • The alias overlay - the third argument of the ->join() call - does not necessarily have to be set to a different name than the table name itself here. Using pages_language_overlay as third argument and not specifying a different name would do. Aliases are mostly useful if a join to the same table is needed: SELECT `something` FROM `tt_content` JOIN `tt_content` `content2` ON .... Aliases additionally become handy to increase readability of ->where() expressions.
  • The RestrictionBuilder added additional WHERE conditions for both involved tables! Table sys_language obviously only specifies a 'disabled' => 'hidden' as enableColumns in its TCA ctrl section, while table pages_language_overlay specifies deleted, hidden, starttime and stoptime fields.

A more complex example with two joins. The first join points to the first table again using an alias to resolve a language overlay scenario. The second join uses the alias name of the first join target as left side:

// SELECT `tt_content_orig`.`sys_language_uid`
// FROM `tt_content`
// INNER JOIN `tt_content` `tt_content_orig` ON `tt_content`.`t3_origuid` = `tt_content_orig`.`uid`
// INNER JOIN `sys_language` `sys_language` ON `tt_content_orig`.`sys_language_uid` = `sys_language`.`uid`
// WHERE
//     (`tt_content`.`colPos` = 1)
//     AND (`tt_content`.`pid` = 42)
//     AND (`tt_content`.`sys_language_uid` = 2)
//     AND ... RestrictionBuilder TCA restrictions for tables tt_content and sys_language ...
// GROUP BY `tt_content_orig`.`sys_language_uid`
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$constraints = [
   $queryBuilder->expr()->eq('tt_content.colPos', $queryBuilder->createNamedParameter(1, \PDO::PARAM_INT)),
   $queryBuilder->expr()->eq('tt_content.pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)),
   $queryBuilder->expr()->eq('tt_content.sys_language_uid', $queryBuilder->createNamedParameter(2, \PDO::PARAM_INT)),
];
$queryBuilder
   ->select('tt_content_orig.sys_language_uid')
   ->from('tt_content')
   ->join(
      'tt_content',
      'tt_content',
      'tt_content_orig',
      $queryBuilder->expr()->eq(
         'tt_content.t3_origuid',
         $queryBuilder->quoteIdentifier('tt_content_orig.uid')
      )
   )
   ->join(
      'tt_content_orig',
      'sys_language',
      'sys_language',
      $queryBuilder->expr()->eq(
         'tt_content_orig.sys_language_uid',
         $queryBuilder->quoteIdentifier('sys_language.uid')
      )
   )
   ->where(...$constraints)
   ->groupBy('tt_content_orig.sys_language_uid')
   ->execute();

Further remarks:

  • ->join() and innerJoin are identical. They create an INNER JOIN query, this is identical to a JOIN query.
  • ->leftJoin() creates a LEFT JOIN query, this is identical to a LEFT OUTER JOIN query.
  • ->rightJoin() creates a RIGHT JOIN query, this is identical to a RIGT OUTER JOIN query.
  • Calls on join() methods are only considered for ->select() and ->count() type queries. ->delete(), ->insert() and update() do not support joins, those query parts are ignored and do not end up in the final statement.
  • The argument of ->getQueryBuilderForTable() should be the left most main table.
  • A join of two tables that are configured to different connections will throw an exception. This restricts which tables can be configured to different database endpoints. It is possible to test the connection objects of involved tables for equality and implement a fallback logic in PHP if they are different.
orderBy() and addOrderBy()

Add ORDER BY to a ->select() statement. Both ->orderBy() and ->addOrderBy() require a field name as first argument:

// SELECT * FROM `sys_language` ORDER BY `sorting` ASC
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->getRestrictions()->removeAll();
$languageRecords = $queryBuilder
   ->select('*')
   ->from('sys_language')
   ->orderBy('sorting')
   ->execute()
   ->fetchAll();

Remarks:

  • ->orderBy() resets any previously specified orders. It doesn't make sense to call it after a previous ->orderBy() or ->addOrderBy() again.
  • Both methods need a field name or a table.fieldName or a tableAlias.fieldName as first argument, in the above example calling ->orderBy('sys_language.sorting') would have been identical. All identifiers are quoted automatically.
  • The second, optional argument of both methods specifies the sorting order. The two allowed values are ASC and DESC where ASC is default and can be omited.
  • To create a chain of orders, use ->orderBy() and then multiple ->addOrderBy() calls. Calling ->orderBy('header')->addOrderBy('bodytext')->addOrderBy('uid', 'DESC') creates ORDER BY `header` ASC, `bodytext` ASC, `uid` DESC
groupBy() and addGroupBy()

Add GROUP BY to a ->select() statement. Each argument to the methods is a single identifier:

// GROUP BY `pages_language_overlay`.`sys_language_uid`, `sys_language`.`uid`
->groupBy('pages_language_overlay.sys_language_uid', 'sys_language.uid');

Remarks:

  • Similar to ->select() and ->where() both methods are variadic and take any number of arguments, argument unpacking is supported: ->groupBy(...$myGroupArray)
  • Each argument is either a direct field name GROUP BY `bodytext`, a table.fieldName or a tableAlias.fieldName and will be properly quoted.
  • ->groupBy() resets any previously set group specification and should be called only once per statement.
setMaxResults() and setFirstResult()

Add LIMIT to restrict number of records and OFFSET for pagination query parts. Both methods should be called only once per statement:

// SELECT * FROM `sys_language` LIMIT 2 OFFSET 4
$queryBuilder
   ->select('*')
   ->from('sys_language')
   ->setMaxResults(2)
   ->setFirstResult(4)
   ->execute();

Remarks:

  • It's allowed to call ->setMaxResults() but not to call ->setFirstResult().
  • It is possible to call ->setFirstResult() without calling setMaxResults(): This equals to "Fetch everything, but leave out the first n records". Internally, LIMIT will be added by doctrine-dbal and set to a very high value.
getSQL()

Method ->getSQL() returns the created query statement as string. It is incredibly useful during development to verify the final statement is executed just as a developer expects it:

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->select('*')->from('sys_language');
debug($queryBuilder->getSQL());
$statement = $queryBuilder->execute();

Remarks:

  • This is debugging code. Take proper actions to ensure those calls do not end up in production!
  • The method is typically called directly before ->execute() to output the final statement.
  • Casting a QueryBuilder object to (string) has the same effect as calling ->getSQL(), the explicit call using the method should be preferred to simplify a search operation for this kind of debugging statements, though.
  • The method is a simple way to see which restrictions the RestrictionBuilder added.
  • doctrine-dbal always creates prepared statements: Any value that is added via ->createNamedParameter() creates a placeholder that is later substituted when the real query is fired via ->execute(). ->getSQL() does not show those values, instead the placeholder names are displayed, usually with a string like :dcValue1. There is no simple solution to show the fully replaced query from within the framework, but you can go for ->getParameters() to see the array of parameters used to replace these placeholders within the query.
getParameters()

Method ->getParameters() returns the values for the prepared statement placeholders in an array. It is incredibly useful during development to verify the final statement is executed just as a developer expects it:

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->select('*')->from('sys_language');
debug($queryBuilder->getParameters());
$statement = $queryBuilder->execute();

Remarks:

  • This is debugging code. Take proper actions to ensure those calls do not end up in production!
  • The method is typically called directly before ->execute() to output the final values for the statement.
  • doctrine-dbal always creates prepared statements: Any value that added via ->createNamedParameter() creates a placeholder that is later substituted when the real query is fired via ->execute(). ->getparameters() does not show the statement or those placeholders, instead the values are displayed, usually within an array using keys like :dcValue1. There is no simple solution to show the fully replaced query from within the framework, but you can go for ->getSQL() to see the string with placeholders used as a prepared statement.
execute()

Compile and fire the final query statement. This is usually the last call on a QueryBuilder object. The method has two possible return values: On success, it either returns a Statement object representing the result set of ->select() and ->count() queries, or it returns an integer representing the number of affected rows for ->insert(), ->update() and ->delete() queries.

If the query fails for whatever reason (for instance if the database connection was lost or if the query contains a syntax error), a \Doctrine\DBAL\DBALException is thrown. It is most often bad habit to catch and suppress this exception since it indicates a runtime or a program error. Both should bubble up. See the coding guidelines for more information on proper exception handling.

expr()

Return an instance of the ExpressionBuilder. This object is used to create complex WHERE query parts and JOIN expressions:

// SELECT `uid` FROM `tt_content` WHERE (`uid` > 42)
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->gt('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Remarks:

  • This object is stateless and can be called and worked on as often as needed. It however bound to the specific connection a statement is created for and is thus only available through the QueryBuilder which is specific for one connection, too.
  • Never re-use the ExpressionBuilder, especially not between multiple QueryBuilder objects, always get an instance of the ExpressionBuilder by calling ->expr().
createNamedParameter()

Create a placeholder for a prepared statement field value. Always use that when dealing with user input in expressions to make the statement SQL injection safe:

// SELECT * FROM `tt_content` WHERE (`bodytext` = 'kl\'aus')
$searchWord = "kl'aus"; // $searchWord = GeneralUtility::_GP('searchword');
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->getRestrictions()->removeAll();
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter($searchWord))
   )
   ->execute();

The above example shows the importance of using ->createNamedParameter(): The search word kl'aus is "tainted" and would break the query if not channeled through ->createNamedParameter() which quotes the backtick and makes the value SQL injection safe.

Not convinced? Suppose the code would look like this:

// NEVER EVER DO THIS!
$_POST['searchword'] = "'foo' UNION SELECT username FROM be_users";
$searchWord = GeneralUtility::_GP('searchword');
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->getRestrictions()->removeAll();
   this fails with syntax error to prevent copy and paste
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      // MASSIVE SECURITY ISSUE DEMONSTRATED HERE, USE ->createNamedParameter() ON $searchWord!
      $queryBuilder->expr()->eq('bodytext', $searchWord)
    );

Mind the missing ->createNamedParameter() in the ->eq() expression on given value! This code would happily execute the statement SELECT uid FROM `tt_content` WHERE `bodytext` = 'foo' UNION SELECT username FROM be_users; returning a list of backend user names!

Rules:

  • Always use ->createNamedParameter() around any input, no matter where it comes from.

  • The second argument of ->expr() is always either a call to ->createNamedParameter() or ->quoteIdentifier().

  • The second argument of ->createNamedParameter() specifies the type of input. For string, this can be omitted, but it is good practice to add \PDO::PARAM_INT for integers or similar for other field types. This is currently no strict rule, but following this will reduces headaches in the future, especially for DBMS that are not as relaxed as mysql when it comes to field types. The PDO constants can be used for simple types like bool, string, null, lob and integer. Additionally, the two constants Connection::PARAM_INT_ARRAY and Connection::PARAM_STR_ARRAY can be used if an array of strings or integers is handled, for instance in an IN() expression.

  • Keep the ->createNamedParameter() as close as possible to the expression. Do not structure your code in a way that it first quotes something and only later stuffs the already prepared names into the expression. Having ->createNamedParameter() directly within the created expression is much less error prone and easier to review. This is a general rule: Sanitizing input must be as close as possible to the "sink" where a value is submitted to a lower part of the framework. This paradigm should be followed for other quote operations like htmlspecialchars() or GeneralUtility::quoteJSvalue(), too. Sanitizing should be directly obvious at the very place where it is important:

    // DO
    $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
    $queryBuilder->getRestrictions()->removeAll();
    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
           $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter($searchWord))
       )
    
    // DON'T DO, this is much harder to track:
    $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
    $myValue = $queryBuilder->createNamedParameter($searchWord);
    // Imagine much more code here
    $queryBuilder->getRestrictions()->removeAll();
    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
           $queryBuilder->expr()->eq('bodytext', $myValue)
       )
    
quoteIdentifier() and quoteIdentifiers()

->quoteIdentifier() must be used if not a value is handled, but a field name. The quoting is different in those cases and typically ends up with backticks ` instead of ticks ':

// SELECT `uid` FROM `tt_content` WHERE (`header` = `bodytext`)
// Return list of rows where header and bodytext values are identical
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('header', $queryBuilder->quoteIdentifier('bodytext'))
   );

The method quotes single field names or combinations of table names or table aliases with field names:

// Single field name: `bodytext`
->quoteIdentifier('bodytext');
// Table name and field name: `tt_content`.`bodytext`
->quoteIdentifier('tt_content.bodytext')
// Table alias and field name: `foo`.`bodytext`
->from('tt_content', 'foo')->quoteIdentifier('foo.bodytext')

Remarks:

  • Similar to ->createNamedParameter() this method is crucial to prevent SQL injections. The same rules apply here.
  • Method ->set() for UPDATE statements expects their second argument to be a field value by default and quotes them using ->createNamedParameter() internally. In case a field should be set to the value of another field, this quoting can be turned off and an explicit call to ->quoteIdentifier() must be added.
  • Internally, ->quoteIdentifier() is automatically called on all method arguments that must be a field name. For instance, ->quoteIdentifier() is called on all arguments given to ->select().
  • ->quoteIdentifiers() (mind the plural) can be used to quote multiple field names at once. While that method is 'public` and thus exposed as API method, this is mostly useful internally only.
escapeLikeWildcards()

Helper method to quote % characters within a search string. This is helpful in ->like() and ->notLike() expressions:

// SELECT `uid` FROM `tt_content` WHERE (`bodytext` LIKE '%kl\\%aus%')
$searchWord = 'kl%aus';
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->like(
         'bodytext',
         $queryBuilder->createNamedParameter('%' . $queryBuilder->escapeLikeWildcards($searchWord) . '%')
      )
   );

Warning

Even with using ->escapeLikeWildcards(), the value must again be encapsulated in a ->createNamedParameter() call. Only calling ->escapeLikeWildcards() does not make the value SQL injection safe!

getRestrictions(), setRestrictions(), resetRestrictions()

API methods to deal with the RestrictionBuilder.

Connection

An instance of class TYPO3\CMS\Core\Database\Connection is retrieved from the ConnectionPool by calling ->getConnectionForTable() and handing over the table name a query should executed on.

The class extends the basic doctrine-dbal Doctrine\DBAL\Connection class and is mainly used internally within the TYPO3 CMS framework to establish, maintain and terminate connections to single database endpoints. Those internal methods are not scope of this documentation since an extension developer usually doesn't have to deal with that.

For an extension developer however, the class provides a list of "short-hand" methods that allow dealing with "simple" query cases, without the complexity of the QueryBuilder. Using those methods typically ends up in rather short and easily readable code. The methods have in common that they support only "equal" comparisons in WHERE conditions, that all fields and values are fully quoted automatically and the created queries are executed right away.

Note

The Connection object is designed to work on a single table only. If queries to multiple tables should be performed, the object must not be re-used. Instead, a single Connection instance should be retrieved via ConnectionPool per target table. However, it is allowed to use one Connection object for multiple queries to the same table.

insert()

Creates and executes an INSERT INTO statement. A (slightly simplified) example from the Registry API:

// INSERT INTO `sys_registry` (`entry_namespace`, `entry_key`, `entry_value`) VALUES ('aoeu', 'aoeu', 's:3:\"bar\";')
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_registry')
   ->insert(
      'sys_registry',
      [
         'entry_namespace' => $namespace,
         'entry_key' => $key,
         'entry_value' => serialize($value)
      ]
   );

Well, that should be rather obvious: First argument is the table name to insert a row into, second argument is an array of key/value pairs. All keys are quoted to field names and all values are quoted to string values.

It is possible to add another array as third argument to specify how single values are quoted. This is useful if date or numbers or similar should be inserted. The example below quotes the first value to an integer and the second one to a string:

// INSERT INTO `sys_log` (`userid`, `details`) VALUES (42, 'klaus')
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_log')
   ->insert(
      'sys_log',
      [
         'userid' => (int)$userId,
         'details' => (string)$details,
      ],
      [
         Connection::PARAM_INT,
         Connection::PARAM_STR,
      ]
   );

insert() returns the number of affected rows. Guess what? That's the number 1 ... In case something goes wrong a \Doctrine\DBAL\DBALException is raised.

Note

A list of allowed field types for proper quoting can be found in the TYPO3\CMS\Core\Database\Connection class and its base class \Doctrine\DBAL\Connection

bulkInsert()

INSERT multiple rows at once. An example from the test suite:

$connection->bulkInsert(
   'aTestTable',
   [
      ['aField' => 'aValue'],
      ['aField' => 'anotherValue']
   ],
   [
      'aField'
   ]
);

First argument is the table to insert table into, second argument is an array of rows, third argument is the list of field names. Similar to ->insert() it is optionally possible to add another argument to specify quoting details, if omitted, everything will be quoted to strings.

Note

mysql is rather forgiving when it comes to insufficient field quoting: Inserting a string to an int field will not raise an error and mysql will adapt internally. However, other dbms are not that relaxed and may raise errors. It is good practice to specify field types for each field, especially if they are not strings. Doing so right away will reduce the number of raised bugs if people run your extension an anything else than mysql.

update()

Create and execute an UPDATE statement. The example from FAL's ResourceStorage sets a storage to offline:

// UPDATE `sys_file_storage` SET `is_online` = 0 WHERE `uid` = '42'
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_file_storage')
   ->update(
      'sys_file_storage',
      ['is_online' => 0],
      ['uid' => (int)$this->getUid()],
      [Connection::PARAM_INT]
   );

First argument is the table an update should be executed on, the second argument is an array of key/value pairs to set, the third argument is an array of "equal" where statements that are combined with AND, the (optional) fourth argument specifies the type of values to be updated similar to ->insert() and bulkInsert().

Note the third argument WHERE `foo` = 'bar' only supports equal =. For more complex stuff the QueryBuilder has to be used.

The method returns the number of affected rows.

delete()

Execute a DELETE query using equal conditions in WHERE, example from BackendUtility to mark rows as no longer locked by a user:

// DELETE FROM `sys_lockedrecords` WHERE `userid` = 42
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_lockedrecords')
   ->delete(
      'sys_lockedrecords',
      ['userid' => (int)42],
      [Connection::PARAM_INT]
    );

First argument is the table name, second argument is a list of AND combined WHERE conditions as array, third argument specifies the quoting of WHERE values. There is a pattern ;)

Note

TYPO3 CMS uses a "soft delete" approach for many tables. Instead of directly deleting a rows in the database, a field - often called deleted - is set from 0 to 1. Executing a DELETE query circumvents this and really removes rows from a table. For most tables, it is better to use the DataHandler API to handle deletes instead of executing such low level queries directly.

truncate()

Empty a table, removing all rows. Usually much quicker than a ->delete() of all rows. This typically resets "auto increment primary keys" to zero. Use with care:

// TRUNCATE `cache_treelist`
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('cache_treelist')
   ->truncate('cache_treelist');
count()

A COUNT query. Again, this methods becomes handy if very simple COUNT statements are to be executed, the example returns tha number of active rows from table tt_content that have their bodytext field set to klaus:

// SELECT COUNT(*)
// FROM `tt_content`
// WHERE
//     (`bodytext` = 'klaus')
//     AND (
//         (`tt_content`.`deleted` = 0)
//         AND (`tt_content`.`hidden` = 0)
//         AND (`tt_content`.`starttime` <= 1475621940)
//         AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1475621940))
//     )
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content');
$rowCount = $connection->count(
   '*',
   'tt_content',
   ['bodytext' => 'klaus']
);

First argument is the field to count on, usually * or uid. Second argument is the table name, third argument is an array of WHERE equal conditions combined with AND.

Remarks:

  • ->count() of Connection returns the number directly as integer, in contrast to the method of the QueryBuilder, there is no need to call ->fetchColumns(0) or similar.
  • The third argument expects all WHERE values to be strings, each single expression is combined with AND.
  • The RestrictionBuilder kicks in and adds additional WHERE conditions based on TCA settings.
  • Field names and values are quoted automatically.
  • If anything more complex than a simple equal condition on WHERE is needed, the QueryBuilder methods are a better choice: Next to ->select(), the ->count() query is often the least useful method of the Connection object.
select()

Creates and executes a simple SELECT query based on equal conditions. Its usage is limited, the RestrictionBuilder kicks in and key/value pairs are automatically quoted:

// SELECT `entry_key`, `entry_value` FROM `sys_registry` WHERE `entry_namespace` = 'my_extension'
$resultRows = GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_registry')
   ->select(
      ['entry_key', 'entry_value'],
      'sys_registry',
      ['entry_namespace' => 'my_extension']
   );

Remarks:

  • In contrast to the other short-hand methods, ->select() returns a Statement object ready to ->fetch() single rows or to ->fetchAll()
  • The method accepts a series of further arguments to specify GROUP BY, ORDER BY, LIMIT and OFFSET query parts.
  • For non-trivial SELECT queries, it is often better to switch to the according method of the QueryBuilder object.
  • The RestrictionBuilder adds default WHERE restrictions. If those restrictions do not apply to the query needs, it is required to switch to the QueryBuilder->select() method for fine-grained WHERE manipulation.
lastInsertId()

Returns the uid of the last ->insert() statement. Useful if this id needs to be used afterwards directly:

$databaseConnectionForPages = $connectionPool->getConnectionForTable('myTable');
$databaseConnectionForPages->insert(
   'myTable',
   [
      'pid' => 0,
      'title' => 'Home',
   ]
);
$pageUid = (int)$databaseConnectionForPages->lastInsertId('pages');

Remarks:

  • ->lastInsertId($tableName) needs the table name as first argument. While this is optional, you should always supply the table name for DBAL compatibility with engines like postgres.
  • If the auto increment field name is not uid, the second argument with the name of this field must be supplied. For casual TYPO3 tables, uid is ok and the argument can be left out.
createQueryBuilder()

The QueryBuilder should not be re-used for multiple different queries. However, it sometimes becomes handy to first fetch a Connection object for a specific table and to execute a simple query, and to create a QueryBuilder for a more complex query from this connection object later. The methods usefulness is limited however and no good example within the core can be found at the time of this writing.

The method can be helpful in loops to save some precious code characters, too:

$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable($myTable);
foreach ($someList as $aListValue) {
   $myResult = $connection->createQueryBuilder
      ->select('something')
      ->from('whatever')
      ->where(...)
      ->execute()
      ->fetchAll();
}

ExpressionBuilder

The ExpressionBuilder class is responsible to dynamically create SQL query parts for WHERE and JOIN ON conditions, functions like ->min() may also be used in SELECT parts.

It takes care of building query conditions while ensuring table and column names are quoted within the created expressions / SQL fragments. It is a facade to the actual doctrine-dbal ExpressionBuilder.

The ExpressionBuilder is used within the context of the QueryBuilder to ensure queries are being build based on the requirements of the database platform in use.

An instance of the ExpressionBuilder is retrieved from the QueryBuilder object:

$expressionBuilder = $queryBuilder->expr();

It is good practice to not assign an instance of the ExpressionBuilder to a variable but to use it within the code flow of the QueryBuilder context directly:

$rows = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content')
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      // `bodytext` = 'klaus' AND `header` = 'peter'
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
      $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('peter'))
   )
   ->execute()
   ->fetchAll();

Warning

It is crucially important to quote values correctly to not introduce SQL injection attack vectors to your application. See the section of the QueryBuilder for details.

Junctions
  • ->andX() conjunction
  • ->orX() disjunction

Combine multiple single expressions with AND or OR. Nesting is possible, both methods are variadic and take any number of argument which are all combined. It usually doesn't make much sense to hand over zero or only one argument, though.

A core example to find a sys_domain record:

// WHERE
//     (`sys_domain`.`pid` = `pages`.`uid`)
//     AND (
//        (`sys_domain`.`domainName` = 'example.com')
//        OR
//        (`sys_domain`.`domainName` = 'example.com/')
//     )
$queryBuilder->where(
   $queryBuilder->expr()->eq('sys_domain.pid', $queryBuilder->createNamedParameter('pages.uid', \PDO::PARAM_INT)),
   $queryBuilder->expr()->orX(
      $queryBuilder->expr()->eq('sys_domain.domainName', $queryBuilder->createNamedParameter($domain)),
      $queryBuilder->expr()->eq('sys_domain.domainName', $queryBuilder->createNamedParameter($domain . '/'))
   )
)
Comparisons

A set of methods to create various comparison expressions or SQL functions:

  • ->eq($fieldName, $value) "equal" comparison =
  • ->neq($fieldName, $value) "not equal" comparison !=
  • ->lt($fieldName, $value) "less than" comparison <
  • ->lte($fieldName, $value) "less than or equal" comparison <=
  • ->gt($fieldName, $value) "greater than" comparison >
  • ->gte($fieldName, $value) "greater than or equal" comparison >=
  • ->isNull($fieldName) "IS NULL" comparison
  • ->isNotNull($fieldName) "IS NOT NULL" comparison
  • ->like($fieldName) "LIKE" comparison
  • ->notLike($fieldName) "NOT LIKE" comparison
  • ->in($fieldName, $valueArray) "IN ()" comparison
  • ->notIn($fieldName, $valueArray) "NOT IN ()" comparison
  • ->inSet($fieldName, $value) "FIND_IN_SET('42', aField)" Find a value in a comma separated list of values
  • ->bitAnd($fieldName, $value) A bitwise AND operation &

Remarks and warnings:

Examples:

// `bodytext` = 'foo' - string comparison
->eq('bodytext', $queryBuilder->createNamedParameter('foo'))

// `tt_content`.`bodytext` = 'foo'
->eq('tt_content.bodytext', $queryBuilder->createNamedParameter('foo'))

// `aTableAlias`.`bodytext` = 'foo'
->eq('aTableAlias.bodytext', $queryBuilder->createNamedParameter('foo'))

// `uid` = 42 - integer comparison
->eq('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))

// `uid` >= 42
->gte('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))

// `bodytext` LIKE 'klaus'
->like(
   'bodytext',
   $queryBuilder->createNamedParameter($queryBuilder->escapeLikeWildcards('klaus'))
)

// `bodytext` LIKE '%klaus%'
->like(
   'bodytext',
   $queryBuilder->createNamedParameter('%' . $queryBuilder->escapeLikeWildcards('klaus') . '%')
)

// `uid` IN (42, 0, 44) - properly sanitized, mind the intExplode and PARAM_INT_ARRAY
->in(
   'uid',
   $queryBuilder->createNamedParameter(
      GeneralUtility::intExplode(',', '42, karl, 44', true),
      Connection::PARAM_INT_ARRAY
   )
)

// `CType` IN ('media', 'multimedia') - properly sanitized, mind the PARAM_STR_ARRAY
->in(
   'CType',
   $queryBuilder->createNamedParameter(
      ['media', 'multimedia'],
      Connection::PARAM_STR_ARRAY
   )
)
Aggregate functions

Aggregate functions used in SELECT parts, often combined with GROUP BY. First argument is the field name (or table name / alias with field name), second argument an optional alias.

  • ->min($fieldName, $alias = NULL) "MIN()" calculation
  • ->max($fieldName, $alias = NULL) "MAX()" calculation
  • ->avg($fieldName, $alias = NULL) "AVG()" calculation
  • ->sum($fieldName, $alias = NULL) "SUM()" calculation
  • ->count($fieldName, $alias = NULL) "COUNT()" calculation

Examples:

// Calculate the average creation timestamp of all rows from tt_content
// SELECT AVG(`crdate`) AS `averagecreation` FROM `tt_content`
$result = $queryBuilder
   ->addSelectLiteral(
      $queryBuilder->expr()->avg('crdate', 'averagecreation')
   )
   ->from('tt_content')
   ->execute()
   ->fetch();

// Distinct list of all existing endtime values from tt_content
// SELECT `uid`, MAX(`endtime`) AS `maxendtime` FROM `tt_content` GROUP BY `endtime`
$statement = $queryBuilder
   ->select('uid')
   ->addSelectLiteral(
      $queryBuilder->expr()->max('endtime', 'maxendtime')
   )
   ->from('tt_content')
   ->groupBy('endtime')
   ->execute();
Various Expressions
TRIM

Using the TRIM expression makes sure fields get trimmed on database level. See the examples below to get a better idea of what can be done.

$queryBuilder->expr()->comparison(
    $queryBuilder->expr()->trim($fieldName),
    ExpressionBuilder::EQ,
    $queryBuilder->createNamedParameter('', \PDO::PARAM_STR)
);

The call to $queryBuilder->expr()-trim() can be one of the following:

  • trim('fieldName') results in TRIM("tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_LEADING, 'x') results in TRIM(LEADING "x" FROM "tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_TRAILING, 'x') results in TRIM(TRAILING "x" FROM "tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_BOTH, 'x') results in TRIM(BOTH "x" FROM "tableName"."fieldName")
LENGTH

The LENGTH string function can be used to return the length of a string in bytes, method signature is fieldName with optional alias ->length(string $fieldName, string $alias = null)

$queryBuilder->expr()->comparison(
    $queryBuilder->expr()->length($fieldName),
    ExpressionBuilder::GT,
    $queryBuilder->createNamedParameter(0, \PDO::PARAM_INT)
);

RestrictionBuilder

Database tables in TYPO3 CMS that can be administrated in the backend come with TCA definitions that specify how single fields and rows of the table should be handled and displayed by the framework.

The ctrl section of a tables TCA array specifies optional framework internal handling of soft deletes and language overlays: For instance, if a row in the backend is deleted using the page or list module, many tables are configured to not entirely drop that row from the table, instead a field (often deleted) is set from zero to one for that row. Similar mechanics kick in for start- and endtime as well as language and workspace overlays. See the ['ctrl'] chapter in the TCA reference for details on this topic.

These mechanics however come with a price tag attached to it: Extension developers dealing with low-level query stuff must take care overlayed or deleted rows are not in the result set of a casual query.

This is where this "automatic restriction" stuff kicks in: The construct is created on top of native doctrine-dbal as TYPO3 CMS specific extension. It automatically adds WHERE expressions that suppress rows which are marked as deleted or exceeded their "active" life cycle. All that is based on the TCA configuration of the affected table.

Rationale

A developer may ask why she has to go through all this and why this additional stuff is added on a low-level query layer, when "just a simple query" should be fired. The construct implements some important design goals:

  • Simple: Query creation should be easy to use without forcing a developer thinking too much about those nasty TCA details.
  • Cope with developer laziness: If the framework would force a developer to always add casual restrictions for each and every query, this is easy to forget. We're all lazy, are we?
  • Security: If in doubt, it is better to show a little too less than too much. It is much better to deal with a customer who complains some records are not shown than to show too many records. The former is "just a bug" while the latter can easily escalate to a serious privilege escalation security issue.
  • Automatic query upgrades: If a table was designed without soft-delete in the first place and later a deleted flag is added and registered in TCA, queries executed on that table will automatically upgrade and add the according deleted = 0 restriction.
  • Handing over restriction details to the framework: Having the restriction expressions done by the framework gives it the opportunity to change details without breaking extension code. This may very well happen in the future and having a happy little upgrade path for such cases in place may become very handy later.
  • Flexibility: The class construct is created in a way that allows developers to extend or substitute it with own restrictions if that is useful to model the domain in question.
Main construct

The restriction builder is called whenever a SELECT or COUNT query is executed through either the QueryBuilder or Connection. The QueryBuilder allows manipulation of those restrictions while the simplified Connection class does not. If a query deals with multiple tables in a join, restrictions for all affected tables are added.

Each single restriction like a DeletedRestriction or a StartTimeRestriction is modeled as a single class implementing the QueryRestrictionInterface. Each restriction looks up in TCA if it should kick in. If so, it adds according expressions to the WHERE clause when the final statement is compiled.

Multiple restrictions can be grouped in containers which implement the QueryRestrictionContainerInterface.

The DefaultRestrictionContainer is always added by ... uuhm ... default: It adds the DeletedRestriction, the HiddenRestriction, the StartTimeRestriction and the EndTimeRestriction. Note this is true for all contexts a query is executed in: It does not matter whether a query is created from within a frontend, a backend or a cli call, they all add the DefaultRestrictionContainer if not explicitly told otherwise by an extension developer.

Note

Having this DefaultRestrictionContainer used everywhere is the second iteration of that code construct:

The first variant automatically added restrictions based on context. For instance, a query fired by a call that is executed in the backend did not add the hidden flag, while a query fired from within a frontend call did so. We quickly figured this ends up in a huge mess: The distinction between frontend, backend and cli is not that sharp in TYPO3, as example the frontend behaves much more like a backend call if the admin panel is used.

The currently active variant is much easier: It always adds sane defaults everywhere, a developer only has to deal with details if they don't fit. The core team hopes this approach is a good balance between hidden magic, security, transparency and convenience.

Restrictions
  • DeletedRestriction: (default) Evaluates ['ctrl']['delete'], adds for instance AND deleted = 0 if TCA['aTable']['ctrl']['delete'] = 'deleted' is specified.
  • HiddenRestriction: (default) Evaluates ['ctrl']['enablecolumns']['disabled'], adds AND hidden = 0 if hidden is specified as field name.
  • StartTimeRestriction: (default) Evaluates ['ctrl']['enablecolumns']['starttime'], typically adds something like AND (`tt_content`.`starttime` <= 1475580240).
  • EndTimeRestriction: (default) Evaluates ['ctrl']['enablecolumns']['endtime'].
  • FrontendGroupRestriction: Evaluates ['enablecolumns']['fe_group'].
  • RootlevelRestriction: Match records on root level, adds AND (`pid` = 0)
  • BackendWorkspaceRestriction: Determines the current workspace a backend user is working in and adds a couple of restrictions to select only records of that workspace if the table supports workspaced records.
  • FrontendWorkspaceRestriction: Restriction to filter records for fronted workspaces preview.
QueryRestrictionContainer
  • DefaultRestrictionContainer: Add DeletedRestriction, HiddenRestriction, StartTimeRestriction and EndTimeRestriction. This container is always added if not told otherwise.
  • FrontendRestrictionContainer: Adds DeletedRestriction, HiddenRestriction, StartTimeRestriction, EndTimeRestriction, FrontendWorkspaceRestriction and FrontendGroupRestriction. This container should be be added by a developer to a query if creating query statements in frontend context or if handling frontend stuff from within cli calls.
Examples

Often the default restrictions are sufficient. Nothing needs to be done in those cases.

However, many backend modules still want to show disabled records and remove the starttime and endtime restrictions to allow administration of those records for an editor. A typical setup from within a backend module:

// SELECT `uid`, `bodytext` FROM `tt_content` WHERE (`pid` = 42) AND (`tt_content`.`deleted` = 0)
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
// Remove all restrictions but add DeletedRestriction again
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
$result = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter($pid, \PDO::PARAM_INT)))
   ->execute()
   ->fetchAll();

The DeletedRestriction should be kept in almost all cases. Usually, the only extension that dismiss that flag is the recycler module to list and resurrect deleted records. Any object implementing the QueryRestrictionInterface can be given to ->add(). This allows extensions to deliver own restrictions.

An alternative to the recommended way of first removing all restrictions and then adding needed ones again (using ->removeAll(), then ->add()) is to kick specific restrictions with a call to ->removeByType():

// Remove starttime and endtime, but keep hidden and deleted
$queryBuilder
   ->getRestrictions()
   ->removeByType(StartTimeRestriction::class)
   ->removeByType(EndTimeRestriction::class);

In the frontend it is often needed to swap the DefaultRestrictionContainer with the FrontendRestrictionContainer:

// Kick default restrictions and add list of default frontend restrictions
$queryBuilder->setRestrictions(GeneralUtility::makeInstance(FrontendRestrictionContainer::class));

Note that ->setRestrictions() resets any previously specified restrictions. Any class instance implementing QueryRestrictionContainerInterface can be given to ->setRestrictions(). This allows extensions to deliver and use an own set of restrictions for own query statements if needed.

Tip

It can be very helpful to debug the final statements created by the RestrictionBuilder using debug($queryBuilder->getSQL()) right before the final call to $queryBuilder->execute(). Just take care these calls do not end up in production code.

Statement

A Statement object is returned by QueryBuilder->execute() for ->select() and ->count() query types and by Connection->select() and Connection->count() calls.

The object represents a query result set and comes with methods to ->fetch() single rows or to ->fetchAll() of them. Additionally, it can also be used to execute a single prepared statement with different values multiple times. This part is however not widely used within the TYPO3 CMS core yet, and thus not fully documented here.

Note

The name "Statement" instead of "Result" can be puzzling at first glance: The class represents a prepared statement that can be executed multiple times with different values and then returns multiple different result sets. From this point of view "Statement" fits much better than "Result".

Warning

The return type of single field values is NOT type safe! If selecting a value from a field that is defined as int, the Statement result may very well return that as PHP string. This is true for other database column types like FLOAT, DOUBLE and others. This is an issue with the database drivers used below, it may happen that MySQL returns an integer value for an int field, while MSSQL returns a string. In general, the application must take care of an according type cast on their own to reach maximum DBMS compatibility.

fetch()

Fetch next row from a result statement. Usually used in while() loops. Typical example:

// Fetch all records from tt_content on page 42
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute();
while ($row = $statement->fetch()) {
   // Do something useful with that single $row
}

->fetch() returns arrays with single field / values pairs until the end of the result set is reached which then returns false and thus breaks the while loop.

fetchAll()

Returns an array containing all of the result set rows by implementing the same while loop as above internally. Using that method saves some precious code characters but is more memory intensive if the result set is large with lots of rows and lot of data since big arrays are carried around in PHP:

// Fetch all records from tt_content on page 42
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$rows = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute()
   ->fetchAll();
fetchColumn()

Returns a single column from the next row of a result set, other columns from that result row are discarded. This method is especially handy for QueryBuilder->count() queries. The Connection->count() implementation does exactly that to return the number of rows directly:

// Get the number of tt_content records on pid 42 into variable $numberOfRecords
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$numberOfRecords = $queryBuilder
   ->count('uid')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute()
   ->fetchColumn(0);
rowCount()

Returns the number of rows affected by the last execution of this statement. Use that method instead of counting the number of records in a ->fetch() loop manually.

Warning

->rowCount() works well with DELETE, UPDATE and INSERT queries. However, it does NOT return a valid number for SELECT queries on some DBMS. Never use ->rowCount() on SELECT queries. This may work with MySOL, but fails with other databases like SQLite.

Re-use prepared Statement()

Doctrine usually prepares a statement first, and then executes it with given parameters. Implementing prepared statements depends on the given driver. For instance, the native mysql driver mysqli does implement prepared statements, while the pdo driver of mysql pdo_mysql does not, at least in some scenarios. A driver not properly implementing prepared statements fall back to a direct execution of given query.

There is an API to make real use of prepared statements that becomes handy if the same query is executed with different arguments over and over again. The example below prepares a statement to the pages table and executes it twice with different arguments:

$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('pages');
$queryBuilder = $connection->createQueryBuilder();
$queryBuilder->getRestrictions()->removeAll();
$sqlStatement = $queryBuilder->select('uid')
    ->from('pages')
    ->where($queryBuilder->expr()->eq('uid', $queryBuilder->createPositionalParameter(0, \PDO::PARAM_INT)))
    ->getSQL();
$statement = $connection->executeQuery($sqlStatement, [ 24 ]);
$result1 = $statement->fetch();
$statement->bindValue(1, 25);
$statement->execute();
$result2 = $statement->fetch();

Looking at a mysql debug log:

Prepare SELECT `uid` FROM `pages` WHERE `uid` = ?
Execute SELECT `uid` FROM `pages` WHERE `uid` = '24'
Execute SELECT `uid` FROM `pages` WHERE `uid` = '25'

The log shows one statement preparation with two executions.

QueryHelper

The class contains miscellaneous helper methods to build syntactically valid SQL queries.

Most helper methods are required to deal with legacy data where the format of the input is not strict enough to reliably use the SQL parts in queries directly.

The whole class is marked as @internal, should not be used by extension authors and may - if things go wrong - change at will. The class will hopefully vanish mid-term. However, there may be situations when the class methods can become handy if extension authors migrate their own extensions away from TYPO3_DB to doctrine-dbal. In practice, the core will most likely add proper deprecations to single methods if they are target of removal later.

Extension developers may keep this class in mind for migration, but must not use methods for new code created from scratch. Apart from that, as can be seen below, using those methods often ends up in rather ugly code.

The migration benefits are the only reason the methods are documented here.

Warning

Using those methods raise the risk of SQL injections, especially for methods like ->stripLogicalOperatorPrefix() since its input string tends to come from user supplied input and is sometimes added as WHERE expression without further quoting. Keep a special eye on those scenarios!

parseOrderBy()

Some parts of the core framework allow string definitions like ORDER BY sorting for instance in TCA and TypoScript. The method rips those strings apart and prepares them to be fed to QueryBuilder->orderBy():

// 'ORDER BY aField ASC,anotherField, aThirdField DESC'
// ->
// [ ['aField', 'ASC'], ['anotherField', null], ['aThirdField', 'DESC'] ]
$uglyOrderBy = 'ORDER BY aField ASC,anotherField, aThirdField DESC'
foreach (QueryHelper::parseOrderBy((string)$uglyOrderBy) as $orderPair) {
   list($fieldName, $order) = $orderPair;
   $queryBuilder->addOrderBy($fieldName, $order);
}
parseGroupBy()

Parses GROUP BY strings ready to be added via QueryBuilder->groupBy(), similar to ->parseOrderBy():

// 'GROUP BY be_groups.title, anotherField'
// ->
// ['be_groups.title', 'anotherField']
$uglyGroupBy = 'GROUP BY be_groups.title, anotherField';
$queryBuilder->groupBy(QueryHelper::parseGroupBy($uglyGroupBy));
parseTableList()

Parse a table list, possibly prefixed with FROM, and explode it into and array of arrays where each item consists of a tableName and an optional alias name, ready to be put into QueryBuilder->from():

// 'FROM aTable a,anotherTable, aThirdTable AS c',
// ->
// [ ['aTable', 'a'], ['anotherTable', null], ['aThirdTable', 'c'] ]
$uglyTableString = 'FROM aTable a,anotherTable, aThirdTable AS c;
foreach (QueryHelper::parseTableList($uglyTableString) as $tableNameAndAlias) {
   list($tableName, $tableAlias) = $tableNameAndAlias;
   $queryBuilder->from($tableName, $tableAlias);
}
parseJoin()

Split a JOIN SQL fragment into table name, alias and join conditions:

// 'aTable AS `anAlias` ON anAlias.uid = anotherTable.uid_foreign'
// ->
// [
//     'tableName' => 'aTable',
//     'tableAlias' => 'anAlias',
//     'joinCondition' => 'anAlias.uid = anotherTable.uid_foreign'
// ],
$uglyJoinString = 'aTable AS `anAlias` ON anAlias.uid = anotherTable.uid_foreign';
$joinParts = QueryHelper::parseJoin($uglyJoinString);
$queryBuilder->join(
   $leftTable,
   $joinParts['tableName'],
   $joinParts['tableAlias'],
   $joinParts['joinCondition']
);
stripLogicalOperatorPrefix()

Removes the prefixes AND / OR from an input string.

Those prefixes are added in doctrine-dbal via QueryBuilder->where(), QueryBuilder->orWhere(), ExpressionBuilder->andX() and friends. Some parts of the TYPO3 framework however carry SQL fragments prefixed with AND or OR around and it's not always possible to easily get rid of those. The method helps by killing those prefixes before they are handed over to the doctrine API:

// 'AND 1=1'
// ->
// '1=1'
$uglyWherePart = 'AND 1=1'
$queryBuilder->where(
   // WARNING: High risk of possible SQL injection here, take additional actions!
   QueryHelper::stripLogicalOperatorPrefix($uglyWherePart)
);
getDateTimeFormats()

Just a left over method from the old TYPO3_DB DatabaseConnection class. Of little to no use for extension authors. This one is hopefully one of the first methods to vanish from the class.

Migrating from TYPO3_DB

This chapter is for those poor souls who want to migrate old and busted $GLOBALS['TYPO3_DB'] calls to new hotness doctrine-dbal based API.

It tries to give some hints on typical pitfalls and areas a special eye should be kept on.

Migration of a single extension is finished if a search for $GLOBALS['TYPO3_DB'] does not return hits anymore. This search is the most simple entry point to see which areas need work.

Compare raw queries

The main goal during migration is usually to fire a logically identical query. One recommended and simple approach to verify this is to note down and compare the queries at the lowest possible layer. In $GLOBALS['TYPO3_DB'], the final query statement is usually retrieved by removing the exec_ part from the method name, in doctrine method QueryBuilder->getSQL() can be used:

// Inital code:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery('*', 'index_fulltext', 'phash=' . (int)$phash);

// Remove 'exec_' and debug SQL:
debug($GLOBALS['TYPO3_DB']->SELECTquery('*', 'index_fulltext', 'phash=' . (int)$phash));
// Returns:
'SELECT * FROM index_fulltext WHERE phash=42'

// Migrate to doctrine and debug SQL:
// 'SELECT * FROM index_fulltext WHERE phash=42'
$queryBuilder->select('*')
->from('index_fulltext')
->where(
   $queryBuilder->expr()->eq('phash', $queryBuilder->createNamedParameter($pash, \PDO::PARAM_INT))
);
debug($queryBuilder->getSQL());

The above example returns the exact same query as before. This is not always as trivial to see since WHERE clauses are often in a different order. This especially happens if the RestrictionBuilder is involved. Since the restrictions are crucial and can easily go wrong it is advised to keep an eye on those where parts during transition.

enableFields() and deleteClause()

BackendUtility::deleteClause() adds deleted=0 if ['ctrl']['deleted'] is specified in the table's TCA. The method call should be removed during migration. If there is no other restriction method involved in the old call like enableFields(), the migrated code typically removes all doctrine default restrictions and just adds the DeletedRestriction again:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, TSconfig',
   'pages',
   'TSconfig != \'\''
      . BackendUtility::deleteClause('pages'),
   'pages.uid'
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('pages');
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
$res = $queryBuilder->select('uid', 'TSconfig')
   ->from('pages')
   ->where($queryBuilder->expr()->neq('TSconfig', $queryBuilder->createNamedParameter('')))
   ->groupBy('uid')
   ->execute();

BackendUtility::versioningPlaceholderClause('pages') is typically substituted with the BackendWorkspaceRestriction. Example very similar to the above one:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, TSconfig',
   'pages',
   'TSconfig != \'\''
      . BackendUtility::deleteClause('pages')
      . BackendUtility::versioningPlaceholderClause('pages'),
   'pages.uid'
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('pages');
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class))
   ->add(GeneralUtility::makeInstance(BackendWorkspaceRestriction::class));
$res = $queryBuilder->select('uid', 'TSconfig')
   ->from('pages')
   ->where($queryBuilder->expr()->neq('TSconfig', $queryBuilder->createNamedParameter('')))
   ->groupBy('uid')
   ->execute();

BackendUtility::BEenableFields() in combination with BackendUtility::deleteClause() adds the same calls as the DefaultRestrictionContainer. No further configuration needed:

// Before:
$GLOBALS['TYPO3_DB']->exec_SELECTgetRows(
   'title, content, crdate',
   'sys_news',
   '1=1'
      . BackendUtility::BEenableFields($systemNewsTable)
      . BackendUtility::deleteClause($systemNewsTable)
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)
   ->getQueryBuilderForTable('sys_news');
   ->select('title', 'content', 'crdate')
   ->from('sys_news')
   ->execute();

cObj->enableFields() in frontend context is typically directly substituted with FrontendRestrictionContainer:

// Before:
$GLOBALS['TYPO3_DB']->exec_SELECTquery(
   '*', $table,
   'pid=' . (int)$pid
      . $this->cObj->enableFields($table)
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable($table);
$queryBuilder->setRestrictions(GeneralUtility::makeInstance(FrontendRestrictionContainer::class));
$queryBuilder->select('*')
   ->from($table)
   ->where(
      $queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter($pid, \PDO::PARAM_INT))
   )
);
From ->exec_UDATEquery() to ->update()

Most often, the easiest way to migrate a $GLOBALS['TYPO3_DB']->exec_UDATEquery() is to use $connection->update():

// Before:
$database->exec_UPDATEquery(
    'aTable', // table
    'uid = 42', // where
    [ 'aField' => 'newValue' ] // value array
);

// After:
$connection->update(
    'aTable', // table
    [ 'aField' => 'newValue' ], // value array
    [ 'uid' => 42 ] // where
);

Warning

If switching from exec_UPDATEquery() to update, the order of arguments change, where and values are swapped!

Result set iteration

The exec_* calls return a resource object that is typically iterated over using sql_fetch_assoc(). This is typically changed to ->fetch() on the Statement object:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(...);
while ($row = $GLOBALS['TYPO3_DB']->sql_fetch_assoc($res)) {
   // Do something
}

// After:
$statement = $queryBuilder->execute();
while ($row = $statement->fetch()) {
   // Do something
}
sql_insert_id()

It is sometimes needed to fetch the new uid of a just added record to further work with that row. In TYPO3_DB this was done with a call to ->sql_insert_id() after a ->exec_INSERTquery() call on the same resource. ->lastInsertId() can be used instead:

// Before:
$GLOBALS['TYPO3_DB']->exec_INSERTquery(
   'pages',
   [
      'pid' => 0,
      'title' => 'Home',
   ]
);
$pageUid = $GLOBALS['TYPO3_DB']->sql_insert_id();

// After:
$databaseConnectionForPages = $connectionPool->getConnectionForTable('pages');
$databaseConnectionForPages->insert(
   'pages',
   [
      'pid' => 0,
      'title' => 'Home',
    ]
);
$pageUid = (int)$databaseConnectionForPages->lastInsertId('pages');
fullQuoteStr()

->fullQuoteStr() is rather straight changed to a ->createNamedParameter(), typical case:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, title',
   'tt_content',
   'bodytext = ' . $GLOBALS['TYPO3_DB']->fullQuoteStr('horst')
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'title')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('horst'))
   )
   ->execute();
ext_tables.sql

The schema migrator that compiles ext_tables.sql files from all loaded extensions and compares them with current schema definitions in the database has been fully rewritten. It mostly should work as before, some specific fields however tend to grow a little larger on mysql platforms than before. This usually shouldn't have negative side effects, typically no ext_tables.sql changes are needed when migrating an extension to the new query API.

extbase QueryBuilder

The extbase internal QueryBuilder used in Repositories still exists and works a before. There is usually no manual migration needed. It is theoretically possible to use the doctrine based query builder object in extbase which can become handy since the new one is much more feature rich, but that topic didn't yet fully settle in the core and no general recommendation can be given yet.

Various tips and tricks

  • Use Find usages of PhpStorm for examples! The source code of the core is a great way to learn how specific methods of the API are used. In PhpStorm it is extremely helpful to right click on a single method and list all method usages with Find usages. This is especially handy to quickly see usage examples of complex methods like join() from the QueryBuilder.

  • INSERT, UPDATE and DELETE statements are often easier to read and write using the Connection object instead of the QueryBuilder.

  • SELECT DISTINCT aField is not supported but can be substituted with a ->groupBy('aField').

  • getSQL() and execute() can be used after each other during development to simplify debugging:

    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
          $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
       );
    debug($queryBuilder->getSql());
    $statement = $queryBuilder->execute();
    
  • In contrast to the old API based on $GLOBALS['TYPO3_DB'], doctrine-dbal will throw exceptions if something goes wrong when calling execute(). The exception type is a \Doctrine\DBAL\DBALException which can be caught and transferred to a better error message if the application has to expect query errors. Note this is not good habit and often indicates an architectural flaw of the application at a different layer.

  • count() query types using the QueryBuilder typically call ->fetchColumn(0) to receive the count value. The count() method of Connection object does that automatically and returns the count value result directly.

Internationalization

Internationalization is achieved in TYPO3 with the use of files containing localizable strings. Since TYPO3 CMS 4.6, the XLIFF format is used. It is described further in this section. Until TYPO3 CMS 4.5, a proprietary format called "locallang XML" was used.

Introduction to XLIFF

The XML Localisation Interchange File Format (or XLIFF) is an OASIS-blessed standard format for translations.

Note

In a nutshell an XLIFF document contains one or more <file> elements. Each file element usually corresponds to a source (file or database table) and contains the source of the localizable data. Once translated, the corresponding localized data for one, and only one, locale is added.

Localizable data are stored in <trans-unit> elements. The <trans-unit> contains a <source> element to store the source text and a (non-mandatory) <target> element to store the translated text.

Note that having several <file> elements in the same XLIFF document is not supported by the TYPO3 CMS Core.

Keep in mind that the default language is always considered to be english, even when you have changed your typo3 backend to another language, so source-language must always be source-language="en".

Basics

Here is a sample XLIFF file:

<?xml version="1.0" encoding="UTF-8"?>
<xliff version="1.0" xmlns="urn:oasis:names:tc:xliff:document:1.1">
   <file source-language="en" datatype="plaintext" original="messages" date="2011-10-18T18:20:51Z" product-name="my-ext">
      <header/>
      <body>
         <trans-unit id="headerComment" xml:space="preserve">
            <source>The default Header Comment.</source>
         </trans-unit>
         <trans-unit id="generator" xml:space="preserve">
            <source>The "Generator" Meta Tag.</source>
         </trans-unit>
      </body>
   </file>
</xliff>

The translated file is very similar. If the original file was named locallang.xlf, the translated file for German (code "de") will be named de.locallang.xlf. Note that the original file must always be in english, so it is not allowed to create a file with the prefix "en" e.g. en.locallang.xlf. Inside the file itself, a <target-language> attribute is added in the <file> tag to indicate the translation language ("de" in our example). Then for each <source> tag there's a sibling <target> tag containing the translated string.

Here is what the translation of our sample file could look like:

<xliff version="1.0" xmlns="urn:oasis:names:tc:xliff:document:1.1">
   <file source-language="en" target-language="de" datatype="plaintext" original="messages" date="2011-10-18T18:20:51Z" product-name="my-ext">
      <header/>
      <body>
         <trans-unit id="headerComment" xml:space="preserve">
            <source>The default Header Comment.</source>
            <target>Der Standard-Header-Kommentar.</target>
         </trans-unit>
         <trans-unit id="generator" xml:space="preserve">
            <source>The "Generator" Meta Tag.</source>
            <target>Der "Generator"-Meta-Tag.</target>
         </trans-unit>
      </body>
   </file>
</xliff>

Contrary to "locallang XML" files, only one language can be stored per file. Each translation in a different language goes to an additional file.

File locations and naming

The files follow the same naming conventions as the "locallang XML" files, except they use extension "xlf" instead of "xml".

In the TYPO3 Core, XLIFF files are located in the various system extensions as needed. The system extension "lang" provides several general purpose files plus the classes related to the localization API.

In Extbase-based extensions, XLIFF files are expected to be located in Resources/Private/Language. The main file (locallang.xlf) will be loaded automatically and available in the controller and Fluid views without further work needed. Other files will need to be referred to explicitly.

As mentioned above, the translation files follow the same naming conventions, but are prepended with the language code and a dot. They are stored alongside the default language files.

Translating XLIFF files

This sections highlights the different ways to translate XLIFF files.

The TYPO3 translation server

The TYPO3 community manages an official translation server, running Pootle. Localization files in English are uploaded on that server and translations are packaged nightly. They are fetched in the TYPO3 CMS backend, via the Extension Manager (or the new "Language" module since version 6.0).

It is not the point of this manual to go into the details of the translation process. More information can be found in the TYPO3 wiki.

Translating locally

Using Virtaal, it is possible to translate XLIFF files locally. Virtaal is an open source, cross-platform application.

Virtaal screenshot

Translating with Virtaal, with suggestions from other software

Translating files locally is useful for extensions which are not meant to be published or for creating custom translations.

Custom translations

The $GLOBALS['TYPO3_CONF_VARS']['SYS']['locallangXMLOverride'] allows to override both locallang-XML and XLIFF files. Actually this is not just about translations. Default language files can also be overridden. In the case of XLIFF files, the syntax is as follows (to be placed in an extension's ext_localconf.php file):

$GLOBALS['TYPO3_CONF_VARS']['SYS']['locallangXMLOverride']['EXT:cms/locallang_tca.xlf'][] = 'EXT:examples/Resources/Private/Language/custom.xlf';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['locallangXMLOverride']['de']['EXT:news/Resources/Private/Language/locallang_modadministration.xlf'][] = 'EXT:examples/Resources/Private/Language/Overrides/de.locallang_modadministration.xlf';

The first line shows how to override a file in the default language, the second how to override a German ("de") translation. The German language file looks like this:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<xliff version="1.0">
   <file source-language="en" datatype="plaintext" original="messages" date="2013-03-09T18:44:59Z" product-name="examples">
      <header/>
      <body>
         <trans-unit id="pages.title_formlabel" xml:space="preserve">
            <source>Most important tile</source>
            <target>Wichtigster Titel</target>
         </trans-unit>
      </body>
   </file>
</xliff>

and the result can be easily seen in the backend:

Custom label

Custom translation in the TYPO3 backend

Important

  • Please note that you do not have to copy the full reference file, but only the labels you want to translate.
  • The path to the file to override must be expressed as EXT:foo/bar/.... For the extension "xlf" or "xml" can be used interchangeably. The TYPO3 Core will try both anyway, but using "xlf" is more correct and future-proof.

Attention

The following is a bug but must be taken as a constraint for now:

  • The files containing the custom labels must be located inside an extension. Other locations will not be considered.
  • The original translation needs to exist in typo3temp/l10n/ or next to the base translation file in extensions, for example in typo3conf/ext/myext/Resources/Private/Language/.
Custom languages

Going further it is even possible - since TYPO3 CMS 4.6 - to add custom languages to the TYPO3 backend and create the translations locally using XLIFF files.

First of all, the language must be declared:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['localization']['locales']['user'] = array(
    'gsw_CH' => 'Swiss German',
);

This new language does not need to be entirely translated. It can be defined as falling back to another language, so that only differing labels need be translated:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['localization']['locales']['dependencies'] = array(
   'gsw_CH' => array('de_AT', 'de'),
);

In this case we define that "gsw_CH" (which is the official code for "Schwiizertüütsch" - that is, "Swiss German") can fall back on "de_AT" (another custom translation) and then on "de".

The translations have to be stored in the appopriate folder, in this case typo3conf/l10n/gsw_CH.

The very least you need is to translate the label containing the name of the language itself, so that it appears in the user preferences. In our example this would be in file typo3conf/l10n/gsw_CH/setup/mod/gsw_CH.locallang.xlf.

<?xml version='1.0' encoding='utf-8'?>
<xliff version="1.0">
   <file source-language="en" target-language="gsw_CH" datatype="plaintext" original="messages" product-name="setup">
      <header/>
      <body>
         <trans-unit id="lang_gsw_CH" approved="yes">
            <source>Swiss German</source>
            <target state="translated">Schwiizertüütsch</target>
         </trans-unit>
      </body>
   </file>
</xliff>
User Settings screenshot

The new language appears in the user preferences

Note

Any language will always fall back on the default one (i.e. English) when a translation is not found. A custom language will fall back on its "parent" language automatically. Thus - in our second example of de_AT (German for Austria) - no fallback would have to be defined for "de_AT" if it were just falling back on "de".

Custom translation servers

With the use of XLIFF and the freely available Pootle translation server, companies and individuals may easily set up a custom translation server for their extensions.

There is a signal that can be caught to change the translation server URL to use. The first step is to register one's code for handling the signal. Such code would be placed in an extension's ext_localconf.php file:

$signalSlotDispatcher = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Extbase\\SignalSlot\\Dispatcher');

$signalSlotDispatcher->connect(
   version_compare(TYPO3_version, '7.0', '<')
      ? 'TYPO3\\CMS\\Lang\\Service\\UpdateTranslationService'
      : 'TYPO3\\CMS\\Lang\\Service\\TranslationService',
   'postProcessMirrorUrl',
   'Company\\Extension\Slots\\CustomMirror',
   'postProcessMirrorUrl'
);

The class (slot) which receives the signal (EXT:myext/Classes/Slots/CustomMirror.php) could look something like:

<?php
namespace Company\Extensions\Slots;
class CustomMirror {

   /** @var string */
   static protected $extKey = 'myext';

   public function postProcessMirrorUrl($extensionKey, &$mirrorUrl) {
      if ($extensionKey === self::$extKey) {
         $mirrorUrl = 'http://mycompany.tld/typo3-packages/';
      }
   }
}

Note that the mirror URL is passed as a reference, so that it can be modified. In the above example, the URL is changed only for a given extension, but of course it could be changed on a more general basis.

On the custom translation server side, the structure needs to be:

https://mycompany.tld/typo3-packages/
`-- <first-letter-of-extension-key>
   `-- <second-letter-of-extension-key>
      `-- <extension-key>-l10n
         |-- <extension-key>-l10n-de.zip
         |-- <extension-key>-l10n-fr.zip
         |-- <extension-key>-l10n-it.zip
         `-- <extension-key>-l10n.xml

hence in our example:

https://mycompany.tld/typo3-packages/
`-- m
   `-- y
      `-- myext-l10n
         |-- myext-l10n-de.zip
         |-- myext-l10n-fr.zip
         |-- myext-l10n-it.zip
         `-- myext-l10n.xml

And the myext-l10n.xml file contains something like:

<?xml version="1.0" standalone="yes" ?>
<TERlanguagePackIndex>
   <meta>
      <timestamp>1374841386</timestamp>
      <date>2013-07-26 14:23:06</date>
   </meta>
   <languagePackIndex>
      <languagepack language="de">
         <md5>1cc7046c3b624ba1fb1ef565343b84a1</md5>
      </languagepack>
      <languagepack language="fr">
         <md5>f00f73ae5c43cb68392e6c508b65de7a</md5>
      </languagepack>
      <languagepack language="it">
         <md5>cd59530ce1ee0a38e6309544be6bcb3d</md5>
      </languagepack>
   </languagePackIndex>
</TERlanguagePackIndex>

Programming with workspaces in mind

The concept of workspaces needs attention from extension programmers. The implementation of workspaces is however made so that no critical problems can appear with old extensions;

  • First of all the "Live workspace" is no different from how TYPO3 has been working for years so that will be supported out of the box (except placeholder records must be filtered out in the frontend with t3ver_state != , see below).
  • Secondly, all permission related issues are implemented in DataHandler so the worst your users can experience is an error message.

However, you probably want to update your extension so that in the backend the current workspace is reflected in the records shown and the preview of content in the frontend works as well. Therefore this chapter has been written with instructions and insight into the issues you are facing.

Frontend challenges in general

For the frontend the challenges are mostly related to creating correct previews of content in workspaces. For most extensions this will work transparently as long as they use the API functions in TYPO3 to request records from the system.

The most basic form of a preview is when a live record is selected and you lookup a future version of that record belonging to the current workspace of the logged in backend user. This is very easy as long as a record is selected based on its "uid" or "pid" fields which are not subject to versioning; You simply call sys_page->versionOL() after record selection.

However, when other fields are involved in the where clause it gets dirty. This happens all the time! For instance, all records displayed in the frontend must be selected with respect to "enableFields" configuration! What if the future version is hidden and the live version is not? Since the live version is selected first (not hidden) and then overlaid with the content of the future version (hidden) the effect of the hidden field we wanted to preview is lost unless we also check the overlaid record for its hidden field (->versionOL() actually does this). But what about the opposite; if the live record was hidden and the future version not? Since the live version is never selected the future version will never have a chance to display itself! So we must first select the live records with no regard to the hidden state, then overlay the future version and eventually check if it is hidden and if so exclude it. The same problem applies to all other "enableFields", future versions with "delete" flags and current versions which are invisible placeholders for future records. Anyway, all that is handled by the \TYPO3\CMS\Frontend\Page\PageRepository class which includes functions for "enableFields" and "deleted" so it will work out of the box for you. But as soon as you do selection based on other fields like email, username, alias etc. it will fail.

Summary

Challenge: How to preview elements which are disabled by "enableFields" in the live version but not necessarily in the offline version. Also, how to filter out new live records with t3ver_state set to 1 (placeholder for new elements) but only when not previewed.

Solution: Disable check for enableFields/where_del_hidden on live records and check for them in versionOL on input record.

Frontend implementation guidelines

  • Any place where enableFields() are not used for selecting in the frontend you must at least check that t3ver_state != 1 so placeholders for new records are not displayed.
  • Make sure never to select any record with pid = -1! (offline records - related to versioning).
  • If you need to detect preview mode for versioning and workspaces you can read these variables:
    • $GLOBALS['TSFE']->sys_page->versioningPreview: If true, you are allowed to display previews of other record versions.
    • $GLOBALS['TSFE']->sys_page->versioningWorkspaceId: Will tell you the id of the workspace of the current backend user. Used for preview of workspaces.
  • Use these API functions for support of version previews in the frontend:
Function Description
$GLOBALS['TSFE']->sys_page->versionOL($table, &$row, $unsetMovePointers=FALSE)

Versioning Preview Overlay.

Generally ALWAYS used when records are selected based on uid or pid. If records are selected on other fields than uid or pid (e.g. "email = ....") then usage might produce undesired results and that should be evaluated on individual basis.

Principle: Record online! => Find offline?

Example:

This is how simple it is to use this record in your frontend plugins when you do queries directly (not using API functions already using them):

$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(...);
while (($row = $GLOBALS['TYPO3_DB']->sql_fetch_assoc($res))) {
   $GLOBALS['TSFE']->sys_page->versionOL($table,$row);

   if (is_array($row)) {
      // ...
   }
   // ...
}

When the live record is selected, call ->versionOL() and make sure to check if the input row (passed by reference) is still an array.

The third argument, $unsetMovePointers = FALSE, can be set to TRUE when selecting records for display ordered by their position in the page tree. Difficult to explain easily, so only use this option if you don't get a correct preview of records that has been moved in a workspace (only for "element" type versioning)

$GLOBALS['TSFE']->sys_page->fixVersioningPid()

Finding online PID for offline version record.

Will look if the "pid" value of the input record is -1 (it is an offline version) and if the table supports versioning; if so, it will translate the -1 PID into the PID of the original record

Used whenever you are tracking something back, like making the root line. In fact, it is currently only used by the root line function and chances are that you will not need this function often.

Principle: Record offline! => Find online?

Frontend scenarios impossible to preview

These issues are not planned to be supported for preview:

  • Lookups and searching for records based on other fields than uid, pid or "enableFields" will never reflect workspace content since overlays happen to online records after they are selected.

    • This problem can largely be avoided for versions of new records because versions of a "New"-placeholder can mirror certain fields down onto the placeholder record. For the tt\_content table this is configured as

      shadowColumnsForNewPlaceholders'=> 'sys\_language\_uid,l18n\_parent,colPos,header'
      

      so that these fields used for column position, language and header title are also updated in the placeholder thus creating a correct preview in the frontend.

    • For versions of existing records the problem is in reality reduced a lot because normally you don't change the column or language fields after the record is first created anyway! But in theory the preview can fail.

    • When changing the type of a page (e.g. from "Standard" to "External URL") the preview might fail in cases where a look up is done on the :code`doktype` field of the live record.

      • Page shortcuts might not work properly in preview.
      • Mount Points might not work properly in preview.
  • It is impossible to preview the value of count(*) selections since we would have to traverse all records and pass them through ->versionOL() before we would have a reliable result!

  • In \TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController::getPageShortcut(), sys_page->getMenu() is called with an additional WHERE clause which will not respect if those fields are changed for a future version. This could be the case other places where getmenu() is used (but a search shows it is not a big problem). In this case we will for now accept that a wrong shortcut destination can be experienced during previews.

Backend challenges

The main challenge in the backend is to reflect how the system will look when the workspace gets published. To create a transparent experience for backend users we have to overlay almost every selected record with any possible new version it might have. Also when we are tracking records back to the page tree root point we will have to correct pid-values. All issues related to selecting on fields other than pid and uid also relates to the backend as they did for the frontend.

Backend module access

You can restrict access to backend modules by using $MCONF['workspaces'] in the conf.php files. The variable is a list of keywords defining where the module is available:

$MCONF['workspaces'] = online,offline,custom

You can also restrict function menu items to certain workspaces if you like. This is done by an argument sent to the function \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::insertModuleFunction(). See that file for more details.

Detecting current workspace

You can always check what the current workspace of the backend user is by reading $GLOBALS['BE_USER']->workspace. If the workspace is a custom workspace you will find its record loaded in $GLOBALS['BE_USER']->workspaceRec.

The values for workspaces is either 0 (online/live) or the uid of the corresponding entry in the sys_workspace table.

Using DataHandler with workspaces

Since admin users are also restricted by the workspace it is not possible to save any live records when in a workspace. However for very special occasions you might need to bypass this and to do so, you can set the instance variable \TYPO3\CMS\Core\DataHandling\DataHandler::bypassWorkspaceRestrictions to TRUE. An example of this is when users are updating their user profile using the "User Tool > User Settings" module; that actually allows them to save to a live record (their user record) while in a draft workspace.

Moving in workspaces

TYPO3 4.2 and beyond supports moving for "Element" type versions in workspaces. Technically this works by creating a new online placeholder record (like for new elements in a workspace) in the target location with t3ver_state = 3 (move-to placeholder) and a field, t3ver_move_id, holding the uid of the record to move (source record) upon publishing. In addition, a new version of the source record is made and has t3ver_state = 4 (move-to pointer). This version is simply necessary in order for the versioning system to have something to publish for the move operation.

So in summary, two records are created for a move operation in a workspace: The placeholder (online, with t3ver_state = 3 and t3ver_move_id set) and a new version (t3ver_state = 4) of the online source record (the one being moved).

When the version of the source is published a look up will be made to see if a placeholder exists for a move operation and if so the record will take over the pid / "sortby" value upon publishing.

Preview of move operations is almost fully functional through the \TYPO3\CMS\Frontend\Page\PageRepository::versionOL() and \TYPO3\CMS\Backend\Utility\BackendUtility::workspaceOL() functions. When the online placeholder is selected it simply looks up the source record, overlays any version on top and displays it. When the source record is selected it should simply be discarded in case shown in context where ordering or position matters (like in menus or column based page content). This is done in the appropriate places.

Digital assets management (FAL)

Contents:

Introduction

This part of the Core API document contains details about the File Abstraction Layer (FAL), TYPO3 CMS' toolbox for handling media. It explains its architecture and concepts and details what a web site administrator should know about FAL maintenance and permissions.

Content related assets - mostly videos and images - are accessible through a file abstraction layer API and never referenced directly throughout the system.

The API abstracts physical file assets storage within the system. It allows to store, manipulate and access assets with different Digital Assets Management Systems transparently within the system, allows high availability cloud storages and assets providers. Assets can be enriched with meta data like description information, authors, and copyright. This information is stored in local database tables and all access to assets used for instance in content elements or managed through the backend uses the FAL API.

Finally this manual provides a number of examples showing how to use the File Abstraction Layer in your own code.

Basic Concepts

This chapter presents the general concepts underlying the TYPO3 CMS File Abstraction Layer (FAL). The whole point of FAL - as its name implies - is to provide information about files abstracted with regards to their actual nature and storage.

Information about files is stored inside database tables and using a given file is mostly about creating a database relation to the record representing that file.

Storages and drivers

Every file belongs to a storage, which is a very general concept encompassing any kind of place where a file can be stored: a local file system, a remote server or a cloud-based resource. Accessing these different places requires an appropriate driver.

Each storage relies on a driver to provide the user with the ability to use and manipulate the files that exist in the storage. By default TYPO3 CMS provides only a local file system driver.

A new TYPO3 CMS installation comes with a predefined storage, using the local file system driver and pointing to the fileadmin/ directory.

Files and metadata

For each available file in all present storages, there exists a corresponding database record in table "sys_file", which contains basic information about the file (name, path, size, etc.), and an additional record in table "sys_file_metadata", designed to hold a large variety of additional information about the file (metadata such as title, description, width, height, etc.).

Tip

Although FAL is part of the TYPO3 CMS Core, there is a system extension called "filemetadata", which is not installed by default. It extends the "sys_file_metadata" table with fields such as copyright notice, author name, location, etc.

File references

Whenever a file is used - for example an image attached to a content element - a reference is created in the database between the file and the content element. This reference can hold additional information like an alternative title to use for this file just for this reference.

This central reference table ("sys_file_reference") makes it easy to track every place where a file is used inside a TYPO3 CMS installation.

All these elements are explored in greater depth in the chapter about FAL components.

Architecture

This chapter provides an in-depth look into the architecture of FAL.

Overview

The FAL architecture consists of three layers:

Usage Layer
This layer is comprised of the File References, which represent relations to files from any structure that may use them (pages, content elements or any custom structure defined by extensions).
Storage Layer
This layer is made of several parts. First of all there are the files and their associated metadata. Then each file is associated with a Storage.
Driver Layer

This layer is the deepest one. It consists of the Drivers, managing the actual access to and manipulation of the files. It is invisible from both the frontend and the backend, as it works just in the background.

Indeed Drivers are explicitly not part of the public interface. Developers will only interact with File, Folder, FileReference or Storage objects, but never with a Driver object, unless actually developing one.

This layered architecture makes it easy to use different Drivers for accessing files, while maintaining a consistent interface for both developers (in terms of API) and end users (via the backend).

Folders

The actual storage structure depends on which Driver each Storage is based on. When using the local file system Driver provided by the TYPO3 CMS Core, a Storage will correspond to some existing folder on the local storage system (e.g. hard drive). Other Drivers may use virtual structures.

By default, a Storage pointing to the fileadmin folder is created automatically in every TYPO3 CMS installation.

Processed files

Inside each Storage there will be a folder named _processed_ which contains all resized images, be they rendered in the frontend or thumbnails from the backend. The name of this folder is not hard-coded. It can be defined as a property of the Storage. It may even point to a different storage.

Defining a location for processed files

Editing a File Storage to define a location for processed files

Migrated files

When upgrading from a pre-FAL installation (i.e. a TYPO3 CMS version older than 6.0), files will have been moved from various locations (but generally the uploads folder and its sub-folders) to a folder named _migrated in the default Storage (or other Storages if you had several). Such a folder may also be used by custom processes provided by extensions.

This folder contains active files from your older TYPO3 CMS installation. It should not be deleted unless you are sure that you are not using any of these files anymore. It would be advisable to move all files out if this folder over time and into a more explicit structure.

Database structure

This chapter lists the various tables related to FAL and highlights some of their important fields.

sys_file

This table is used to store basic information about each file. Some important fields:

storage
Id of the storage where the file is stored.
identifier
A string which should uniquely identify a file within its storage. Duplicate identifiers are possible, but will create a confusion. For the local file system driver, the identifier is the path to the file, relative to the storage root (starting with a slash and using a slash as directory delimiter).
name
The name of the file. For the local file system driver, this will be the current name of the file in the file system.
sha1
A hash of the file's content. This is used to detect whether a file has changed or not.
metadata
Foreign side of the "sys_file_metadata" relation. Always "0" in the database, but necessary for the TCA of the "sys_file".
sys_file_metadata

This table is used to store metadata about each file. It has a one-to-one relationship with table "sys_file". Contrary to the basic information stored in "sys_file", the content of the table "sys_file_metadata" can be translated.

Most fields are really just additional information. The most important one is:

file
Id of the sys_file record of the file the metadata is related to.

The "sys_file_metadata" table is extended by system extension "filemetadata". In particular, it adds the necessary definitions to categorize files with system categories.

sys_file_reference

This table is used to store all references between files and whatever other records they are used in, typically pages and content elements. The most important fields are:

uid_local
Id of the file.
uid_foreign
Id of the related record.
tablenames
Name of the table containing the related record.
fieldname
Name of the field of the related record where the relation was created.
table_local
Always "sys_file".
title

When a file is referenced, normally its title is used (for whatever purpose, like displaying a caption for example). However it is possible to define a title in the reference, which will be used instead of the original file's title.

The fields "description", "alternative" and "downloadname" obey the same principle.

sys_file_processedfile

This table is similar to "sys_file", but for "temporary" files, like image previews. This table does not have a TCA representation, as it is only written for using direct SQL queries in the source code.

sys_file_collection

FAL offers the possibility to create File Collections, which can then be used for various purposes. By default, they can be used with the "File links" content element.

The most important fields are:

type
The type of the Collection. A Collection can be based on hand-picked files, a folder or categories.
files
The list of selected files. The relationship between files and their Collection is also stored in "sys_file_reference".
storage
The chosen storage, for folder-type Collections.
folder
The chosen folder, for folder-type Collections.
category
The chosen categories, for category-type Collections.
sys_file_storage

This table is used to store the Storages available in the installation. The most important fields are:

driver
The type of Driver used for the storage.
configuration
The Storage configuration with regards to its Driver. This is a FlexForm field and the current options depend on the selected Driver.
sys_filemounts

File Mounts are not specifically part of the FAL (they existed long before), but their definition is based on Storages. Each File Mount is related to a specific storage. The most important fields are:

base
Id of the storage the File Mount is related to.
path
Folder which will actually be mounted (absolute path, considering that / is the root of the selected Storage).
Components

FAL consists of a number of components that interact with each other. Each component has a clear role in the architecture, which is detailed in this section.

Files and Folders

The Files and Folders are facades representing files and folders or whatever equivalent there is in the system the Driver is connecting to (it could be categories from Digital Asset Management tool, for example). They are tightly coupled with the Storage, which they use to actually perform any actions. For example a copying action ($file->copyTo($targetFolder)) is technically not implemented by the \TYPO3\CMS\Core\Resource\File object itself but in the Storage and Driver.

Apart from the shorthand methods to the action methods of the Storage, the Files and Folders are pretty lightweight objects with properties (and related getters and setters) for obtaining information about their respective file or folder on the file system, such as name or size.

A File can be indexed, which makes it possible to reference the file from any database record in order to use it, but also speeds up obtaining cached information such as various metadata or other file properties like size or file name.

A File may be referenced by its uid in the sys_file table, but will also often be referred to by its identifier, which is the path to the file from the root of the Storage the file belongs to. The combined identifier includes the File's identifier prepended by the Storage's uid and a colon (:). Example: 1:/path/to/file/filename.foo.

File References

A \TYPO3\CMS\Core\Resource\FileReference basically represents a usage of a File in a specific location, e.g. as an image attached to a content element ("tt_content") record. A FileReference always references a real, underlying File, but can add context-specific information such as a caption text for an image when used at a specific location.

In the database, each FileReference is represented by a record in the sys_file_reference table.

Creating a reference to a file requires the file to be indexed first, as the reference is done through the normal record relation handling of TYPO3 CMS.

Note

Technically, the \TYPO3\CMS\Core\Resource\FileReference implements the same interface as the \TYPO3\CMS\Core\Resource\File itself. So you have all the methods and properties of a File available in the FileReference as well. This makes it possible to use both files and references to them.

Additionally, there is a property "originalFile" on the FileReference which lets you get information about the underlying file (e.g. $fileReference->getOriginalFile()->getName()).

Storage

The Storage is the focal point of the FAL architecture. Even though it doesn't do the actual low-level actions on a File (that's up to the Driver), it still does the largest part of the logic.

Among the many things done by the Storage layer are:

  • the capabilities check (is the driver capable of writing a file to the target location?)
  • the action permission checks (is the user allowed to do file actions at all?)
  • the user mount permission check (do the user's file mount restrictions allow reading the target file and writing to the target folder?)
  • communication with the Driver (it is the ONLY object that does so)
  • logging and throwing of exceptions for successful and unsuccessful file operations (although some exceptions are also thrown in other layers if necessary, of course)

The Storage essentially works with \TYPO3\CMS\Core\Resource\File and \TYPO3\CMS\Core\Resource\Folder objects.

Drivers

The driver does the actual actions on a file (e.g. moving, copying, etc.). It can rely on the Storage having done all the necessary checks before, so it doesn't need to worry about permissions and other rights.

In the communication between Storage and Driver, the Storage hands over identifiers to the Driver where appropriate. For example, the copyFileWithinStorage() method of the Driver API has the following method signature:

/**
 * Copies a file *within* the current storage.
 * Note that this is only about an inner storage copy action,
 * where a file is just copied to another folder in the same storage.
 *
 * @param string $fileIdentifier
 * @param string $targetFolderIdentifier
 * @param string $fileName
 * @return string the Identifier of the new file
 */
public function copyFileWithinStorage($fileIdentifier, $targetFolderIdentifier, $fileName);
The File Index

Indexing a file creates a database record for the file, containing meta-information both about the file (file-system properties) and from the file (e.g. EXIF information for images). Collecting file-system data is done by the Driver, while all additional properties have to be fetched by additional services.

This distinction is important because it makes clear that FAL does in fact two things: It manages files in terms of assets we use in our Content Management System. In that regard, files are not different from any other content, like texts. On the other hand, it also manages files in terms of a representation of such an asset. While the former thing only uses the contents, the latter heavily depends on the file itself and thus is considered low-level, driver-dependent stuff.

Managing the asset properties of a file (related to its contents) is not done by the Storage/Driver combination, but by services that build on these low-level parts.

Technically, both indexed and non-indexed files are represented by the same object type (\TYPO3\CMS\Core\Resource\File), but being indexed is nevertheless an important step for a file.

Note

An object of an indexed file could theoretically even live without its Storage as long as it is only about querying the object for file properties, as all these properties reside in the database and are read from there when constructing the object. This is currently not the case, as Files are always retrieved via Storages.

Collections

Collections are groups of files defined in various ways. They can be picked up individually, by the selection of a folder or by the selection of one or more categories. Collections can be used by content elements or plugins for various needs.

The TYPO3 CMS Core makes usage of collections for the "File Links" content object type.

Services

The File Abstraction Layer also comes with a number of services:

\TYPO3\CMS\Core\Resource\Service\FileProcessingService

This service processes files to generate previews or scaled/cropped images. These two functions are known as task types and are identified by class constants.

The task which generates preview images is used in most places in the backend where thumbnails are called for. It is identified by constant \TYPO3\CMS\Core\Resource\ProcessedFile::CONTEXT_IMAGEPREVIEW.

The other task is about cropping and scaling an image, typically for frontend output. It is dentified by constant \TYPO3\CMS\Core\Resource\ProcessedFile::CONTEXT_IMAGECROPSCALEMASK).

The configuration for \TYPO3\CMS\Core\Resource\ProcessedFile::CONTEXT_IMAGECROPSCALEMASK is the one used for the imgResource function, but only taking the crop, scale and mask settings into account.

\TYPO3\CMS\Core\Resource\Service\MagicImageService
This service creates resized ("magic") images as can be used in the Rich-Text Editor, for example.
\TYPO3\CMS\Core\Resource\Service\UserFileInlineLabelService
This service is called to generate the label of a "sys_file_reference" entry, i.e. what will appear in the header of an IRRE element.
\TYPO3\CMS\Core\Resource\Service\UserFileMountService
This service provides a single public method which builds a list of folders (and subfolders, recursively) inside any given Storage. It is used when defining File Mounts.
\TYPO3\CMS\Core\Resource\Service\UserStorageCapabilityService
This service provides a single public method, which is used for rendering the "is_public" field of a Storage, given that assessing this property may not be obvious depending on the underlying Driver.
Signals and slots

FAL comes with a series of signals that offer the opportunity to hook into FAL processes at a variety of points.

They are listed below with some explanation, in particular when they are sent (if the name is not explicit enough) and what parameters the corresponding slot will receive. They are grouped by emitting class.

Most signals exist in pairs, one being sent before a given operation, the other one after.

Note

Unless mentioned otherwise, mentions of class "File" below actually refer to the \TYPO3\CMS\Core\Resource\FileInterface interface.

"Folder" objects actually refer to the \TYPO3\CMS\Core\Resource\Folder class.

\TYPO3\CMS\Core\Resource\ResourceStorage

All signals are identified by constants of the \TYPO3\CMS\Core\Resource\ResourceStorageInterface interface.

SIGNAL_SanitizeFileName
The sanitize file name operation aims at removing characters from file names which are not allowed by the underlying Driver. The slot receives the file name and the target folder.
SIGNAL_PreFileAdd
Receives the target file name, the target folder (as a Folder instance) and the local file path.
SIGNAL_PostFileAdd
Receives the File instance corresponding to the newly stored file and the target folder (as a Folder instance).
SIGNAL_PostFileCreate
Receives the identifier of the newly created file and the target folder (as a Folder instance).
SIGNAL_PreFileCopy
Receives a File instance for the file to be copied and the target folder (as a Folder instance).
SIGNAL_PostFileCopy
Receives a File instance for the file that was copied (i.e. the original file) and the target folder (as a Folder instance).
SIGNAL_PreFileMove
Receives a File instance for the file to be moved and the target folder (as a Folder instance).
SIGNAL_PostFileMove
Receives a File instance for the file that was moved, the target folder and the original folder the file was in (both as Folder instances).
SIGNAL_PreFileDelete
Receives a File instance for the file to be deleted.
SIGNAL_PostFileDelete
Receives a File instance for the file that was deleted.
SIGNAL_PreFileRename
Receives a File instance for the file to be renamed and the sanitized new name.
SIGNAL_PostFileRename
Receives a File instance for the file that was renamed and the sanitized new name.
SIGNAL_PreFileReplace
Receives a File instance for the file to be replaced and the path to the local file that will replace it.
SIGNAL_PostFileReplace
Receives a File instance for the file that was replaced and the path to the local file that has replaced it.
SIGNAL_PostFileSetContents
Receives a \TYPO3\CMS\Core\Resource\AbstractFile instance for the file whose content was changed and the content itself (as a string).
SIGNAL_PreFolderAdd
Receives the name of the new folder and a reference to the parent folder, if any (as a Folder instance).
SIGNAL_PostFolderAdd
Receives the newly created folder (as a Folder instance).
SIGNAL_PreFolderCopy
Receives references to the folder to copy and the parent target folder (both as \TYPO3\CMS\Core\Resource\FolderInterface instances) and the sanitized name for the copy.
SIGNAL_PostFolderCopy
Receives references to the original folder and the parent target folder (both as \TYPO3\CMS\Core\Resource\FolderInterface instances) and the identifier of the newly copied folder.
SIGNAL_PreFolderMove
Receives references to the folder to move and the parent target folder (both as Folder instances) and the sanitized target name.
SIGNAL_PostFolderMove
Receives references to the folder to move and the parent target folder (both as Folder instances), the identifier of the moved folder and a reference to the original parent folder (as a Folder instance).
SIGNAL_PreFolderDelete
Receives a reference to the folder to delete (as a Folder instance).
SIGNAL_PostFolderDelete
Receives a reference to the deleted folder (as a Folder instance).
SIGNAL_PreFolderRename
Receives a reference to the folder to be renamed (as a Folder instance) and the sanitized new name.
SIGNAL_PostFolderRename
Receives a reference to the renamed folder (as a Folder instance) and the new identifier of the renamed folder.
SIGNAL_PreGeneratePublicUrl

This signal makes it possible to influence the construction of a resource's public URL. If the slot defines the URL, it is kept as is and the rest of the URL generation process is ignored.

It receives a reference to the instance for which the URL should be generated (as a \TYPO3\CMS\Core\Resource\ResourceInterface instance), a boolean flag indicating whether the URL should be relative to the current script or absolute and a reference to the public URL (which is null at this point, but can be then modified by the slot).

\TYPO3\CMS\Core\Resource\ResourceFactory

The signal is identified by a constant of the \TYPO3\CMS\Core\Resource\ResourceFactoryInterface interface.

SIGNAL_PostProcessStorage
This signal is emitted by method \TYPO3\CMS\Core\Resource\ResourceFactory::getStorageObject() after a Storage object has been fetched. The slot receives a reference to the Storage.
\TYPO3\CMS\Core\Resource\Index\FileIndexRepository
recordCreated
Receives an array containing the information collected about the file whose index (i.e. "sys_file" table entry) was just created.
recordUpdated
Receives an array containing the information collected about the file whose index (i.e. "sys_file" table entry) was just updated.
recordDeleted
Receives the uid of the file (i.e. "sys_file" table entry) which was deleted.
recordMarkedAsMissing
Receives the uid of the file (i.e. "sys_file" table entry) which was marked as missing.
\TYPO3\CMS\Core\Resource\Index\MetaDataRepository
recordPostRetrieval
This signal is emitted after metadata has been retrieved for a given file. The slot receives the metadata as an \ArrayObject instance.
recordCreated
Receives an array containing the metadata collected about the file just after it has been inserted into the "sys_file_metadata" table.
recordUpdated
This signal is emitted after metadata for a given file has been updated. The slot receives the metadata as an array containing all metadata fields (and not just the updated ones).
recordDeleted
Receives the uid of the file whose metadata has just been deleted.
\TYPO3\CMS\Core\Resource\Service\FileProcessingService

All signals are identified by constants of the \TYPO3\CMS\Core\Resource\Service\FileProcessingService class.

SIGNAL_PreFileProcess
This signal is emitted before a file is processed. The slot receives a reference to the processed file and to the original file (both as File instances), a string defining the type of task being executed and an array containing the configuration for that task.
SIGNAL_PostFileProcess
This signal is emitted after a file has been processed. The slot receives a reference to the processed file and to the original file (both as File instances), a string defining the type of task being executed and an array containing the configuration for that task.

See the section about Services for more information about this class.

Administration

Permissions

Permissions in the File Abstraction Layer are the result of a combination of various mechanisms.

System permissions

System permissions are strictly enforced and may prevent an action no matter what component triggered them.

Administrators always have full access. The only reason they might not have access is that the underlying file system or storage service does not allow access to a resource (e.g. some file is read-only in the local file system).

File mounts

Files mounts (discussed in the Getting Started Tutorial) restrict users to a certain folder in a certain Storage. This is an obvious permission restriction: users will never be able to act on a file or folder outside of their allotted file mounts.

User permissions

User permissions for files can be set in the "Fileoperation permissions" section of the Backend User or Backend User Group records.

It is also possible to set permissions using User TSconfig (defined either at Backend User or Backend User Group level). The TSconfig way is recommended because it allows for more flexibility (see below).

The default permissions for backend users and backend user groups are read-only:

permissions.file.default {
   addFile      = 0
   readFile     = 1
   writeFile    = 0
   copyFile     = 0
   moveFile     = 0
   renameFile   = 0
   unzipFile    = 0
   deleteFile   = 0
   addFolder    = 0
   readFolder   = 1
   writeFolder  = 0
   copyFolder   = 0
   moveFolder   = 0
   renameFolder = 0
   deleteFolder = 0
   recursivedeleteFolder = 0
}

If no permissions are defined in TSconfig, the settings in the Backend User and in the Backend User Group record are taken into account and treated as default permissions for all Storages.

User permissions per storage

Using User TSconfig it is possible to set different permissions for different Storages. This syntax uses the uid of the targeted Storage record.

The following example grants all permission for the Storage with uid "1":

permissions.file.storage.1 {
   addFile      = 1
   readFile     = 1
   writeFile    = 1
   copyFile     = 1
   moveFile     = 1
   renameFile   = 1
   unzipFile    = 1
   deleteFile   = 1
   addFolder    = 1
   readFolder   = 1
   writeFolder  = 1
   copyFolder   = 1
   moveFolder   = 1
   renameFolder = 1
   deleteFolder = 1
   recursivedeleteFolder = 1
}

Note

Configured permissions for a specific Storage take precedence over default permissions.

User permissions details

This model for permissions behaves very similar to permission systems on Unix and Linux systems. Folders are seen as a collection of files and folders. If you want to change that collection by adding, removing or renaming files or folders you need to have write permissions for the folder as well. If you only want to change the content of a file you need write permissions for the file but not for the containing folder.

Here is the detail of what the various permission options mean:

addFile
Create new files, upload files.
readFile
Show content of files.
writeFile
Edit or save contents of files, even if NO write permissions to folders are granted.
copyFile
Allow copying of files; needs writeFolder permissions for the target folder.
moveFile
Allow moving files; needs writeFolder permissions for source and target folders.
renameFile
Allow renaming files; needs writeFolder permissions.
unzipFile
Allow unzipping a file; needs writeFolder permissions on the target folder.
deleteFile
Delete a file; needs writeFolder permissions.
addFolder
Add or create new folders; needs writeFolder permissions for the parent folder.
readFolder
List contents of folder.
writeFolder
Permission to change contents of folder (add files, rename files, add folders, rename folders). Changing contents of existing files is not governed by this permission!
copyFolder
Needs writeFolder permissions for the target folder.
moveFolder
Needs writeFolder permissions for both target and source folder (because it is removed from the latter, which changes the folder).
renameFolder
Needs writeFolder permissions (because it changes the folder itself and also the containing folder's contents).
deleteFolder
Remove an (empty) folder; needs write folder permissions.
recursivedeleteFolder
Remove a folder even if it has contents; needs write folder permissions.
Default upload folder

When nothing else is defined, any file uploaded by a user will end up in fileadmin/user_upload. With User TSconfig, it is possible to define a different default upload folder for each backend user or user group. Example:

options.defaultUploadFolder = 3:users/uploads/

See the full reference.

There are a number of circumstances where it might be convenient to change the default upload folder. A hook exists to provide maximum flexibility in that regard. For example, take a look at extension default_upload_folder, which makes it possible to define a default upload folder for a given field of a given table (using custom TSconfig).

Frontend permissions

System extension "filemetadata" adds a "fe_groups" field to the "sys_file_metadata" table. This makes it possible to attach frontend permissions to files. However these permissions are not enforced in any way by the TYPO3 CMS Core. It is up to extension developers to create tools which make use of these permissions.

As an example, you may want to take a look at extension fal_securedownload which also makes use of the "Is publicly available?" property of File Storages.

File Storages

File Storages have a few properties which deserve more explanations.

The Access tab of a File Storage

Special properties in the Access tab of a File Storage

Is browsable?
If this box is not checked, the Storage will not be browsable by users via the FILE > Filelist module, not via the link browser window.
Is publicly available?

When this box is unchecked, the "publicUrl" property of files is replaced by an eID call pointing to a file dumping script provided by the TYPO3 CMS Core. The public URL looks something like index.php?eID=dumpFile&t=f&f=1230&token=135b17c52f5e718b7cc94e44186eb432e0cc6d2f. Behind the scene, class \TYPO3\CMS\Core\Controller\FileDumpController is invoked to manage the download. The class itself does not implement any access check, but provides a hook for doing so.

Warning

This does not magically protect your files if they are within your web root (e.g. below the fileadmin folder). They will still be available to anyone who knows the path to the file. To implement a strict access restriction the Storage must point to some path outside the web root or the folder it points to must contain web server restrictions to block direct access to the files it contains (for example, in an Apache .htaccess file).

Is writable?
This property simply enables to make any Storage read-only.
Is online?

A Storage which is not online cannot be accessed in any way. This flag will generally not be set voluntarily. It will be set automatically when someone tries to access files in that Storage but the underlying Driver detects that the files are not accessible (for whatever reason particular to that Driver, but generally because some third-party storage service is not available at the time).

The important thing to note is that a Storage must be turned online again manually.

Maintenance

There are various maintenance tasks which can be performed to maintain a healthy TYPO3 CMS installation with the File Abstraction Layer.

Scheduler tasks

Two base tasks provided by the Scheduler are related to the File Abstraction Layer.

File Abstraction Layer: Update storage index

This task goes through a Storage and makes sures that every file is properly indexed. When files are manipulated only via the TYPO3 CMS backend, they are always indexed. However if files get added via other means (e.g. FTP) or if some Storages are based on drivers accessing remote systems, it is crucial to run this task regularly so that the TYPO3 CMS installation knows about all existing files in order to make them available to users.

This task is defined per Storage.

File Abstraction Layer: Extract metadata in storage

This task goes through all files in a Storage and updates their metadata. Again this is especially important when files can be manipulated by other means or actually reside on external systems.

This task is defined per Storage.

Processed files

If you change some graphics-related settings, it may be necessary to force a regeneration of all processed files. This can be achieved by deleting all existing processed files via the Install Tool.

Cleaning up processed files

Removing all processed files in the Install Tool

This cleanup is also good if processed files have accumulated for a long time. Many of them may then be obsolete.

Using FAL

This chapter explains the principles on how to use FAL in various contexts, like the frontend or during extension or TYPO3 CMS Core development, by the way of references or useful examples for common use-cases.

Using FAL in the frontend
TypoScript

Using FAL relations in the frontend via TypoScript is achieved using the FILES content object, which is described in details in the TypoScript Reference.

Fluid
The ImageViewHelper

If you have the uid of a File Reference, you can use it directly in the \TYPO3\CMS\Fluid\ViewHelpers\ImageViewHelper:

<f:image src="xxx" treatIdAsReference="1" />

where xxx is the uid of the File Reference.

Get File Properties

If you have a file reference and want to get its properties like Metadata, you have to access "originalResource" first. Example:

{filereference.originalResource.title}
{filereference.originalResource.description}
{filereference.originalResource.publicUrl}

Note: Some metadata fields, like title and description, can be entered either in the referenced file itself or in the reference or both. TYPO3 automatically merges both sources when you access originalResource in Fluid. So originalResource returns the merged value. Values which are entered in the reference will override values from the file itself.

FLUIDTEMPLATE

More often the File Reference information will not be available explicitly. The FLUIDTEMPLATE content object has a dataProcessing property which can be used to call up the \TYPO3\CMS\Frontend\DataProcessing\FilesProcessor class, whose task it is to load all media referenced for the current database record being processed.

This requires first a bit of TypoScript:

lib.carousel = FLUIDTEMPLATE
lib.carousel {
                file = EXT:extension/Resources/Private/Templates/Carousel.html
                dataProcessing.10 = TYPO3\CMS\Frontend\DataProcessing\FilesProcessor
                dataProcessing.10 {
                        references {
                                table = tt_content
                                fieldName = image
                        }
                        as = images
                }
        }
}

This will fetch all Files related to the content element being rendered (referenced in the "image" field) and make them available in a variable called images. This can then be used in the Fluid template:

<f:for each="{images}" as="image">
        <div class="slide">
                <f:image image="{image.originalFile}" />
        </div>
</f:for>
TCA definition

This chapter explains how to create a field that makes it possible to create relations to files.

TYPO3 CMS provides a convenient API for this. Let's look at the TCA configuration the "image" field of the "tt_content" table for example (with some parts skipped).

'image' => array(
        'label' => 'LLL:EXT:lang/locallang_general.xlf:LGL.images',
        'config' => \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::getFileFieldTCAConfig('image', array(
                'appearance' => array(
                        'createNewRelationLinkTitle' => 'LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xlf:images.addFileReference'
                ),
                // custom configuration for displaying fields in the overlay/reference table
                // to use the imageoverlayPalette instead of the basicoverlayPalette
                'foreign_types' => array(
                        ...
                )
        ), $GLOBALS['TYPO3_CONF_VARS']['GFX']['imagefile_ext'])
),

The API call is \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::getFileFieldTCAConfig(). The first argument is the name of the current field, the second argument is an override configuration array, the third argument is the list of allowed file extensions and the fourth argument is the list of disallowed file extensions. All arguments but the first are optional.

A call to \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::getFileFieldTCAConfig() will generate a standard TCA configuration for an inline-type field, with relation to the "sys_file" table via the "sys_file_reference" table as "MM" table.

The override configuration array (the second argument) can be used to tweak this default TCA definition. Any valid property from the "config" section of inline-type fields can be used.

Note

Such FAL-enabled fields can also be used inside FlexForms, but there's no API to generate the code in such a case.

On the database side, the corresponding field needs just store an integer, as is usual for relations field:

CREATE TABLE tt_content (
        ...
        image int(11) unsigned DEFAULT '0' NOT NULL,
        ...
);
The ResourceFactory class

The \TYPO3\CMS\Core\Resource\ResourceFactory is the workhorse of the File Abstraction Layer from a coding point of view. It contains a number of utility methods, some of which are described here, some others which appear in the other code samples provided in this chapter.

Getting the default Storage

Of all available Storages, one may be marked as default. This is the Storage that will be used for any operation whenever no Storage has been explicitly chosen or defined (for example, when not using a combined identifier).

$resourceFactory = \TYPO3\CMS\Core\Resource\ResourceFactory::getInstance();
$storage = $resourceFactory->getDefaultStorage();

Note

This may return null if no default Storage exists.

Getting any Storage

The ResourceFactory should also be used when retrieving any Storage. You should not try to instantiate directly a \TYPO3\CMS\Core\Resource\StorageRepository and call its findByUid() method.

$resourceFactory = \TYPO3\CMS\Core\Resource\ResourceFactory::getInstance();
$storage = $resourceFactory->getStorageObject(3);
Working with Files, Folders and File References

This chapter provides some examples about interacting with File, Folder and FileReference objects.

Getting a file

A file can be retrieved using its uid:

$resourceFactory = \TYPO3\CMS\Core\Resource\ResourceFactory::getInstance();
$file = $resourceFactory->getFileObject(4);

or its combined identifier:

$resourceFactory = \TYPO3\CMS\Core\Resource\ResourceFactory::getInstance();
$file = $resourceFactory->getFileObjectFromCombinedIdentifier('1:/foo.txt');
Copying a file
$storageUid = 17;
$someFileIdentifier = 'templates/images/banner.jpg';
$someFolderIdentifier = 'website/images/';

$resourceFactory = \TYPO3\CMS\Core\Resource\ResourceFactory::getInstance();
$storage = $resourceFactory->getStorageObject($storageUid);

// $file returns a TYPO3\CMS\Core\Resource\File object
$file = $storage->getFile($someFileIdentifier);
// $folder returns a TYPO3\CMS\Core\Resource\Folder object
$folder = $storage->getFolder($someFolderIdentifier);

// returns the TYPO3\CMS\Core\Resource\File object of the new, copied file
$copiedFile = $file->copyTo($folder);
Adding a file

This example adds a new file in the root folder of the default Storage:

$resourceFactory = \TYPO3\CMS\Core\Resource\ResourceFactory::getInstance();
$storage = $resourceFactory->getDefaultStorage();
$newFile = $storage->addFile(
      '/tmp/temporary_file_name.foo',
      $storage->getRootLevelFolder(),
      'final_file_name.foo'
);
Creating a file reference
In the backend context

In the backend or command-line context, it is possible to create file references using the normal \TYPO3\CMS\Core\DataHandling\DataHandler processes.

Assuming you have the "uid" of both the File and whatever other item you want to create a relation to, the following code will create the "sys_file_reference" entry and the relation to the other item (in this case a "tt_content" record).

$resourceFactory = ResourceFactory::getInstance();
$fileObject = $resourceFactory->getFileObject((int)$file);
$contentElement = BackendUtility::getRecord(
        'tt_content',
        (int)$element
);
// Assemble DataHandler data
$newId = 'NEW1234';
$data = array();
$data['sys_file_reference'][$newId] = array(
        'table_local' => 'sys_file',
        'uid_local' => $fileObject->getUid(),
        'tablenames' => 'tt_content',
        'uid_foreign' => $contentElement['uid'],
        'fieldname' => 'image',
        'pid' => $contentElement['pid']
);
$data['tt_content'][$contentElement['uid']] = array(
        'image' => $newId
);
// Get an instance of the DataHandler and process the data
/** @var DataHandler $dataHandler */
$dataHandler = GeneralUtility::makeInstance(DataHandler::class);
$dataHandler->start($data, array());
$dataHandler->process_datamap();
// Error or success reporting
if (count($dataHandler->errorLog) === 0) {
    // Handle success
} else {
    // Handle errors
}

The above example comes from the "examples" extension (reference: https://github.com/TYPO3-Documentation/TYPO3CMS-Code-Examples/blob/master/Classes/Controller/ModuleController.php).

For another table than "tt_content", you need to define the "pid" explicitly when creating the relation:

$data['tt_address'][$address['uid']] = array(
    'pid' => $address['pid'],
    'image' => 'NEW1234' // changed automatically
);
In the frontend context

In a frontend context, the \TYPO3\CMS\Core\DataHandling\DataHandler class cannot be used and there is no specific API to create a File Reference. You are on your own.

The simplest solution is to simply create a database entry into table "sys_file_reference" by using directly the database connection class provided by TYPO3 CMS.

A cleaner solution using Extbase requires far more work. An example can be found here: https://github.com/helhum/upload_example

Getting referenced files

This snippet shows how to retrieve FAL items that have been attached to some other element, in this case the "media" field of the "pages" table:

$fileRepository = GeneralUtility::makeInstance(\TYPO3\CMS\Core\Resource\FileRepository::class);
$fileObjects = $fileRepository->findByRelation('pages', 'media', $uid);

where $uid is the id of some page. The return value is an array of \TYPO3\CMS\Core\Resource\FileReference objects.

Listing files in a folder

These would be the shortest steps to get the list of files in a given folder: get the Storage, get a Folder object for some path in that Storage (path relative to Storage root), finally retrieve the files.

$resourceFactory = \TYPO3\CMS\Core\Resource\ResourceFactory::getInstance();
$defaultStorage = $resourceFactory->getDefaultStorage();
$folder = $defaultStorage->getFolder('/some/path/in/storage/');
$files = $defaultStorage->getFilesInFolder($folder);
Working with collections

The \TYPO3\CMS\Core\Resource\ResourceFactory class provides a convenience method to retrieve a File Collection.

$resourceFactory = ResourceFactory::getInstance();
$collection = $resourceFactory->getCollectionObject(1);
// Load the contents of the collection
$collection->loadContents();

In this example, we retrieve and load the content from the File Collection with a uid of "1". Any Collection implements the \Iterator interface, which means that a Collection can be looped over (once its content has been loaded). Thus if the above code passed the $collection variable to a Fluid view, you could do the following:

<ul>
        <f:for each="{collection}" as="file">
                <li>{file.title}</li>
        </f:for>
</ul>

Caching framework

Since TYPO3 CMS 4.3, the core contains a data caching framework which supports a wide variety of storage solutions and options for different caching needs. Each cache can be configured individually and can implement its own specific storage strategy. Major parts of the system are backported from TYPO3 Flow and are kept in sync between the two systems.

The caching framework exists to help speeding up TYPO3 sites, especially heavily loaded ones. It is possible to move all caches to a dedicated cache server with specialized cache systems like the Redis key-value store (a so called NoSQL database).

Since TYPO3 CMS 4.6, the caching framework is always enabled, the old and unflexible approach to cache content is gone. This document covers settings for TYPO3 CMS 6.0 and beyond.

Quick start for Integrators

This section gives come simple instructions for getting started with using the caching framework without giving the whole details under the hood.

Change specific cache options

By default, most core caches use the database backend. Default cache configuration is defined in typo3/sysext/core/Configuration/DefaultConfiguration.php and can be overridden in LocalConfiguration.php.

If specific settings should be applied to the configuration, they should be added to LocalConfiguration.php. All settings in LocalConfiguration.php will be merged with DefaultConfiguration.php. The easiest way to see the final cache configuration is to use the TYPO3 Backend module Admin Tools > Configuration > $TYPO3_CONF_VARS.

Example for a configuration of redis cache backend on redis database number 42 instead of the default database backend with compression for the pages cache:

return array(
// ...
   'SYS' => array(
   // ...
      'caching' => array(
         // ...
         'cache_pages' => array(
            'backend' => 'TYPO3\CMS\Core\Cache\Backend\RedisBackend',
            'options' => array(
               'database' => 42,
            ),
         ),
      ),
   ),
);
Garbage collection task

Most cache backends do not have an internal system to remove old cache entries that exceeded their lifetime. A cleanup must be triggered externally to find and remove those entries, otherwise caches could grow to arbitrary size. This could lead to a slow website performance, might sum up to significant hard disk or memory usage and could render the server system unusable.

It is advised to always enable the scheduler and run the "Caching framework garbage collection" task to retain clean and small caches. This housekeeping could be done once a day when the system is otherwise mostly idle.

Configuration

Caches are configured in the array $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']. The basic structure is predefined in typo3/sysext/core/Configuration/DefaultConfiguration.php, and consists of the single section:

  • cacheConfigurations: Registry of all configured caches. Each cache is identified by its array key. Each cache can have the sub keys frontend, backend and options to configure the used frontend, backend and possible backend options.
Cache configurations

Unfortunately in TYPO3 CMS, all ext_localconf.php files are loaded after the instance specific configuration from LocalConfiguration.php and AdditionalConfiguration.php. This enables extensions to overwrite cache configuration already done for the instance. Any extension should avoid this situation and should just define the very required minimum of cache configuration. This boils down to define just the array key to populate a new cache to the system. Without further configuration, the cache system fall back to default backend and default frontend settings:

if (!is_array($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'] = array();
}

Extensions like extbase define default caches this way, giving administrators full freedom for specific and possibly quicker setups (eg. a memory driven cache for the extbase reflection cache).

Administrators can overwrite specific settings of the cache configuration in LocalConfiguration.php, example configuration to switch cache_pages to the redis backend using database 3:

return array(
    'SYS' => array(
        'caching' => array(
            'cacheConfigurations' => array(
                'cache_pages' => array(
                    'backend' => 'TYPO3\CMS\Core\Cache\Backend\RedisBackend',
                    'options' => array(
                        'database' => 3,
                    ),
                ),
            ),
        ),
    ),
);

Some backends have mandatory as well as optional parameters (which are documented below). If not all mandatory options are defined, the specific backend will throw an exception if accessed.

How to disable specific caches

During development, it can be convenient to disable certain caches. This is especially helpful since TYPO3 CMS 4.6 for central caches like the language or autoloader cache. This can be achieved by using the null backend (see below) as storage backend.

Warning

Do not use this in production, it will strongly slow down the system!

Example entry to switch the runtime cache to use the null backend:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']
   ['cache_runtime']['backend'] = 'TYPO3\CMS\Core\Cache\Backend\NullBackend';

Caching framework architecture

Basic knowhow

The caching framework can handle multiple caches with different configurations. A single cache consists of any number of cache entries.

A single cache entry is defined by these fields:

  • identifier: A string as unique identifier within this cache. Used to store and retrieve entries.
  • data: The data to be cached.
  • lifetime: A lifetime in seconds of this cache entry. An entry can not be retrieved from cache if lifetime expired.
  • tags: Additional tags (an array of strings) assigned to the entry. Used to remove specific cache entries.

Tip

The difference between identifier and tags is quite simple: an identifier uniquely identifies a cache entry, and a tag is additional data applied to an entry (used for cache eviction). Thus, an identifier refers to a single cache entry to store and retrieve an entry, and a tag can refer to multiple cache entries.

About the identifier

The identifier is used to store ("set") and retrieve ("get") entries from the cache and holds all information to differentiate entries from each other. For performance reasons, it should be quick to calculate.

Suppose there is an resource-intensive extension added as a plugin on two different pages. The calculated content depends on the page on which it is inserted and if a user is logged in or not. So, the plugin creates at maximum four different content outputs, which can be cached in four different cache entries:

  • page 1, no user logged in
  • page 1, a user is logged in
  • page 2, no user logged in
  • page 2, a user is logged in

To differentiate all entries from each other, the identifier is built from the page ID where the plugin is located, combined with the information whether a user is logged in. These are concatenated and hashed. In PHP this could look like this:

$identifier = sha1((string)$this->getPageUid() . (string)$this->isUserLoggedIn());

Tip

sha1 is a good hash algorithm in this case, as collisions are extremely unlikely. It scales O(n) with the input length.

When the plugin is accessed, the identifier is calculated early in the program flow. Next, the plugin looks up for a cache entry with this identifier. If such an entry exists, the plugin can return the cached content, else it calculates the content and stores a new cache entry with this identifier.

In general, the identifier is constructed from all dependencies which specify an unique set of data. The identifier should be based on information which already exist in the system at the point of its calculation. In the above scenario the page id and whether or not a user is logged in are already determined during the frontend bootstrap and can be retrieved from the system quickly.

About tags

Tags are used to drop specific cache entries when some information they are based on is changed.

Suppose the above plugin displays content based on different news entries. If one news entry is changed in the backend, all cache entries which are compiled from this news row must be dropped to ensure that the frontend renders the plugin content again and does not deliver old content on the next frontend call.

If - for example - the plugin uses news number one and two on one page, and news one on another page, the related cache entries should be tagged with these tags:

  • page 1, tags news_1, news_2
  • page 2, tag news_1

If entry 2 is changed, a simple backend logic (probably a hook in DataHandler) could be created, which drops all cache entries tagged with news_2. In this case the first entry would be invalidated while the second entry still exists in the cache after the operation.

While there is always exactly one identifier for each cache entry, an arbitrary number of tags can be assigned to an entry and one specific tag can be assigned to multiple cache entries. All tags a cache entry has are given to the cache when the entry is stored ("set").

Caches in the TYPO3 Core

The TYPO3 core defines and uses several caching framework caches by default. This section gives an overview of default caches, its usage and behaviour. If not stated otherwise, the default database backend with variable frontend is used.

Since TYPO3 CMS 6.2, the various caches are organized in groups. Three groups currently exist:

pages
Frontend-related caches.
system
Low-level caches. Flushing low-level caches should be avoided as much as possible, as rebuilding them requires significant resources.
all
All other caches.

Cache clearing commands can be issued to target a particular group. If a cache does not belong to a group, it will be flushed when the "all" group is flushed, but such caches should normally be transient anyway.

There are TSconfig options for permissions corresponding to each group.

The following caches exist in the TYPO3 CMS Core:

  • cache_core
    • Core cache for compiled php code. It should not be used by extensions.
    • Uses PhpFrontend with the SimpleFileBackend for maximum performance.
    • Stores core internal compiled PHP code like concatenated ext_tables.php and ext_localconf.php files, autoloader and sprite configuration PHP files.
    • This cache is instantiated very early during bootstrap and can not be re configured by instance specific LocalConfiguration.php or similar.
    • Cache entries are located in directory typo3temp/Cache/Code/cache_code. The full directory and any file in this directory can be safely removed and will be re-created upon next request. This is especially useful during development
    • group: system
  • cache_classes
    • Maps class names (and potentially one or more aliases) to the location of the class files in the filesystem. This cache is used by the class loader in order to require the correct class file when that class needs to be instantiated.
    • group: system
  • cache_hash
    • Stores several key-value based cache entries, mostly used during frontend rendering.
    • groups: all, pages
  • cache_pages
    • The frontend page cache. Stores full frontend pages.
    • Content is compressed by default to reduce database memory and storage overhead.
    • groups: all, pages
  • cache_pagesection
    • Used to store "parts of a page", for example used to store Typoscript snippets and compiled frontend templates.
    • Content is compressed by default to reduce database memory and storage overhead.
    • groups: all, pages
  • cache_runtime
    • Runtime cache to store data specific for current request.
    • Used by several core parts during rendering to re-use already calculated data.
    • Valid for one request only.
    • Can be re-used by extensions that have similar caching needs.
  • cache_rootline
    • Cache for rootline calculations.
    • Quick and simple cache dedicated for core usage, Should not be re-used by extensions.
    • groups: all, pages
  • l10n
    • Cache for the localized labels.
    • group: system
  • extbase_reflection
    • Contains detailed information about a class' member variables and methods.
    • group: system

Tip

In rare cases, for example when classes that are required during the bootstrap process are introduced (usually when working on the TYPO3 core), cache clearings requests themselves might throw fatal errors. The solution here is to manually remove the cache files from typo3temp/Cache/Code/.

Garbage collection task

The core system provides a Scheduler task to collect the garbage of all cache backends. This is important for backends like the database backend that do not remove old cache entries and tags internally. It is highly recommended to add this Scheduler task and run it once in a while (maybe once a day at night) for all used backends that do not delete entries which exceeded their lifetime on their own to free up memory or hard disk space.

Cache API

The caching framework architecture is based on the following classes:

  • \TYPO3\CMS\Core\Cache\CacheManager: Returns the cache frontend of a specific cache. This is the main class used by core and extensions to access the instance of a specific cache. Handles configuration settings and default configuration.
  • \TYPO3\CMS\Core\Cache\Frontend\FrontendInterface: Main interface to handle cache entries of a specific cache. Different frontends and further interfaces exist to handle different data types.
  • \TYPO3\CMS\Core\Cache\Backend\BackendInterface: Main interface that every valid storage backend must implement. Several backends and further interfaces exist to specify specific backend capabilities. Some frontends require backends to implement additional interfaces.

Cache frontends

Frontend API

All frontends must implement the API defined in interface TYPO3\CMS\Core\Cache\Frontend\FrontendInterface. All operations on a specific cache must be done with these methods. The frontend object of a cache is the main object any cache manipulation is done with, usually the assigned backend object should not be used directly.

Method Description
getIdentifier Returns the cache identifier.
getBackend Returns the backend instance of this cache. It is seldom needed in usual code.
set Sets/overwrites an entry in the cache.
get Returns the cache entry for the given identifier.
has Checks for existence of a cache entry. Do no use this prior to get() since get() returns NULL if an entry does not exist.
remove Removes the entry for the given identifier from the cache.
flushByTag Flushes all cache entries which are tagged with the given tag.
collectGarbage Calls the garbage collection method of the backend. This is important for backends which are unable to do this internally (like the DB backend).
isValidEntryIdentifier Checks if a given identifier is valid.
isValidTag Checks if a given tag is valid.
requireOnce PhpFrontend only Requires a cached PHP file directly.
Available frontends

Currenly three different frontends are implemented. The main difference is the data types which can be stored using a specific frontend.

String Frontend

The string frontend accepts strings as data to be cached.

Variable Frontend

Strings, arrays and objects are accepted by this frontend. Data is serialized before it is passed to the backend.

Note

Since version 4.5, the igbinary serializer is used transparently (if available in the system), which speeds up both serialization and unserialization while also reducing data size.

Tip

The variable frontend is the most frequently used frontend and handles the widest range of data types. While it can also handle string data, the string frontend should be used if the cache needs to store strings, if only to avoid the additional serialization done by the variable frontend.

PHP Frontend

This is a special frontend to cache PHP files. It extends the string frontend with the method requireOnce() which allows PHP files to be require()'d if a cache entry exists. This can be used by extensions to cache and speed up loading of calculated PHP code and becomes handy if a lot of reflection and dynamic PHP class construction is done.

A backend to be used in combination with the PHP frontend must implement the interface TYPO3\CMS\Core\Cache\Backend\PhpCapableBackendInterface. Currently the file backend and the simple file backend fulfill this requirement.

Note

The PHP frontend can only be used to cache PHP files. It does not work with strings, arrays or objects. It is not intended as a page content cache.

Cache backends

A variety of storage backends exists. They have different characteristics and can be used for different caching needs. The best backend depends on a given server setup and hardware, as well as cache type and usage. A backend should be chosen wisely, as a wrong decision could end up actually slowing down a TYPO3 installation.

Backend API

All backends must implement at least interface TYPO3\CMS\Core\Cache\Backend\BackendInterface. All operations on a specific cache must be done with these methods. There are several further interfaces that can be implemented by backends to declare additional capabilities. Usually, extension code should not handle cache backend operations directly, but should use the frontend object instead.

Method Description
setCache Reference to the frontend which uses the backend. This method is mostly used internally.
set Save data in the cache.
get Load data from the cache.
has Checks if a cache entry with the specified identifier exists.
remove Remove a cache entry with the specified identifier.
flush Remove all cache entries.
collectGarbage Does garbage collection.
flushByTag TaggableBackendInterface only Removes all cache entries which are tagged by the specified tag.
findIdentifiersByTag TaggableBackendInterface only Finds and returns all cache entry identifiers which are tagged by the specified tag.
requireOnce PhpCapableBackendInterface only Loads PHP code from the cache and require_onces it right away.
freeze FreezableBackendInterface only Freezes this cache backend.
isFrozen FreezableBackendInterface only Tells if this backend is frozen.
Common options
Option Description Mandatory Type Default
defaultLifetime Default lifetime in seconds of a cache entry if it is not specified for a specific entry on set() No integer 3600
Database Backend

This is the main backend suitable for most storage needs. It does not require additional server daemons nor server configuration.

The database backend does not automatically perform garbage collection. Instead the Scheduler garbage collection task should be used.

It stores data in the configured database (usually MySQL) and can handle large amounts of data with reasonable performance. Data and tags are stored in two different tables, every cache needs its own set of tables. In terms of performance the database backend is already pretty well optimized and should be used as default backend if in doubt. This backend is the default backend if no backend is specifically set in the configuration.

The core takes care of creating and updating required database tables "on the fly".

Note

However, caching framework tables which are not needed anymore are not deleted automatically. That is why the database analyzer in the install tool will propose you to rename/delete caching framework tables after you changed the caching backend to a non-database one.

For caches with a lot of read and write operations, it is important to tune the MySQL setup. The most important setting is innodb_buffer_pool_size. A generic goal is to give MySQL as much RAM as needed to have the main table space loaded completely in memory.

The database backend tends to slow down if there are many write operations and big caches which do not fit into memory because of slow harddrive seek and write performance. If the data table grows too big to fit into memory, it is possible to compress given data transparently with this backend, which often shrinks the amount of needed space to 1/4 or less. The overhead of the compress/uncompress operation is usually not high. A good candidate for a cache with enabled compression is the core pages cache: it is only read or written once per request and the data size is pretty large. The compression should not be enabled for caches which are read or written multiple times during one request.

InnoDB issues

The database backend for MySQL uses InnoDB tables. Due to the nature of InnoDB, deleting records does not reclaim the actual disk space. E.g. if the cache uses 10GB, cleaning it will still keep 10GB allocated on the disk even though phpMyAdmin will show 0 as the cache table size. To reclaim the space, turn on the MySQL option file_per_table, drop the cache tables and re-create them using the Install tool. This does not by any mean that you should skip the scheduler task. Deleting records still improves performance.

Options
Option Description Mandatory Type Default
compression Whether or not data should be compressed with gzip. This can reduce size of the cache data table, but incurs CPU overhead for compression and decompression. No boolean false
compressionLevel Gzip compression level (if the compression option is set to true). The default compression level is usually sufficient. - -1: Default gzip compression (recommended) - 0: No compression - 9: Maximum compression (costs a lot of CPU) No integer from -1 to 9 -1
Memcached Backend

Memcached is a simple, distributed key/value RAM database. To use this backend, at least one memcached daemon must be reachable, and the PECL module "memcache" must be loaded. There are two PHP memcached implementations: "memcache" and "memcached". Currently, only memcache is supported by this backend.

Warning and design constraints

Memcached is a simple key-value store by design . Since the caching framework needs to structure it to store the identifier-data-tags relations, for each cache entry it stores an identifier->data, identifier->tags and a tag->identifiers entry.

This leads to structural problems:

  • If memcache runs out of memory but must store new entries, it will toss some other entry out of the cache (this is called an eviction in memcached speak).
  • If data is shared over multiple memcache servers and some server fails, key/value pairs on this system will just vanish from cache.

Both cases lead to corrupted caches. If, for example, a tags->identifier entry is lost, dropByTag() will not be able to find the corresponding identifier->data entries which should be removed and they will not be deleted. This results in old data delivered by the cache. Additionally, there is currently no implementation of the garbage collection that could rebuild cache integrity.

It is important to monitor a memcached system for evictions and server outages and to clear clear caches if that happens.

Furthermore memcache has no sort of namespacing. To distinguish entries of multiple caches from each other, every entry is prefixed with the cache name. This can lead to very long runtimes if a big cache needs to be flushed, because every entry has to be handled separately and it is not possible to just truncate the whole cache with one call as this would clear the whole memcached data which might even hold non TYPO3 related entries.

Because of the mentioned drawbacks, the memcached backend should be used with care or in situations where cache integrity is not important or if a cache has no need to use tags at all. Currently, the memcache backend implements the TaggableBackendInterface, so the implementation does allow tagging, even if it is not advised to used this backend together with heavy tagging.

Warning

Since memcached has no sort of namespacing and access control, this backend should not be used if other third party systems have access to the same memcached daemon for security reasons. This is a typical problem in cloud deployments where access to memcache is cheap (but could be read by third parties) and access to databases is expensive.

Options
Option Description Mandatory Type Default
servers

Array of used memcached servers. At least one server must be defined. Each server definition is a string, allowed syntaxes:

  • hostname or IP: TCP connect to host on memcached default port (usually 11211, defined by PHP ini variable memcache.default_port)
  • hostname:port: TCP connect to host on port
  • tcp://hostname:port: Same as above
  • unix:///path/to/memcached.sock: Connect to memcached server using unix sockets
Yes array  
compression Enable memcached internal data compression. Can be used to reduce memcached memory consumption, but adds additional compression / decompression CPU overhead on the related memcached servers. No boolean false
Redis Backend

Redis is a key-value storage/database. In contrast to memcached, it allows structured values. Data is stored in RAM but it allows persistence to disk and doesn't suffer from the design problems of the memcached backend implementation. The redis backend can be used as an alternative to the database backend for big cache tables and helps to reduce load on database servers this way. The implementation can handle millions of cache entries each with hundreds of tags if the underlying server has enough memory.

Redis is known to be extremely fast but very memory hungry. The implementation is an option for big caches with lots of data because most important operations perform O(1) in proportion to the number of (redis) keys. This basically means that the access to an entry in a cache with a million entries is not slower than to a cache with only 10 entries, at least if there is enough memory available to hold the complete set in memory. At the moment only one redis server can be used at a time per cache, but one redis instance can handle multiple caches without performance loss when flushing a single cache.

The garbage collection task should be run every once in a while to find and delete old tags.

The implementation is based on the PHP phpredis module, which must be available on the system.

Note

It is important to monitor the redis server and tune its settings to the specific caching needs and hardware capabilities. There are several articles on the net and the redis configuration file contains some important hints on how to speed up the system if it reaches bounds. A full documentation of available options is far beyond this documentation.

Options
Option Description Mandatory Type Default
hostname IP address or name of redis server to connect to. No string 127.0.0.1
port Port of the redis daemon. No integer 6379
persistentConnection Activate a persistent connection to redis server. This could be a benefit under high load cloud setups. No boolean false
database Number of the database to store entries. Each cache should use its own database, otherwise all caches sharing a database are flushed if the flush operation is issued to one of them. Database numbers 0 and 1 are used and flushed by the core unit tests and should not be used if possible. No integer 0
password

Password used to connect to the redis instance if the redis server needs authentication.

Warning

The password is sent to the redis server in plain text.

No string  
compression Whether or not data compression with gzip should be enabled. This can reduce cache size, but adds some CPU overhead for the compression and decompression operations in PHP. No boolean false
compressionLevel

Set gzip compression level to a specific value. The default compression level is usually sufficient.

  • -1: Default gzip compression (recommended)
  • 0: No compression
  • 9: Maximum compression (but more CPU overhead)
No integer from -1 to 9 -1
APC Backend

APC is mostly known as an opcode cache for PHP source files but can be used to store user data in shared memory as well. Its main advantage is that data can be shared between different PHP processes and requests. All calls directly access shared memory. This makes this backend lightning fast for get() and set() operations. It can be an option for relatively small caches (few dozens of megabytes) which are read and written very often and becomes handy if APC is used as opcode cache anyway.

The implementation is very similar to the memcached backend implementation and suffers from the same problems if APC runs out of memory. Garbage collection is currently not implemented. In its latest version, APC will fail to store data with a PHP warning if it runs out of memory. This may change in the future. Even without using the cache backend, it is advisable to increase the memory cache size of APC to at least 64MB when working with TYPO3, simply due to the large number of PHP files to be cached. A minimum of 128MB is recommended when using the additional content cache. Cache TTL for file and user data should be set to zero (disabled) to avoid heavy memory fragmentation.

Note

It is not advisable to use the APC backend in shared hosting environments for security reasons. The user cache in APC is not aware of different virtual hosts. Basically every PHP script which is executed on the system can read and write any data to this shared cache, given data is not encapsulated or namespaced in any way. Only use the APC backend in environments which are completely under your control and where no third party can read or tamper your data.

Xcache Backend

Xcache is a PHP opcode cache similar to APC. It can also store in-memory key/value user data.

The cache backend implementation is nearly identical to the implementation of APC backend and has the same design constraints.

Important

Xcache does not work in command-line context. The Xcache backend implementation is constructed to silently discard any cache operation if in CLI context. That means if Xcache backend is used, it is of no effect in CLI.

Furthermore, it is important to set the PHP ini value xcache.var_size to a value (eg. 100M) that is big enough to store the needed data. The usage of this capacity should be monitored.

(Available since TYPO3 CMS 6.1)

Wincache backend

Wincache is a PHP opcode cache similar to APC, but dedicated to the Windows OS platform. Similar to APC, the cache can also be used as in-memory key/value cache.

The cache backend implementation is nearly identical to the implementation of APC backend and has the same design constrains.

File Backend

The file backend stores every cache entry as a single file to the file system. The lifetime and tags are added after the data part in the same file.

This backend is the big brother of the Simple file backend and implements additional interfaces. Like the simple file backend it also implements the PhpCapableInterface, so it can be used with PhpFrontend. In contrast to the simple file backend it furthermore implements TaggableInterface and FreezableInterface.

A frozen cache does no lifetime check and has a list of all existing cache entries that is reconstituted during initialization. As a result, a frozen cache needs less file system look ups and calculation time if accessing cache entries. On the other hand, a frozen cache can not manipulate (remove, set) cache entries anymore. A frozen cache must flush the complete cache again to make cache entries writable again. Freezing caches is currently not used in TYPO3 CMS core. It can be an option for code logic that is able to calculate and set all possible cache entries during some initialization phase, to then freeze the cache and use those entries until the whole thing is flushed again. This can be useful especially if caching PHP code.

In general, the backend was specifically optimized to cache PHP code, the get and set operations have low overhead. The file backend is not very good with tagging and does not scale well with the number of tags. Do not use this backend if cached data has many tags.

Warning

The performance of flushByTag() is bad and scales just O(n).

On the contrary performance of get() and set() operations. is good and scales well. Of course if many entries have to be handled, this might still slow down after a while and a different storage strategy should be used (e.g. RAM disks, battery backed up RAID systems or SSD hard disks).

Options
Option Description Mandatory Type Default
cacheDirectory The directory where the cache files are stored. By default it is assumed that the directory is below TYPO3_DOCUMENT_ROOT. However, an absolute path can be selected, too. Every cache should be assigned its own directory, otherwise flushing of one cache would flush all other caches within the same directory as well. No string typo3temp/cache/
Simple File Backend

The simple file backend is the small brother of the file backend. In contrast to most other backends, it does not implement the TaggableInterface, so cache entries can not be tagged and flushed by tag. This improves the performance if cache entries do not need such tagging. TYPO3 CMS core uses this backend for its central core cache (that hold autoloader cache entries and other important cache entries). The core cache is usually flushed completely and does not need specific cache entry eviction.

PDO Backend

The PDO backend can be used as a native PDO interface to databases which are connected to PHP via PDO. It is an alternative to the database backend if a cache should be stored in a database which is otherwise only supported by TYPO3 dbal to reduce the parser overhead.

The garbage collection is implemented for this backend and should be called to clean up hard disk space or memory.

Note

There is currently very little production experience with this backend, especially not with a capable database like Oracle. Any feedback for real life use cases of this cache is appreciated.

Options
Option Description Mandatory Type Default
dataSourceName

Data source name for connecting to the database. Examples:

  • mysql:host=localhost;dbname=test
  • sqlite:/path/to/sqlite.db
  • sqlite::memory
Yes string  
username Username for the database connection. No string  
password Password to use for the database connection. No string  
Transient Memory Backend

The transient memory backend stores data in a PHP array. It is only valid for one request. This becomes handy if code logic needs to do expensive calculations or must look up identical information from a database over and over again during its execution. In this case it is useful to store the data in an array once and just lookup the entry from the cache for consecutive calls to get rid of the otherwise additional overhead. Since caches are available system wide and shared between core and extensions they can profit from each other if they need the same information.

Since the data is stored directly in memory, this backend is the quickest backend available. The stored data adds to the memory consumed by the PHP process and can hit the memory_limit PHP setting.

Null Backend

The null backend is a dummy backend which doesn't store any data and always returns false on get(). This backend becomes handy in development context to practically "switch off" a cache.

Developer information

This chapter is targeted at extension authors who want to use the caching framework for their needs. It is about how to use the framework properly. For details about its inner working, please refer to the section about architecture.

Example usages can be found throughout the TYPO3 CMS Core, in particular in system extension "core" and "extbase".

Cache registration and usage

Registration of a new cache should be done in ext_localconf.php. The example below just defines an empty sub-array in cacheConfigurations. Neither frontend nor backend are defined, meaning that the cache manager will choose the default variable frontend and the database backend by default.

if (!is_array($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'] = array();
}

Tip

The is_array() check is done to enable administrators to overwrite configuration of caches in LocalConfiguration.php. During bootstrap, any ext_localconf.php is loaded after DefaultConfiguration.php and AdditionalConfiguration.php are loaded, so it is important to make sure that the administrator did not already set any configuration of the extensions cache.

If special settings are needed, for example a specific backend (like the transient memory backend), it can be defined with an additional line below the cache array declaration. The extension documentation should hint an integrator about specific caching needs or setups in this case.

Tip

Extensions should not force specific settings, therefore the selection is again encapsulated in a if (!isset()) check to allow administrators to overwrite those settings. It is recommended to set up a cache configuration with sane defaults, but administrators should always be able to overwrite them for whatever reason.

if (!is_array($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'] = array();
}
if (!isset($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache']['backend'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\TransientMemoryBackend';
}

To get an instance of a cache, GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Cache\\CacheManager')->getCache('myext_mycache') should be used. The cache manager will return the fully initialized cache instance:

$myCacheInstance = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Cache\\CacheManager')->getCache('myext_mycache');
Cache access logic

Cache usage patterns are usually wrappers around the main code sections. Here is some example code:

protected function getCachedMagic() {
    $cacheIdentifier = $this->calculateCacheIdentifier();
    $cache = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Cache\\CacheManager')->getCache('myext_mycache');

    // If $entry is null, it hasn't been cached. Calculate the value and store it in the cache:
    if (($entry = $cache->get($cacheIdentifier)) === FALSE) {
        $entry = $this->calculateMagic();

        // [calculate lifetime and assigned tags]

        // Save value in cache
        $cache->set($cacheIdentifier, $entry, $tags, $lifetime);
    }
    return $entry;
}

Tip

It isn't needed to call has() before accessing cache entries with get() as the latter returns FALSE if no entry exists.

Using the system log

Note

A new logging API was introduced in TYPO3 CMS 6.0. It is far more flexible than the old one described here, but is not yet in use in the Core. Thus this section remains valid if you want to write to the "sys_log" table. Otherwise please consider using the new API.

Writing to the system log is done using the backend user object, which writes to the "sys_log" table:

$this->BE_USER->writelog($type, $action, $error, $details_nr, $details, $data, $table, $recuid, $recpid,$event_pid, $NEWid);

Here is a description of the arguments to this function call, and corresponding database fields in table "sys_log":

Field Type Var Description
type tinyint $type

Value telling which module in TYPO3 set the log entry. The type values are paired with an action-integer which is telling in more detail what the event was. Here type and action values are arranged hierarchically (type on first level, action on second level):


  • 1 : \TYPO3\CMS\Core\DataHandling\DataHandler ("TYPO3 Core Engine" where database records are manipulated)
    • Action values are:
      • 0 = no category
      • 1 = new record
      • 2 = update record
      • 3 = delete record
      • 4 = move record
      • 5 = check/evaluate
  • 2 : "tce_file" (File handling in fileadmin/ and absolute filemounts)
    • Action values are for various file handling types like upload, rename, edit etc.
  • 3 : System (e.g. sys_history save)
  • 4 : Modules: This is the mode you may use for extensions having backend module functionality. Probably you would like to use BE_USER->simplelog() for your extensions.
  • 254 : Personal settings changed
  • 255 : Login or Logout action
    • 1 = login
    • 2 = logout
    • 3 = failed login (+ errorcode 3)
    • 4 = failure_warning_email sent
action tinyint $action

See "type" above

When not available, use value "0"

error tinyint $error

Error level:

  • 0 = message, a notice of an action that happened.
  • 1 = error, typically a permission problem for the user
  • 2 = System Error, something which should not happen for technical reasons.
  • 3 = Security notice, like login failures
details_nr tinyint $details_nr

Number of "detail" message. This number should be unique for the combination of type/action

-1 is a temporary detail number you can use while developing and error messages are not fixed yet.

0 is a value that means the message is not supposed to be translated

>= 1 means the message is fixed and ready for translation.

details tinytext $details

The log message text (in english). By identification through type/action/details_nr this can be translated through the localization system.

If you insert "%s" markers in the details message and set $data to an array the first 5 entries (keys 0-4) from $data will substitute the markers sequentially (using sprintf).

log_data tinyblob $data Data that follows the log entry. Can be an array. See "details" for more info.
tablename varchar(40) $table Table name. Special field used by tce_main.php.
recuid int $recuid Record UID. Special field used by tce_main.php.
recpid int $recpid Record PID. Special field used by tce_main.php. [OBSOLETE; not used anymore.]
event_pid int $event_pid The page ID (pid) where the event occurred. Used to select log-content for specific pages.
NEWid varchar(20) $NEWid Special field used by tce_main.php. NEWid string of newly created records.
tstamp int - EXEC_TIME of event, UNIX time in seconds.
uid int - Unique ID for log entry, automatically inserted
userid int - User ID of backend user, automatically set for you
IP varchar(39) - REMOTE_ADDR of client
workspace int - Workspace ID

Making logging simple

While it is nice to have log message categorized and numbered during development and sometimes beyond that point a simpler logging API is necessary. Therefore you can also call this function:

BE_USER->simplelog($message, $extKey='', $error=0);

All you need is to set $message to store a log message. If you call it from an extension it is good practice to also supply the extension key. Finally you can add the error number (according to the table above) if you need to signal an error.

Logging with TYPO3

TYPO3 Logging consists of the following components:

  • A Logger that receives the log message and related details, like a severity
  • A LogRecord model which encapsulates the data
  • Configuration of the logging system
  • Writers which write the log records to different targets (like file, database, rsyslog server, etc.)
  • Processors which add more detailed information to the log record.

Contents:

Quickstart

Instantiate a logger for the current class:

/** @var $logger \TYPO3\CMS\Core\Log\Logger */
$logger = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\CMS\Core\Log\LogManager')->getLogger(__CLASS__);

Log a simple message:

$logger->info('Everything went fine.');
$logger->warning('Something went awry, check your configuration!');

Provide additional information with the log message:

$logger->error(
  'This was not a good idea',
  array(
    'foo' => $bar,
    'bar' => $foo,
  )
);

$logger->warning() etc. are only shorthands - you can also call $logger->log() directly and pass the severity level:

$logger->log(
   \TYPO3\CMS\Core\Log\LogLevel::CRITICAL,
   'This is an utter failure!'
);

TYPO3 has the FileWriter enabled by default, so all log entries are written to a file. If the filename is not set, then the file will contain a hash like typo3temp/logs/typo3_<hash>.log, for example typo3temp/logs/typo3_7ac500bce5.log.

A sample output looks like this:

Fri, 08 Mar 2013 09:45:00 +0100 [INFO] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": Everything went fine.
Fri, 08 Mar 2013 09:45:00 +0100 [WARNING] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": Something went awry, check your configuration!
Fri, 08 Mar 2013 09:45:00 +0100 [ERROR] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": This was not a good idea - {"foo":"bar","bar":{}}
Fri, 08 Mar 2013 09:45:00 +0100 [CRITICAL] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": This is an utter failure!

Logger

Instantiation

The LogManager enables an auto-configured usage of loggers in your PHP code by reading the logging configuration and setting the minimum severity level of the Logger accordingly.

/** @var $logger \TYPO3\CMS\Core\Log\Logger */
$logger = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\CMS\Core\Log\LogManager')->getLogger(__CLASS__);

Using __CLASS__ as name for the logger is recommended to enable logging configuration based on the class hierarchy.

Log() method

\TYPO3\CMS\Core\Log\Logger provides a central point for submitting log messages, the log() method:

$logger->log($level, $message, $data);

which takes three parameters:

Parameter Type Description
$level Type integer

One of either:

  • \TYPO3\CMS\Core\Log\LogLevel::EMERGENCY
  • \TYPO3\CMS\Core\Log\LogLevel::ALERT
  • \TYPO3\CMS\Core\Log\LogLevel::CRITICAL
  • \TYPO3\CMS\Core\Log\LogLevel::ERROR
  • \TYPO3\CMS\Core\Log\LogLevel::WARNING
  • \TYPO3\CMS\Core\Log\LogLevel::NOTICE
  • \TYPO3\CMS\Core\Log\LogLevel::INFO
  • \TYPO3\CMS\Core\Log\LogLevel::DEBUG
$message Type string The log message itself.
$data Type array Optional parameter, can contain additional data, which is added to the log record in the form of an array.

An early return in the log() method prevents unneeded computation work to be done. So you are safe to call $logger->debug() frequently without slowing down your code too much. The Logger will know by its configuration, what the most explicit severity level is.

As next step, all registered Processors are notified. They can modify the log records or add extra information.

The Logger then forwards the log records to all of its configured Writers, which will then persist the log record.

Shorthand methods

For each of the severity levels mentioned above, a shorthand method exists in \TYPO3\CMS\Core\Log\Logger, like

  • $logger->debug($message, array $data = array());
  • $logger->info($message, array $data = array());
  • $logger->notice($message, array $data = array());
  • etc.

Configuration of the Logging system

Instantiation of Loggers is configuration-free, as the LogManager automatically applies its configuration.

The Logger configuration is read from $GLOBALS['TYPO3_CONF_VARS']['LOG'], which contains an array reflecting the namespace and class hierarchy of your TYPO3 project.

Example:

To apply a configuration for all Loggers within the \TYPO3\CMS\Core\Cache namespace, the configuration is read from $GLOBALS['TYPO3_CONF_VARS']['LOG']['TYPO3']['CMS']['Core']['Cache']. So every logger requested for classes like \TYPO3\CMS\Core\Cache\CacheFactory, \TYPO3\CMS\Core\Cache\Backend\NullBackend, etc. will get this configuration applied. The same holds for the old pseudo-namespaces with underscore separator which are still common in extensions.

Configuring Logging for extensions works the same. If an extension uses namespaces, the syntax for the configuration is as above.

For older extensions, configuration is searched for in $GLOBALS['TYPO3_CONF_VARS']['LOG']['tx'] or $GLOBALS['TYPO3_CONF_VARS']['LOG']['Tx'] to differentiate extension classes from Core classes (as extension class names start with tx or Tx).

Writer configuration

The Log Writer configuration is read from the subkey writerConfiguration of the configuration array:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'] = array(
    // configuration for ERROR level log entries
  \TYPO3\CMS\Core\Log\LogLevel::ERROR => array(
      // add a FileWriter
    'TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter' => array(
        // configuration for the writer
      'logFile' => 'typo3temp/logs/typo3_7ac500bce5.log'
    )
  )
);

The above configuration applies to all log entries of level "ERROR" or above.

To apply a special configuration for the controllers of the examples extension, use the following configuration:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['Documentation']['Examples']['Controller']['writerConfiguration'] = array(
   // configuration for WARNING severity, including all
   // levels with higher severity (ERROR, CRITICAL, EMERGENCY)
    \TYPO3\CMS\Core\Log\LogLevel::WARNING => array(
     // add a SyslogWriter
        'TYPO3\\CMS\\Core\\Log\\Writer\\SyslogWriter' => array(),
    ),
);

This overwrites the default configuration shown in the first example for classes located in the namespace \Documentation\Examples\Controller.

For extension "foo" with key "tx_foo" (not using namespaces), the configuration would be located at:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['Tx']['Foo']['writerConfiguration'] = array(
   // ...
);

An arbitrary number of writers can be added for every severity level (INFO, WARNING, ERROR, ...). The configuration based on severity levels is applied to log entries of the particular severity level plus all levels with a higher severity. Thus, a log messages created with $logger->warning() will be affected by a writerConfiguration for \TYPO3\CMS\Core\Log\LogLevel::DEBUG, \TYPO3\CMS\Core\Log\LogLevel::INFO, \TYPO3\CMS\Core\Log\LogLevel::NOTICE and \TYPO3\CMS\Core\Log\LogLevel::WARNING. For the above example code that means:

  • Calling $logger->warning($msg); will result in $msg being written to the computer's syslog on top of the default configuration.
  • Calling $logger->debug($msg); will result in $msg being written only to the default log file (typo3temp/logs/typo3.log).

For a list of writers shipped with the TYPO3 Core see the section about Log Writers.

Processor configuration

Similar to the writer configuration, log record processors can be configured on a per-class and per-namespace basis from the subkey processorConfiguration

$GLOBALS['TYPO3_CONF_VARS']['LOG']['Documentation']['Examples']['Controller']['processorConfiguration'] = array(
    // configuration for ERROR level log entries
  \TYPO3\CMS\Core\Log\LogLevel::ERROR => array(
      // add a MemoryUsageProcessor
    'TYPO3\\CMS\\Core\\Log\\Processor\\MemoryUsageProcessor' => array(
      'formatSize' => TRUE
    )
  )
);

For a list of processors shipped with the TYPO3 Core, see the section about Log Processors.

The LogRecord model

All logging data is modeled using \TYPO3\CMS\Core\Log\LogRecord.

This model has the following properties:

requestId
A unique identifier for each request which is created by the TYPO3 bootstrap.
created
The micro-timestamp when the record is created.
component
The name of the logger which created the LogRecord, usually the fully qualified class name where the Logger has been instantiated.
level
An integer severity level from \TYPO3\CMS\Core\Log\LogLevel.
message
The log message string.
data
Any additional data, encapsulated within an array.

The API to create a new instance of LogRecord is \TYPO3\CMS\Core\Log\Logger:log() or one of the shorthand methods.

LogRecord implements the ArrayAccess interface so that the properties can be accessed like a native array, for example: $logRecord['requestId']. It also implements a __toString() method for your convenience, which returns the log records as a simplified string.

A LogRecord can be processed using LogProcessors or LogWriters. LogProcessors are meant to add values to the data property of LogRecord. For example, if you would like to add a stack trace, use \TYPO3\CMS\Core\Log\Processor\IntrospectionProcessor.

LogWriters are used to write a LogRecord to a particular target, for example a log file.

Log Writers

The purpose of a log writer is (usually) to save all log records into a persistent storage, like a log file, a database table, or to a remote syslog server.

Different log writers offer possibilities to log into different targets. Custom log writers can extend the functionality shipped with TYPO3 core.

Built-in Log Writers

This section describes the log writers shipped with the TYPO3 core. Some writers have options to allow customization of the particular writer. See the Configuration section for how to use these options.

DatabaseWriter

The database writer logs into a database table. This table has to reside in the database used by TYPO3 and is not automatically created.

Option Mandatory Description Default
logTable no Database table sys_log

Warning

The Admin Tools > Log module is not adapted to the records written by the DatabaseWriter into the sys_log table. If you write such records there, you will not be able to see them using that module.

Tip: There's a tool for viewing such records in the TYPO3 backend at github.com/vertexvaar/logs.

Example of a CREATE TABLE statement for logTable:

#
# Table structure for table 'tx_myextname_log'
#
# The KEY on request_id is optional
#
CREATE TABLE tx_myextname_log (
        request_id varchar(13) DEFAULT '' NOT NULL,
        time_micro double(16,4) NOT NULL default '0.0000',
        component varchar(255) DEFAULT '' NOT NULL,
        level tinyint(1) unsigned DEFAULT '0' NOT NULL,
        message text,
        data text,

        KEY request (request_id)
);

Warning

If you are using a MariaDB Galera Cluster you should definitely add a primary key field to the database definition, since it is required by Galera (this can be a normal uid field as known from other tables): MariaDB Galera Cluster - Known Limitations.

FileWriter

The file writer logs into a log file, one log record per line. If the log file does not exist, it will be created (including parent directories, if needed). Please make sure that your web server has write-permissions to that path and it is below the root directory of your web site (defined by PATH_site). The filename is appended with a hash, that depends on the encryption key. If $GLOBALS['TYPO3_CONF_VARS']['SYS']['generateApacheHtaccess'] is set, an .htaccess file is added to the directory. It protects your log files from being accessed from the web. If the log file is not set, then TYPO3 will use a filename containing a random hash, like typo3temp/logs/typo3_7ac500bce5.log.

Option Mandatory Description Default
logFile no Path to log file typo3temp/logs/typo3_<hash>.log like for example typo3temp/logs/typo3_7ac500bce5.log
PhpErrorLogWriter

Logs into the PHP error log using error_log()

SyslogWriter

Logs into the syslog (Unix only).

Option Mandatory Description Default
facility no Syslog Facility to log into. USER
Custom Log Writers

Custom log writers can be added through extensions. Every log writer has to implement the interface \TYPO3\CMS\Core\Log\Writer\WriterInterface. It is suggested to extend the abstract class \TYPO3\CMS\Core\Log\Writer\AbstractWriter which allows you use configuration options by adding the corresponding properties and setter methods.

Please keep in mind that TYPO3 will silently continue operating, in case a log writer is throwing an exception while executing the writeLog() method. Only in the case that all registered writers fail, the log entry plus additional information will be added to the configured fallback logger (which defaults to the PhpErrorLog writer).

Log Processors

The purpose of a log processor is (usually) to modify a log record or add more detailed information to it.

Log processors allow to manipulate log records without changing the code where the log method actually is called (inversion of control). This enables you to add any information from outside the scope of the actual calling function, for example webserver environment variables. The TYPO3 core ships some basic log processors, but more can be added with extensions.

Built-in Log Processors

This section describes the log processors shipped with the TYPO3 core. Some processors have options to allow customization of the particular processor. See the Configuration section for how to use these options.

IntrospectionProcessor

The introspection processor adds backtrace data about where the log event was triggered.

By default the following parameters from the original function call are added:

file
absolute path to the file.
line
line number.
class
class name.
function
function name.

If appendFullBackTrace is set, the full backtrace stack is added instead.

Option Mandatory Description Default
appendFullBackTrace no Adds a full backtrace stack to the log. TRUE
shiftBackTraceLevel no Removes the given number of entries from the top of the backtrace stack. 0
MemoryUsageProcessor

The memory usage processor adds the amount of used memory to the log record (result from memory_get_usage()).

Option Mandatory Description Default
realMemoryUsage no Use real size of memory allocated from system instead of emalloc() value. TRUE
formatSize no Whether the size is formatted with GeneralUtility::formatSize() TRUE
MemoryPeakUsageProcessor

The memory peak usage processor adds the peak amount of used memory to the log record (result from memory_get_peak_usage()).

Option Mandatory Description Default
realMemoryUsage no Use real size of memory allocated from system instead of emalloc() value. TRUE
formatSize no Whether the size is formatted with GeneralUtility::formatSize() TRUE
WebProcessor

The web processor adds selected webserver environment variables to the log record, i.e. all possible values from \TYPO3\CMS\Core\Utility\GeneralUtility::getIndpEnv('_ARRAY').

Custom Log Processors

Custom log processors can be added through extensions. Every log processor has to implement the interface \TYPO3\CMS\Core\Log\Processor\ProcessorInterface. It is suggested to extend the abstract class \TYPO3\CMS\Core\Log\Processor\AbstractProcessor which allows you use configuration options by adding the corresponding properties and setter methods.

Please keep in mind that TYPO3 will silently continue operating, in case a log processor is throwing an exception while executing the processLogRecord() method.

Error and Exception Handling

Since version 4.3.0 TYPO3 comes with a build-in error and exception handling system. Admins can configure how errors and exceptions should be displayed in the backend and in the frontend. See the examples below.

Configuration

All configuration options related to error and exception handling are part of $TYPO3_CONF_VARS[SYS]:

Key Data type Description
displayErrors integer

Configures whether PHP errors should be displayed.

  • 0 = Do not display any PHP error messages. Overrides the value of "exceptionalErrors" and sets it to 0 (= no errors are turned into exceptions),the configured productionExceptionHandler is used as exception handler
  • 1 = Display error messages with the registered error handler, the configured debugExceptionHandler is used as exception handler
  • -1 = Default setting. With this option, you can override the PHP setting display_errors. If devIPmask matches the users IP address the configured debugExceptionHandler is used for exceptions, if not productionExceptionHandler will be used.
errorHandler string

Classname to handle PHP errors. Leave empty to disable error handling.

Default: \TYPO3\CMS\Core\Error\ErrorHandler. This class will register itself as error handler. It is able to write error messages to all available logging systems in TYPO3. See Logging with TYPO3 for more.

Additionally the errors can be displayed as flash messages in the Backend or in the adminpanel in Frontend. The flash messages in Backend are only displayed if the error and exception handling is in "debug-mode", which is the case when the configured "debugExceptionHandler" is registered as exception handler (see: $TYPO3_CONF_VARS[SYS][displayErrors]).

Errors which are registered as "exceptionalErrors" will be turned into exceptions (to be handled by the configured exceptionHandler).

errorHandlerErrors integer

The E_* constant that will be handled by the error handler

Default: E_ALL ^ E_NOTICE

exceptionalErrors integer

The E_* constant that will be handled as an exception by the error handler.

Default: E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER\_ERROR ^ E_USER\_NOTICE ^ E_USER\_WARNING (4341) and "0" if $TYPO3_CONF_VARS[SYS][displayErrors] = 0.

Refer to the PHP documentation for more details on this value.

productionExceptionHandler string

Classname to handle exceptions that might happen in the TYPO3-code.

Leave empty to disable exception handling.

Default: \TYPO3\CMS\Core\Error\ProductionExceptionHandler. This exception handler displays a nice error message when something went wrong. The error message is logged to the configured logs.

Note

The configured productionExceptionHandler is used if $TYPO3_CONF_VARS[SYS][displayErrors] is set to "0" or to "-1" and $TYPO3_CONF_VARS[SYS][devIPmask] doesn't match.

debugExceptionHandler string

Classname to handle exceptions that might happen in the TYPO3 code.

Leave empty to disable exception handling.

Default: \TYPO3\CMS\Core\Error\DebugExceptionHandler. This exception handler displays the complete stack trace of any encountered exception. The error message and the stack trace is logged to the configured logs.

Note

The configured debugExceptionHandler is used if $TYPO3_CONF_VARS[SYS][displayErrors] is set to "1" or if $TYPO3_CONF_VARS[SYS][displayErrors] is "-1" or "2" and the $TYPO3_CONF_VARS[SYS][devIPmask] matches.

syslogErrorReporting integer

Configures which PHP errors should be logged to the configured syslogs (see: [SYS][systemLog]). If set to "0" no PHP errors are logged to the syslog.

Default: E_ALL ^ E_NOTICE (6135).

belogErrorReporting integer

Configures which PHP errors should be logged to the "sys_log" table (extension: belog). If set to "0" no PHP errors are logged to the "sys_log" table.

Default: E_ALL ^ E_NOTICE (6135).

systemLog boolean Configures whether logging is enabled. Logging itself can be configured in detail via the Logging Framework.

The table below shows which values can be set by the user and which are set by TYPO3.

Values in plain text can be changed in LocalConfiguration.php.

Values in bold are set by TYPO3.

displayErrors errorHandlerErrors exceptionalErrors errorHandler devIPmask exceptionHandler
display_errors
(php_ini)
-1 E_ALL ^ E_NOTICE E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE TYPO3CMS CoreError ErrorHandler Matters If devIPmask matches: debugExceptionHandler Not changed
If devIPmask doesn't match: productionExceptionHandler Not changed
0 E_ALL ^ E_NOTICE 0 (no errors are turned into exceptions) TYPO3CMS CoreError ErrorHandler Doesn't matter production ExceptionHandler 0 (Off)
1 E_ALL ^ E_NOTICE E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE ^ E_USER_WARNING TYPO3CMS CoreError ErrorHandler Doesn't matter debugException Handler 1 (On)
2 E_ALL ^ E_NOTICE E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE ^ E_USER_WARNING TYPO3CMS CoreError ErrorHandler Matters If devIPmask matches: debugExceptionHandler 1 (On)
If devIPmask doesn't match: productionExceptionHandler 0 (Off)

Tip

Search for php error calculator in the web.

PHP constants:

    1  E_ERROR            Fatal run-time errors.
    2  E_WARNING           Run-time warnings (non-fatal errors).
    4  E_PARSE             Compile-time parse errors.
    8  E_NOTICE            Run-time notices.
   16  E_CORE_ERROR        Fatal errors that occur during PHP's initial startup.
   32  E_CORE_WARNING      Warnings (non-fatal errors) that occur during PHP's initial startup.
   64  E_COMPILE_ERROR     Fatal compile-time errors.
  128  E_COMPILE_WARNING   Compile-time warnings (non-fatal errors).
  256  E_USER_ERROR        User-generated error message.
  512  E_USER_WARNING      User-generated warning message.
 1024  E_USER_NOTICE       User-generated notice message.
 2048  E_STRICT            Enable to have PHP suggest changes to your code.
 4096  E_RECOVERABLE_ERROR Catchable fatal error. It indicates that a probably dangerous error occurred, but did not leave the Engine in an unstable state. If the error is not caught by a user defined handle (see also set_error_handler()), the application aborts as it was an E_ERROR.         Since PHP 5.2.0
 8192  E_DEPRECATED        Run-time notices.
16384  E_USER_DEPRECATED   User-generated warning message.
----------------------------------------------------------------------------
32767  E_ALL               (Use ~0 in the code to set all bits.)

Defaults:

30711                      SYS/belogErrorReporting  = E_ALL & ~(E_STRICT | E_NOTICE)

30466                      SYS/errorHandlerErrors  = E_WARNING     | E_USER_ERROR        | E_USER_WARNING |
                                                     E_USER_NOTICE | E_RECOVERABLE_ERROR | E_DEPRECATED   |
                                                     E_USER_DEPRECATED

30711                      SYS/syslogErrorReporting = E_ALL & ~(E_STRICT | E_NOTICE)

Typical in TYPO3 for production:

20480                      SYS/exceptionalErrors   = E_RECOVERABLE_ERROR | E_USER_DEPRECATED

Typical in TYPO3 for development:

28674                      SYS/exceptionalErrors   = E_WARNING | E_RECOVERABLE_ERROR | E_DEPRECATED | E_USER_DEPRECATED

The following sections highlight the roles and goals of the various classes related to error and exception handling. Examples and custom handlers are also discussed.

Error Handler

Class \TYPO3\CMS\Core\Error\ErrorHandler is the default error handler in TYPO3.

Functions:

  • Can be registered for all PHP errors or for only a subset of the PHP errors which will then be handled by an error handler.
  • Displays error messages as flash messages in the Backend (if exceptionHandler is set to \TYPO3\CMS\Core\Error\DebugExceptionHandler). Since flash messages are integrated in the Backend template, PHP messages will not destroy the Backend layout.
  • Displays errors as TsLog messages in the adminpanel.
  • Logs error messages to \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() which is able to write error messages to a file, to the web server's error_log, the system's log and it can send error- and exception-messages by mail. \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() offers a hook and can be extended by userdefined logging methods.
  • Logs error messages to the sys_log table. Logged errors are displayed in the belog extension (Admin Tools > Log). This will work only with an existing DB connection.

Production Exception Handler

Functionality of the \TYPO3\CMS\Core\Error\ProductionExceptionHandler:

  • Shows brief exception message ("Oops, an error occurred!") using TYPO3\CMS\Core\Controller\ErrorPageController and its attendant template.
  • Logs exception messages via the logging API.
  • Logs exception messages to the sys_log table. Logged errors are displayed in the belog extension (Admin Tools > Log). This will only work with an existing DB connection.

Debug Exception Handler

Functions of \TYPO3\CMS\Core\Error\DebugExceptionHandler:

  • Shows detailed exception messages and full trace of an exception.
  • Logs exception messages via the TYPO3 logging framework.
  • Logs exception messages to the sys_log table. Logged errors are displayed in the belog extension (Admin Tools > Log). This will work only if there is an existing DB connection.

Examples

Debugging and development setup

Very verbose configuration which logs and displays all errors and exceptions.

In LocalConfiguration.php:

'SYS' => array(
   'displayErrors' => '1',
   'devIPmask' => '*',
   'errorHandler' => 'TYPO3\\CMS\\Core\\Error\\ErrorHandler',
   'errorHandlerErrors' => E_ALL ^ E_NOTICE,
   'exceptionalErrors' => E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE ^ E_USER_WARNING,
   'debugExceptionHandler' => 'TYPO3\\CMS\\Core\\Error\\DebugExceptionHandler',
   'productionExceptionHandler' => 'TYPO3\\CMS\\Core\\Error\\DebugExceptionHandler',
   'systemLogLevel' => '0',
   'systemLog' => true,
),

In .htaccess:

php_flag display_errors on
php_flag log_errors on
php_value error_log /path/to/php_error.log
Production setup

Example for a production configuration which displays only errors and exceptions if the devIPmask matches. Errors and exceptions are only logged if their level is at least 2 (=Warning).

In LocalConfiguration.php:

'SYS' => array(
   'displayErrors' => '2',
   'devIPmask' => '[your.IP.address]',
   'errorHandler' => 'TYPO3\\CMS\\Core\\Error\\ErrorHandler',
   'systemLogLevel' => '2',
   'systemLog' => true,
   'syslogErrorReporting' => E_ALL ^ E_NOTICE ^ E_WARNING,
   'belogErrorReporting' => '0',
),

In .htaccess:

php_flag display_errors off
php_flag log_errors on
php_value error_log /path/to/php_error.log
Performance setup

Since the error and exception handling and also the logging need some performance, here's an example how to disable error and exception handling completely.

In LocalConfiguration.php:

'SYS' => array(
   'displayErrors' => '0',
   'devIPmask' => '',
   'errorHandler' => '',
   'debugExceptionHandler' => '',
   'productionExceptionHandler' => '',
   'systemLog' => false,
   'syslogErrorReporting' => '0',
   'belogErrorReporting' => '0',
),

In .htaccess:

php_flag display_errors off
php_flag log_errors off

How to extend the error and exception handling

If you want to register your own error or exception handler, simply include the class and insert its name into "productionExceptionHandler", "debugExceptionHandler" or "errorHandler":

$TYPO3_CONF_VARS['SYS']['errorHandler'] = 'myOwnErrorHandler';
$TYPO3_CONF_VARS['SYS']['debugExceptionHandler'] = 'myOwnDebugExceptionHandler';
$TYPO3_CONF_VARS['SYS']['productionExceptionHandler'] = 'myOwnProductionExceptionHandler';

An error or exception handler class must register an error (exception) handler in its constructor. Have a look at the files in EXT:core/Classes/Error/ to see how this should be done.

If you want to use the built-in error and exception handling but extend it with your own functionality, simply derive your class from the error and exception handling classes shipped with TYPO3 and register this class as error (exception) handler:

class tx_postExceptionsOnTwitter extends \TYPO3\CMS\Core\Error\DebugExceptionHandler {
    function echoExceptionWeb(Exception $exception) {
        $this->postExceptionsOnTwitter($exception);
    }
    function postExceptionsOnTwitter($exception) {
        // do it ;-)
    }
}
$TYPO3_CONF_VARS['SYS']['debugExceptionHandler'] = 'tx_postExceptionsOnTwitter';
$TYPO3_CONF_VARS['SYS']['productionExceptionHandler'] = 'tx_postExceptionsOnTwitter';

Using the system registry

The purpose of the registry (introduced in TYPO3 4.3) is to hold key- value pairs of information. You can actually think of it being an equivalent to the Windows registry (just not as complicated).

You might use the registry to store information that your script needs to store across sessions or request.

An example would be a setting that needs to be altered by a PHP script, which currently is not possible with TypoScript.

Another example: The scheduler system extension stores when it ran the last time. The reports system extension then checks that value, in case it determines that the scheduler hasn't run for a while it issues a warning. While this might not be of great use to anyone with an actual cron job set up for the scheduler, it is of use for users that have to run the scheduler tasks by hand due to missing access to a cron job.

The registry is not meant to store things that are supposed to go into a session or a cache, use the appropriate API for these instead.

The registry table (sys_registry)

Here's a description of the fields found in the sys_registry table:

Field Type Description
uid int Primary key, needed for replication and also useful as an index.
entry_namespace varchar (128)

Represents an entry's namespace. In general the namespace is an extension key starting with "tx_", a user script's prefix "user_", or "core" for entries that belong to the core.

The point of namespaces is that entries with the same key can exist inside different namespaces.

entry_key varchar (255) The entry's key. Together with the namespace the key is unique for the whole table. The key can be any string to identify the entry. It's recommended to use dots as dividers if necessary. This way the naming is similar to the already known syntax in TypoScript.
entry_value blob The entry's actual value. The value is stored as a serialized string, thus you can even store arrays or objects in a registry entry – it's not recommended though. Using phpMyAdmin's Show BLOB option you can check the value in that field although being stored as a binary.

The registry API