TYPO3 Core APIs

Previous Key:doc_core_api
Version:latest
Language:en
Description:Reference to the Core APIs of TYPO3, e.g. main classes, Extension API, RTE API.
Keywords:tsref, typoscript, reference, forDevelopers, forAdvanced
Copyright:2000-2017
Author:Documentation Team
Email:documentation@typo3.org
License:Open Publication License available from www.opencontent.org/openpub/
Rendered:2017-05-16 14:52

The content of this document is related to TYPO3 CMS, a GNU/GPL CMS/Framework available from www.typo3.org

Official Documentation

This document is included as part of the official TYPO3 documentation. It has been approved by the TYPO3 Documentation Team following a peer- review process. The reader should expect the information in this document to be accurate - please report discrepancies to the Documentation Team (documentation@typo3.org). Official documents are kept up-to-date to the best of the Documentation Team's abilities.

Core Manual

This document is a Core Manual. Core Manuals address the built in functionality of TYPO3 CMS and are designed to provide the reader with in- depth information. Each Core Manual addresses a particular process or function and how it is implemented within the TYPO3 source code. These may include information on available APIs, specific configuration options, etc.

Core Manuals are written as reference manuals. The reader should rely on the Table of Contents to identify what particular section will best address the task at hand.

Sitemap

Introduction

Overview

TYPO3 is known for its extensibility. However to really benefit from this power, a complete documentation is needed. "Core APIs" and its companion, "Inside TYPO3", aim to provide such information to developers and administrators. Not all areas are covered with the same amount of details, but at least some pointers are provided.

"Inside TYPO3" contains the overall introduction to the architecture of the TYPO3 core. It also contains API descriptions to a certain degree but mostly in the form of examples and short table listings. "Core APIs" goes into much more detail about such APIs and covers subjects more closely related to development.

These documents do not contain any significant information about the frontend of TYPO3. Creating templates, setting up TypoScript objects etc. is not the scope of these documents; they are about the backend part of the core only.

The TYPO3 Documentation Team hopes that these two documents, "Inside TYPO3" and "TYPO3 Core APIs", will form a complete picture of the TYPO3 Core architecture, the backend and be the reference of choice in your work with TYPO3. It took Kasper more than a year to get the first version published and we've tried to maintain it as best we could.

What's new

This version is updated for TYPO3 CMS 6.2.

Many recent changes in the TYPO3 CMS code base were documented in this new version of Core APIs, some dating back to version 6.1. The release of a new LTS version was the opportunity to put extra efforts into the manuals.

Highlights from new features brought by TYPO3 CMS 6.2:

  • an Application Context, backported from TYPO3 Flow.
  • changes to the caching framework, in particuler the new cache groups.
  • well-known folder t3lib is now gone and so is constant PATH_t3lib.
  • a new API for registering AJAX handlers which provides CSRF protection (documentation yet missing, but will come very soon).
  • the system categories API has matured and the related chapter was extended. In particular, it is now possible to have more than one categories field per table.
  • usage of flash messages in Extbase has changed.
  • it is possible to define a custom mirror to fetch extension translations from.

Code examples

Many of the code examples found in this document come from the TYPO3 Core itself.

Quite a few others come from the "examples" extension which is available in the TER. You can install it if you want to try out these examples yourself and use them as a basis for your own stuff.

Yet some other examples just belong to this manual. Some may be moved to the "examples" extension at some later stage.

Feedback

For general questions about the documentation get in touch by writing to documentation@typo3.org .

If you find a bug in this manual, please be so kind as to check the online version on https://docs.typo3.org/typo3cms/CoreApiReference/. From there you can hit the "Edit me on GitHub" button in the top right corner and submit a pull request via GitHub. Alternatively you can just file an issue using the bug tracker: https://github.com/TYPO3-Documentation/TYPO3CMS-Reference-CoreApi/issues.

Maintaining high quality documentation requires time and effort and the TYPO3 Documentation Team always appreciates support. If you want to support us, please join the documentation mailing list/forum (http://forum.typo3.org/index.php/f/44/).

Credits

This manual was originally written by Kasper Skårhøj. It was further maintained, refreshed and expanded by François Suter.

Dedication

I want to dedicate this document to the people in the TYPO3 community who have the discipline to do the boring job of writing documentation for their extensions or contribute to the TYPO3 documentation in general. It's great to have good coders, but it's even more important to have coders with character to carry their work through till the end - even when it means spending days writing good documents. Go for completeness!

- kasper

Extension Architecture

Introduction

TYPO3 can be extended in nearly any direction without loosing backwards compatibility. The Extension API provides a powerful framework for easily adding, removing, installing and developing such extensions to TYPO3. This is in particular powered by the Extension Manager (EM) inside TYPO3 and the online TYPO3 Extension Repository (TER) found at typo3.org for easy sharing of extensions.

"Extensions" is a general term in TYPO3 which covers many kinds of additions to TYPO3. The main types are:

  • Plugins which play a role on the website itself, e.g. a discussion board, guestbook, shop, etc. It is normally enclosed in a PHP class and invoked through a USER or USER_INT cObject from TypoScript. A plugin is an extension in the frontend.
  • Modules are backend applications which have their own entry in the main menu. They require a backend login and work inside the framework of the backend. We might also call something a module if it exploits any connectivity of an existing module, that is if it simply adds itself to the function menu of existing modules. A module is an extension in the backend.
  • Services are libraries that provide a given service through a clearly defined API. A service may exist both in the frontend and the backend. Please refer to the TYPO3 Services Reference for more information about this type of extension.
  • Distributions are fully packaged TYPO3 CMS web installations, complete with files, templates, extensions, etc. Distributions are covered in their own chapter.

Extensions and the Core

Extensions are designed in a way so that extensions can supplement the core seamlessly. This means that a TYPO3 system will appear as "a whole" while actually being composed of the core application and a set of extensions providing various features. This philosophy allows TYPO3 to be developed by many individuals without loosing fine control since each developer will have a special area (typically a system extension) of responsibility which is effectively encapsulated.

So, at one end of the spectrum system extensions make up what is known as "TYPO3" to the outside world. At the other end, extensions can be entirely specific to a given project and contain only files and functionality related to a single implementation.

Files and locations

Files

An extension consists of:

  1. a directory named by the extension key (which is a worldwide unique identification string for the extension)
  2. standard files with reserved names for configuration related to TYPO3 (of which most are optional, see list below)
  3. any number of additional files for the extension itself.

Reserved filenames

This list of filenames are all reserved filenames in the root directory of extensions. None of them are required but for example you cannot have a TYPO3 extension recognized by TYPO3 without the "ext_emconf.php" file etc. You can read more details like that in the table below.

In general, do not introduce your own files in the root directory of extensions with the name prefix "ext_".

Filename Description
ext_emconf.php

Definition of extension properties.


Name, category, status etc. Used by the EM. The content of this file is described in more details below. Note that it is auto-written by EM when extensions are imported from the repository.

Note

If this file is not present the EM will not find the extension.

ext_localconf.php

Addition to LocalConfiguration.php which is included if found. Should contain additional configuration of $TYPO3_CONF_VARS and may include additional PHP class files.


All ext_localconf.php files of included extensions are included right after the typo3conf/LocalConfiguration.php file has been included and database constants defined. Therefore you cannot setup database name, username, password though, because database constants are defined already at this point.

Note

Observe rules for content of these files. See section on caching below.

ext_tables.php

Included if found. Contains extensions of existing tables, declaration of modules, backend styles etc. All code in such files is included after all the default definitions provided by the Core.


Since TYPO3 CMS 6.1, definition of new database tables should be done entirely in Configuration/TCA/(name of the table).php. These files are expected to contain the full TCA of the given table (as an array) and simply return it (with a return statement).


Since TYPO3 CMS 6.2, customizations of existing tables should be done entirely in Configuration/TCA/Overrides/(name of the table).php. This way the TCA changes are cached.

ext_tables.sql

SQL definition of database tables.


This file should contain a table-structure dump of the tables used by the extension. It is used for evaluation of the database structure and is therefore important to check and update the database when an extension is enabled.If you add additional fields (or depend on certain fields) to existing tables you can also put them here. In that case insert a CREATE TABLE structure for that table, but remove all lines except the ones defining the fields you need.


The ext_tables.sql file may not necessarily be "dumpable" directly to MySQL (because of the semi-complete table definitions allowed defining only required fields, see above). But the EM or Install Tool can handle this. The only very important thing is that the syntax of the content is exactly like MySQL made it so that the parsing and analysis of the file is done correctly by the EM.

ext_tables_static+adt.sql

Static SQL tables and their data.


If the extension requires static data you can dump it into a sql-file by this name.Example for dumping mysql data from bash (being in the extension directory):

mysqldump --password=[password] [database name] [tablename] --add-drop-table > ./ext_tables_static.sql

--add-drop-table will make sure to include a DROP TABLE statement so any data is inserted in a fresh table.


You can also drop the table content using the EM in the backend.

Note

The table structure of static tables needs to be in the ext_tables.sql file as well - otherwise an installed static table will be reported as being in excess in the EM!

ext_typoscript_constants.txt

Preset TypoScript constants. Will be included in the constants section of all TypoScript templates.

Warning

Use such a file if you absolutely need to load some TS (because you would get serious errors without it). Otherwise static templates or usage of the Extension Management API are preferred.

ext_typoscript_setup.txt

Preset TypoScript setup. Will be included in the setup section of all TypoScript templates.

Warning

Use such a file if you absolutely need to load some TS (because you would get serious errors without it). Otherwise static templates or usage of the Extension Management API are preferred.

ext_conf_template.txt

Extension Configuration template.


Configuration code in TypoScript syntax setting up a series of values which can be configured for the extension in the EM. Read more about the file format here.


If this file is present the EM provides you with an interface for editing the configuration values defined in the file. The result is written as a serialized array to LocalConfiguration.php in the variable $TYPO3_CONF_VARS['EXT']['extConf'][ *extension_key* ]


If you want to do user processing before the content from the configuration form is saved (or shown for that sake) there is a hook in the EM which is configurable with $TYPO3_CONF_VARS ['SC_OPTIONS']['typo3/mod/tools/em/index.php']['tsStyleConfigForm'][] = " *function reference* "

ext_icon.gif, ext_icon.png or ext_icon.svg

Extension Icon


18x16 GIF, PNG or SVG icon for the extension.

Note

Extension icon will look nicer when provided as vector graphics (SVG) rather than bitmaps (GIF or PNG).

class.ext_update.php

Local Update tool class


If this file is found it will install a new menu item, "UPDATE", in the EM when looking at details for the extension. When this menu item is selected the class inside of this file (named "ext_update") will be instantiated and the method "main()" will be called and expected to return HTML content.


Also you must add the function "access()" and make it return a boolean value whether or not the menu item should be shown. This feature is meant to let you disable the update tool if you can somehow detect that it has already been run and doesn't need to run again.The point of this file is to give extension developers the possibility to provide an update tool if their extensions in newer versions require some updates to be done.

ext_autoload.php

Since TYPO3 CMS 4.3, it is possible to declare classes in this file so that they will be automatically detected by the TYPO3 autoloader. This means that it is not necessary to require the related class files anymore. See the Autoloading chapter for more details.


Not needed anymore since TYPO3 CMS 6.1, when using namespaces.

Reserved folders

The current standard for files location - except for the special files mentioned above - is inspired by TYPO3 Flow. It is necessary to use such structure in Extbase-based extensions and recommended for all extensions anyway.

In order to use Namespaces, class files must be located in a Classes folder.

Refer to the Extbase and Fluid book for more information on extension structure. Also look at the "examples" extension.

The Extension Builder extension will create the right structure for you. It is described below:

Classes/Controller
Contains MVC Controller classes.
Classes/Domain/Model
Contains MVC Domain model classes.
Classes/Domain/Repository
Contains data repository classes.
Classes/ViewHelpers
Helper classes used in the views.
Configuration/TypoScript
TypoScript static setup (setup.txt) and constants (constants.txt). Use subfolders if your have several static templates.
Configuration/TCA
One file per database table, using the name of the table for the file, plus ".php". Only for new tables.
Configuration/TCA/Overrides
For extending existing tables, one file per database table, using the name of the table for the file, plus ".php".
Documentation
Contains the manual in reStructuredText format (read more on the topic).
Resources/Private/Language
XLIFF files for localized labels.
Resources/Private/Layouts
Main layouts for the views.
Resources/Private/Partials
Partial templates for repetitive use.
Resources/Private/Templates
One template per action, stored in a folder named after each Controller.
Resources/Public/Css
Any CSS file used by the extension.
Resources/Public/Images
Any images used by the extension.
Resources/Public/JavaScript
Any JS file used by the extension.
Tests/Unit
Contains unit testing classes.
Legacy structure

The structure of older extensions was not so clearly defined, but it generally adhered to the following conventions:

Filename Description
pi*/ Typical folder for a frontend plugin class.
mod*/ Typical folder for a backend module.
sv*/ Typical folder for a service.
res*/

Extensions normally consist of other files: Classes, images, html- files etc. Files not related to either a frontend plugin (pi/) or backend module (mod/) might be put in a subfolder of the extension directory named "res/" (for "resources") but you can do it as you like (inside of the extension directory that is).The "res/" folder content will be listed as files you can select in the configuration interface.

Files in this folder can also be selected in a selector box if you set up Extension configuration in a "ext_conf_template.txt" file.

System, Global and Local extensions

The files for an extension are located in a folder named by the extension key . The location of this folder can be either inside typo3/sysext/, typo3/ext/ or typo3conf/ext/.

The extension must be programmed so that it does automatically detect where it is located and can work from all three locations. If it is not possible to make the extension that flexible, it is possible to lock its installation requirement to one of these locations in the ext_emconf.php file (see "lockType").

Local extensions

Local extensions are located in the typo3conf/ext/ directory.

This is where to put extensions which are local for a particular TYPO3 installation. The typo3conf directory is always local, containing local configuration (e.g. LocalConfiguration.php), local modules etc. If you put an extension here it will be available for a single TYPO3 installation only. This is a "per-database" way to install an extension.

Note

Local extension can successfully be symlinked to other local extensions on a server as long as they are running under the same TYPO3 source version (which would typically also be symlinked). This method is useful for maintenance of the same local extension running under several sites on a server.

Global extensions

Global extensions are located in the typo3/ext/ directory.

This is a "per-server" way to install an extension; they are global for the TYPO3 source code on the web server. These extensions will be available for any TYPO3 installation sharing the source code.

Note

This features has not been consistently supported in recent versions of TYPO3, so you may encounter problems when using it.

System extensions

System extensions are located in the typo3/sysext/ directory.

This is system default extensions which cannot and should not be updated by the EM. They are distributed with TYPO3 core source code and generally understood to be a part of the core system.

Loading precedence

Local extensions take precedence which means that if an extension exists both in typo3conf/ext/ and typo3/ext/ the one in typo3conf/ext/ is loaded. Likewise global extension takes precedence over system extensions. This means that extensions are loaded in the order of priority local-global-system.

In effect you can therefore have - say - a "stable" version of an extension installed in the global dir (typo3/ext/) which is used by all your projects on a server sharing source code, but on a single experimental project you can import the same extension in a newer "experimental" version and for that particular project the locally available extension will be used instead.

Choosing an extension key

The "extension key" is a string uniquely identifying the extension. The folder where the extension resides is named by this string. The string can contain characters a-z0-9 and underscore. No uppercase characters should be used (keeps folder-,file- and table/field-names in lowercase). Furthermore the name must not start with an "tx" or "u" (this is prefixes used for modules) and because backend modules related to the extension should be named by the extension name without underscores, the extension name must still be unique even if underscores are removed (underscores are allowed to make the extension key easily readable).

The naming conventions of extension keys are automatically validated by the registration at the repository, so you have nothing to worry about here.

There are two ways to name an extension:

  • Project specific extensions (not generally usable or shareable): Select any name you like and prepend it "user_" (which is the only allowed use of a key starting with "u"). This prefix denotes that this extension is a local one which does not come from the central TYPO3 Extension Repository or is ever intended to be shared. Probably this is an "adhoc" extension you have made for some special occasion.
  • General extensions: Register an extension name online at the TYPO3 Extension Repository. Your extension name will automatically be validated and you are sure to have a unique name returned which nobody else in the world uses. This makes it very easy to share your extension later on with every one else, because it ensures that no conflicts with other extension will happen. But by default a new extension you make is defined "private" which means nobody else but you have access to it until you permit it to be public. It's free of charge to register an extension name. By definition all code in the TYPO3 Extension Repository is covered by the GPL license because it interfaces with TYPO3. You should really consider making general extensions!

Tip

It is far easier to settle for the right extension key from the beginning. Changing it later involves a cascade of name changes to tables, modules, configuration files etc. Think carefully.

About GPL and extensions

Remember that TYPO3 is GPL software and at the same moment you extend TYPO3 your extensions are legally covered by GPL. This does not force you to share your extension, but it should inspire you to do so and legally you cannot prevent anyone who gets hold of your extension code from using it and further develop it.The TYPO3 Extension API is designed to make sharing of your work easy as well as using others work easy. Remember TYPO3 is Open Source Software and we rely on each other in the community to develop it further.

Important

It's also your responsibility to make sure that all content of your extensions is legally covered by GPL. The webmaster of TYPO3.org reserves the right to kick out any extension without notice that is reported to contain non-GPL material.

Security

You are responsible for security issues in your extensions. People may report security issues either directly to you or to the TYPO3 Security Team. Whatever the case you should get in touch with the Security Team which will validate the security fixes. They will also include information about your (fixed) extension in their next Security bulletin. If you don't respond to requests from the Security Team, your extension will be forcibly removed from the TYPO3 Extension Repository.

More details on the security team's policy on handling security issues can be found at http://typo3.org/teams/security/extension-security-policy/.

Registering an extension key

Before starting a new extension you should register an extension key on typo3.org (unless you plan to make an implementation-specific extension – of course – which it does not make sense to share).

Go to typo3.org, log in with your (pre-created) username / password and go to Extensions > Extension Keys and click on the "Register keys" tab. On that page you can enter the key name you want to register.

The extension registration form

The extension registration form on typo3.org.

Naming conventions

Based on the extension key of an extension these naming conventions should be followed:

  General Example User-specific Example

Extension key

(Lowercase "alnum" + underscores. )

Assigned by the TYPO3 Extension Repository. cool_shop Determined by yourself, but prefixed "user_" user_my_shop
Database tables and fields Prefix with "tx_[ key ]_" where key is without underscores!

Prefix: tx_coolshop_

Examples:

tx_coolshop_products

tx_coolshop_categories

Prefix with "[ key ]_"

Prefix: user_my_shop_

Examples:

user_my_shop_products

user_my_shop_categories

Backend module

(Names are always without underscores!)

Name: The extension key name without underscores, prefixed "tx" txcoolshop Name: No underscores, prefixed "u" uMyShop or umyshop or ...

For frontend PHP classes, follow the same conventions as for database tables and field, but prepend class file names with class.

You may also want to refer to the TYPO3 Core Coding Guidelines for more on general naming conventions in TYPO3.

Tip

If you study the naming conventions above closely you will find that they are complicated due to varying rules for underscores in key names. Sometimes the underscores are stripped off, sometimes not.

The best practice you can follow is to avoid using underscores in your extensions keys at all! That will make the rules simpler. This is highly encouraged.

Note on "old" extensions:

Some the "classic" extensions from before the extension structure came about do not comply with these naming conventions. That is an exception made for backwards compatibility. The assignment of new keys from the TYPO3 Extension Repository will make sure that any of these old names are not accidentially reassigned to new extensions.

Further, some of the classic plugins (tt_board, tt_guest etc) users the "user_" prefix for their classes as well.

Extending "extensions classes"

As a standard procedure you should include the "class extension code" even in your own extensions. This is placed at the bottom of every class file:

if (defined('TYPO3_MODE') && isset($GLOBALS['TYPO3_CONF_VARS'][TYPO3_MODE]['XCLASS']['ext/myext/pi1/class.tx_myext_pi1.php'])) {
        include_once($GLOBALS['TYPO3_CONF_VARS'][TYPO3_MODE]['XCLASS']['ext/myext/pi1/class.tx_myext_pi1.php']);
}

Normally the key used as example here ("ext/myext/pi1/class.tx_myext_pi1.php") would be the full path to the script relative to the PATH_site constant. However because modules are required to work from both typo3/sysext/, typo3/ext/ and typo3conf/ext/ it is a policy that any path before "ext/" is omitted.

Installing extensions

There are only two (possibly three) steps involved in using extensions with TYPO3:

  1. You must import it.

    This simply means to copy the extensions files into the correct directory in either typo3/ext/ (global) or typo3conf/ext/ (local). More commonly you import an extension directly from the online TYPO3 Extension Repository (TER). When an extension is found located in one of the extension locations, it is available to the system.

    The Extension Manager (EM) should take care of this process, including updates to newer versions if needed.Notice that backend modules will have their "conf.php" file modified in the install process depending on whether they are installed locally or globally!

  2. You must install it.

    An extension is loaded only if its state is set to active in the PackageStates.php file. Extensions are loaded in the order they appear in this list.

    An enabled extension is always global to the TYPO3 Installation - you cannot disable an extension from being loaded in a particular branch of the page tree.The EM takes care enabling extensions. It's highly recommended that the EM is doing this, because the EM will make sure the priorities, dependencies and conflicts are managed according to the extension characteristics, including clearing of the cache-files if any.

  3. You might need to configure it.

    Certain extensions may allow you to configure some settings. Again the EM is able to handle the configuration of the extensions based on a certain API for this. Any settings - if present - configured for an extension are available as an array in the variable $TYPO3_CONF_VARS["EXT"]["extConf"][extension key].

Loaded extensions are registered in a global variable, $TYPO3_LOADED_EXT, available in both frontend and backend of TYPO3.

This is how the data structure for an extension in this array looks:

$TYPO3_LOADED_EXT[extension key] = array(
        "type" =>                S, G, L for system, global or local type of availability.
        "siteRelPath" => Path of extension dir relative to the PATH_site constant
                                e.g. "typo3/ext/my_ext/" or "typo3conf/ext/my_ext/"
        "typo3RelPath" => Path of extension dir relative to the "typo3/" admin folder
                                e.g. "ext/my_ext/" or "../typo3conf/ext/my_ext/"
        "ext_localconf" => Contains absolute path to 'ext_localconf.php' file if present
        "ext_tables" => [same]
        "ext_tables_sql" => [same]
        "ext_tables_static+adt.sql" => [same]
        "ext_typoscript_constants.txt" => [same]
        "ext_typoscript_setup.txt" => [same]
        "ext_typoscript_editorcfg.txt" => [same]
)

The order of the registered extensions in this array corresponds to the order they were listed in PackageStates.php.

The inclusion of ext_tables.php or ext_localconf.php files (see next chapter) is done by traversing (a copy of) the $TYPO3_LOADED_EXT array.

Declaration file

The ext_emconf.php is the single most important file in an extension. Without it, the Extension Manager (EM) will not detect the extension, much less be able to install it. This file contains a declaration of what the extension is or does for the EM. The only thing included is an associative array, $EM_CONF[extension key]. The keys are described in the table below.

When extensions are imported from the online repository this file is written anew! So don't put any custom stuff in there - only change values in the $EM_CONF array if needed.

Key Data type Description
title string, required The name of the extension in English.
description string, required Short and precise description in English of what the extension does and for whom it might be useful.
category string

Which category the extension belongs to:

  • be

    Backend (Generally backend-oriented, but not a module)

  • module

    Backend modules (When something is a module or connects with one)

  • fe

    Frontend (Generally frontend oriented, but not a "true" plugin)

  • plugin

    Frontend plugins (Plugins inserted as a "Insert Plugin" content element)

  • misc

    Miscellaneous stuff (Where not easily placed elsewhere)

  • services

    Contains TYPO3 services

  • templates

    Contains website templates

  • example

    Example extension (Which serves as examples etc.)

  • doc

    Documentation (e.g. tutorials, FAQ's etc.)

  • distribution

    Distribution, an extension kickstarting a full site

constraints array

List of requirements, suggestions or conflicts with other extensions or TYPO3 or PHP version. Here's how a typical setup might look:

'constraints' => array(
    'depends' => array(
        'typo3' => '4.5.0-6.1.99',
        'php' => '5.3.0-5.5.99'
    ),
    'conflicts' => array(
        'dam' => ''
    ),
    'suggests' => array(
        'tt_news' => '2.5.0-0.0.0'
    )
)
depends
List of extensions that this extension depends on. Extensions defined here will be loaded before the current extension.
conflicts
List of extensions which will not work with this extension.
suggests

List of suggestions of extensions that work together or enhance this extension. Extensions defined here will be loaded before the current extension. Dependencies take precedence over suggestions.

Note: If a "suggested" extension depends on the current extension (directly or indirectly), the suggestion is not taken into account for loading order calculation. Read more at Forge #57825.

The above example indicated that the extension depends on a version of TYPO3 between 4.5 and 6.1 (as only bug and security fixes are integrated into TYPO3 when the last digit of the version changes, it is safe to assume it will be compatible with any upcoming version of the corresponding branch, thus .99). Also the extension has been tested and is known to work properly with PHP 5.3, 5.4 and 5.5. It will conflict with the DAM (any version) and it is suggested that it might be worth installing "tt_news" (version at least 2.5.0).

state string

Which state is the extension in

  • alpha

    Alpha state is used for very initial work, basically the state is has during the very process of creating its foundation.

  • beta

    Under current development. Beta extensions are functional but not complete in functionality. Most likely beta-extensions will not be reviewed.

  • stable

    Stable extensions are complete, mature and ready for production environment. You will be approached for a review. Authors of stable extensions carry a responsibility to be maintain and improve them.

  • experimental

    Experimental state is useful for anything experimental - of course. Nobody knows if this is going anywhere yet... Maybe still just an idea.

  • test

    Test extension, demonstrates concepts etc.

  • obsolete

    The extension is obsolete or deprecated. This can be due to other extensions solving the same problem but in a better way or if the extension is not being maintained anymore.

  • excludeFromUpdates

    This state makes it impossible to update the extension through the extension manager (neither by the Update mechanism, nor by uploading a newer version to the installation). This is very useful if you made local changes to an extension for a specific installation and don't want any admin to overwrite them.

    New since TYPO3 4.3.

uploadfolder boolean If set, then the folder named "uploads/tx_[extKey-with-no- underscore]" should be present!
createDirs list of strings Comma list of directories to create upon extension installation.
clearCacheOnLoad boolean If set, the EM will request the cache to be cleared when this extension is loaded.
author string Author name (Use a-z)
author_email email address Author email address
author_company string Author company (if any company sponsors the extension).
docPath string

Path to documentation. This has never been fully supported neither by the TER nor by the Extension Manager. The documentation is expected to be in folder doc when using OpenOffice/LibreOffice format and in folder Documentation when using reStructuredText (recommended). See Adding documentation for more information.

Deprecated

CGLcompliance keyword

Compliance level that the extension claims to adhere to. A compliance defines certain coding guidelines, level of documentation, technical requirements (like XHTML, DBAL usage etc).

Deprecated

CGLcompliance_note string

Any remarks to the compliance status. Might describe some minor incompatibilities or other reservations.

Deprecated

private boolean

If set, this version of the extension is not included in the public list!

Not supported anymore

download_password string

If set, this password must additionally be specified if people want to access (import or see details for) this the extension.

Not supported anymore

version main.sub.dev Version of the extension. Automatically managed by EM / TER. Format is [int].[int].[int]

Configuration files

Files ext_tables.php and ext_localconf.php are the two most important files for the execution of extensions within TYPO3. They contain configuration used by the system on almost every request. They should therefore be optimized for speed.

  • ext_localconf.php is always included in global scope of the script, either frontend or backend.

    While you can put functions and classes into the script, it is a really bad practice because such classes and functions would always be loaded. It is better to have them included only as needed.

    So stick to changing values in TYPO3_CONF_VARS only!

  • ext_tables.php is not always included in global scope (in the frontend)

    It should still not contain functions and classes as it still very often loaded.

  • Use the API of class ExtensionManagementUtility for tasks such as adding tables, merging information into arrays, etc.

  • Before the inclusion of any of the two files, the variables $_EXTKEY is set to the extension key and $_EXTCONF is set to the configuration from $TYPO3_CONF_VARS["EXT"]["extConf"][extension key]. We recommend to not use $_EXTKEY, current planning is to deprecate this variable in the future.

  • $TYPO3_LOADED_EXT[extension key] contains information about whether the module is loaded as local, global or system type, including the proper paths you might use, absolute and relative.

  • Your ext_tables.php and ext_localconf.php file must be designed so that they can safely be read and subsequently imploded into one single file with all the other configuration scripts!

  • You must never use a "return" statement in the files global scope - that would make the cached script concept break.

  • You must never use a "use" statement in the files global scope - that would make the cached script concept break and could conflict with other extensions.

  • You should not rely on the PHP constant __FILE__ for detection of include path of the script - the configuration might be executed from a cached script and therefore such information should be derived from e.g. \TYPO3\CMS\Core\Utility\GeneralUtility::getFileAbsFileName() or ExtensionManagementUtility::extPath().

Best practice for ext_tables.php and ext_localconf.php

It is a good practice to use call_user_func with an closure function. The following example contains the complete code:

<?php

if (!defined('TYPO3_MODE')) {
   die('Access denied.');
}

call_user_func(function () {
   // Add your code here
});

In most cases, the file ext_tables.php is no longer needed, since most of the code can be placed in Configuration\TCA\*.php files.

Configuration options

In the ext_conf_template.txt file configuration options for an extension can be defined. They will be accessible in the TYPO3 backend from the Extension Manager.

There's a specific syntax to declare these options properly, which is similar to the one used for TypoScript constants (see "Declaring constants for the Constant editor" in "TypoScript Syntax and In-depth Study"). This syntax applies to the comment line that should be placed just before the constant. Consider the following example (taken from system extension "rsaauth"):

# cat=basic/enable; type=string; label=Path to the temporary directory:This directory will contain...
temporaryDirectory =

First a category (cat) is defined ("basic") with the subcategory "enable". Then a type is given ("string") and finally a label, which is itself split (on the colon ":") into a title and a description (this should actually be a localized string). The above example will be rendered like this in the EM:

Configuration screen for the rsaauth extension

The configuration tab displays all options from a single category. A selector is available to switch between categories. Inside an option screen, options are grouped by subcategory. At the bottom of the screenshot, the label – split between header and description – is visible. Then comes the field itself, in this case an input, because the option's type is "string".

Once you saved the configuration in the ExtensionManager, it will be stored in $GLOBALS['TYPO3_CONF_VARS']['EXT']['extConf']['your_extension_key'] as a serialized array.

To fetch the value of temporaryDirectory from the example above, you could simply use:

$extensionConfiguration = unserialize($GLOBALS['TYPO3_CONF_VARS']['EXT']['extConf']['your_extension_key']);
$temporaryDirectory = $extensionConfiguration['temporaryDirectory'];

Or even better use the API to get the information merged with the default settings if the settings have not been saved yet:

/** @var \TYPO3\CMS\Extensionmanager\Utility\ConfigurationUtility $configurationUtility */
$configurationUtility = $this->objectManager->get('TYPO3\CMS\Extensionmanager\Utility\ConfigurationUtility');
$extensionConfiguration = $configurationUtility->getCurrentConfiguration('themes');

You can also define nested options using the TypoScript notation:

directories {
   # cat=basic/enable; type=string; label=Path to the temporary directory
   tmp =
   # cat=basic/enable; type=string; label=Path to the cache directory
   cache =
}

This will result in a multidimensional array:

$extensionConfiguration = unserialize($GLOBALS['TYPO3_CONF_VARS']['EXT']['extConf']['your_extension_key']);
$extensionConfiguration['directories.']['tmp']
$extensionConfiguration['directories.']['cache']

Important

Notice the dot at the end of the directories key. This notation must be used for every grouping key and is a convention of the TypoScript parser.

Extending the $TCA array

Being a PHP array, the Table Configuration Array can be easily extended. It can be accessed as the global variable $GLOBALS['TCA']. TYPO3 also provides APIs for making this simpler.

Storing the changes

There are various ways to store changes to $GLOBALS['TCA']. They depend - partly - on what you are trying to achieve and - a lot - on the version of TYPO3 CMS which you are targeting.

There are two main ways to store your changes to the TCA: inside an extension or straight in the typo3conf folder. Both are described below in more details.

Storing in extensions

The advantage of putting your changes inside an extension is that they are nicely packaged in a self-contained entity which can be easily deployed on multiple servers.

The drawback is that the extension loading order must be finely controlled. Indeed if your extension modifies another extension, your extension must be loaded after the extension you are modifying. This can be achieved by registering that other extension as a dependency of yours. See the description of constraints in Core APIs.

For more information about an extension's structure, please refer to the extension architecture chapter in Core APIs.

Storing in ext_tables.php files

Until TYPO3 CMS 6.1 (still supported for 6.2) changes to $GLOBALS['TCA'] are packaged into an extension's ext_tables.php file. This is strongly discouraged in more recent versions of TYPO3 CMS.

Storing in the Overrides folder

Since TYPO3 CMS 6.2 (6.2.1 to be precise) changes to $GLOBALS['TCA'] must be stored inside a folder called Configuration/TCA/Overrides with one file per modified table. These files are named along the pattern <tablename>.php.

Thus if you want to customize the TCA of tx_foo_domain_model_bar, you'd create the file Configuration/TCA/Overrides/tx_foo_domain_model_bar.php.

The advantage of this method is that all such changes are incorporated into $GLOBALS['TCA'] before it is cached. This is thus far more efficient.

Important

Be aware that you cannot extend the TCA of extensions if it was configured within its ext_tables.php file, usually containing the "ctrl" section referencing a "dynamicConfigFile". Please ask the extension author to switch to the Configuration/TCA/<tablename>.php setup.

Important

Only TCA-related changes should go into Configuration/TCA/Overrides files. Some API calls may be okay as long as they also manipulate only $GLOBALS['TCA']. For example, it is fine to register a plugin with \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addPlugin() in Configuration/TCA/Overrides/tt_content.php because that API call only modifies $GLOBALS['TCA'] for table "tt_content".

Changing the TCA "on the fly"

It is also possible to perform some special manipulations on $GLOBALS['TCA'] right before it is stored into cache, thanks to the tcaIsBeingBuilt signal. This signal was introduced in TYPO3 CMS 6.2.1.

Customization examples

Many extracts can be found throughout the manual, but this section provides more complete examples.

Example 1: extending the fe_users table

The "examples" extension adds two fields to the "fe_users" table. Here's the complete code, taken from file Configuration/TCA/Overrides/fe_users.php:

<?php
if (!defined('TYPO3_MODE')) {
        die ('Access denied.');
}


// Add some fields to FE Users table to show TCA fields definitions
// USAGE: TCA Reference > $GLOBALS['TCA'] array reference > ['columns'][fieldname]['config'] / TYPE: "select"
$temporaryColumns = array (
        'tx_examples_options' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options',
                'config' => array (
                        'type' => 'select',
                        'items' => array (
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.0', '1'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.1', '2'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.2', '--div--'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.3', '3'),
                        ),
                        'size' => 1,
                        'maxitems' => 1,
                )
        ),
        'tx_examples_special' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_special',
                'config' => array (
                        'type' => 'user',
                        'size' => '30',
                        'userFunc' => 'Documentation\\Examples\\Userfuncs\\Tca->specialField',
                        'parameters' => array(
                                'color' => 'blue'
                        )
                )
        ),
);

\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns(
        'fe_users',
        $temporaryColumns
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addToAllTCAtypes(
        'fe_users',
        'tx_examples_options, tx_examples_special'
);

First of all, the fields that we want to add are detailed according to the $GLOBALS['TCA'] syntax for columns. This configuration is stored in the $temporaryColumns array.

After that come two additional steps:

  • first the columns are actually added to the table by using \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns().
  • then the fields are added to the "types" definition of the "fe_users" table by using \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addToAllTCAtypes(). It is possible to be more fine-grained.

This does not create the corresponding fields in the database. The new fields must also be defined in the ext_tables.sql file of the extension:

CREATE TABLE fe_users (
        tx_examples_options int(11) DEFAULT '0' NOT NULL,
        tx_examples_special varchar(255) DEFAULT '' NOT NULL
);

Warning

The above statement uses the SQL CREATE TABLE statement. This is the way TYPO3 expects it to be. The Extension Manager will automatically transform this into a ALTER TABLE statement when it detects that the table already exists.

By default new fields are added at the bottom of the form when editing a record from that table. If the table uses tabs, new fields are added at the bottom of the "Extended" tab (this tab is created if it does not exist). The following screenshot shows the placement of the two new fields when editing a "fe_users" record:

New fields for fe\_users table

The new fields added at the bottom of the "Extended" tab

The next example shows how to place a field more precisely.

Example 2: extending the tt_content table

In this second example, we will add a "No print" field to all content element types. First of all, we add its SQL definition in ext_tables.sql:

CREATE TABLE tt_content (
        tx_examples_noprint tinyint(4) DEFAULT '0' NOT NULL
);

Then we add it to the $GLOBALS['TCA'] in Configuration/TCA/Overrides/tt_content.php:

$temporaryColumn = array(
        'tx_examples_noprint' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:tt_content.tx_examples_noprint',
                'config' => array (
                        'type' => 'check',
                )
        )
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns(
        'tt_content',
        $temporaryColumn
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addFieldsToPalette(
        'tt_content',
        'visibility',
        'tx_examples_noprint',
        'after:linkToTop'
);

The code is mostly the same as in the first example, but the last line is very different and requires an explanation. The "pages" and "tt_content" use palettes extensively for all fields and not just for secondary options, for increased flexibility. So in this case we use addFieldsToPalette() instead of addToAllTCAtypes(). We need to specify the palette's key as the second argument (visibility). Precise placement of the new field is achieved with the fourth parameter (after:linkToTop). This will place the "no print" field right after the "link to top" field, instead of putting it in the "Extended" tab.

The result is the following:

New fields for tt\_content table

The new field added next to an existing one

Note

Obviously this new field will no magically exclude a content element from being printed. For it to have any effect, it must be used during the rendering by modifying the TypoScript used to render the "tt_content" table. Although this is outside the scope of this manual, here is an example of what you could do, for the sake of showing a complete process.

Assuming you are using "css_styled_content" (which is installed by default), you could add the following TypoScript to your template:

tt_content.stdWrap.outerWrap = <div class="noprint">|</div>
tt_content.stdWrap.outerWrap.if.isTrue.field = tx_examples_noprint

This will wrap a "div" tag with a "noprint" class around any content element that has its "No print" checkbox checked. The final step would be to declare the appropriate selector in the print-media CSS file so that "noprint" elements don't get displayed.

This is just an example of how the effect of the "No print" checkbox can be ultimately implemented. It is meant to show that just adding the field to the $GLOBALS['TCA'] is not enough.

Verifying the $TCA

You may find it necessary – at some point – to verify the full structure of the $GLOBALS['TCA'] in your TYPO3 installation. The System > Configuration module makes it possible to have an overview of the complete $GLOBALS['TCA'], with all customizations taken into account.

The Configuration module

Checking the existence of the new field via the Configuration module

If you cannot find your new field, it probably means that you have made some mistake.

This view is also useful when trying to find out where to insert a new field, to explore the combination of types and palettes that may be used for the table that we want to extend.

The Extension Manager (EM)

Extensions are managed from the Extension Manager inside TYPO3 by "admin" users. The module is located at "Admin tools > Ext Manager" and offers a menu with options to see loaded extensions (those that are installed or activated), available extensions on the server and the possibility to import extensions from online resources, typically the TER (TYPO3 Extension Repository) located at typo3.org.

The Extension Manager (from TYPO3 7.6 to TYPO3 8.7)

Interface of the Extension Manager (from TYPO3 7.6 to TYPO3 8.6) showing all available extensions.

The interface is really easy to use. You just click the +/- icon to the left of an extension in order to install it and follow the instructions.

Creating a new extension

This chapter is not a tutorial about how to create an Extension. It only aims to be a list of steps to perform and key information to remember.

First you have to register an extension key. This is the unique identifier for your extension.

Kickstarting the extension

Although it is possible to write every single line of an extension from scratch, there is tool which makes it easier to start. It is called "Extension builder" (key: "extension_builder") and can be installed from TER.

The Extension Builder comes with its own BE module:

A view from the Extension Builder

The Domain Modeller screen of the Extension Builder. The comfort of building your model with drag and drop.

Note that this tool is not a complete editor. It helps you creating the scaffolding of your extension, generating the necessary files. It's then up to you to fill these with the relevant code.

Warning

The Extension Builder has some possibility to preserve code, but it should still be used with care.

After the extension is written to your computer's disk you will be able to install it locally and start using it.

Please refer to the Extension Builder's manual for more information.

Creating a new distribution

This chapter describes the main steps in creating a new distribution. It should not be considered as a full fledge tutorial.

Concept of distributions

Distributions are full TYPO3 CMS websites ready to be unpacked. They provide an easy quickstart for using TYPO3 CMS.

A distribution takes care of the following parts:

  • Deliver initial database data
  • Deliver fileadmin files
  • Deliver configuration for a package
  • Hook into the process after saving configuration to trigger actions dependent on configuration values
  • Deliver dependent extensions (e.g., customized versions or extensions not available through TER)

Kickstarting the distribution

A distribution is a special kind of extension. The first step is thus to create a new extension. Start by registering an extension key, which will be the unique identifier of your distribution.

Next create the Extension declaration file as usual, except for the "category" property which must be set to distribution.

Configuring the distribution display in the EM

You should provide two preview images for your distribution. Provide a small 220x150 pixels for the list in the extension manager as Resources/Public/Images/Distribution.png and a larger 300x400 pixels welcome image as Resources/Public/Images/DistributionWelcome.png. The welcome image is displayed in the distribution detail view inside the extension manager.

Fileadmin files

Create the following folder structure inside your extension:

  • Initialisation
  • Initialisation/Files

All the files inside that second folder will be copied to fileadmin/<extkey> during installation, where "extkey" is the extension key of your distribution.

Database data

The database data is delivered as TYPO3 CMS export data.t3d. Generate this file by exporting your whole installation from the tree root with the import/export module. Make sure to include all tables in the export.

The file has to be name data.t3d and must be located in the Initialisation folder.

Distribution configuration

A distribution is technically handled as an extension. Therefore your can make use of all configuration options as needed.

After saving the configuration, the signal afterExtensionConfigurationWrite is dispatched. You may use this to alter your website configuration (e.g. color scheme) on the fly.

Delivering custom dependencies

Normally extension dependencies are setup in the Extension declaration file.

However sometimes, extensions are not available in the TYPO3 Extension Repository (TER). Therefore, a distribution can act as its own extension repository. Add unpacked extensions to Initialisation/Extensions/ to provide dependencies. Your main extension has to be dependent on these extension as normal dependencies in ext_emconf.php.

Extensions delivered inside an extension have the highest priority when extensions need to be fetched.

Caution

This will not overwrite extensions already present in the system.

Test your distribution

To test your distribution, simply copy your extension to an empty TYPO3 CMS installation and try to install it from the Extension Manager.

Warning

It is not enough to clean all files and the page tree if you want to try again to install your distribution. Indeed, TYPO3 CMS remembers that it previously imported your distribution and will skip any known files. Make sure to clean the table "sys_registry" if you want to work around that.

More information

Some additional backgrounds can be retrieved from the blueprint for this feature.

Adding documentation

If you plan to upload your extension to the TYPO3 Extension Repository (TER), you should first consider adding a documentation to your extension. A documentation will help users and administrators to quickly install and configure your extension and give it more weight.

The documentation platform https://docs.typo3.org centralizes documentation for every project. It supports two different kind of documentation:

  1. (recommended) A Sphinx project, stored within EXT:extkey/Documentation/
  2. A simple README file stored as EXT:extkey/README.rst as seen on Github

Sphinx project

Sphinx is the official format for official TYPO3 documentation. A Sphinx-based documentation is a set of plain text files making up the chapters or sections of the documentation. It uses a markup language called "reStructuredText" (reST).

Advantages of this new documentation format are numerous:

  • Output formats: Sphinx projects may be automatically rendered as HTML or TYPO3-branded PDF.
  • Cross-references: It is easy to cross-reference other chapters and sections of other manuals (either TYPO3 references or extension manuals).
  • Multilingual: Unlike OpenOffice, Sphinx projects may be easily localized and automatically presented in the most appropriate language to TYPO3 users.
  • Collaboration: As the documentation is plain text, it is easy to work as a team on the same manual or quickly review changes using any versioning system.

Although it is possible to write every single line of a Sphinx-based documentation from scratch, the TYPO3 community provides tools that help write and manage Sphinx projects:

  • The extension "Sphinx" (Sphinx Python Documentation Generator and Viewer) installs a local Sphinx environment to view, edit and compile documentation in the backend of your TYPO3 website. It can be installed from the TYPO3 Extension Repository (TER) like any other extension.
  • The Sphinx extension is able to convert existing OpenOffice manuals (manual.sxw) into Sphinx projects with just one click.
  • An example manual is available on the TYPO3 Documentation Github repository.
  • The Extension Builder provides a skeleton documentation based on the above-mentioned Git repository.
  • A good primer to get started using the reStructuredText markup.

README.rst

A "README.rst" is a simple text file stored at the root of your extension directory and briefly describing the purpose of your extension. It is best suited when installing or using your extension is straightforward. The format of this file is reStructuredText, as for chapters of a Sphinx project.

Tip

In TYPO3 6.2, the system extension "documentation" is using such a simple manual.

Other resources

Beyond the general overview given in this chapter, other sections in this manual will be of particular interest to extension developers:

TYPO3 API

The source is the documentation! (General wisdom)

The TYPO3 APIs are first and foremost documented inside of the source scripts. It would be impossible to maintain documentation at more than one location given the fact that things change and sometimes fast. This chapter describes the most important elements of the API. Some other elements have their own chapter further on.

In general the TYPO3 Active Contributors Team and all contributors aim to produce well-documented source code. As such the best way to look up a particular class or method is to use the online API browser at http://api.typo3.org/.

Namespaces

Since version 6.0, TYPO3 CMS uses PHP namespaces for all classes in the Core.

The general structure of namespaces is the following:

\{VendorName}\{PackageName}\({CategoryName}\)*{ClassName}

For the Core, the vendor name is TYPO3\CMS and the package name corresponds to a system extension.

All classes must be located inside the Classes folder at the root of the (system) extension. The category name may contain several segments that correspond to the path inside the Classes folder.

Finally the class name is the same as the corresponding file name, without the .php extension.

"UpperCamelCase" is used for all segments.

Tip

See the chapter about 'ClassAliasMap.php' in the 6.2 documentation.. It may help you with migrating code from old to new conventions.

Core example

The good old t3lib_div class has been renamed to:

\TYPO3\CMS\Core\Utility\GeneralUtility

This means that the class is now found in the "core" system extension, in folder Classes/Utility, in a file named GeneralUtility.php.

Usage in extensions

Extension developers are free to use their own vendor name. Important: It may consist of one segment only. Vendor names must start with an uppercase character and are usually written in UpperCamelCase style. In order to avoid problems with different filesystems, only the characters a-z, A-Z, 0-9 and the dash sign "-" are allowed for package names – don't use special characters:

// good vendor name:
\Webcompany

// wrong vendor name:
\Web\Company

Attention

The vendor name TYPO3\CMS is reserved and may not be used by extensions!

The package name corresponds to the extension key. Underscores in the extension key are removed in the namespace and replaced by upper camel-case. So extension key:

weird-name_examples

would become:

Weird-nameExamples

in the namespace.

As mentioned above, all classes must be located in the Classes folder inside your extension. All sub-folders translate to a segment of the category name and the class name is the file name without the .php extension.

Looking at the "examples" extension, class:

examples/Classes/Controller/DefaultController.php

corresponds to namespace:

\Documentation\Examples\Controller\DefaultController

Inside the class, the namespace is declared as:

<?php
namespace Documentation\Examples\Controller;

Namespaces in Extbase

When registering components in Extbase, the vendor name must be used on top of the extension key.

For a backend module:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::registerModule(
    '<vendorName>.<ExtensionName>',
    // ...
);

For a frontend module:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::configurePlugin(
    '<vendorName>.<ExtensionName>',
    // ...
);

Important

  • Do not forget the dot after the vendor name.
  • Do not use dots inside the vendor name.

Namespaces for test classes

As for ordinary classes, namespaces for test classes start with a vendor name followed by the extension key.

All test classes reside in a Tests folder and thus the third segment of the namespace must be "Tests". Unit tests are located in a Unit folder which is the fourth segment of the namespace. Any further subfolders will be subsequent segments.

So a test class in EXT:foo_bar_baz/Tests/Unit/Bla/ will have as namespace \Vendor\FooBarBaz\Tests\Unit\Bla.

Creating instances

When creating instances using \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance() the leading backslash must be omitted and all other backslashes escaped, even when using single quotes. Thus the following code is correct:

$contentObject = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::class);

There is no need to use require() or include() statements. All classes that follow namespace conventions will automatically located and included by the autoloader.

References

For more information about PHP namespaces in general, you may want to refer to the PHP documentation and in particular the Namespaces FAQ.

Autoloading

The autoloader takes care of finding classes in TYPO3. It is closely related to \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance() which takes care of singleton and XCLASS handling.

As a developer you should always instantiate classes either through \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance() or with the Extbase ObjectManager (which internally uses makeInstance() again).

Autoloading classes since TYPO3 7.x

TYPO3 6.2 was still delivered with a couple of different autoloaders, that all had different approaches and rules to find a class. This led to the naming conventions in and outside extbase and the optional ext_autoload.php file to load classes that didn't follow the conventions. Since TYPO3 7.0 all this is gone and there is only a single autoloader left, the one of composer. No matter if you run TYPO3 in composer mode or not, TYPO3 uses the composer autoloader to resolve all class file locations. However, the autoloader is little bit more sophisticated in composer mode as it then supports PSR-4 autoloading.

Loading classes without composer mode

This means, you did not install TYPO3 via a require-statement inside your composer.json. It's a regular old-school install where the TYPO3 source and the symlinks (typo3/index.php) are setup manually. In this case, every time you install an extension, the autoloader scans the whole extension directory for classes. No matter if they follow any convention at all. There is just one rule. Put each class into its own file. The generated classmap is a huge array with a mapping of classnames to their location on the disk.

Example:

<?php

// autoload_classmap.php @generated by TYPO3

$typo3InstallDir = PATH_site;

return array(
   'Schnitzler\\Templavoila\\Clipboard\\Clipboard' => $typo3InstallDir . 'typo3conf/ext/templavoila/Classes/Clipboard/Clipboard.php',
   'tx_templavoila_pi1' => $typo3InstallDir . 'typo3conf/ext/templavoila/Compatibility/class.tx_templavoila_pi1.php',
   ...
);

This method is failsafe unless the autoload information cannot be written. In this case, check the install tool for warnings and make sure that typo3temp is writable.

Troubleshooting:

If your classes cannot be found, try the following approaches.

  • Dump the class loading information manually with the following command: php typo3/cli_dispatch.phpsh extbase extension:dumpclassloadinginformation
  • If that command itself fails, please (manually) uninstall the extension and simply try reinstalling it (via the extension manager).
  • If you are still not lucky, the issue is definitely on your side and you should double check the write permissions on typo3temp.

Loading classes with composer mode

In composer mode, the autoloader checks for (classmap and PSR-4) autoloading information inside your extensions' composer.json. If you do not provide any information, the autoloader falls back to the classmap autoloading like in non composer mode.

Troubleshooting:

  • Dump the class loading information manually via composer dumpautoload and check that the autoload information is updated. Typically you would check vendor/composer to hold files like autoload_classmap.php and autoload_psr4.php etc.

Example:

$ tree vendor/composer
.
├── ClassLoader.php
├── LICENSE
├── autoload_classmap.php
├── autoload_files.php
├── autoload_namespaces.php
├── autoload_psr4.php
├── autoload_real.php
├── autoload_static.php
├── include_paths.php
└── installed.json

Best practices

  • If you didn't do so before, have a look at the PSR-4 standard. It defines very good rules for naming classes and the files they reside in. Really, read the specs and start using PSR-4 in your projects. It's unlikely that there will be any other more advanced standard in the near future in the PHP world. PSR-4 is the way to go and you should embrace it.
  • Even if you do not use composer mode and the class mapping of the autoloader allows you to use whatever you want, stick to PSR-4. It's not only a very good standard to find classes, but it will also help organizing your code.
  • PSR-4 is all about namespaces. No matter if you like namespaces or not, use them. Namespaces exist since PHP 5.3, so you will be able to use them in any modern TYPO3 project due to the minimum PHP requirements of TYPO3 itself.

Tip

PSR-4 is a standard that has been develop by the PHP Framework Interop Group (FIG). PSR-4 is an advanced standard for autoloading php classes and replaces PSR-0. If you want to know more about the PHP FIG in general and PSR-4 in specific, please visit http://www.php-fig.org/psr/psr-4/.

Bootstrapping

TYPO3 CMS has a clean bootstrapping process driven mostly by class \TYPO3\CMS\Core\Core\Bootstrap. This class contains a host of methods each responsible for a little step along the initialization of a full TYPO3 process, be it the backend or other contexts.

Some contexts add their own bootstrap class (like the command line, which additionally requires \TYPO3\CMS\Core\Core\CliBootstrap.

Note

The frontend's bootstrapping process is not yet fully encapsulated in a bootstrap class.

Warning

This boostrapping API is internal and may change any time in the near future even in minor updates. It is thus discouraged to use it in third party code. Choose this solution only if other extensbility features such as Hooks, Signals or XCLASS are not enough to reach your goals.

One can see the bootstrapping process in action in file typo3/init.php:

define('TYPO3_MODE', 'BE');

require 'sysext/core/Classes/Core/Bootstrap.php';

\TYPO3\CMS\Core\Core\Bootstrap::getInstance()
   ->baseSetup('typo3/')
   ->redirectToInstallToolIfLocalConfigurationFileDoesNotExist('../')
   ->startOutputBuffering()
   ->loadConfigurationAndInitialize()
   ->loadTypo3LoadedExtAndExtLocalconf(TRUE)
   ->applyAdditionalConfigurationSettings()
   ->initializeTypo3DbGlobal()
   ->checkLockedBackendAndRedirectOrDie()
   ->checkBackendIpOrDie()
   ->checkSslBackendAndRedirectIfNeeded()
   ->checkValidBrowserOrDie()
   ->loadExtensionTables(TRUE)
   ->initializeSpriteManager()
   ->initializeBackendUser()
   ->initializeBackendUserMounts()
   ->initializeLanguageObject()
   ->initializeModuleMenuObject()
   ->initializeBackendTemplate()
   ->endOutputBufferingAndCleanPreviousOutput()
   ->initializeOutputCompression();

Note that most methods of the Bootstrap class must be called in a precise order. It is perfectly possible to define one's own bootstrapping process, but care should be taken about the call order.

Also note that all bootstrapping methods return the instance of the Bootstrap class itself, allowing calls to be chained.

Application Context

Each request, no matter if it runs from the command line or through HTTP, runs in a specific application context. TYPO3 CMS provides exactly three built-in contexts:

  • Production (default) - should be used for a live site
  • Development - used for development
  • Testing - is only used internally when executing TYPO3 core tests. It must not be used otherwise.

The context TYPO3 runs in is specified through the environment variable TYPO3_CONTEXT. It can be set on the command line:

# run the TYPO3 CMS CLI commands in development context
TYPO3_CONTEXT=Development ./typo3/cli_dispatch.phpsh

or be part of the web server configuration:

# In your Apache configuration, you usually use:
SetEnv TYPO3_CONTEXT Development

# Set context with mod_rewrite
# Rules to set ApplicationContext based on hostname
RewriteCond %{HTTP_HOST} ^dev\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Development]

RewriteCond %{HTTP_HOST} ^staging\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Production/Staging]

RewriteCond %{HTTP_HOST} ^www\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Production]
# In your Nginx configuration, you can pass the context as a fastcgi parameter
location ~ \.php$ {
   include         fastcgi_params;
   fastcgi_index   index.php;
   fastcgi_param   TYPO3_CONTEXT  Development/Dev;
   fastcgi_param   SCRIPT_FILENAME  $document_root$fastcgi_script_name;
}
Custom Contexts

In certain situations, more specific contexts are desirable:

  • a staging system may run in a Production context, but requires a different set of credentials than the production server.
  • developers working on a project may need different application specific settings but prefer to maintain all configuration files in a common Git repository.

By defining custom contexts which inherit from one of the three base contexts, more specific configuration sets can be realized.

While it is not possible to add new "top-level" contexts at the same level like Production and Testing, you can create arbitrary sub-contexts, just by specifying them like <MainContext>/<SubContext>.

For a staging environment a custom context Production/Staging may provide the necessary settings while the Production/Live context is used on the live instance.

Note

This even works recursively, so if you have a multiple-server staging setup, you could use the context Production/Staging/Server1 and Production/Staging/Server2 if both staging servers needed different configuration.

Attention

Testing Is reserved for internal use when executing TYPO3 core functional and unit tests It must not be used otherwise. Instead sub-contexts must be used: Production/Testing or Development/Testing

Usage Example

The current Application Context is set very early in the bootstrap process can be accessed through public API for example in the AdditionalConfiguration.php file to automatically set different configuration for different contexts.

In file typo3conf/AdditionalConfiguration.php:

switch (\TYPO3\CMS\Core\Utility\GeneralUtility::getApplicationContext()) {
   case 'Development':
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 1;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '*';
      break;
   case 'Production/Staging':
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 0;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '192.168.1.*';
      break;
   default:
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 0;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '127.0.0.1';
}

Variables and Constants

After TYPO3's bootstrap sequence has completed, a number of global variables, constants and classes available to any script.

The column "Avail. in FE" is an indicator that tells you if the constant, variable or class mentioned is also available to scripts running under the frontend of the "cms" extension.

Constants

Constants normally define paths and database information. These values are global and cannot be changed when they are first defined. This is why constants are used for such vital information.

These constants are defined at various points during the bootstrap sequence.

Note

To make the table below a bit more compact, namespaces were left out. Here are the fully qualified class names referred to below:

Table 1: Traditional List
Constant Defined in Description Avail. in FE
TYPO3_MODE \TYPO3\CMS\Backend\Console\Application::defineLegacyConstants() \TYPO3\CMS\Backend\Http\Application::defineLegacyConstants() \TYPO3\CMS\Core\Console\CommandApplication::defineLegacyConstants() \TYPO3\CMS\Frontend\Http\Application::defineLegacyConstants() \TYPO3\CMS\Install\Http\Application::defineLegacyConstants() Mode of TYPO3: Set to either "FE" or "BE" depending on frontend or backend execution and context.

Yes

value = "FE"

TYPO3_OS SystemEnvironmentBuilder::defineBaseConstants() Operating systen; Windows = "WIN", other = "" (presumed to be some sort of Unix) Yes
PATH_thisScript SystemEnvironmentBuilder::definePaths() Abs. path to current script. Yes
TYPO3_mainDir SystemEnvironmentBuilder::definePaths() This is the directory of the backend administration for the sites of this TYPO3 installation. Hardcoded to typo3/. Must be a subdirectory to the website. See elsewhere for descriptions on how to change the default admin directory, typo3/, to something else. Yes
PATH_typo3 SystemEnvironmentBuilder::definePaths() Abs. path of the TYPO3 admin dir (PATH_site + TYPO3_mainDir). No
PATH_site SystemEnvironmentBuilder::definePaths() Absolute path to directory with the frontend (one directory above PATH_typo3) Yes
PATH_typo3conf SystemEnvironmentBuilder::definePaths() Absolute TYPO3 configuration path (local, not part of source). Yes
TYPO3_DLOG Bootstrap::defineLoggingAndExceptionConstants() If true, calls to \TYPO3\CMS\Core\Utility\GeneralUtility::devLog() can be made in both frontend and backend; This is event logging which can help to track debugging in general. Yes
TYPO3_version SystemEnvironmentBuilder::defineBaseConstants() The TYPO3 version, as a "x.y.z" number. Development versions will be either "x.y.z-dev" for stable versions or "x.y-dev" for the current master. Yes
TYPO3_branch SystemEnvironmentBuilder::defineBaseConstants() The TYPO3 version Branch, as a "x.y" number. Without the patch level. Yes
Table 2: Base Constants

Check \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::defineBaseConstants() for updates.

String constants
Constant Value Description
NUL chr(0) A null
TAB chr(9) A tabulator
LF chr(10) A linefeed
CR chr(13) A carriage return
SUB chr(26) A sub (substitute) character
CRLF CR + LF Carriage return + linefeed pair
Operating system identifier
Constant Value Description
TYPO3_OS self::getTypo3Os()) Either "WIN" or empty string
Service error constants
Constant Value Description
T3_ERR_SV_GENERAL -1 General error - something went wrong
T3_ERR_SV_NOT_AVAIL -2 During execution it showed that the service is not available and should be ignored. The service itself should call $this->setNonAvailable()
T3_ERR_SV_WRONG_SUBTYPE -3 Passed subtype is not possible with this service
T3_ERR_SV_NO_INPUT -4 Passed subtype is not possible with this service
T3_ERR_SV_FILE_NOT_FOUND -20 File not found which the service should process
T3_ERR_SV_FILE_READ -21 File not readable
T3_ERR_SV_FILE_WRITE -22 File not writable
T3_ERR_SV_PROG_NOT_FOUND -40 Passed subtype is not possible with this service
T3_ERR_SV_PROG_FAILED -41 Passed subtype is not possible with this service

Global variables

Note

Variables in italics may be set in a script prior to the bootstrap process so they are optional.

Note

To make the table below a bit more compact, namespaces were left out. Here are the fully qualified class names referred to below:

  • "SystemEnvironmentBuilder" = \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder
  • "Bootstrap" = \TYPO3\CMS\Core\Core\Bootstrap
  • "PackageManager" = \TYPO3\CMS\Core\Package\PackageManager
Global variable Defined in Description Avail. in FE
$TYPO3_CONF_VARS typo3/sysext/core/Configuration/DefaultConfiguration.php TYPO3 configuration array. Please refer to file typo3/sysext/core/Configuration/DefaultConfigurationDescription.php where each option is described in detail in the comments. The same comments are also available in the Install Tool when you choose "All Configuration". Yes
$TYPO3_LOADED_EXT PackageManager::loadPackageManagerStatesFromCache() PackageManager::initializeCompatibilityLoadedExtArray() Array with all loaded extensions listed with a set of paths. You can check if an extension is loaded by the function \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::isLoaded($key) where $key is the extension key. Yes
$TYPO3_DB Bootstrap::initializeTypo3DbGlobal()

An instance of the TYPO3 DB wrapper class, \TYPO3\CMS\Core\Database\DatabaseConnection. Formerly (before 8.2) this object had to be used for all interaction with the database.

Attention

You should NOT use this anymore. Use Doctrine instead!

Yes
$EXEC_TIME SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables() Is set to time() so that the rest of the script has a common value for the script execution time. YES
$SIM_EXEC_TIME SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables() Is set to $EXEC_TIME but can be altered later in the script if we want to simulate another execution-time when selecting from e.g. a database (used in the frontend for preview of future and past dates) Yes
$PARSETIME_START SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables() Time in milliseconds right after inclusion of the configuration. No
$PAGES_TYPES typo3/sysext/core/ext_tables.php See Page types (occasionally)
$TCA Bootstrap::loadExtensionTables() See TCA Reference Yes, partly
$TBE_MODULES typo3/sysext/core/ext_tables.php The backend main/sub-module structure. See section elsewhere plus source code of class \TYPO3\CMS\Backend\Module\ModuleLoader which also includes some examples. (occasionally)
$TBE_STYLES typo3/sysext/core/ext_tables.php Contains information related to BE skinning. (will be removed on CMS 9) (occasionally)
$T3_SERVICES SystemEnvironmentBuilder::initializeGlobalVariables() Global registration of services. Yes
$T3_VAR SystemEnvironmentBuilder::initializeGlobalVariables()

Space for various internal global data storage in TYPO3. Each key in this array is a data space for an application. Keys currently defined for use is:

['callUserFunction'] + ['callUserFunction_classPool']: Used by \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction to store singleton objects.

['getUserObj'] : Used by \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj to store singleton objects.

['RTEobj'] : Used to hold the current RTE object if any. See \TYPO3\CMS\Backend\Utility\BackendUtility.

['ext'][ extension-key ] : Free space for extensions.

Yes
$BE_USER Bootstrap::initializeBackendUser() Backend user object. See Backend User Object. (depends)
$TBE_MODULES_EXT [In ext_tables.php files of extensions] Used to store information about modules from extensions that should be included in "function menus" of real modules. See the Extension API for details. (occasionally)
$TCA_DESCR [tables.php files] Can be set to contain file references to local lang files containing TCA_DESCR labels. See section about Context Sensitive Help. No
Exploring global variables

Many of the global variables described above can be inspected using the Admin Tools > Configuration module.

Warning

This module is always viewed in the BE context. Variables defined only in the FE context will not be visible there.

The Configuration module in Admin Tools

Viewing the $TCA array using the Admin Tools > Configuration module

Backend User Object

The backend user of a session is always available to the backend scripts as the global variable $BE_USER. The object is created in \TYPO3\CMS\Core\Core\Bootstrap::initializeBackendUser() and is an instance of the class \TYPO3\CMS\Core\Authentication\BackendUserAuthentication (which extends \TYPO3\CMS\Core\Authentication\AbstractUserAuthentication).

In addition to $BE_USER one other global variables is of interest - $FILEMOUNTS, holding an array with the File mounts of the $BE_USER.

Checking user access

The $BE_USER object is mostly used to check user access right, but contains other helpful information. This is presented here by way of a few examples:

Checking access to current backend module

$MCONF is module configuration and the key $MCONF['access'] determines the access scope for the module. This function call will check if the $BE_USER is allowed to access the module and if not, the function will exit with an error message.

$BE_USER->modAccess($MCONF, 1);
Checking access to any backend module

If you know the module key you can check if the module is included in the access list by this function call:

$BE_USER->check('modules', 'web_list');

Here access to the module "Web > List" is checked.

Access to tables and fields?

The same function ->check() can actually check all the ->groupLists inside $BE_USER. For instance:

Checking modify access to the table "pages":

$BE_USER->check('tables_modify', 'pages');

Checking read access to the table "tt_content":

$BE_USER->check('tables_select', 'tt_content');

Checking if a table/field pair is allowed explicitly through the "Allowed Excludefields":

$BE_USER->check('non_exclude_fields', $table . ':' . $field);
Is "admin"?

If you want to know if a user is an "admin" user (has complete access), just call this method:

$BE_USER->isAdmin();
Read access to a page?

This function call will return true if the user has read access to a page (represented by its database record, $pageRec):

$BE_USER->doesUserHaveAccess($pageRec, 1);

Changing the "1" for other values will check other permissions:

  • use "2" for checking if the user may edit the page
  • use "4" for checking if the user may delete the page.
Is a page inside a DB mount?

Access to a page should not be checked only based on page permissions but also if a page is found within a DB mount for ther user. This can be checked by this function call ($id is the page uid):

$BE_USER->isInWebMount($id)
Selecting readable pages from database?

If you wish to make a SQL statement which selects pages from the database and you want it to be only pages that the user has read access to, you can have a proper WHERE clause returned by this function call:

$BE_USER->getPagePermsClause(1);

Again the number "1" represents the "read" permission; "2" is "edit" and "4" is delete permission. The result from the above query could be this string:

((pages.perms_everybody & 1 = 1)OR(pages.perms_userid = 2 AND pages.perms_user & 1 = 1)OR(pages.perms_groupid in (1) AND pages.perms_group & 1 = 1))
Saving module data

This stores the input variable $compareFlags (an array!) with the key "tools_beuser/index.php/compare"

$compareFlags = \TYPO3\CMS\Core\Utility\GeneralUtility::_GP('compareFlags');
$BE_USER->pushModuleData('tools_beuser/index.php/compare', $compareFlags);
Getting module data

This gets the module data with the key "tools_beuser/index.php/compare" (lasting only for the session)

$compareFlags = $BE_USER->getModuleData('tools_beuser/index.php/compare', 'ses');
Getting TSconfig

This function can return a value from the "User TSconfig" structure of the user. In this case the value for "options.clipboardNumberPads":

$BE_USER->getTSConfigVal('options.clipboardNumberPads');
Getting the username

The full "be_users" record of a authenticated user is available in $BE_USER->user as an array. This will return the "username":

$BE_USER->user['username']
Get User Configuration value

The internal ->uc array contains options which are managed by the User Tools > User Settings module (extensions "setup"). These values are accessible in the $BE_USER->uc array. This will return the current state of "Condensed mode" for the user:

$BE_USER->uc['condensedMode']

TYPO3 Core Engine (TCE)

Introduction

Database

The TYPO3 Core Engine is the class that handles all *data* writing to database tables configured in $TCA. In addition the class handles commands such as copy, move, delete. It will handle undo/history and versioning of records and everything will be logged to the sys_log. And it will make sure that write permissions are evaluated correctly for the user trying to write to the database. Generally, any processing specific option in the $TCA array is handled by TCE.

Using TCE for manipulation of the database content in the $TCA-configured tables guarantees that the data integrity of TYPO3 is respected. This cannot be safely guaranteed if you write to $TCA-configured database tables directly. It will also manage the relations to files and other records.

TCE requires a backend login to work. This is due to the fact that permissions are observed (of course) and thus TCE needs a backend user to evaluate against. This means you cannot use DataHandler from the frontend scope. Thus writing to tables (such as a guestbook) will have to be done from the frontend without DataHandler.

The features of the $TCA are described in the TCA Reference.

Files

TCE also has a part for handling files. The file operations are normally performed in the File > List module where you can manage a directory on the server by copying, moving, deleting and editing files and directories. The file operations are managed by two core classes, \TYPO3\CMS\Core\Utility\File\BasicFileUtility and \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility.

Database: DataHandler basics (formerly known as TCEmain)

When you are using TCE from your backend applications you need to prepare two arrays of information which contain the instructions to DataHandler (\TYPO3\CMS\Core\DataHandling\DataHandler) of what actions to perform. They fall into two categories: data and commands.

"Data" is when you want to write information to a database table or create a new record.

"Commands" is when you want to move, copy or delete a record in the system.

The data and commands are created as multidimensional arrays and to understand the API of DataHandler you simply need to understand the hierarchy of these two arrays.

Commands Array

Syntax:

$cmd[ tablename ][ uid ][ command ] = value

Description of keywords in syntax:

Key Data type Description
tablename string Name of the database table. Must be configured in $TCA array, otherwise it cannot be processed.
uid integer The UID of the record that is manipulated. This is always an integer.
command string (command keyword)

The command type you want to execute.

Note

Only one command can be executed at a time for each record! The first command in the array will be taken.

See table below for :ref:`command keywords and values <tce-command-keywords>`

value mixed

The value for the command

See table below for :ref:`command keywords and values <tce-command-keywords>`

Command keywords and values
Command Data type Value
copy integer

The significance of the value depends on whether it is positive or negative:

  • Positive value: The value points to a page UID. A copy of the record (and possibly child elements/tree below) will be inserted inside that page as the first element.
  • Negative value: The (absolute) value points to another record from the same table as the record being copied. The new record will be inserted on the same page as that record and if $TCA[...]['ctrl']['sortby'] is set, then it will be positioned after.
  • Zero value: Record is inserted on tree root level.
move integer Works like "copy" but moves the record instead of making a copy.
delete 1

Value should always be "1"

This action will delete the record (or mark the record "deleted" if configured in $TCA).

undelete 1

Value should always be "1".

This action will set the deleted-flag back to 0.

localize integer

Pointer to a sys_language uid to localize the record into. Basically a localization of a record is making a copy of the record (possibly excluding certain fields defined with l10n_mode) but changing relevant fields to point to the right sys language / original language record.

Requirements for a successful localization is this:

  • [ctrl] options "languageField" and "transOrigPointerField" must be defined for the table
  • A sys_language record with the given sys_language_uid must exist.
  • The record to be localized by currently be set to "Default" language and not have any value set for the transOrigPointerField either.
  • There cannot exist another localization to the given language for the record (looking in the original record PID).

Apart from this, ordinary permissions apply as if the user wants to make a copy of the record on the same page.

version array

Versioning action.

Keys:

  • [action] : Keyword determining the versioning action. Options are:
    • "new": Indicates that a new version of the record should be created.Additional keys, specific for "new" action:
      • [treeLevels]: (Only pages) Integer, -1 to 4, indicating the number of levels of the page tree to version together with a page. This is also referred to as the versioning type:-1 ("element") means only the page record gets versioned (default)0 ("page") means the page + content tables (defined by ctrl-flag versioning_followPages )>0 ("branch") means the the whole branch is versioned ( full copy of all tables), down to the level indicated by the value (1= 1 level down, 2= 2 levels down, etc.)The treeLevel is recorded in the field t3ver_swapmode and will be observed when the record is swapped during publishing.
      • [label]: Indicates the version label to apply. If not given, a standard label including version number and date is added.
    • "swap": Indicates that the current online version should be swapped with another.Additional keys, specific for "swap" action:
      • [swapWith]: Indicates the uid of the record to swap current version with!
      • [swapIntoWS]: Boolean, indicates that when a version is published it should be swapped into the workspace of the offline record.
    • "clearWSID": Indicates that the workspace of the record should be set to zero (0). This removes versions out of workspaces without publishing them.
    • "flush": Completely deletes a version without publishing it.
    • "setStage": Sets the stage of an element. Special feature: The id- key in the array can be a comma list of ids in order to perform the stageChange over a number of records. Also, the internal variable ->generalComment (also available through :file:`tce_db.php` as "&generalComment") can be used to set a default comment for all stage changes of an instance of tcemain. Additional keys for this action is:
      • [stageId]: Values are: -1 (rejected), 0 (editing, default), 1 (review), 10 (publish)
      • [comment]: Comment string that goes into the log.
Examples of commands:
$cmd['tt_content'][54]['delete'] = 1;    // Deletes tt_content record with uid=54
$cmd['pages'][1203]['copy'] = -303;   //Copies page id=1203 to the position after page 303
$cmd['pages'][1203]['move'] = 303;  // Moves page id=1203 to the first position in page 303
Data Array

Syntax:

$data[tablename][uid][fieldname] = value

Description of keywords in syntax:

Key Data type Description
tablename string Name of the database table. Must be configured in $TCA array, otherwise it cannot be processed.
uid mixed The UID of the record that is modified. If the record already exists, this is an integer. If you're creating new records, use a random string prefixed with "NEW", e.g. "NEW7342abc5e6d".
fieldname string Name of the database field you want to set a value for. Must be configure in $TCA[ tablename ]['columns']
value string

Value for "fieldname".

Important

Always make sure $this->stripslashes_values is false before using DataHandler.)

Note

For FlexForms the data array of the FlexForm field is deeper than three levels. The number of possible levels for FlexForms is infinite and defined by the data structure of the FlexForm. But FlexForm fields always end with a "regular value" of course.

Examples of Data submission

This creates a new page titled "The page title" as the first page inside page id 45:

$data['pages']['NEW9823be87'] = array(
    'title' => 'The page title',
    'subtitle' => 'Other title stuff',
    'pid' => '45'
);

This creates a new page titled "The page title" right after page id 45 in the tree:

$data['pages']['NEW9823be87'] = array(
    'title' => 'The page title',
    'subtitle' => 'Other title stuff',
    'pid' => '-45'
);

This creates two new pages right after each other, located right after the page id 45:

$data['pages']['NEW9823be87'] = array(
    'title' => 'Page 1',
    'pid' => '-45'
);
$data['pages']['NEWbe68s587'] = array(
    'title' => 'Page 2',
    'pid' => '-NEW9823be87'
);

Notice how the second "pid" value points to the "NEW..." id placeholder of the first record. This works because the new id of the first record can be accessed by the second record. However it works only when the order in the array is as above since the processing happens in that order!

This creates a new content record with references to existing and one new system category:

$data['sys_category']['NEW9823be87'] = array(
    'title' => 'New category',
    'pid' => 1,
);
$data['tt_content']['NEWbe68s587'] = array(
    'header' => 'Look ma, categories!',
    'pid' => 45,
    'categories' => array(
        1,
        2,
        'NEW9823be87', // You can also use placeholders here
    ),
);

This updates the page with uid=9834 to a new title, "New title for this page", and no_cache checked:

$data['pages'][9834] = array(
    'title' => 'New title for this page',
    'no_cache' => '1'
);
Clear cache

TCE also has an API for clearing the cache tables of TYPO3:

Syntax:

$tce->clear_cacheCmd($cacheCmd);
$cacheCmd values Description
[integer] Clear the cache for the page id given.
"all"

Clears all cache tables (cache_pages, cache_pagesection, cache_hash).

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.all".

"pages"

Clears all pages from cache_pages.

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.pages".

"temp_CACHED" Clears the temp_CACHED files in typo3conf/.
Hook for cache post-processing

You can configure cache post-processing with a user defined PHP function. Configuration of the hook can be done from ext_localconf.php. An example might look like:

$TYPO3_CONF_VARS['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['clearCachePostProc'][] = 'myext_cacheProc->proc';
require_once(\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::extPath('myext') . 'class.myext_cacheProc.php');
Flags in DataHandler

There are a few internal variables you can set prior to executing commands or data submission. These are the most significant:

Internal variable Data type Description
->deleteTree Boolean

Sets whether a page tree branch can be recursively deleted.

If this is set, then a page is deleted by deleting the whole branch under it (user must have delete permissions to it all). If not set, then the page is deleted only if it has no branch.

Default is false.

->copyTree Integer

Sets the number of branches on a page tree to copy.

If 0 then branch is not copied. If 1 then pages on the 1st level is copied. If 2 then pages on the second level is copied, and so on.

Default is zero.

->reverseOrder Boolean

If set, the data array is reversed in the order, which is a nice thing if you're creating a whole bunch of new records.

Default is zero.

->copyWhichTables list of strings (tables)

This list of tables decides which tables will be copied. If empty then none will. If "*" then all will (that the user has permission to of course).

Default is "*".

->stripslashes_values boolean

If set, then all values will be passed through stripslashes(). This has been the default since the birth of TYPO3 in times when input from POST forms were always escaped an needed to be unescaped. Today this is deprecated and values should be passed around without escaped characters.

Warning

It is highly recommended to set this value to zero every time the class is used!

If you set this value to false you can pass values as-is to the class and it is most like that this is what you want. Otherwise you would have to pass all values through addslashes() first.

Default is (currently) "1" (true) but might be changed in the future!

Using DataHandler in scripts

It's really easy to use the class \TYPO3\CMS\Core\DataHandling\DataHandler in your own scripts. All you need to do is include the class, build a $data/$cmd array you want to pass to the class and call a few methods.

Important

Mind that these scripts have to be run in the backend scope! There must be a global $BE_USER object.

In your script you simply insert this line to include the class:

What follows are a few code listings with comments which will provide you with enough knowledge to get started. It is assumed that you have populated the $data and $cmd arrays correctly prior to these chunks of code. The syntax for these two arrays is explained in the previous chapter.

DataHandler examples
Submitting data

This is the most basic example of how to submit data into the database. It is four lines. Line 1 instantiates the class, line 2 defines that values will be provided without escaped characters (recommended!), line 3 registers the $data array inside the class and initializes the class internally! Finally line 4 will execute the data submission.

1
2
3
4
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->start($data, array());
$tce->process_datamap();
Executing commands

The most basic way of executing commands. Line 1 creates the object, line 2 defines that values will be provided without escaped characters (recommended), line 3 registers the $cmd array inside the class and initializes the class internally! Finally line 4 will execute the commands.

1
2
3
4
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->start(array(), $cmd);
$tce->process_cmdmap();
Clearing cache

In this example the cache clearing API is used. No data is submitted, no commands executed. Still you will have to initialize the class by calling the start() method (which will initialize internal variables).

Note

Clearing a given cache is possible only for users that are "admin" or have specific permissions to do so.

1
2
3
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->start(array(), array());
$tce->clear_cacheCmd('all');

Since TYPO3 CMS 6.2, caches are organized in groups. Clearing "all" caches will actually clear caches from the "all" group and not really all caches. Check the caching framework architecture section for more details about available caches and groups.

Complex data submission

Imagine the $data array something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$data = array(
    'pages' => array(
        'NEW_1' => array(
            'pid' => 456,
            'title' => 'Title for page 1',
        ),
        'NEW_2' => array(
            'pid' => 456,
            'title' => 'Title for page 2',
        ),
    )
);

This aims to create two new pages in the page with uid "456". In the follow code this is submitted to the database. Notice how line 3 reverses the order of the array. This is done because otherwise "page 1" is created first, then "page 2" in the same PID meaning that "page 2" will end up above "page 1" in the order. Reversing the array will create "page 2" first and then "page 1" so the "expected order" is preserved.

To insert a record after a given record, set the other record's negative uid as pid in the new record you're setting as data.

Apart from this line 6 will send a "signal" that the page tree should be updated at the earliest occasion possible. Finally, the cache for all pages is cleared in line 7.

1
2
3
4
5
6
7
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->reverseOrder = 1;
$tce->start($data, array());
$tce->process_datamap();
\TYPO3\CMS\Backend\Utility\BackendUtility::setUpdateSignal('updatePageTree');
$tce->clear_cacheCmd('pages');
Both data and commands executed with alternative user object

In this case it is shown how you can use the same object instance to submit both data and execute commands if you like. The order will depend on the order of line 4 and 5.

In line 3 the start() method is called, but this time with the third possible argument which is an alternative $BE_USER object. This allows you to force another backend user account to create stuff in the database. This may be useful in certain special cases. Normally you should not set this argument since you want TCE to use the global $BE_USER.

1
2
3
4
5
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->stripslashes_values = 0;
$tce->start($data, $cmd, $alternative_BE_USER);
$tce->process_datamap();
$tce->process_cmdmap();

The "tce_db.php" API

This script is a gateway for POST forms to class \TYPO3\CMS\Core\DataHandling\DataHandler. It has historically been the script to which data was posted when you wanted to update something in the database.

Today it is used for editing by only a few scripts, actually only the "Quick Edit" module in "Web>Page" (frontend). The standard forms you find in TYPO3 are normally rendered and handled by alt_doc.php which includes \TYPO3\CMS\Core\DataHandling\DataHandler on its own.

For commands it is still used from various locations.

You can send data to this file either as GET or POST vars where POST takes precedence. The variable names you can use are:

GP var name Data type Description
data array

Data array on the form [tablename][uid][fieldname] = value.

Typically it comes from a POST form which submits a form field like <input name="data[tt_content][123][header]" value="This is the headline" />.

cmd array

Command array on the form [tablename][uid][command] = value. This array may get additional data set internally based on clipboard commands send in CB var!

Typically this comes from GET vars passed to the script like &cmd[tt\_content][123][delete]=1 which will delete Content Element with UID 123.

cacheCmd string Cache command sent to ->clear_cacheCmd
redirect string Redirect URL. Script will redirect to this location after performing operations (unless errors has occurred)
flags array Accepts options to be set in TCE object. Currently it supports "reverseOrder" (boolean).
mirror array Example: [mirror][table][11] = '22,33' will look for content in [data][table][11] and copy it to [data][table][22] and [data][table][33].
prErr boolean If set, errors will be printed on screen instead of redirection. Should always be used, otherwise you will see no errors if they happen.
CB array Clipboard command array. May trigger changes in "cmd".
vC string Verification code
uPT string Update Page Tree Trigger. If set and the manipulated records are pages then the update page tree signal will be set.

File functions basics

File operations in the TCE are handled by the class \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility which extends \TYPO3\CMS\Core\Utility\File\BasicFileUtility. The instructions for file manipulation are passed to this class as a multidimensional array.

Files Array

Syntax:

$file[ command ][ index ][ key ] = value

Description of keywords in syntax:

Key Data type Description
command string (command keyword)

The command type you want to execute.

See table below for :ref:`command keywords, keys and values<tce-file-keywords>`

index integer Integer index in the array which separates multiple commands of the same type.
key string

Depending on the command type. The keys will carry the information needed to perform the action. Typically a "target" key is used to point to the target directory or file while a "data" key carries the data.

See table below for :ref:`command keywords, keys and values<tce-file-keywords>`

value string

The value for the command

See table below for :ref:`command keywords, keys and values<tce-file-keywords>`

Command keywords and values
Command Keys Value
delete "data" "data" = Absolute path to the file/folder to delete
copy

"data"

"target"

"altName"

"data" = Absolute path to the file/folder to copy

"target" = Absolute path to the folder to copy to (destination)

"altName" = (boolean): If set, a new filename is made by appending numbers/unique-string in case the target already exists.

move

"data"

"target"

"altName"

(Exactly like copy, just replace the word "copy" with "move")
rename

"data"

"target"

"data" = New name, max 30 characters alphanumeric

"target" = Absolute path to the target file/folder

newfolder

"data"

"target"

"data" = Folder name, max 30 characters alphanumeric

"target" = Absolute path to the folder where to create it

newfile

"data"

"target"

"data" = New filename

"target" = Absolute path to the folder where to create it

editfile

"data"

"target"

"data" = The new content

"target" = Absolute path to the target file

upload

"data"

"target"

upload_$id

"data" = ID-number (points to the global var that holds the filename- ref ($_FILES["upload_" . $id]["name"]).

"target" = Absolute path to the target folder (destination)

upload_$id = File reference. $id must equal value of file[upload][...][data]!

See \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility::func_upload().

unzip

"data"

"target"

"data" = Absolute path to the zip-file. (file extension must be "zip")

"target" = The absolute path to the target folder (destination) (if not set, default is the same as the zip-file)

It is unlikely that you will need to use this internally in your scripts like you will need \TYPO3\CMS\Core\DataHandling\DataHandler. It is fairly uncommon to need the file manipulations in own scripts unless you make a special application. Therefore the most typical usage of this API is from tce_file.php and the core scripts that are activated by the "File > List" module.

However, if you need it this is an example (taken from tce_file.php) of how to initialize the usage.

1
2
3
4
5
6
7
    // Initializing:
$this->fileProcessor = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Utility\\File\\ExtendedFileUtility');
$this->fileProcessor->init($FILEMOUNTS, $TYPO3_CONF_VARS['BE']['fileExtensions']);
$this->fileProcessor->init_actionPerms($BE_USER->user['fileoper_perms']);

$this->fileProcessor->start($this->file);
$this->fileProcessor->processData();

Line 2 makes an instance of the class and line 3 initializes the object with the filemounts of the current user and the array of allow/deny file extensions in web-space and ftp-space (see below). Then the file operation permissions are loaded from the user object in line 4. Finally, the file command array is loaded in line 6 (and internally additional configuration takes place from $TYPO3_CONF_VARS!). In line 7 the command map is executed.

Web-space, FTP-space and $TYPO3_CONF_VARS['BE']['fileExtensions']

The control of file extensions goes in two categories. Webspace and ftpspace. Webspace is folders accessible from a web browser (below TYPO3_DOCUMENT_ROOT) and ftpspace is everything else.

The control is done like this: if an extension matches 'allow' then the check returns true. If not and an extension matches 'deny' then the check return false. If no match at all, returns true.

You list extensions comma-separated. If the value is a '*' every extension is matched. If no file extension, true is returned if 'allow' is '*', false if 'deny' is '*' and true if none of these matches. This (default) configuration below accepts everything in ftpspace and everything in webspace except php files:

$TYPO3_CONF_VARS['BE']['fileExtensions'] = array (
    'webspace' => array('allow' => '', 'deny' => 'php'),
    'ftpspace' => array('allow' => '*', 'deny' => '')
);

The "tce_file.php" API

This script serves as the file administration part of the TYPO3 Core Engine. It's a gateway for TCE (TYPO3 Core Engine) file-handling through POST forms. It uses \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility for the manipulation of the files.

This script is used from the File > List module where you can rename, create, delete etc. files and directories on the server.

You can send data to this file either as GET or POST vars where POST takes precedence. The variable names you can use are:

GP var name Data type Description
file array

Array of file operations. See previous information about basic file functions.

This could typically be a GET var like &file[delete][0][data]=[absolute file path] or a POST form field like:

"<input type="text" name="file[newfolder][0][data]" value=""/>
<input type="hidden" name="file[newfolder][0][target]"
value="[absolute path to folder to create in]"/>"
redirect string Redirect URL. Script will redirect to this location after performing operations.
CB array Clipboard command array. May trigger changes in "file"
vC string Verification code
overwriteExistingFiles boolean If existing files should be overridden.

FormEngine

Introduction

Looking at TYPO3's main constructs from an abstract position, the system splits into three most important pillars:

DataHandler
TYPO3\CMS\Core\DataHandling\...: Construct taking care of persisting data into the database. The DataHandler takes an array representing one or more records, inserts, deletes or updates them in the database and takes care of relations between multiple records. If editing content in the backend, this construct does all main database munging. DataHandler is fed by some controller that most often gets GET or POST data from FormEngine.
FormEngine
TYPO3\CMS\Backend\Form\...: FormEngine renders records, usually in the backend. It creates all the HTML needed to edit complex data and data relations. Its GET or POST data is then fed to the DataHandler by some controller.
Frontend rendering
TYPO3\CMS\Frontend\...: Render the website frontend. The frontend rendering, usually based on TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController uses TypoScript and / or Fluid to process and render database content into the frontend.

The glue between these three pillars is TCA (Table Configuration Array): It defines how database tables are constructed, which localization or workspace facilities exist, how it should be displayed in the backend, how it should be written to the database, and - next to TypoScript - which behaviour it has in the frontend.

This chapter is about FormEngine. It is important to understand this construct is based on TCA and is usually used in combination with the DataHandler. However, FormEngine is constructed in a way that it can work without DataHandler: A controller could use the FormEngine result and process it differently. Furthermore, all dependencies of FormEngine are abstracted and may come from "elsewhere", still leading to the form output known for casual records.

This makes FormEngine an incredible flexible construct. The basic idea is "feed something that looks like TCA and render forms that have the full power of TCA but look like all other parts of the backend".

The FormEngine code base has been significantly refactored in TYPO3 CMS version 7 and version 8 to be much more flexible, more easy to use and extend, and much more powerful than before. This is an ongoing process and some areas still need a major overhaul. The current state of the documentation aims to explain the main constructs of FormEngine and gives an insight on how to re-use, adapt and extend it with extensions. The core team expects to see more usages of FormEngine within core itself and within extensions in the future, and encourages developers to solve feature needs based on FormEngine. With the ongoing changes, those areas that may need code adaptions in the foreseeable future have notes within the documentation and developers should be available to adapt with younger cores. Watch out for breaking changes if using FormEngine and updating core.

Main rendering workflow

This is done by example. The details to steer and how to use only sub-parts of the rendering chain are explained in more detail in the following sections.

Editing a record in the backend - often from within the Page or List module - triggers the EditDocumentController by routing definitions using getModuleUrl() and handing over which record of which table should be edited. This can be an existing record, or it could be a command to create the form for a new record. The EditDocumentController is the main logic triggered whenever an editor changes a record!

The EditDocumentController has two main jobs: Trigger rendering of one or multiple records via FormEngine, and hand over any given data by a FormEngine POST result over to the DataHandler to persist stuff in the database.

The rendering part of the EditDocumentController job splits into these parts:

  • Initialize main FormEngine data array using POST or GET data to specify which specific record(s) should be edited.
  • Select which group of DataProviders should be used.
  • Trigger FormEngine DataCompiler to enrich the initialized data array with further data by calling all data providers specified by selected data provider group.
  • Hand over DataCompiler result to an entry "render container" of FormEngine and receive a result array.
  • Take result array containing HTML, CSS and JavaScript details and put them into FormResultCompiler which hands them over to the PageRenderer.
  • Let the PageRenderer output its compiled result.
Main FormEngine workflow

The controller does two distinct things here: First, it initializes a data array and lets it get enriched by data providers of FormEngine which add all information needed for the rendering part. Then feed this data array to the rendering part of FormEngine to end up with a result array containing all HTML, CSS and JavaScript.

In code, this basic workflow looks like this:

$formDataGroup = GeneralUtility::makeInstance(TcaDatabaseRecord::class);
$formDataCompiler = GeneralUtility::makeInstance(FormDataCompiler::class, $formDataGroup);
$nodeFactory = GeneralUtility::makeInstance(NodeFactory::class);
$formResultCompiler = GeneralUtility::makeInstance(FormResultCompiler::class);
$formDataCompilerInput = [
    'tableName' => $table,
    'vanillaUid' => (int)$theUid,
    'command' => $command,
];
$formData = $formDataCompiler->compile($formDataCompilerInput);
$formData['renderType'] = 'outerWrapContainer';
$formResult = $nodeFactory->create($formData)->render();
$formResultCompiler->mergeResult($formResult);

This basically means the main FormEngine concept is a two-fold process: First create an array to gather all render-relevant information, then call the render engine using this array to come up with output.

This two-fold process has a number of advantages:

  • The data compiler step can be regulated by a controller to only enrich with stuff that is needed in any given context. This part is supported by encapsulating single data providers in data groups, single data providers can be omitted if not relevant in given scope.
  • Data providing and rendering is split: Controllers could re-use the rendering part of FormEngine while all or parts of the data providers are omitted, or their data comes from "elsewhere". Furthermore, controllers can re-use the data providing part of FormEngine and output the result in an entirely different way than HTML. The latter is for instance used when FormEngine is triggered for a TCA tree by an ajax call and thus outputs a JSON array.
  • The code constructs behind "data providing" and "rendering" can be different to allow higher re-use and more flexibility with having the "data array" as main communication base in between. This will become more obvious in the next sections where it is shown that data providers are a linked list, while rendering is a tree.

Data compiling

This is the first step of FormEngine. The data compiling creates an array containing all data the rendering needs to come up with a result.

A basic call looks like this:

$formDataGroup = GeneralUtility::makeInstance(TcaDatabaseRecord::class);
$formDataCompiler = GeneralUtility::makeInstance(FormDataCompiler::class, $formDataGroup);
$formDataCompilerInput = [
   'tableName' => $table,
   'vanillaUid' => (int)$theUid,
   'command' => $command,
];
$formData = $formDataCompiler->compile($formDataCompilerInput);

The above code is a simplified version of the relevant part of the EditDocumentController. This controller knows by its GET or POST parameters which record ("vanillaUid") of which specific table ("tableName") should be edited (command="edit") or created (command="new"), and sets this as init data to the DataCompiler. The controller also knows that it should render a full database record and not only parts of it, so it uses the TcaDatabaseRecord data provider group to trigger all data providers relevant for this case. By calling ->compile() on this data group, all providers configured for this group are called after each other, and formData ends up with a huge array of data record details.

So, what happens here in detail?

  • Variable $formDataCompilerInput maps input values to keys specified by FormDataCompiler as "init" data.
  • FormDataCompiler returns a unified array of data. This array is enriched by single data providers.
  • A data provider group is a list of single data providers for a specific scope and enriches the array with information.
  • Each data provider is called by the DataGroup to add or change data in the array.

The variable $formData roughly consists of this data after calling $formDataCompiler->compile():

  • A validated and initialized list of current database row field variables.
  • A processed version of $TCA['givenTable'] containing only those columns fields current user has access to.
  • A processed list of items for single fields like select and group types.
  • A list of relevant localizations.
  • Information of expanded inline record details if needed.
  • Resolved flex form data structures and data.
  • A lot more

Basic goal of this step is to create an array in a specified format with all data needed by the render-part of FormEngine. A controller initializes this with init data, and then lets single data providers fetch additional data and write it to the main array. The deal is here that the data within that array is not structured in an arbitrary way, and each single data provider only adds data the render part of FormEngine understands and needs later. This is why the main array keys are restricted: The main array is initialized by FormDataCompiler, and each DataProvider can only add data to sub-parts of that array.

Note

The main data array is prepared by FormDataCompiler, each key is well documented in this class. To find out which data is expected to reside in this array, those comments are worth a look.

Note

It may happen in future versions of FormEngine with core version 9, the responsibility for the main structure and integrity of the data array is moved away from FormDataCompiler into the single FormDataGroup. This may even obsolete the FormDataCompiler altogether.

Data groups and providers

So we have this empty data array pre-set with data by a controller and then initialized by FormDataCompiler, which it hands over to a specific FormDataGroup. What are these data providers now? Data providers are single classes that add or change data within the data array. They are called in a chain after each other. A FormDataGroup has the responsibility to find out which specific single data providers should be used and calls them in a specific order.

Data compiling by multiple providers

Why we need this?

  • Which data providers are relevant depends on the specific scope: For instance, if editing a full database based record, one provider fetches the according row from the database and initializes $data['databaseRow']. But if flex form data is calculated, the flex form values fetched from table fields directly. So, while the DatabaseEditRow data provider is needed in the first case, it's not needed or even counter productive in the second case. The FormDataGroup's are used to manage providers for specific scopes.
  • FormDataGroups know which providers should be used in a specific scope. They usually fetch a list of providers from some global configuration array. Extensions can add own providers in this configuration array for further data munging.
  • Single data providers have dependencies to each other and must be executed in a specific order. For Instance, the PageTsConfig of a record can only be determined if the rootline of a record has been determined, which can only happen after the pid of a given record has been consolidated, which relies on the record being fetched from database. This makes data providers a linked list and it is task of a FormDataGroup to manage the correct order.

Main data groups:

TcaDatabaseRecord
List of providers used if rendering a database based record.
FlexFormSegment
List of data providers used to prepare flex form data and flex form section container data.
TcaInputPlaceholderRecord
List of data providers used to prepare placeholder values for type=input and type=text fields.
InlineParentRecord
List of data providers used to prepare data needed if an inline record is opened from within an ajax call.
OnTheFly
A special data group that can be initialized with a list of to-execute data providers directly. In contrast to the others, it does not resort the data provider list by its dependencies and does not fetch the list of data providers from a global config. Used in the core at a couple of places where a small number of data providers should be called right away without being extensible.

Note

It is a good idea set a breakpoint at the form data result returned by the DataCompiler and to have a look at the data array to get an idea on what is in this array after compiling.

Let's have a closer look at the data providers. The main TcaDatabaseRecord group consists mostly of three parts:

Main record data and dependencies
  • Fetch record from DB or initialize a new row depending on $data['command'] being "new" or "edit", set row as $data['databaseRow']
  • Add userTs and pageTsConnfig to data array
  • Add table TCA as $data['processedTca']
  • Determine record type value
  • Fetch record translations and other details and add to data array
Single field processing
  • Process values and items of simple types like type=input, type=radio, type=check and so on. Validate their databaseRow values and validate and sanitize their processedTca settings.
  • Process more complex types that may have relations to other tables like type=group and type=select, set possible selectable items in $data['processedTca'] of the according fields, sanitize their TCA settings.
  • Process type=inline and type=flex fields and prepare their child fields by using new instances of FormDataCompiler and adding their result to $data['processedTca']
Post process after single field values are prepared
  • Execute display conditions and remove fields from $data['processedTca'] that shouldn't be shown.
  • Determine main record title and set as $data['recordTitle']
Extending data groups with own providers

The base set of DataProviders for all DataGroups is defined within typo3/sysext/core/Configuration/DefaultConfiguration.php in section ['SYS']['formEngine']['formDataGroup'], and ends up in variable $GLOBALS['TYPO3_CONF_VARS'] after core bootstrap. The provider list can be read top-down, so the DependencyOrderingService typically does not resort this list to a different order.

Adding an own provider to this list means adding an array key to that array having a specification where the new data provider should be added in the list. This is done by the arrays depends and before.

As an example, extension "news" uses an own data provider to do additional flex form data structure preparation. The core internal flex preparation is already split into two providers: TcaFlexPrepare determines the data structure and parses it, TcaFlexProcess uses the prepared data structure, processes values and applies defaults if needed. The data provider from extension news hooks in between these two to add some own preparation stuff. The registration happens with this code in ext_localconf.php:

// Modify flexform fields since core 8.5 via formEngine: Inject a data provider
// between TcaFlexPrepare and TcaFlexProcess
if (\TYPO3\CMS\Core\Utility\VersionNumberUtility::convertVersionNumberToInteger(TYPO3_version) >= 8005000) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['formDataGroup']['tcaDatabaseRecord']
    [\GeorgRinger\News\Backend\FormDataProvider\NewsFlexFormManipulation::class] = [
        'depends' => [
            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class,
        ],
        'before' => [
            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexProcess::class,
        ],
    ];
}

This is pretty powerful since it allows extensions to hook in additional stuff at any point of the processing chain, and it does not depend on the load order of extensions.

Limitations:
  • It is not easily possible to "kick out" an existing provider if other providers have dependencies to them - which is usually the case.
  • It is not easily possible to substitute an existing provider with an own one.

Note

It may happen that the core splits or deletes the one or the other DataProvider in the future. If then an extension has a dependency to a removed provider, the DependencyOrderingService which takes care of the sorting throws an exception. There is currently no good solution in the core on how to mitigate this issue.

Adding data to data array

Most custom data providers change or add existing data within the main data array. Typical use case is an additional record initialization for specific fields in $data['databaseRow'] or additional items somewhere within $data['processedTca']. The main data array is documented in FormDataCompiler->initializeResultArray().

Sometimes, own DataProvider need to add additional data that does not fit into existing places. In those cases, they can add stuff in $data['customData']. This key is not filled with data by core DataProviders and a place for extensions to add things. Those data components can be used in own code parts of the rendering later. It is advisable to prefix own data in $data['customData'] with some unique key (for instance the extension name) to not collide with other data a different extension may add.

Rendering

This is the second step of the processing chain: The rendering part gets the data array prepared by FormDataCompiler and creates a result array containing HTML, CSS and JavaScript. This is then post-processed by a controller to feed it to the PageRenderer or to create an ajax response.

The rendering is a tree: The controller intitializes this by setting one container as renderType entry point within the data array, then hands over the full data array to the NodeFactory which looks up a class responsible for this renderType, and calls render() on it. A container class creates only a fraction of the full result, and delegates details to another container. The second one does another detail and calls a third one. This continues to happen until a single field should be rendered, at which point an element class is called taking care of one element.

Render tree example

Each container creates some "outer" part of the result, calls some sub-container or element, merges the sub-result with its own content and returns the merged array up again. The data array is given to each sub class along the way, and containers can add further render relevant data to it before giving it "down". The data array can not be given "up" in a changed way again. Inheritance of a data array is always top-bottom. Only HTML, CSS or JavaScript created by a sub-class is returned by the sub-class "up" again in a "result" array of a specified format.

class SomeContainer extends AbstractContainer
{
    public function render()
    {
        $result = $this->initializeResultArray();
        $data = $this->data;
        $data['renderType'] = 'subContainer';
        $childArray = $this->nodeFactory->create($data)->render();
        $resultArray = $this->mergeChildReturnIntoExistingResult($result, $childArray, false);
        $result['html'] = '<h1>A headline</h1>' . $childArray['html'];
        return $result;
    }
}

Above example lets NodeFactory find and compile some data from "subContainer", and merges the child result with its own. The helper methods initializeResultArray() and mergeChildReturnIntoExistingResult() help with combining CSS and JavaScript.

An upper container does not directly create an instance of a sub node (element or container) and never calls it directly. Instead, a node that wants to call a sub node only refers to it by a name, sets this name into the data array as $data['renderType'] and then gives the data array to the NodeFactory which determines an appropriate class name, instantiates and initializes the class, gives it the data array, and calls render() on it.

Note

The SingleFieldContainer and FlexFormElementContainer will probably vanish with core version 9.

Note

Data set by containers and given down to children will likely change in core version 9: All fields not registered in the main data array of FormDataCompiler and only added within containers will move into section renderData. Furthermore, it is planned to remove parameterArray and substitute it with something better. This will affect most elements and will probably break a lot of these elements.

Class inheritance
Main render class inheritance

All classes must implement NodeInterface to be routed through the NodeFactory. The AbstractNode implements some basic helpers for nodes, the two classes AbstractContainer and AbstractFormElement implement helpers for containers and elements respectively.

The call concept is simple: A first container is called, which either calls a container below or a single element. A single element never calls a container again.

NodeFactory

The NodeFactory plays an important abstraction role within the render chain: Creation of child nodes is always routed through it, and the NodeFactory takes care of finding and validating the according class that should be called for a specific renderType. This is supported by an API that allows registering new renderTypes and overriding existing renderTypes with own implementations. This is true for all classes, including containers, elements, fieldInformation, fieldWizards and fieldControls. This means the child routing can be fully adapted and extended if needed. It is possible to transparently "kick-out" a core container and to substitute it with an own implementation.

As example, the TemplaVoila implementation needs to add additional render capabilities of the flex form rendering to add for instance an own multi-language rendering of flex fields. It does that by overriding the default flex container with own implementation:

// Default registration of "flex" in NodeFactory:
// 'flex' => \TYPO3\CMS\Backend\Form\Container\FlexFormEntryContainer::class,

// Register language aware flex form handling in FormEngine
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1443361297] = [
    'nodeName' => 'flex',
    'priority' => 40,
    'class' => \TYPO3\CMS\Compatibility6\Form\Container\FlexFormEntryContainer::class,
];

This re-routes the renderType "flex" to an own class. If multiple registrations for a single renderType exist, the one with highest priority wins.

Note

The NodeFactory uses $data['renderType']. This has been introduced with core version 7 in TCA, and a couple of TCA fields actively use this renderType. However, it is important to understand the renderType is only used within the FormEngine and type is still a must-have setting for columns fields in TCA. Additionally, type can not be overridden in columnsOverrides. Basically, type specifies how the DataHandler should put data into the database, while renderType specifies how a single field is rendered. This additionally means there can exist multiple different renderTypes for a single type, and it means it is possible to invent a new renderType to render a single field differently, but still let the DataHandler persist it the usual way.

Adding a new renderType in ext_localconf.php

// Add new field type to NodeFactory
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1487112284] = [
    'nodeName' => 'selectTagCloud',
    'priority' => '70',
    'class' => \MyVendor\CoolTagCloud\Form\Element\SelectTagCloudElement::class,
];

And use it in TCA for a specific field, keeping the full database functionality in DataHandler together with the data preparation of FormDataCompiler, but just routing the rendering of that field to the new element:

$GLOBALS['TCA']['myTable']['columns']['myField'] = [
    'label' => 'Cool Tag cloud',
    'config' => [
        'type' => 'select',
        'renderType' => 'selectTagCloud',
        'foreign_table' => 'tx_cooltagcloud_availableTags',
    ],
];

The above examples are a static list of nodes that can be changed by settings in ext_localconf.php. If that is not enough, the NodeFactory can be extended with a resolver that is called dynamically for specific renderTypes. This resolver gets the full current data array at runtime and can either return NULL saying "not my job", or return the name of a class that should handle this node.

An example of this are the core internal rich text editors. Both "ckeditor" and "rtehtmlarea" register a resolver class that are called for node name "text", and if the TCA config enables the editor, and if the user has enabled rich text editing in his user settings, then the resolvers return their own RichTextElement class names to render a given text field:

// Register FormEngine node type resolver hook to render RTE in FormEngine if enabled
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeResolver'][1480314091] = [
    'nodeName' => 'text',
    'priority' => 50,
    'class' => \TYPO3\CMS\RteCKEditor\Form\Resolver\RichTextNodeResolver::class,
];

The trick is here that "ckeditor" registers his resolver with ah higher priority (50) than "rtehtmlarea" (40), so the "ckeditor" resolver is called first and wins if both extensions are loaded and if both return a valid class name.

Result array

The array returned by every node looks like:

[
    'html' => '',
    'additionalInlineLanguageLabelFiles' => [],
    'stylesheetFiles' => [],
    'requireJsModules' => [],
]

CSS and language labels (which can be used in JS) are added with their file names in format EXT:extName/path/to/file. JavaScript is added only via RequireJS modules, the registration allows an init method to be called if the module is loaded by the browser.

Note

The result array actually contains a couple of more fields, but those will vanish with further FormEngine refactoring steps. If using them, be prepared to adapt extensions later.

Node expansion

The "node expansion" classes FieldControl, FieldInformation and FieldWizard are called by containers and elements and allow "enriching" containers and elements. Which enrichments are called can be configured via TCA.

This API is the substitution of the old "TCA wizards array" and has been introduced with core version 8.

FieldInformation
Additional information. In elements, their output is shown between the field label and the element itself. They can not add functionality, but only simple and restricted HTML strings. No buttons, no images. An example usage could be an extension that auto-translates a field content and outputs an information like "Hey, this field was auto-filled for you by an automatic translation wizard. Maybe you want to check the content".
FieldWizard
Wizards shown below the element. "enrich" an element with additional functionality. The localization wizard and the file upload wizard of type=group fields are examples of that.
FieldControl
"Buttons", usually shown next to the element. For type=group the "list" button and the "element browser" button are examples. A field control must return an icon identifier.

Currently, all elements usually implement all three of these, except in cases where it does not make sense. This API allows adding functionality to single nodes, without overriding the whole node. Containers and elements can come with default expansions (and usually do). TCA configuration can be used to add own stuff. On container side the implementation is still basic, only OuterWrapContainer and InlineControlContainer currently implement FieldInformation and FieldWizard.

See the TCA reference ctrl section for more information on how to configure these for containers in TCA.

Example. The InputTextElement (standard input element) defines a couple of default wizards and embeds them in its main result HTML:

class InputTextElement extends AbstractFormElement
{
    protected $defaultFieldWizard = [
        'localizationStateSelector' => [
            'renderType' => 'localizationStateSelector',
        ],
        'otherLanguageContent' => [
            'renderType' => 'otherLanguageContent',
            'after' => [
                'localizationStateSelector'
            ],
        ],
        'defaultLanguageDifferences' => [
            'renderType' => 'defaultLanguageDifferences',
            'after' => [
                'otherLanguageContent',
            ],
        ],
    ];

    public function render()
        $resultArray = $this->initializeResultArray();

        $fieldWizardResult = $this->renderFieldWizard();
        $fieldWizardHtml = $fieldWizardResult['html'];
        $resultArray = $this->mergeChildReturnIntoExistingResult($resultArray, $fieldWizardResult, false);

        $mainFieldHtml = [];
        $mainFieldHtml[] = '<div class="form-control-wrap">';
        $mainFieldHtml[] =  '<div class="form-wizards-wrap">';
        $mainFieldHtml[] =      '<div class="form-wizards-element">';
        // Main HTML of element done here ...
        $mainFieldHtml[] =      '</div>';
        $mainFieldHtml[] =      '<div class="form-wizards-items-bottom">';
        $mainFieldHtml[] =          $fieldWizardHtml;
        $mainFieldHtml[] =      '</div>';
        $mainFieldHtml[] =  '</div>';
        $mainFieldHtml[] = '</div>';

        $resultArray['html'] = implode(LF, $mainFieldHtml);
        return $resultArray;
    }
}

This element defines three wizards to be called by default. The renderType concept is re-used, the values localizationStateSelector are registered within the NodeFactory and resolve to class names. They can be overridden and extended like all other nodes. The $defaultFieldWizards are merged with TCA settings by the helper method renderFieldWizards(), which uses the DependencyOrderingService again.

It is possible to:

  • Override existing expansion nodes with own ones from extensions, even using the resolver mechanics is possible.
  • It is possible to disable single wizards via TCA
  • It is possible to add own expansion nodes at any position relative to the other nodes by specifying "before" and "after" in TCA.

Database Access

Contents:

Introduction

Database queries in TYPO3 are done with an API based on doctrine-dbal. The API is provided by the system extension core which is always loaded and thus always available.

Extension authors can use this low-level API to manage query operations directly on the configured DBMS.

Doctrine-dbal is feature rich. Drivers for various target systems enable TYPO3 to run on a long list of ANSI SQL compatible DBMS. If used properly, queries created with this API are translated to the specific database engine by doctrine without an extension developer taking care of that specifically.

The API provided by the core is basically a pretty small and lightweight facade in front of doctrine-dbal that adds some convenient methods as well as some TYPO3 CMS specific sugar. The facade additionally provides methods to retrieve specific connection objects per configured database connection based on the table that is queried. This enables instance administrators to configure different database engines for different tables while this is transparent for extension developers.

doctrine-dbal has been introduced with TYPO3 CMS version 8 and substitutes the old API based on $GLOBALS['TYPO3_DB']. Extension authors are encouraged to switch away from TYPO3_DB to the new API. A dedicated chapter helps with typical migration questions. With database abstraction being built in doctrine-dbal the old and optional extensions dbal and adodb are obsolete.

This document does not outline each and every single method the API provides. It sticks to those that are commonly used in extensions and some parts like the rewritten schema migrator are left out since they are usually of little to no interest for extensions.

Understanding Doctrine-Dbal and Doctrine-Orm

Doctrine is a two-fold project with doctrine-dbal being the low-level database abstraction and query building interface to specific database engines, while doctrine-orm is a high-level object relational mapping on top of doctrine-dbal.

The TYPO3 CMS core - only - implements the dbal part. doctrine-orm is neither required nor implemented nor used at the time of this writing.

Low-level and high-level database calls

This documentation is about low-level database calls. In many cases it is better to use higher level API's like the DataHandler or extbase repositories and to let the framework handle persistence details internally.

Tip

Always remember the high-level database calls and use them when appropriate!

Credits

Implementing the doctrine-dbal API into TYPO3 has been a huge project in 2016. Special thanks goes to awesome Mr. Morton Jonuschat for the initial design, integration and support and to more than 40 different people who actively contributed to migrate more than 1700 calls from TYPO3_DB-style to Doctrine within half ayear. This was a huge community achievement, thanks everyone involved!

Configuration

Configuring doctrine-dbal for TYPO3 CMS is all about specifying the single database endpoints and handing over connection credentials. The frameworks supports the parallel usage of multiple database connections, a specific connection is mapped depending on its table name. The table space can be seen as a transparent layer that determines which specific connection is chosen for a query to a single or a group of tables: It allows "swapping-out" single tables from the Default connection to point them to a different database endpoint.

As with other central configuration options, the database endpoint and mapping configuration happens within typo3conf/LocalConfiguration.php and ends up in $GLOBALS['TYPO3_CONF_VARS'] after core bootstrap. The specific sub-array is $GLOBALS['TYPO3_CONF_VARS']['DB'].

A typical, basic example using only the Default connection with a single database endpoint:

// LocalConfiguration.php
// [...]
'DB' => [
   'Connections' => [
      'Default' => [
         'charset' => 'utf8',
         'dbname' => 'theDatabaseName',
         'driver' => 'mysqli',
         'host' => 'theHost',
         'password' => 'theConnectionPassword',
         'port' => 3306,
         'user' => 'theUser',
      ],
   ],
],
// [...]

Remarks:

  • The Default connection must be configured, this can not be left out or renamed.
  • For mysqli, if the host is set to localhost and if the default PHP options in this area are not changed, the connection will be socket based. This saves a little overhead. To force a TCP/IP based connection even for localhost, the IPv4 or IPv6 address 127.0.0.1 and ::1/128 respectively must be used as host value.
  • The connect options are hand over to doctrine-dbal without much manipulation from TYPO3 CMS side. Please refer to the doctrine connection docs for a full overview of settings.
  • If charset option is not specified it defaults to utf8.
  • The option wrapperClass is used by the TYPO3 CMS framework to "hang in" the extended Connection class TYPO3\CMS\Database\Connection as main facade around doctrine-dbal.

A slightly more complex example with two connections, mapping the sys_log table to a different endpoint:

// LocalConfiguration.php
// [...]
'DB' => [
   'Connections' => [
      'Default' => [
         'charset' => 'utf8',
         'dbname' => 'default_dbname',
         'driver' => 'mysqli',
         'host' => 'default_host',
         'password' => '***',
         'port' => 3306,
         'user' => 'default_user',
      ],
      'Syslog' => [
         'charset' => 'utf8',
         'dbname' => 'syslog_dbname',
         'driver' => 'mysqli',
         'host' => 'syslog_host',
         'password' => '***',
         'port' => 3306,
         'user' => 'syslog_user',
      ],
   ],
   'TableMapping' => [
      'sys_log' => 'Syslog'
   ]
],
// [...]

Remarks:

  • The array key Syslog is just a name, it can be different but it's good practice to give it a useful speaking name.
  • It is possible to map multiple tables to a different endpoint by adding further table name / connection name pairs to TableMapping.
  • Mind this "connection per table" approach is limited: If in the above example a join query that spans over different connections is fired, an exception is raised. It is up to the administrator to group affected tables to the same connection in those cases, or a developer should implement some fallback logic to suppress the join().

Attention

At the time of this writing (TYPO3 CMS version 8.6), there are still some known issues with connections to databases other than mysql or mariadb. Core internal tests to postgresql and mssql still fail. This however should improve with younger versions.

Furthermore, the TYPO3 CMS installer supports only a single mysql or mariadb connection at the moment and the connection details can not be properly edited within the All configuration section of the install tool.

The core team hopes this situation settles until the final release of TYPO3 v8 LTS.

Basic CRUD

A list of basic usage examples of the query API. This is just a kickstart. Details on the single methods are found in the following chapters, especially QueryBuilder and Connection.

Note

The examples use the shorthand syntax for class names. Please refer to Class overview for the full namespace.

INSERT a row

A straight insert to a table:

GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->insert(
        'tt_content',
        [
            'pid' => (int)42,
            'bodytext' => 'bernd',
        ]
    );
INSERT INTO `tt_content` (`pid`, `bodytext`) VALUES ('42', 'bernd')
SELECT a single row

Straight fetch of a single row from tt_content table:

$uid = 4
$row = GeneralUtility::makeInstance(ConnectionPool::class)
    ->getConnectionForTable('tt_content')
    ->select(
        ['uid', 'pid', 'bodytext'], // fields to select
        'tt_content', // from
        [ 'uid' => (int)$uid ] // where
    )
    ->fetch();

Result in $row:

array(3 items)
   uid => 4 (integer)
   pid => 35 (integer)
   bodytext => 'some content' (12 chars)

The engine quotes field names, adds default TCA restrictions like "deleted=0", and prepares a query executed with this final statement:

SELECT `uid`, `pid`, `bodytext`
    FROM `tt_content`
    WHERE (`uid` = '4')
        AND ((`tt_content`.`deleted` = 0)
        AND (`tt_content`.`hidden` = 0)
        AND (`tt_content`.`starttime` <= 1473447660)
        AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1473447660)))

Note

Default restrictions deleted, hidden, startime and endtime based on TCA setting of a table are only applied to select() calls, they are not added for delete() or other query types.

SELECT multiple rows with some WHERE magic

Advanced query using the QueryBuilder and manipulating the default restrictions:

$uid = 4;
// Get a query builder for a query on table "tt_content"
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
// Remove all default restrictions (delete, hidden, starttime, stoptime), but add DeletedRestriction again
$queryBuilder->getRestrictions()
    ->removeAll()
    ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
// Execute a query with "bodytext=klaus OR uid=4" and proper quoting
$rows = $queryBuilder
    ->select('uid', 'pid', 'bodytext')
    ->from('tt_content')
    ->where(
        $queryBuilder->expr()->orX(
            $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
            $queryBuilder->expr()->eq('uid', $queryBuilder->createNamedParameter($uid, \PDO::PARAM_INT))
        )
    )
    ->execute()
    ->fetchAll();

Result in $rows:

array(2 items)
   0 => array(3 items)
      uid => 4 (integer)
      pid => 35 (integer)
      bodytext => 'bernd' (5 chars)
   1 => array(3 items)
      uid => 366 (integer)
      pid => 13 (integer)
      bodytext => 'klaus' (5 chars)

The executed query looks like:

SELECT `uid`, `pid`, `bodytext`
    FROM `tt_content`
    WHERE ((`bodytext` = 'klaus') OR (`uid` = 4))
        AND (`tt_content`.`deleted` = 0)
UPDATE multiple rows
GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->update(
        'tt_content',
        [ 'bodytext' => 'bernd' ], // set
        [ 'bodytext' => 'klaus' ] // where
    );
UPDATE `tt_content` SET `bodytext` = 'bernd' WHERE `bodytext` = 'klaus'
DELETE a row
GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->delete(
        'tt_content', // from
        [ 'uid' => (int)4711 ] // where
    );
DELETE FROM `tt_content` WHERE `uid` = '4711'

Class overview

Doctrine provides a set of php objects to represent, create and handle SQL queries and their results. The basic class structure was slightly enriched by TYPO3 to add CMS specific features. Extension authors will typically interact with these classes and objects:

Connection
TYPO3\CMS\Core\Database\Connection: Object representing a specific connection to one connected database. Provides "shortcut" methods for simple standard queries like SELECT or UPDATE. An instance of the QueryBuilder can be retrieved to build more complex queries.
ConnectionPool
TYPO3\CMS\Core\Database\ConnectionPool: Main entry point for extensions to retrieve a specific connection a query should be executed on. Typically used to return a Connection or a QueryBuilder object.
ExpressionBuilder
TYPO3\CMS\Core\Database\Query\Expression\ExpressionBuilder: Object to model complex expressions. Mainly used for WHERE and JOIN conditions.
QueryBuilder
TYPO3\CMS\Core\Database\Query\QueryBuilder: Object to create all sort of complex queries executed on a specific connection. Provides the main CRUD methods for select(), delete() and friends.
QueryHelper
TYPO3\CMS\Core\Database\Query\QueryHelper: Set of static helper methods that can simplify the transition from old TYPO3_DB based code to the doctrine base API.
Restriction ...
TYPO3\CMS\Core\Database\Query\Restriction\...: Set of classes that add expressions like "deleted=0" to a query based on TCA settings of a table. This automatically adds TYPO3 specific restrictions like starttime and endtime, as well as deleted and hidden flags. Further restrictions for language overlays and workspaces are available. This documentation refers to these classes as the RestrictionBuilder.
Statement
Doctrine\DBAL\Driver\Statement: Result object retrieved if a SELECT or COUNT query has been executed. Single rows are returned as array by calling ->fetch() until the method returns false.

ConnectionPool

TYPO3's interface to execute queries via doctrine-dbal typically starts by asking the ConnectionPool for a QueryBuilder or a Connection object, handing over the table name to be queried:

// Get a query builder for a table
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tx_myext_comments');
// or
// Get a connection for a table
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tx_myext_comments');

The QueryBuilder is the default workhorse object used by extension authors to express complex queries, while a Connection instance can be used as shortcut to deal with some simple query cases and little written down code.

Pooling

TYPO3 can handle multiple connections to different database endpoints at the same time. This can be configured on a per-table basis in $TYPO3_CONF_VARS. It allows running tables on different databases, without an extension developer taking care of that.

The ConnectionPool implements this feature: It looks up a configured table-to-database mapping and can return a Connection or a QueryBuilder instance for that specific connection. Those objects internally know which target connection they are dealing with and will for instance quote field names accordingly.

The transparency of tables to different database endpoints is limited, though:

Executing a table JOIN between two tables that point to different connections will throw an exception. This restriction may in practice create implicit "groups" of tables that need to point to one connection at once if an extension or the TYPO3 core joins those tables.

This can turn out as a headache if multiple different extensions use for instance the core category or collection API with their mm table joins between core internal tables and their extension's counterparts.

That situation is not easy to deal with. At the time of this writing the core development will eventually implement some non-join fallbacks for typical cases that would be good to decouple, though.

Tip

In case joins cannot be decoupled but still affected tables must run on different databases, and if the code can not be easily adapted, some DBMS like PostgreSQL allow executing those queries by having own connection handlers to different other endpoints on its own.

QueryBuilder

The QueryBuilder is a rather huge class that takes care of the main query dealing.

An instance can get hold of by calling the ConnectionPool->getQueryBuilderForTable() and handing over the table. Never instantiate and initialize the QueryBuilder directly via makeInstance()!

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('aTable');

This documentation does not mention every single available method but sticks to those used in casual queries and normal code flow. There are a couple of not mentioned methods, most of them are either very seldom used or marked as internal. Extension authors typically don't have to deal with anything not mentioned here.

Warning

From security point of view, the documentation of ->createNamedParameter() and ->quoteIdentifier() are an absolute must read and follow section. Make very sure this is understood and use this for each and every query to prevent SQL injections!

The QueryBuilder comes with a happy little list of small methods:

  • Set type of query: ->select(), ->count(), ->update(), ->insert() and delete()
  • Prepare WHERE conditions
  • Manipulate default WHERE restrictions added by TYPO3 for ->select()
  • Add LIMIT, GROUP BY and other SQL stuff
  • ->execute() a query and retrieve a Statement (a query result) object

Most methods of the QueryBuilder return $this and can be chained:

$queryBuilder->select('uid')->from('pages');

Note

The QueryBuilder holds internal state and should not be re-used for different queries: Use one query builder per query. Get a fresh one by calling $connection->createQueryBuilder() if the same table is affected, or use $connectionPool->getQueryBuilderForTable() for a query on to a different table. Don't worry, creating those object instances is rather quick.

select() and addSelect()

Create a SELECT query.

Select all fields:

// SELECT *
$queryBuilder->select('*')

->select() and a number of other methods of the QueryBuilder are variadic and can handle any number of arguments. For ->select(), every argument is interpreted as a single field name to select:

// SELECT `uid`, `pid`, `aField`
$queryBuilder->select('uid', 'pid', 'aField');

Argument unpacking can be used if the list of fields is available as array already:

$fields = ['uid', 'pid', 'aField', 'anotherField'];
$queryBuilder->select(...$fields);

->select() supports AS and quotes identifiers automatically. This can become especially handy in join() operations:

// SELECT `tt_content`.`bodytext` AS `t1`.`text`
$queryBuilder->select('tt_content.bodytext AS t1.text')

->select() sets the list of fields that should be selected and ->addSelect() can add further items to an existing list.

Mind that ->select() resets any formerly registered list and does not append. Thus, it usually doesn't make much sense to call select() twice in a code flow, or to call it after an ->addSelect(). The methods ->where() and ->andWhere() share the same behavior.

A useful combination of ->select() and ->addSelect() can be:

$queryBuilder->select(...$defaultList);
if ($needAdditionalFields) {
   $queryBuilder->addSelect(...$additionalFields);
}

Calling ->execute() on a ->select() query returns a Statement object. To receive single rows a ->fetch() loop on that object is used, or ->fetchAll() to return a single array with all rows. A typical code flow of a SELECT query looks like:

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->execute();
while ($row = $statement->fetch()) {
   // Do something with that single row
   debug($row);
}

Note

->select() and ->count() queries trigger TYPO3 CMS magic that adds further default where clauses if the queried table is also registered via $GLOBALS['TCA']. See the RestrictionBuilder section for details on that topic.

count()

Create a COUNT query, a typical usage:

// SELECT COUNT(`uid`) FROM `tt_content` WHERE (`bodytext` = 'klaus')
//     AND ((`tt_content`.`deleted` = 0) AND (`tt_content`.`hidden` = 0)
//     AND (`tt_content`.`starttime` <= 1475580240)
//     AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1475580240)))
$count = $queryBuilder
   ->count('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
    )
   ->execute()
   ->fetchColumn(0);

Remarks:

  • Similar to the ->select() query type, ->count() automatically triggers RestrictionBuilder magic that adds default deleted, hidden, starttime and endtime restrictions if that is defined in TCA.
  • Similar to ->select() query types, ->execute() with ->count() returns a Statement object. To fetch the number of rows directly, use ->fetchColumn(0).
  • First argument to ->count() is required, typically ->count(*) or ->count('uid') is used, the field name is automatically quoted.
  • There is no support for DISTINCT, a ->groupBy() has to be used instead.
  • If combining ->count() with a ->groupBy(), the result may return multiple rows. The order of those rows depends on the used DBMS. To ensure same order of result rows on multiple different databases, a ->groupBy() should thus always be combined with a ->orderBy().
delete()

Create a DELETE FROM query. The method requires the table name to drop data from. Classic usage:

// DELETE FROM `tt_content` WHERE `bodytext` = 'klaus'
$affectedRows = $queryBuilder
   ->delete('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $this->createNamedParameter('klaus'))
   )
   ->execute();

Remarks:

  • For simple cases, it is often easier to write and read if using the ->delete() method of the Connection object.
  • In contrast to ->select(), ->delete() does not add WHERE restrictions like AND `deleted` = 0 automatically.
  • ->delete() does not magically transform a DELETE FROM `tt_content` WHERE `uid` = 4711 to something like UPDATE `tt_content` SET `deleted` = 1 WHERE `uid` = 4711 internally. A soft-delete must be handled on application level code with a dedicated lookup in $GLOBALS['TCA']['theTable']['ctrl']['deleted'] to check if a specific table can handle the soft-delete, together with an ->update() instead.
  • Multi-table delete is not supported: DELETE FROM `table1`, `table2` can not be created.
  • ->delete() ignores ->join()
  • ->delete() ignores setMaxResults(): DELETE with LIMIT does not work.
update() and set()

Create an UPDATE query. Typical usage:

// UPDATE `tt_content` SET `bodytext` = 'peter' WHERE `bodytext` = 'klaus'
$queryBuilder
   ->update('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $this->createNamedParameter('klaus')
   )
   ->set('bodytext', 'peter')
   ->execute();

->update() requires the table to update as first argument and a table alias as optional second argument. The table alias can then be used in ->set() and ->where() expressions:

// UPDATE `tt_content` `t` SET `t`.`bodytext` = 'peter' WHERE `u`.`bodytext` = 'klaus'
$queryBuilder
   ->update('tt_content', 'u')
   ->where(
      $queryBuilder->expr()->eq('u.bodytext', $this->createNamedParameter('klaus')
   )
   ->set('u.bodytext', 'peter')
   ->execute();

->set() requires a field name as first argument and automatically quotes it internally. The second mandatory argument is the value a field should be set to, the value is automatically transformed to a named parameter of a prepared statement. This way, ->set() key/value pairs are automatically SQL injection save by default.

If a field should be set to the value of another field from the row, the quoting needs to be turned off and ->quoteIdentifier() has to be used:

// UPDATE `tt_content` SET `bodytext` = `header` WHERE `bodytext` = 'klaus'
$queryBuilder
   ->update('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->set('bodytext', $queryBuilder->quoteIdentifier('header'), false)
   ->execute();

Remarks:

  • For simple cases, it is often easier to use the ->update() method of the Connection object.
  • ->set() can be called multiple times if multiple fields should be updated.
  • ->set() requires a field name as first argument and automatically quotes it internally.
  • ->set() requires the value a field should be set to as second parameter.
  • ->update() ignores ->join() and ->setMaxResults().
  • The API does not magically add delete = 0 or other restrictions magically.
insert() and values()

Create an INSERT query. Typical usage:

$affectedRows = $queryBuilder
   ->insert('tt_content')
   ->values([
      'bodytext' => 'klaus',
      'header' => 'peter',
   ])
   ->execute();

Remarks:

  • It is often easier to use ->insert() or ->bulkInsert() of the Connection object.
  • ->values() expects an array of key/value pairs. Both keys (field names / identifiers) and values are automatically quoted. In rare cases, quoting of values can be turned off by setting the second argument to false. In those cases the quoting has to be done manually, typically by using ->createNamedParameter() on the values, use with care ...
  • ->execute() after ->insert() returns the number of inserted rows, which is typically 1.
  • QueryBuilder does not contain a method to insert multiple rows at once, use ->bulkInsert() of Connection object instead to achieve that.
from()

->from() is a must have call for ->select() and ->count() query types. ->from() needs a table name and an optional alias name. The method is typically called once per query build and the table name is typically the same as what was given to ->getQueryBuilderForTable(). If the query joins multiple tables, the argument should be the name of the first table within the ->join() chain:

// FROM `myTable`
$queryBuilder->from('myTable');

// FROM `myTable` AS `anAlias`
$queryBuilder->from('myTable', 'anAlias');

->from() can be called multiple times and will create the cartesian product of tables if not restricted by an according ->where() or ->andWhere() expression. In general, it is a good idea to use ->from() only once per query and model multi-table selection with an explicit ->join() instead.

where(), andWhere() and orWhere()

The three methods are used to create WHERE restrictions for SELECT, COUNT, UPDATE and DELETE query types. Each argument is typically an ExpressionBuilder object that will be cast to a string on ->execute():

// SELECT `uid`, `header`, `bodytext`
// FROM `tt_content`
// WHERE
//    (
//       ((`bodytext` = 'klaus') AND (`header` = 'a name'))
//       OR (`bodytext` = 'peter') OR (`bodytext` = 'hans')
//    )
//    AND (`pid` = 42)
//    AND ... RestrictionBuilder TCA restrictions ...
$statement = $queryBuilder
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
      $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('a name'))
   )
   ->orWhere(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('peter')),
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('hans'))
   )
   ->andWhere(
      $queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Note the parenthesis of the above example: ->andWhere() encapsulates both ->where() and ->orWhere() with an additional restriction.

Argument unpacking can become handy with these methods:

$whereExpressions = [
   $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
   $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('a name'))
];
if ($needsAdditionalExpression) {
   $whereExpressions[] = $someAdditionalExpression;
}
$queryBuilder->where(...$whereExpressions);

Remarks:

  • The three methods are variadic. They can handle any number of arguments. If for instance ->where() receives four arguments, they are handled as single expressions, all of them combined with AND.
  • ->where() should be called only once per query and it resets any previously set ->where(), ->andWhere() and ->orWhere() expression. Having a ->where() call after a previous ->where(), ->andWhere() or ->orWhere() typically indicates a bug or a rather weird code flow. Doing so is discouraged.
  • While creating complex WHERE restrictions, ->getSQL() is a helpful debugging friend to verify parenthesis and single query parts.
  • If using only ->eq() expressions, it is often easier to switch to the according Connection object method to simplify quoting and increase readability.
  • It is possible to feed the methods with strings directly, but that is discouraged and typically only used in rare cases where expression strings are created at a different place that can not be resolved easily. In the core, those places are usually combined with QueryHelper::stripLogicalOperatorPrefix() to remove leading AND or OR parts. Using this gives an additional risk of missing or wrong quoting and is a potential security issue. Use with care if ever.
join(), innerJoin(), rightJoin() and leftJoin()

Joining multiple tables in a ->select() or ->count() query is done with one of these methods. Multiple joins are supported by calling the methods more than once. All methods require four arguments: The name of the left side table (or its alias), the name of the right side table, an alias for the right side table name and the join restriction as fourth argument:

// SELECT `sys_language`.`uid`, `sys_language`.`title`
// FROM `sys_language`
// INNER JOIN `pages_language_overlay` `overlay`
//     ON `overlay`.`sys_language_uid` = `sys_language`.`uid`
// WHERE
//     (`overlay`.`pid` = 42)
//     AND (
//          (`overlay`.`deleted` = 0)
//          AND (
//              (`sys_language`.`hidden` = 0) AND (`overlay`.`hidden` = 0)
//          )
//          AND (`overlay`.`starttime` <= 1475591280)
//          AND ((`overlay`.`endtime` = 0) OR (`overlay`.`endtime` > 1475591280))
//     )
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$statement = $queryBuilder
   ->select('sys_language.uid', 'sys_language.title')
   ->from('sys_language')
   ->join(
      'sys_language',
      'pages_language_overlay',
      'overlay',
      $queryBuilder->expr()->eq('overlay.sys_language_uid', $queryBuilder->quoteIdentifier('sys_language.uid'))
   )
   ->where(
      $queryBuilder->expr()->eq('overlay.pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Notes to the above example:

  • The query operates on table sys_language as main table, this table name is given to getQueryBuilderForTable().
  • The query joins table pages_language_overlay as INNER JOIN, giving it the alias overlay.
  • The join condition is `overlay`.`sys_language_uid` = `sys_language`.`uid`. It would have been identical to swap the expression arguments of the fourth ->join() argument ->eq('sys_language.uid', $queryBuilder->quoteIdentifier('overlay.sys_language_uid')).
  • The second argument of the join expression instructs the ExpressionBuilder to quote the value as a field identifier (a field name, here a table/field name combination). Using createNamedParameter() would lead to a quoting as value (' instead of ` in mysql) and the query would fail.
  • The alias overlay - the third argument of the ->join() call - does not necessarily have to be set to a different name than the table name itself here. Using pages_language_overlay as third argument and not specifying a different name would do. Aliases are mostly useful if a join to the same table is needed: SELECT `something` FROM `tt_content` JOIN `tt_content` `content2` ON .... Aliases additionally become handy to increase readability of ->where() expressions.
  • The RestrictionBuilder added additional WHERE conditions for both involved tables! Table sys_language obviously only specifies a 'disabled' => 'hidden' as enableColumns in its TCA ctrl section, while table pages_language_overlay specifies deleted, hidden, starttime and stoptime fields.

A more complex example with two joins. The first join points to the first table again using an alias to resolve a language overlay scenario. The second join uses the alias name of the first join target as left side:

// SELECT `tt_content_orig`.`sys_language_uid`
// FROM `tt_content`
// INNER JOIN `tt_content` `tt_content_orig` ON `tt_content`.`t3_origuid` = `tt_content_orig`.`uid`
// INNER JOIN `sys_language` `sys_language` ON `tt_content_orig`.`sys_language_uid` = `sys_language`.`uid`
// WHERE
//     (`tt_content`.`colPos` = 1)
//     AND (`tt_content`.`pid` = 42)
//     AND (`tt_content`.`sys_language_uid` = 2)
//     AND ... RestrictionBuilder TCA restrictions for tables tt_content and sys_language ...
// GROUP BY `tt_content_orig`.`sys_language_uid`
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$constraints = [
   $queryBuilder->expr()->eq('tt_content.colPos', $queryBuilder->createNamedParameter(1, \PDO::PARAM_INT)),
   $queryBuilder->expr()->eq('tt_content.pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)),
   $queryBuilder->expr()->eq('tt_content.sys_language_uid', $queryBuilder->createNamedParameter(2, \PDO::PARAM_INT)),
];
$queryBuilder
   ->select('tt_content_orig.sys_language_uid')
   ->from('tt_content')
   ->join(
      'tt_content',
      'tt_content',
      'tt_content_orig',
      $queryBuilder->expr()->eq(
         'tt_content.t3_origuid',
         $queryBuilder->quoteIdentifier('tt_content_orig.uid')
      )
   )
   ->join(
      'tt_content_orig',
      'sys_language',
      'sys_language',
      $queryBuilder->expr()->eq(
         'tt_content_orig.sys_language_uid',
         $queryBuilder->quoteIdentifier('sys_language.uid')
      )
   )
   ->where(...$constraints)
   ->groupBy('tt_content_orig.sys_language_uid')
   ->execute();

Further remarks:

  • ->join() and innerJoin are identical. They create an INNER JOIN query, this is identical to a JOIN query.
  • ->leftJoin() creates a LEFT JOIN query, this is identical to a LEFT OUTER JOIN query.
  • ->rightJoin() creates a RIGHT JOIN query, this is identical to a RIGT OUTER JOIN query.
  • Calls on join() methods are only considered for ->select() and ->count() type queries. ->delete(), ->insert() and update() do not support joins, those query parts are ignored and do not end up in the final statement.
  • The argument of ->getQueryBuilderForTable() should be the left most main table.
  • A join of two tables that are configured to different connections will throw an exception. This restricts which tables can be configured to different database endpoints. It is possible to test the connection objects of involved tables for equality and implement a fallback logic in PHP if they are different.
orderBy() and addOrderBy()

Add ORDER BY to a ->select() statement. Both ->orderBy() and ->addOrderBy() require a field name as first argument:

// SELECT * FROM `sys_language` ORDER BY `sorting` ASC
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->getRestrictions()->removeAll();
$languageRecords = $queryBuilder
   ->select('*')
   ->from('sys_language')
   ->orderBy('sorting')
   ->execute()
   ->fetchAll();

Remarks:

  • ->orderBy() resets any previously specified orders. It doesn't make sense to call it after a previous ->orderBy() or ->addOrderBy() again.
  • Both methods need a field name or a table.fieldName or a tableAlias.fieldName as first argument, in the above example calling ->orderBy('sys_language.sorting') would have been identical. All identifiers are quoted automatically.
  • The second, optional argument of both methods specifies the sorting order. The two allowed values are ASC and DESC where ASC is default and can be omited.
  • To create a chain of orders, use ->orderBy() and then multiple ->addOrderBy() calls. Calling ->orderBy('header')->addOrderBy('bodytext')->addOrderBy('uid', DESC') creates ORDER BY `header` ASC, `bodytext` ASC, `uid` DESC
groupBy() and addGroupBy()

Add GROUP BY to a ->select() statement. Each argument to the methods is a single identifier:

// GROUP BY `pages_language_overlay`.`sys_language_uid`, `sys_language`.`uid`
->groupBy('pages_language_overlay.sys_language_uid', 'sys_language.uid');

Remarks:

  • Similar to ->select() and ->where() both methods are variadic and take any number of arguments, argument unpacking is supported: ->groupBy(...$myGroupArray)
  • Each argument is either a direct field name GROUP BY `bodytext`, a table.fieldName or a tableAlias.fieldName and will be properly quoted.
  • ->groupBy() resets any previously set group specification and should be called only once per statement.
setMaxResults() and setFirstResult()

Add LIMIT to restrict number of records and OFFSET for pagination query parts. Both methods should be called only once per statement:

// SELECT * FROM `sys_language` LIMIT 2 OFFSET 4
$queryBuilder
   ->select('*')
   ->from('sys_language')
   ->setMaxResults(2)
   ->setFirstResult(4)
   ->execute();

Remarks:

  • It's allowed to call ->setMaxResults() but not to call ->setFirstResult().
  • It is possible to call ->setFirstResult() without calling setMaxResults(): This equals to "Fetch everything, but leave out the first n records". Internally, LIMIT will be added by doctrine-dbal and set to a very high value.
getSQL()

Method ->getSQL() returns the created query statement as string. It is incredible useful during development to verify the final statement is executed just as a developer expects it:

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->select('*')->from('sys_language');
debug($queryBuilder->getSQL());
$statement = $queryBuilder->execute();

Remarks:

  • This is debugging code. Take proper actions to ensure those calls do not end up in production!
  • The method is typically called directly in front ->execute() to output the final statement.
  • Casting a QueryBuilder object to (string) has the same effect as calling ->getSQL(), the explicit call using the method should be preferred to simplify a search operation for this kind of debugging statements, though.
  • The method is a simple way to see which restrictions the RestrictionBuilder added.
  • doctrine-dbal always creates prepared statements: Any value that added via ->createNamedParameter() creates a placeholder that is later substituted if the real query is fired via ->execute(). ->getSQL() does not show those values, instead the placeholder names are displayed, usually with a string like :dcValue1. There is no simple solution to show the fully replaced query from within the framework.
execute()

Compile and fire the final query statement. This is usually the last call on a QueryBuilder object. The method has two possible return values: On success, it either returns a Statement object representing the result set of ->select() and ->count() queries, or it returns an integer representing the number of affected rows for ->insert(), ->update() and ->delete() queries.

If the query fails for whatever reason (for instance if the database connection was lost or if the query contains a syntax error), a \Doctrine\DBAL\DBALException is thrown. It is most often bad habit to catch and suppress this exception since it indicates a runtime or a program error. Both should bubble up. See the coding guidelines for more information on proper exception handling.

expr()

Return an instance of the ExpressionBuilder. This object is used to create complex WHERE query parts and JOIN expressions:

// SELECT `uid` FROM `tt_content` WHERE (`uid` > 42)
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->gt('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Remarks:

  • This object is stateless and can be called and worked on as often as needed. It however bound to the specific connection a statement is created for and is thus only available through the QueryBuilder which is specific for one connection, too.
  • Never re-use the ExpressionBuilder, especially not between multiple QueryBuilder objects, always get an instance of the ExpressionBuilder by calling ->expr().
createNamedParameter()

Create a placeholder for a prepared statement field value. Always use that when dealing with user input in expressions to make the statement SQL injection safe:

// SELECT * FROM `tt_content` WHERE (`bodytext` = 'kl\'aus')
$searchWord = "kl'aus"; // $searchWord = GeneralUtility::_GP('searchword');
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->getRestrictions()->removeAll();
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter($searchWord))
   )
   ->execute();

The above example shows the importance of using ->createNamedParameter(): The search word kl'aus is "tainted" and would break the query if not channeled through ->createNamedParameter() which quotes the backtick and makes the value SQL injection safe.

Not convinced? Suppose the code would look like this:

// NEVER EVER DO THIS!
$_POST['searchword'] = "'foo' UNION SELECT username FROM be_users";
$searchWord = GeneralUtility::_GP('searchword');
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->getRestrictions()->removeAll();
   this fails with syntax error to prevent copy and paste
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      // MASSIVE SECURITY ISSUE DEMONSTRATED HERE, USE ->createNamedParameter() ON $searchWord!
      $queryBuilder->expr()->eq('bodytext', $searchWord)
    );

Mind the missing ->createNamedParameter() in the ->eq() expression on given value! This code would happily execute the statement SELECT uid FROM `tt_content` WHERE `bodytext` = 'foo' UNION SELECT username FROM be_users; returning a list of backend user names!

Rules:

  • Always use ->createNamedParameter() around any input, no matter where it comes from.

  • The second argument of ->expr() is always either a call to ->createNamedParameter() or ->quoteIdentifier().

  • The second argument of ->createNamedParameter() specifies the type of input. For string, this can be omitted, but it is good practice to add \PDO::PARAM_INT for integers or similar for other field types. This is currently no strict rule, but following this will reduces headaches in the future, especially for DBMS that are not as relaxed as mysql when it comes to field types. The PDO constants can be used for simple types like bool, string, null, lob and integer. Additionally, the two constants Connection::PARAM_INT_ARRAY and Connection::PARAM_STR_ARRAY can be used if an array of strings or integers is handled, for instance in an IN() expression.

  • Keep the ->createNamedParameter() as close as possible to the expression. Do not structure your code in a way that it first quotes something and only later stuffs the already prepared names into the expression. Having ->createNamedParameter() directly within the created expression is much less error prone and easier to review. This is a general rule: Sanitizing input must be as close as possible to the "sink" where a value is submitted to a lower part of the framework. This paradigm should be followed for other quote operations like htmlspecialchars() or GeneralUtility::quoteJSvalue(), too. Sanitizing should be directly obvious at the very place where it is important:

    // DO
    $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
    $queryBuilder->getRestrictions()->removeAll();
    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
           $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter($searchWord))
       )
    
    // DON'T DO, this is much harder to track:
    $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
    $myValue = $queryBuilder->createNamedParameter($searchWord);
    // Imagine much more code here
    $queryBuilder->getRestrictions()->removeAll();
    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
           $queryBuilder->expr()->eq('bodytext', $myValue)
       )
    
quoteIdentifier() and quoteIdentifiers()

->quoteIdentifier() must be used if not a value is handled, but a field name. The quoting is different in those cases and typically ends up with backticks ` instead of ticks ':

// SELECT `uid` FROM `tt_content` WHERE (`header` = `bodytext`)
// Return list of rows where header and bodytext values are identical
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('header', $queryBuilder->quoteIdentifier('bodytext'))
   );

The method quotes single field names or combinations of table names or table aliases with field names:

// Single field name: `bodytext`
->quoteIdentifier('bodytext');
// Table name and field name: `tt_content`.`bodytext`
->quoteIdentifier('tt_content.bodytext')
// Table alias and field name: `foo`.`bodytext`
->from('tt_content', 'foo')->quoteIdentifier('foo.bodytext')

Remarks:

  • Similar to ->createNamedParameter() this method is crucial to prevent SQL injections. The same rules apply here.
  • Method ->set() for UPDATE statements expects their second argument to be a field value by default and quotes them using ->createNamedParameter() internally. In case a field should be set to the value of another field, this quoting can be turned off and an explicit call to ->quoteIdentifier() must be added.
  • Internally, ->quoteIdentifier() is automatically called on all method arguments that must be a field name. For instance, ->quoteIdentifier() is called on all arguments given to ->select().
  • ->quoteIdentifiers() (mind the plural) can be used to quote multiple field names at once. While that method is 'public` and thus exposed as API method, this is mostly useful internally only.
escapeLikeWildcards()

Helper method to quote % characters within a search string. This is helpful in ->like() and ->notLike() expressions:

// SELECT `uid` FROM `tt_content` WHERE (`bodytext` LIKE '%kl\\%aus%')
$searchWord = 'kl%aus';
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->like(
         'bodytext',
         $queryBuilder->createNamedParameter('%' . $queryBuilder->escapeLikeWildcards($searchWord) . '%')
      )
   );

Warning

Even with using ->escapeLikeWildcards(), the value must again be encapsulated in a ->createNamedParameter() call. Only calling ->escapeLikeWildcards() does not make the value SQL injection safe!

getRestrictions(), setRestrictions(), resetRestrictions()

API methods to deal with the RestrictionBuilder.

Connection

An instance of class TYPO3\CMS\Core\Database\Connection is retrieved from the ConnectionPool by calling ->getConnectionForTable() and handing over the table name a query should executed on.

The class extends the basic doctrine-dbal Doctrine\DBAL\Connection class and is mainly used internally within the TYPO3 CMS framework to establish, maintain and terminate connections to single database endpoints. Those internal methods are not scope of this documentation since an extension developer usually doesn't have to deal with that.

For an extension developer however, the class provides a list of "short-hand" methods that allow dealing with "simple" query cases, without the complexity of the QueryBuilder. Using those methods typically ends up in rather short and easily readable code. The methods have in common that they support only "equal" comparisons in WHERE conditions, that all fields and values are fully quoted automatically and the created queries are executed right away.

Note

The Connection object is designed to work on a single table only. If queries to multiple tables should be performed, the object must not be re-used. Instead, a single Connection instance should be retrieved via ConnectionPool per target table. However, it is allowed to use one Connection object for muliple queries to the same table.

insert()

Creates and executes an INSERT INTO statement. A (slightly simplified) example from the Registry API:

// INSERT INTO `sys_registry` (`entry_namespace`, `entry_key`, `entry_value`) VALUES ('aoeu', 'aoeu', 's:3:\"bar\";')
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_registry')
   ->insert(
      'sys_registry',
      [
         'entry_namespace' => $namespace,
         'entry_key' => $key,
         'entry_value' => serialize($value)
      ]
   );

Well, that should be rather obvious: First argument is the table name to insert a row into, second argument is an array of key/value pairs. All keys are quoted to field names and all values are quoted to string values.

It is possible to add another array as third argument to specify how single values are quoted. This is useful if date or numbers or similar should be inserted. The example below quotes the first value to an integer and the second one to a string:

// INSERT INTO `sys_log` (`userid`, `details`) VALUES (42, 'klaus')
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_log')
   ->insert(
      'sys_log',
      [
         'userid' => (int)$userId,
         'details' => (string)$details,
      ],
      [
         Connection::PARAM_INT,
         Connection::PARAM_STR,
      ]
   );

insert() returns the number of affected rows. Guess what? That's the number 1 ... In case something goes wrong a \Doctrine\DBAL\DBALException is raised.

Note

A list of allowed field types for proper quoting can be found in the TYPO3\CMS\Core\Database\Connection class and its base class \Doctrine\DBAL\Connection

bulkInsert()

INSERT multiple rows at once. An example from the test suite:

$connection->bulkInsert(
   'aTestTable',
   [
      ['aField' => 'aValue'],
      ['aField' => 'anotherValue']
   ],
   [
      'aField'
   ]
);

First argument is the table to insert table into, second argument is an array of rows, third argument is the list of field names. Similar to ->insert() it is optionally possible to add another argument to specify quoting details, if omitted, everything will be quoted to strings.

Note

mysql is rather forgiving when it comes to insufficient field quoting: Inserting a string to an int field will not raise an error and mysql will adapt internally. However, other dbms are not that relaxed and may raise errors. It is good practice to specify field types for each field, especially if they are not strings. Doing so right away will reduce the number of raised bugs if people run your extension an anything else than mysql.

update()

Create and execute an UPDATE statement. The example from FAL's ResourceStorage sets a storage to offline:

// UPDATE `sys_file_storage` SET `is_online` = 0 WHERE `uid` = '42'
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_file_storage')
   ->update(
      'sys_file_storage',
      ['is_online' => 0],
      ['uid' => (int)$this->getUid()],
      [Connection::PARAM_INT]
   );

First argument is the table an update should be executed on, the second argument is an array of key/value pairs to set, the third argument is an array of "equal" where statements that are combined with AND, the (optional) fourth argument specifies the type of values to be updated similar to ->insert() and bulkInsert().

Note the third argument WHERE `foo` = 'bar' only supports equal =. For more complex stuff the QueryBuilder has to be used.

The method returns the number of affected rows.

delete()

Execute a DELETE query using equal conditions in WHERE, example from BackendUtility to mark rows as no longer locked by a user:

// DELETE FROM `sys_lockedrecords` WHERE `userid` = 42
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_lockedrecords')
   ->delete(
      'sys_lockedrecords',
      ['userid' => (int)42],
      [Connection::PARAM_INT]
    );

First argument is the table name, second argument is a list of AND combined WHERE conditions as array, third argument specifies the quoting of WHERE values. There is a pattern ;)

Note

TYPO3 CMS uses a "soft delete" approach for many tables. Instead of directly deleting a rows in the database, a field - often called deleted - is set from 0 to 1. Executing a DELETE query circumvents this and really removes rows from a table. For most tables, it is better to use the DataHandler API to handle deletes instead of executing such low level queries directly.

truncate()

Empty a table, removing all rows. Usually much quicker than a ->delete() of all rows. This typically resets "auto increment primary keys" to zero. Use with care:

// TRUNCATE `cache_treelist`
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('cache_treelist')
   ->truncate('cache_treelist');
count()

A COUNT query. Again, this methods becomes handy if very simple COUNT statements are to be executed, the example returns tha number of active rows from table tt_content that have their bodytext field set to klaus:

// SELECT COUNT(*)
// FROM `tt_content`
// WHERE
//     (`bodytext` = 'klaus')
//     AND (
//         (`tt_content`.`deleted` = 0)
//         AND (`tt_content`.`hidden` = 0)
//         AND (`tt_content`.`starttime` <= 1475621940)
//         AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1475621940))
//     )
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content');
$rowCount = $connection->count(
   '*',
   'tt_content',
   ['bodytext' => 'klaus']
);

First argument is the field to count on, usually * or uid. Second argument is the table name, third argument is an array of WHERE equal conditions combined with AND.

Remarks:

  • ->count() of Connection returns the number directly as integer, in contrast to the method of the QueryBuilder, there is no need to call ->fetchColumns(0) or similar.
  • The third argument expects all WHERE values to be strings, each single expression is combined with AND.
  • The RestrictionBuilder kicks in and adds additional WHERE conditions based on TCA settings.
  • Field names and values are quoted automatically.
  • If anything more complex than a simple equal condition on WHERE is needed, the QueryBuilder methods are a better choice: Next to ->select(), the ->count() query is often the least useful method of the Connection object.
select()

Creates and executes a simple SELECT query based on equal conditions. Its usage is limited, the RestrictionBuilder kicks in and key/value pairs are automatically quoted:

// SELECT `entry_key`, `entry_value` FROM `sys_registry` WHERE `entry_namespace` = 'my_extension'
$resultRows = GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_registry')
   ->select(
      ['entry_key', 'entry_value'],
      'sys_registry',
      ['entry_namespace' => 'my_extension']
   );

Remarks:

  • In contrast to the other short-hand methods, ->select() returns a Statement object ready to ->fetch() single rows or to ->fetchAll()
  • The method accepts a series of further arguments to specify GROUP BY, ORDER BY, LIMIT and OFFSET query parts.
  • For non-trivial SELECT queries, it is often better to switch to the according method of the QueryBuilder object.
  • The RestrictionBuilder adds default WHERE restrictions. If those restrictions do not apply to the query needs, it is required to switch to the QueryBuilder->select() method for fine-grained WHERE manipulation.
createQueryBuilder()

The QueryBuilder should not be re-used for multiple different queries. However, it sometimes becomes handy to first fetch a Connection object for a specific table and to execute a simple query, and to create a QueryBuilder for a more complex query from this connection object later. The methods usefulness is limited however and no good example within the core can be found at the time of this writing.

The method can be helpful in loops to save some precious code characters, too:

$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable($myTable);
foreach ($someList as $aListValue) {
   $myResult = $connection->createQueryBuilder
      ->select('something')
      ->from('whatever')
      ->where(...)
      ->execute()
      ->fetchAll();
}

ExpressionBuilder

The ExpressionBuilder class is responsible to dynamically create SQL query parts for WHERE and JOIN ON conditions, functions like ->min() may also be used in SELECT parts.

It takes care of building query conditions while ensuring table and column names are quoted within the created expressions / SQL fragments. It is a facade to the actual doctrine-dbal ExpressionBuilder.

The ExpressionBuilder is used within the context of the QueryBuilder to ensure queries are being build based on the requirements of the database platform in use.

An instance of the ExpressionBuilder is retrieved from the QueryBuilder object:

$expressionBuilder = $queryBuilder->expr();

It is good practice to not assign an instance of the ExpressionBuilder to a variable but to use it within the code flow of the QueryBuilder context directly:

$rows = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content')
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      // `bodytext` = 'klaus' AND `header` = 'peter'
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
      $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('peter'))
   )
   ->execute()
   ->fetchAll();

Warning

It is crucially important to quote values correctly to not introduce SQL injection attack vectors to your application. See the section of the QueryBuilder for details.

Junctions
  • ->andX() conjunction
  • ->orX() disjunction

Combine multiple single expressions with AND or OR. Nesting is possible, both methods are variadic and take any number of argument which are all combined. It usually doesn't make much sense to hand over zero or only one argument, though.

A core example to find a sys_domain record:

// WHERE
//     (`sys_domain`.`pid` = `pages`.`uid`)
//     AND (
//        (`sys_domain`.`domainName` = 'example.com')
//        OR
//        (`sys_domain`.`domainName` = 'example.com/')
//     )
$queryBuilder->where(
   $queryBuilder->expr()->eq('sys_domain.pid', $queryBuilder->createNamedParameter('pages.uid', \PDO::PARAM_INT)),
   $queryBuilder->expr()->orX(
      $queryBuilder->expr()->eq('sys_domain.domainName', $queryBuilder->createNamedParameter($domain)),
      $queryBuilder->expr()->eq('sys_domain.domainName', $queryBuilder->createNamedParameter($domain . '/'))
   )
)
Comparisons

A set of methods to create various comparison expressions or SQL functions:

  • ->eq($fieldName, $value) "equal" comparison =
  • ->neq($fieldName, $value) "not equal" comparison !=
  • ->lt($fieldName, $value) "less than" comparison <
  • ->lte($fieldName, $value) "less than or equal" comparison <=
  • ->gt($fieldName, $value) "greater than" comparison >
  • ->gte($fieldName, $value) "greater than or equal" comparison >=
  • ->isNull($fieldName) "IS NULL" comparison
  • ->isNotNull($fieldName) "IS NOT NULL" comparison
  • ->like($fieldName) "LIKE" comparison
  • ->notLike($fieldName) "NOT LIKE" comparison
  • ->in($fieldName, $valueArray) "IN ()" comparison
  • ->notIn($fieldName, $valueArray) "NOT IN ()" comparison
  • ->inSet($fieldName, $value) "FIND_IN_SET('42', aField)" Find a value in a comma separated list of values
  • ->bitAnd($fieldName, $value) A bitwise AND operation &

Remarks and warnings:

Examples:

// `bodytext` = 'foo' - string comparison
->eq('bodytext', $queryBuilder->createNamedParameter('foo'))

// `tt_content`.`bodytext` = 'foo'
->eq('tt_content.bodytext', $queryBuilder->createNamedParameter('foo'))

// `aTableAlias`.`bodytext` = 'foo'
->eq('aTableAlias.bodytext', $queryBuilder->createNamedParameter('foo'))

// `uid` = 42 - integer comparison
->eq('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))

// `uid` >= 42
->gte('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))

// `bodytext` LIKE 'klaus'
->like(
   'bodytext',
   $queryBuilder->createNamedParameter($queryBuilder->escapeLikeWildcards('klaus'))
)

// `bodytext` LIKE '%klaus%'
->like(
   'bodytext',
   $queryBuilder->createNamedParameter('%' . $queryBuilder->escapeLikeWildcards('klaus') . '%')
)

// `uid` IN (42, 0, 44) - properly sanitized, mind the intExplode and PARAM_INT_ARRAY
->in(
   'uid',
   $queryBuilder->createNamedParameter(
      GeneralUtility::intExplode(',', '42, karl, 44', true),
      Connection::PARAM_INT_ARRAY
   )
)

// `CType` IN ('media', 'multimedia') - properly sanitized, mind the PARAM_STR_ARRAY
->in(
   'CType',
   $queryBuilder->createNamedParameter(
      ['media', 'multimedia'],
      Connection::PARAM_STR_ARRAY
   )
)
Aggregate functions

Aggregate functions used in SELECT parts, often combined with GROUP BY. First argument is the field name (or table name / alias with field name), second argument an optional alias.

  • ->min($fieldName, $alias = NULL) "MIN()" calculation
  • ->max($fieldName, $alias = NULL) "MAX()" calculation
  • ->avg($fieldName, $alias = NULL) "AVG()" calculation
  • ->sum($fieldName, $alias = NULL) "SUM()" calculation
  • ->count($fieldName, $alias = NULL) "COUNT()" calculation

Examples:

// Calculate the average creation timestamp of all rows from tt_content
// SELECT AVG(`crdate`) AS `averagecreation` FROM `tt_content`
$result = $queryBuilder
   ->addSelectLiteral(
      $queryBuilder->expr()->avg('crdate', 'averagecreation')
   )
   ->from('tt_content')
   ->execute()
   ->fetch();

// Distinct list of all existing endtime values from tt_content
// SELECT `uid`, MAX(`endtime`) AS `maxendtime` FROM `tt_content` GROUP BY `endtime`
$statement = $queryBuilder
   ->select('uid')
   ->addSelectLiteral(
      $queryBuilder->expr()->max('endtime', 'maxendtime')
   )
   ->from('tt_content')
   ->groupBy('endtime')
   ->execute();
Various Expressions
TRIM

Using the TRIM expression makes sure fields get trimmed on database level. See the examples below to get a better idea of what can be done.

$queryBuilder->expr()->comparison(
    $queryBuilder->expr()->trim($fieldName),
    ExpressionBuilder::EQ,
    $queryBuilder->createNamedParameter('', \PDO::PARAM_STR)
);

The call to $queryBuilder->expr()-trim() can be one of the following:

  • trim('fieldName') results in TRIM("tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_LEADING, 'x') results in TRIM(LEADING "x" FROM "tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_TRAILING, 'x') results in TRIM(TRAILING "x" FROM "tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_BOTH, 'x') results in TRIM(BOTH "x" FROM "tableName"."fieldName")
LENGTH

The LENGTH string function can be used to return the length of a string in bytes, method signature is fieldName with optional alias ->length(string $fieldName, string $alias = null)

$queryBuilder->expr()->comparison(
    $queryBuilder->expr()->length($fieldName),
    ExpressionBuilder::GT,
    $queryBuilder->createNamedParameter(0, \PDO::PARAM_INT)
);

RestrictionBuilder

Database tables in TYPO3 CMS that can be administrated in the backend come with TCA definitions that specify how single fields and rows of the table should be handled and displayed by the framework.

The ctrl section of a tables TCA array specifies optional framework internal handling of soft deletes and language overlays: For instance, if a row in the backend is deleted using the page or list module, many tables are configured to not entirely drop that row from the table, instead a field (often deleted) is set from zero to one for that row. Similar mechanics kick in for start- and endtime as well as language and workspace overlays. See the ['ctrl'] chapter in the TCA reference for details on this topic.

These mechanics however come with a price tag attached to it: Extension developers dealing with low-level query stuff must take care overlayed or deleted rows are not in the result set of a casual query.

This is where this "automatic restriction" stuff kicks in: The construct is created on top of native doctrine-dbal as TYPO3 CMS specific extension. It automatically adds WHERE expressions that suppress rows which are marked as deleted or exceeded their "active" life cycle. All that is based on the TCA configuration of the affected table.

Rationale

A developer may ask why she has to go through all this and why this additional stuff is added on a low-level query layer, when "just a simple query" should be fired. The construct implements some important design goals:

  • Simple: Query creation should be easy to use without forcing a developer thinking too much about those nasty TCA details.
  • Cope with developer laziness: If the framework would force a developer to always add casual restrictions for each and every query, this is easy to forget. We're all lazy, are we?
  • Security: If in doubt, it is better to show a little too less than too much. It is much better to deal with a customer who complains some records are not shown than to show too many records. The former is "just a bug" while the latter can easily escalate to a serious privilege escalation security issue.
  • Automatic query upgrades: If a table was designed without soft-delete in the first place and later a deleted flag is added and registered in TCA, queries executed on that table will automatically upgrade and add the according deleted = 0 restriction.
  • Handing over restriction details to the framework: Having the restriction expressions done by the framework gives it the opportunity to change details without breaking extension code. This may very well happen in the future and having a happy little upgrade path for such cases in place may become very handy later.
  • Flexibility: The class construct is created in a way that allows developers to extend or substitute it with own restrictions if that is useful to model the domain in question.
Main construct

The restriction builder is called whenever a SELECT or COUNT query is executed through either the QueryBuilder or Connection. The QueryBuilder allows manipulation of those restrictions while the simplified Connection class does not. If a query deals with multiple tables in a join, restrictions for all affected tables are added.

Each single restriction like a DeletedRestriction or a StartTimeRestriction is modeled as a single class implementing the QueryRestrictionInterface. Each restriction looks up in TCA if it should kick in. If so, it adds according expressions to the WHERE clause when the final statement is compiled.

Multiple restrictions can be grouped in containers which implement the QueryRestrictionContainerInterface.

The DefaultRestrictionContainer is always added by ... uuhm ... default: It adds the DeletedRestriction, the HiddenRestriction, the StartTimeRestriction and the EndTimeRestriction. Note this is true for all contexts a query is executed in: It does not matter whether a query is created from within a frontend, a backend or a cli call, they all add the DefaultRestrictionContainer if not explicitly told otherwise by an extension developer.

Note

Having this DefaultRestrictionContainer used everywhere is the second iteration of that code construct:

The first variant automatically added restrictions based on context. For instance, a query fired by a call that is executed in the backend did not add the hidden flag, while a query fired from within a frontend call did so. We quickly figured this ends up in a huge mess: The distinction between frontend, backend and cli is not that sharp in TYPO3, as example the frontend behaves much more like a backend call if the admin panel is used.

The currently active variant is much easier: It always adds sane defaults everywhere, a developer only has to deal with details if they don't fit. The core team hopes this approach is a good balance between hidden magic, security, transparency and convenience.

Restrictions
  • DeletedRestriction: (default) Evaluates ['ctrl']['delete'], adds for instance AND deleted = 0 if TCA['aTable']['ctrl']['delete'] = 'deleted' is specified.
  • HiddenRestriction: (default) Evaluates ['ctrl']['enablecolumns']['disabled'], adds AND hidden = 0 if hidden is specified as field name.
  • StartTimeRestriction: (default) Evaluates ['ctrl']['enablecolumns']['starttime'], typically adds something like AND (`tt_content`.`starttime` <= 1475580240).
  • EndTimeRestriction: (default) Evaluates ['ctrl']['enablecolumns']['endtime'].
  • FrontendGroupRestriction: Evaluates ['enablecolumns']['fe_group'].
  • RootlevelRestriction: Match records on root level, adds AND (`pid` = 0)
  • BackendWorkspaceRestriction: Determines the current workspace a backend user is working in and adds a couple of restrictions to select only records of that workspace if the table supports workspaced records.
  • FrontendWorkspaceRestriction: Restriction to filter records for fronted workspaces preview.
QueryRestrictionContainer
  • DefaultRestrictionContainer: Add DeletedRestriction, HiddenRestriction, StartTimeRestriction and EndTimeRestriction. This container is always added if not told otherwise.
  • FrontendRestrictionContainer: Adds DeletedRestriction, HiddenRestriction, StartTimeRestriction, EndTimeRestriction, FrontendWorkspaceRestriction and FrontendGroupRestriction. This container should be be added by a developer to a query if creating query statements in frontend context or if handling frontend stuff from within cli calls.
Examples

Often the default restrictions are sufficient. Nothing needs to be done in those cases.

However, many backend modules still want to show disabled records and remove the starttime and endtime restrictions to allow administration of those records for an editor. A typical setup from within a backend module:

// SELECT `uid`, `bodytext` FROM `tt_content` WHERE (`pid` = 42) AND (`tt_content`.`deleted` = 0)
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
// Remove all restrictions but add DeletedRestriction again
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
$result = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter($pid, \PDO::PARAM_INT)))
   ->execute()
   ->fetchAll();

The DeletedRestriction should be kept in almost all cases. Usually, the only extension that dismiss that flag is the recycler module to list and resurrect deleted records. Any object implementing the QueryRestrictionInterface can be given to ->add(). This allows extensions to deliver own restrictions.

An alternative to the recommended way of first removing all restrictions and then adding needed ones again (using ->removeAll(), then ->add()) is to kick specific restrictions with a call to ->removeByType():

// Remove starttime and endtime, but keep hidden and deleted
$queryBuilder
   ->getRestrictions()
   ->removeByType(StartTimeRestriction::class)
   ->removeByType(EndTimeRestriction::class);

In the frontend it is often needed to swap the DefaultRestrictionContainer with the FrontendRestrictionContainer:

// Kick default restrictions and add list of default frontend restrictions
$queryBuilder->setRestrictions(GeneralUtility::makeInstance(FrontendRestrictionContainer::class));

Note that ->setRestrictions() resets any previously specified restrictions. Any class instance implementing QueryRestrictionContainerInterface can be given to ->setRestrictions(). This allows extensions to deliver and use an own set of restrictions for own query statements if needed.

Tip

It can be very helpful to debug the final statements created by the RestrictionBuilder using debug($queryBuilder->getSQL()) right before the final call to $queryBuilder->execute(). Just take care these calls do not end up in production :ref:` code.

Statement

A Statement object is returned by QueryBuilder->execute() for ->select() and ->count() query types and by Connection->select() and Connection->count() calls.

The object represents a query result set and comes with methods to ->fetch() single rows or to ->fetchAll() of them. Additionally, it can also be used to execute a single prepared statement with different values multiple times. This part is however not widely used within the TYPO3 CMS core yet, and thus not fully documented here.

Note

The name "Statement" instead of "Result" can be puzzling at first glance: The class represents a prepared statement that can be executed multiple times with different values and then returns multiple different result sets. From this point of view "Statement" fits much better than "Result".

Warning

The return type of single field values is NOT type safe! If selecting a value from a field that is defined as int, the Statement result may very well return that as PHP string. This is true for other database column types like FLOAT, DOUBLE and others. This is an issue with the database drivers used below, it may happen that MySQL returns an integer value for an int field, while MSSQL returns a string. In general, the application must take care of an according type cast on their own to reach maximum DBMS compatibility.

fetch()

Fetch next row from a result statement. Usually used in while() loops. Typical example:

// Fetch all records from tt_content on page 42
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute();
while ($row = $statement->fetch()) {
   // Do something useful with that single $row
}

->fetch() returns arrays with single field / values pairs until the end of the result set is reached which then returns false and thus breaks the while loop.

fetchAll()

Returns an array containing all of the result set rows by implementing the same while loop as above internally. Using that method saves some precious code characters but is more memory intensive if the result set is large with lots of rows and lot of data since big arrays are carried around in PHP:

// Fetch all records from tt_content on page 42
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$rows = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute()
   ->fetchAll();
fetchColumn()

Returns a single column from the next row of a result set, other columns from that result row are discarded. This method is especially handy for QueryBuilder->count() queries. The Connection->count() implementation does exactly that to return the number of rows directly:

// Get the number of tt_content records on pid 42 into variable $numberOfRecords
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$numberOfRecords = $queryBuilder
   ->count('uid')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute()
   ->fetchColumn(0);
rowCount()

Returns the number of rows affected by the last execution of this statement. Use that method instead of counting the number of records in a ->fetch() loop manually.

Warning

->rowCount() works well with DELETE, UPDATE and INSERT queries. However, it does NOT return a valid number for SELECT queries on some DBMS. Never use ->rowCount() on SELECT queries. This may work with MySOL, but fails with other databases like SQLite.

Re-use prepared Statement()

Doctrine usually prepares a statement first, and then executes it with given parameters. Implementing prepared statements depends on the given driver. For instance, the native mysql driver mysqli does implement prepared statements, while the pdo driver of mysql pdo_mysql does not, at least in some scenarios. A driver not properly implementing prepared statements fall back to a direct execution of given query.

There is an API to make real use of prepared statements that becomes handy if the same query is executed with different arguments over and over again. The example below prepares a statement to the pages table and executes it twice with different arguments:

$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('pages');
$queryBuilder = $connection->createQueryBuilder();
$queryBuilder->getRestrictions()->removeAll();
$sqlStatement = $queryBuilder->select('uid')
    ->from('pages')
    ->where($queryBuilder->expr()->eq('uid', $queryBuilder->createPositionalParameter(0, \PDO::PARAM_INT)))
    ->getSQL();
$statement = $connection->executeQuery($sqlStatement, [ 24 ]);
$result1 = $statement->fetch();
$statement->bindValue(1, 25);
$statement->execute();
$result2 = $statement->fetch();

Looking at a mysql debug log:

Prepare SELECT `uid` FROM `pages` WHERE `uid` = ?
Execute SELECT `uid` FROM `pages` WHERE `uid` = '24'
Execute SELECT `uid` FROM `pages` WHERE `uid` = '25'

The log shows one statement preparation with two executions.

QueryHelper

The class contains miscellaneous helper methods to build syntactically valid SQL queries.

Most helper methods are required to deal with legacy data where the format of the input is not strict enough to reliably use the SQL parts in queries directly.

The whole class is marked as @internal, should not be used by extension authors and may - if things go wrong - change at will. The class will hopefully vanish mid-term. However, there may be situations when the class methods can become handy if extension authors migrate their own extensions away from TYPO3_DB to doctrine-dbal. In practice, the core will most likely add proper deprecations to single methods if they are target of removal later.

Extension developers may keep this class in mind for migration, but must not use methods for new code created from scratch. Apart from that, as can be seen below, using those methods often ends up in rather ugly code.

The migration benefits are the only reason the methods are documented here.

Warning

Using those methods raise the risk of SQL injections, especially for methods like ->stripLogicalOperatorPrefix() since its input string tends to come from user supplied input and is sometimes added as WHERE expression without further quoting. Keep a special eye on those scenarios!

parseOrderBy()

Some parts of the core framework allow string definitions like ORDER BY sorting for instance in TCA and TypoScript. The method rips those strings apart and prepares them to be fed to QueryBuilder->orderBy():

// 'ORDER BY aField ASC,anotherField, aThirdField DESC'
// ->
// [ ['aField', 'ASC'], ['anotherField', null], ['aThirdField', 'DESC'] ]
$uglyOrderBy = 'ORDER BY aField ASC,anotherField, aThirdField DESC'
foreach (QueryHelper::parseOrderBy((string)$uglyOrderBy) as $orderPair) {
   list($fieldName, $order) = $orderPair;
   $queryBuilder->addOrderBy($fieldName, $order);
}
parseGroupBy()

Parses GROUP BY strings ready to be added via QueryBuilder->groupBy(), similar to ->parseOrderBy():

// 'GROUP BY be_groups.title, anotherField'
// ->
// ['be_groups.title', 'anotherField']
$uglyGroupBy = 'GROUP BY be_groups.title, anotherField';
$queryBuilder->groupBy(QueryHelper::parseGroupBy($uglyGroupBy));
parseTableList()

Parse a table list, possibly prefixed with FROM, and explode it into and array of arrays where each item consists of a tableName and an optional alias name, ready to be put into QueryBuilder->from():

// 'FROM aTable a,anotherTable, aThirdTable AS c',
// ->
// [ ['aTable', 'a'], ['anotherTable', null], ['aThirdTable', 'c'] ]
$uglyTableString = 'FROM aTable a,anotherTable, aThirdTable AS c;
foreach (QueryHelper::parseTableList($uglyTableString) as $tableNameAndAlias) {
   list($tableName, $tableAlias) = $tableNameAndAlias;
   $queryBuilder->from($tableName, $tableAlias);
}
parseJoin()

Split a JOIN SQL fragment into table name, alias and join conditions:

// 'aTable AS `anAlias` ON anAlias.uid = anotherTable.uid_foreign'
// ->
// [
//     'tableName' => 'aTable',
//     'tableAlias' => 'anAlias',
//     'joinCondition' => 'anAlias.uid = anotherTable.uid_foreign'
// ],
$uglyJoinString = 'aTable AS `anAlias` ON anAlias.uid = anotherTable.uid_foreign';
$joinParts = QueryHelper::parseJoin($uglyJoinString);
$queryBuilder->join(
   $leftTable,
   $joinParts['tableName'],
   $joinParts['tableAlias'],
   $joinParts['joinCondition']
);
stripLogicalOperatorPrefix()

Removes the prefixes AND / OR from an input string.

Those prefixes are added in doctrine-dbal via QueryBuilder->where(), QueryBuilder->orWhere(), ExpressionBuilder->andX() and friends. Some parts of the TYPO3 framework however carry SQL fragments prefixed with AND or OR around and it's not always possible to easily get rid of those. The method helps by killing those prefixes before they are handed over to the doctrine API:

// 'AND 1=1'
// ->
// '1=1'
$uglyWherePart = 'AND 1=1'
$queryBuilder->where(
   // WARNING: High risk of possible SQL injection here, take additional actions!
   QueryHelper::stripLogicalOperatorPrefix($uglyWherePart)
);
getDateTimeFormats()

Just a left over method from the old TYPO3_DB DatabaseConnection class. Of little to no use for extension authors. This one is hopefully one of the first methods to vanish from the class.

Migrating from TYPO3_DB

This chapter is for those poor souls who want to migrate old and busted $GLOBALS['TYPO3_DB'] calls to new hotness doctrine-dbal based API.

It tries to give some hints on typical pitfalls and areas a special eye should be kept on.

Migration of a single extension is finished if a search for $GLOBALS['TYPO3_DB'] does not return hits anymore. This search is the most simple entry point to see which areas need work.

Compare raw queries

The main goal during migration is usually to fire a logically identical query. One recommended and simple approach to verify this is to note down and compare the queries at the lowest possible layer. In $GLOBALS['TYPO3_DB'], the final query statement is usually retrieved by removing the exec_ part from the method name, in doctrine method QueryBuilder->getSQL() can be used:

// Inital code:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery('*', 'index_fulltext', 'phash=' . (int)$phash);

// Remove 'exec_' and debug SQL:
debug($GLOBALS['TYPO3_DB']->SELECTquery('*', 'index_fulltext', 'phash=' . (int)$phash));
// Returns:
'SELECT * FROM index_fulltext WHERE phash=42'

// Migrate to doctrine and debug SQL:
// 'SELECT * FROM index_fulltext WHERE phash=42'
$queryBuilder->select('*')
->from('index_fulltext')
->where(
   $queryBuilder->expr()->eq('phash', $queryBuilder->createNamedParameter($pash, \PDO::PARAM_INT))
);
debug($queryBuilder->getSQL());

The above example returns the exact same query as before. This is not always as trivial to see since WHERE clauses are often in a different order. This especially happens if the RestrictionBuilder is involved. Since the restrictions are crucial and can easily go wrong it is advised to keep an eye on those where parts during transition.

enableFields() and deleteClause()

BackendUtility::deleteClause() adds deleted=0 if ['ctrl']['deleted'] is specified in the table's TCA. The method call should be removed during migration. If there is no other restriction method involved in the old call like enableFields(), the migrated code typically removes all doctrine default restrictions and just adds the DeletedRestriction again:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, TSconfig',
   'pages',
   'TSconfig != \'\''
      . BackendUtility::deleteClause('pages'),
   'pages.uid'
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('pages');
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
$res = $queryBuilder->select('uid', 'TSconfig')
   ->from('pages')
   ->where($queryBuilder->expr()->neq('TSconfig', $queryBuilder->createNamedParameter('')))
   ->groupBy('uid')
   ->execute();

BackendUtility::versioningPlaceholderClause('pages') is typically substituted with the BackendWorkspaceRestriction. Example very similar to the above one:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, TSconfig',
   'pages',
   'TSconfig != \'\''
      . BackendUtility::deleteClause('pages')
      . BackendUtility::versioningPlaceholderClause('pages'),
   'pages.uid'
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('pages');
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class))
   ->add(GeneralUtility::makeInstance(BackendWorkspaceRestriction::class));
$res = $queryBuilder->select('uid', 'TSconfig')
   ->from('pages')
   ->where($queryBuilder->expr()->neq('TSconfig', $queryBuilder->createNamedParameter('')))
   ->groupBy('uid')
   ->execute();

BackendUtility::BEenableFields() in combination with BackendUtility::deleteClause() adds the same calls as the DefaultRestrictionContainer. No further configuration needed:

// Before:
$GLOBALS['TYPO3_DB']->exec_SELECTgetRows(
   'title, content, crdate',
   'sys_news',
   '1=1'
      . BackendUtility::BEenableFields($systemNewsTable)
      . BackendUtility::deleteClause($systemNewsTable)
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)
   ->getQueryBuilderForTable('sys_news');
   ->select('title', 'content', 'crdate')
   ->from('sys_news)
   ->execute();

cObj->enableFields() in frontend context is typically directly substituted with FrontendRestrictionContainer:

// Before:
$GLOBALS['TYPO3_DB']->exec_SELECTquery(
   '*', $table,
   'pid=' . (int)$pid
      . $this->cObj->enableFields($table)
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable($table);
$queryBuilder->setRestrictions(GeneralUtility::makeInstance(FrontendRestrictionContainer::class));
$queryBuilder->select('*')
   ->from($table)
   ->where(
      $queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter($pid, \PDO::PARAM_INT))
   )
);
From ->exec_UDATEquery() to ->update()

Most often, the easiest way to migrate a $GLOBALS['TYPO3_DB']->exec_UDATEquery() is to use $connection->update():

// Before:
$database->exec_UPDATEquery(
    'aTable', // table
    'uid = 42', // where
    [ 'aField' => 'newValue' ] // value array
);

// After:
$connection->update(
    'aTable', // table
    [ 'aField' => 'newValue' ], // value array
    [ 'uid' => 42 ] // where
);

Warning

If switching from exec_UPDATEquery() to update, the order of arguments change, where and values are swapped!

Result set iteration

The exec_* calls return a resource object that is typically iterated over using sql_fetch_assoc(). This is typically changed to ->fetch() on the Statement object:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(...);
while ($row = $GLOBALS['TYPO3_DB']->sql_fetch_assoc($res)) {
   // Do something
}

// After:
$statement = $queryBuilder->execute();
while ($row = $statement->fetch()) {
   // Do something
}
sql_insert_id()

It is sometimes needed to fetch the new uid of a just added record to further work with that row. In TYPO3_DB this was done with a call to ->sql_insert_id() after a ->exec_INSERTquery() call on the same resource. ->lastInsertId() can be used instead:

// Before:
$GLOBALS['TYPO3_DB']->exec_INSERTquery(
   'pages',
   [
      'pid' => 0,
      'title' => 'Home',
   ]
);
$pageUid = $GLOBALS['TYPO3_DB']->sql_insert_id();

// After:
$databaseConnectionForPages = $connectionPool->getConnectionForTable('pages');
$databaseConnectionForPages->insert(
   'pages',
   [
      'pid' => 0,
      'title' => 'Home',
    ]
);
$pageUid = $databaseConnectionForPages->lastInsertId('pages');
fullQuoteStr()

->fullQuoteStr() is rather straight changed to a ->createNamedParameter(), typical case:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, title',
   'tt_content',
   'bodytext = ' . $GLOBALS['TYPO3_DB']->fullQuoteStr('horst')
);

// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'title')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('horst'))
   )
   ->execute();
ext_tables.sql

The schema migrator that compiles ext_tables.sql files from all loaded extensions and compares them with current schema definitions in the database has been fully rewritten. It mostly should work as before, some specific fields however tend to grow a little larger on mysql platforms than before. This usually shouldn't have negative side effects, typically no ext_tables.sql changes are needed when migrating an extension to the new query API.

extbase QueryBuilder

The extbase internal QueryBuilder used in Repositories still exists and works a before. There is usually no manual migration needed. It is theoretically possible to use the doctrine based query builder object in extbase which can become handy since the new one is much more feature rich, but that topic didn't yet fully settle in the core and no general recommendation can be given yet.

Various tips and tricks

  • Use Find usages of PhpStorm for examples! The source code of the core is a great way to learn how specific methods of the API are used. In PhpStorm it is extremely helpful to right click on a single method and list all method usages with Find usages. This is especially handy to quickly see usage examples of complex methods like join() from the QueryBuilder.

  • INSERT, UPDATE and DELETE statements are often easier to read and write using the Connection object instead of the QueryBuilder.

  • SELECT DISTINCT aField is not supported but can be substituted with a ->groupBy('aField').

  • getSQL() and execute() can be used after each other during development to simplify debugging:

    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
          $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
       );
    debug($queryBuilder->getSql());
    $statement = $queryBuilder->execute();
    
  • In contrast to the old API based on $GLOBALS['TYPO3_DB'], doctrine-dbal will throw exceptions if something goes wrong when calling execute(). The exception type is a \Doctrine\DBAL\DBALException which can be caught and transferred to a better error message if the application has to expect query errors. Note this is not good habit and often indicates an architectural flaw of the application at a different layer.

  • count() query types using the QueryBuilder typically call ->fetchColumn(0) to receive the count value. The count() method of Connection object does that automatically and returns the count value result directly.

Programming with workspaces in mind

The concept of workspaces needs attention from extension programmers. The implementation of workspaces is however made so that no critical problems can appear with old extensions;

  • First of all the "Live workspace" is no different from how TYPO3 has been working for years so that will be supported out of the box (except placeholder records must be filtered out in the frontend with t3ver_state != , see below).
  • Secondly, all permission related issues are implemented in DataHandler so the worst your users can experience is an error message.

However, you probably want to update your extension so that in the backend the current workspace is reflected in the records shown and the preview of content in the frontend works as well. Therefore this chapter has been written with instructions and insight into the issues you are facing.

Frontend challenges in general

For the frontend the challenges are mostly related to creating correct previews of content in workspaces. For most extensions this will work transparently as long as they use the API functions in TYPO3 to request records from the system.

The most basic form of a preview is when a live record is selected and you lookup a future version of that record belonging to the current workspace of the logged in backend user. This is very easy as long as a record is selected based on its "uid" or "pid" fields which are not subject to versioning; You simply call sys_page->versionOL() after record selection.

However, when other fields are involved in the where clause it gets dirty. This happens all the time! For instance, all records displayed in the frontend must be selected with respect to "enableFields" configuration! What if the future version is hidden and the live version is not? Since the live version is selected first (not hidden) and then overlaid with the content of the future version (hidden) the effect of the hidden field we wanted to preview is lost unless we also check the overlaid record for its hidden field (->versionOL() actually does this). But what about the opposite; if the live record was hidden and the future version not? Since the live version is never selected the future version will never have a chance to display itself! So we must first select the live records with no regard to the hidden state, then overlay the future version and eventually check if it is hidden and if so exclude it. The same problem applies to all other "enableFields", future versions with "delete" flags and current versions which are invisible placeholders for future records. Anyway, all that is handled by the \TYPO3\CMS\Frontend\Page\PageRepository class which includes functions for "enableFields" and "deleted" so it will work out of the box for you. But as soon as you do selection based on other fields like email, username, alias etc. it will fail.

Summary

Challenge: How to preview elements which are disabled by "enableFields" in the live version but not necessarily in the offline version. Also, how to filter out new live records with t3ver_state set to 1 (placeholder for new elements) but only when not previewed.

Solution: Disable check for enableFields/where_del_hidden on live records and check for them in versionOL on input record.

Frontend implementation guidelines

  • Any place where enableFields() are not used for selecting in the frontend you must at least check that t3ver_state != 1 so placeholders for new records are not displayed.
  • Make sure never to select any record with pid = -1! (offline records - related to versioning).
  • If you need to detect preview mode for versioning and workspaces you can read these variables:
    • $GLOBALS['TSFE']->sys_page->versioningPreview: If true, you are allowed to display previews of other record versions.
    • $GLOBALS['TSFE']->sys_page->versioningWorkspaceId: Will tell you the id of the workspace of the current backend user. Used for preview of workspaces.
  • Use these API functions for support of version previews in the frontend:
Function Description
$GLOBALS['TSFE']->sys_page->versionOL($table, &$row, $unsetMovePointers=FALSE)

Versioning Preview Overlay.

Generally ALWAYS used when records are selected based on uid or pid. If records are selected on other fields than uid or pid (e.g. "email = ....") then usage might produce undesired results and that should be evaluated on individual basis.

Principle: Record online! => Find offline?

Example:

This is how simple it is to use this record in your frontend plugins when you do queries directly (not using API functions already using them):

$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(...);
while (($row = $GLOBALS['TYPO3_DB']->sql_fetch_assoc($res))) {
   $GLOBALS['TSFE']->sys_page->versionOL($table,$row);

   if (is_array($row)) {
      // ...
   }
   // ...
}

When the live record is selected, call ->versionOL() and make sure to check if the input row (passed by reference) is still an array.

The third argument, $unsetMovePointers = FALSE, can be set to TRUE when selecting records for display ordered by their position in the page tree. Difficult to explain easily, so only use this option if you don't get a correct preview of records that has been moved in a workspace (only for "element" type versioning)

$GLOBALS['TSFE']->sys_page->fixVersioningPid()

Finding online PID for offline version record.

Will look if the "pid" value of the input record is -1 (it is an offline version) and if the table supports versioning; if so, it will translate the -1 PID into the PID of the original record

Used whenever you are tracking something back, like making the root line. In fact, it is currently only used by the root line function and chances are that you will not need this function often.

Principle: Record offline! => Find online?

Frontend scenarios impossible to preview

These issues are not planned to be supported for preview:

  • Lookups and searching for records based on other fields than uid, pid or "enableFields" will never reflect workspace content since overlays happen to online records after they are selected.

    • This problem can largely be avoided for versions of new records because versions of a "New"-placeholder can mirror certain fields down onto the placeholder record. For the tt\_content table this is configured as

      shadowColumnsForNewPlaceholders'=> 'sys\_language\_uid,l18n\_parent,colPos,header'
      

      so that these fields used for column position, language and header title are also updated in the placeholder thus creating a correct preview in the frontend.

    • For versions of existing records the problem is in reality reduced a lot because normally you don't change the column or language fields after the record is first created anyway! But in theory the preview can fail.

    • When changing the type of a page (e.g. from "Standard" to "External URL") the preview might fail in cases where a look up is done on the :code`doktype` field of the live record.

      • Page shortcuts might not work properly in preview.
      • Mount Points might not work properly in preview.
  • It is impossible to preview the value of count(*) selections since we would have to traverse all records and pass them through ->versionOL() before we would have a reliable result!

  • In \TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController::getPageShortcut(), sys_page->getMenu() is called with an additional WHERE clause which will not respect if those fields are changed for a future version. This could be the case other places where getmenu() is used (but a search shows it is not a big problem). In this case we will for now accept that a wrong shortcut destination can be experienced during previews.

Backend challenges

The main challenge in the backend is to reflect how the system will look when the workspace gets published. To create a transparent experience for backend users we have to overlay almost every selected record with any possible new version it might have. Also when we are tracking records back to the page tree root point we will have to correct pid-values. All issues related to selecting on fields other than pid and uid also relates to the backend as they did for the frontend.

Backend module access

You can restrict access to backend modules by using $MCONF['workspaces'] in the conf.php files. The variable is a list of keywords defining where the module is available:

$MCONF['workspaces'] = online,offline,custom

You can also restrict function menu items to certain workspaces if you like. This is done by an argument sent to the function \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::insertModuleFunction(). See that file for more details.

Detecting current workspace

You can always check what the current workspace of the backend user is by reading $GLOBALS['BE_USER']->workspace. If the workspace is a custom workspace you will find its record loaded in $GLOBALS['BE_USER']->workspaceRec.

The values for workspaces is either 0 (online/live) or the uid of the corresponding entry in the sys_workspace table.

Using DataHandler with workspaces

Since admin users are also restricted by the workspace it is not possible to save any live records when in a workspace. However for very special occasions you might need to bypass this and to do so, you can set the instance variable \TYPO3\CMS\Core\DataHandling\DataHandler::bypassWorkspaceRestrictions to TRUE. An example of this is when users are updating their user profile using the "User Tool > User Settings" module; that actually allows them to save to a live record (their user record) while in a draft workspace.

Moving in workspaces

TYPO3 4.2 and beyond supports moving for "Element" type versions in workspaces. Technically this works by creating a new online placeholder record (like for new elements in a workspace) in the target location with t3ver_state = 3 (move-to placeholder) and a field, t3ver_move_id, holding the uid of the record to move (source record) upon publishing. In addition, a new version of the source record is made and has t3ver_state = 4 (move-to pointer). This version is simply necessary in order for the versioning system to have something to publish for the move operation.

So in summary, two records are created for a move operation in a workspace: The placeholder (online, with t3ver_state = 3 and t3ver_move_id set) and a new version (t3ver_state = 4) of the online source record (the one being moved).

When the version of the source is published a look up will be made to see if a placeholder exists for a move operation and if so the record will take over the pid / "sortby" value upon publishing.

Preview of move operations is almost fully functional through the \TYPO3\CMS\Frontend\Page\PageRepository::versionOL() and \TYPO3\CMS\Backend\Utility\BackendUtility::workspaceOL() functions. When the online placeholder is selected it simply looks up the source record, overlays any version on top and displays it. When the source record is selected it should simply be discarded in case shown in context where ordering or position matters (like in menus or column based page content). This is done in the appropriate places.

Caching framework

Since TYPO3 CMS 4.3, the core contains a data caching framework which supports a wide variety of storage solutions and options for different caching needs. Each cache can be configured individually and can implement its own specific storage strategy. Major parts of the system are backported from TYPO3 Flow and are kept in sync between the two systems.

The caching framework exists to help speeding up TYPO3 sites, especially heavily loaded ones. It is possible to move all caches to a dedicated cache server with specialized cache systems like the Redis key-value store (a so called NoSQL database).

Since TYPO3 CMS 4.6, the caching framework is always enabled, the old and unflexible approach to cache content is gone. This document covers settings for TYPO3 CMS 6.0 and beyond.

Quick start for Integrators

This section gives come simple instructions for getting started with using the caching framework without giving the whole details under the hood.

Change specific cache options

By default, most core caches use the database backend. Default cache configuration is defined in typo3/sysext/core/Configuration/DefaultConfiguration.php and can be overridden in LocalConfiguration.php.

If specific settings should be applied to the configuration, they should be added to LocalConfiguration.php. All settings in LocalConfiguration.php will be merged with DefaultConfiguration.php. The easiest way to see the final cache configuration is to use the TYPO3 Backend module Admin Tools > Configuration > $TYPO3_CONF_VARS.

Example for a configuration of redis cache backend on redis database number 42 instead of the default database backend with compression for the pages cache:

return array(
// ...
   'SYS' => array(
   // ...
      'caching' => array(
         // ...
         'cache_pages' => array(
            'backend' => 'TYPO3\CMS\Core\Cache\Backend\RedisBackend',
            'options' => array(
               'database' => 42,
            ),
         ),
      ),
   ),
);
Garbage collection task

Most cache backends do not have an internal system to remove old cache entries that exceeded their lifetime. A cleanup must be triggered externally to find and remove those entries, otherwise caches could grow to arbitrary size. This could lead to a slow website performance, might sum up to significant hard disk or memory usage and could render the server system unusable.

It is advised to always enable the scheduler and run the "Caching framework garbage collection" task to retain clean and small caches. This housekeeping could be done once a day when the system is otherwise mostly idle.

Configuration

Caches are configured in the array $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']. The basic structure is predefined in typo3/sysext/core/Configuration/DefaultConfiguration.php, and consists of the single section:

  • cacheConfigurations: Registry of all configured caches. Each cache is identified by its array key. Each cache can have the sub keys frontend, backend and options to configure the used frontend, backend and possible backend options.
Cache configurations

Unfortunately in TYPO3 CMS, all ext_localconf.php files are loaded after the instance specific configuration from LocalConfiguration.php and AdditionalConfiguration.php. This enables extensions to overwrite cache configuration already done for the instance. Any extension should avoid this situation and should just define the very required minimum of cache configuration. This boils down to define just the array key to populate a new cache to the system. Without further configuration, the cache system fall back to default backend and default frontend settings:

if (!is_array($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'] = array();
}

Extensions like extbase define default caches this way, giving administrators full freedom for specific and possibly quicker setups (eg. a memory driven cache for the extbase reflection cache).

Administrators can overwrite specific settings of the cache configuration in LocalConfiguration.php, example configuration to switch cache_pages to the redis backend using database 3:

return array(
    'SYS' => array(
        'caching' => array(
            'cacheConfigurations' => array(
                'cache_pages' => array(
                    'backend' => 'TYPO3\CMS\Core\Cache\Backend\RedisBackend',
                    'options' => array(
                        'database' => 3,
                    ),
                ),
            ),
        ),
    ),
);

Some backends have mandatory as well as optional parameters (which are documented below). If not all mandatory options are defined, the specific backend will throw an exception if accessed.

How to disable specific caches

During development, it can be convenient to disable certain caches. This is especially helpful since TYPO3 CMS 4.6 for central caches like the language or autoloader cache. This can be achieved by using the null backend (see below) as storage backend.

Warning

Do not use this in production, it will strongly slow down the system!

Example entry to switch the phpcode cache (used for the autoloader cache) to use the null backend:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_phpcode']['backend'] = 'TYPO3\CMS\Core\Cache\Backend\NullBackend';

Caching framework architecture

Basic knowhow

The caching framework can handle multiple caches with different configurations. A single cache consists of any number of cache entries.

A single cache entry is defined by these fields:

  • identifier: A string as unique identifier within this cache. Used to store and retrieve entries.
  • data: The data to be cached.
  • lifetime: A lifetime in seconds of this cache entry. An entry can not be retrieved from cache if lifetime expired.
  • tags: Additional tags (an array of strings) assigned to the entry. Used to remove specific cache entries.

Tip

The difference between identifier and tags is quite simple: an identifier uniquely identifies a cache entry, and a tag is additional data applied to an entry (used for cache eviction). Thus, an identifier refers to a single cache entry to store and retrieve an entry, and a tag can refer to multiple cache entries.

About the identifier

The identifier is used to store ("set") and retrieve ("get") entries from the cache and holds all information to differentiate entries from each other. For performance reasons, it should be quick to calculate.

Suppose there is an resource-intensive extension added as a plugin on two different pages. The calculated content depends on the page on which it is inserted and if a user is logged in or not. So, the plugin creates at maximum four different content outputs, which can be cached in four different cache entries:

  • page 1, no user logged in
  • page 1, a user is logged in
  • page 2, no user logged in
  • page 2, a user is logged in

To differentiate all entries from each other, the identifier is built from the page ID where the plugin is located, combined with the information whether a user is logged in. These are concatenated and hashed. In PHP this could look like this:

$identifier = sha1((string)$this->getPageUid() . (string)$this->isUserLoggedIn());

Tip

sha1 is a good hash algorithm in this case, as collisions are extremely unlikely. It scales O(n) with the input length.

When the plugin is accessed, the identifier is calculated early in the program flow. Next, the plugin looks up for a cache entry with this identifier. If such an entry exists, the plugin can return the cached content, else it calculates the content and stores a new cache entry with this identifier.

In general, the identifier is constructed from all dependencies which specify an unique set of data. The identifier should be based on information which already exist in the system at the point of its calculation. In the above scenario the page id and whether or not a user is logged in are already determined during the frontend bootstrap and can be retrieved from the system quickly.

About tags

Tags are used to drop specific cache entries when some information they are based on is changed.

Suppose the above plugin displays content based on different news entries. If one news entry is changed in the backend, all cache entries which are compiled from this news row must be dropped to ensure that the frontend renders the plugin content again and does not deliver old content on the next frontend call.

If - for example - the plugin uses news number one and two on one page, and news one on another page, the related cache entries should be tagged with these tags:

  • page 1, tags news_1, news_2
  • page 2, tag news_1

If entry 2 is changed, a simple backend logic (probably a hook in DataHandler) could be created, which drops all cache entries tagged with news_2. In this case the first entry would be invalidated while the second entry still exists in the cache after the operation.

While there is always exactly one identifier for each cache entry, an arbitrary number of tags can be assigned to an entry and one specific tag can be assigned to multiple cache entries. All tags a cache entry has are given to the cache when the entry is stored ("set").

Caches in the TYPO3 Core

The TYPO3 core defines and uses several caching framework caches by default. This section gives an overview of default caches, its usage and behaviour. If not stated otherwise, the default database backend with variable frontend is used.

Since TYPO3 CMS 6.2, the various caches are organized in groups. Three groups currently exist:

pages
Frontend-related caches.
system
Low-level caches. Flushing low-level caches should be avoided as much as possible, as rebuilding them requires significant resources.
all
All other caches.

Cache clearing commands can be issued to target a particular group. If a cache does not belong to a group, it will be flushed when the "all" group is flushed, but such caches should normally be transient anyway.

There are TSconfig options for permissions corresponding to each group.

The following caches exist in the TYPO3 CMS Core:

  • cache_core
    • Core cache for compiled php code. It should not be used by extensions.
    • Uses PhpFrontend with the SimpleFileBackend for maximum performance.
    • Stores core internal compiled PHP code like concatenated ext_tables.php and ext_localconf.php files, autoloader and sprite configuration PHP files.
    • This cache is instantiated very early during bootstrap and can not be re configured by instance specific LocalConfiguration.php or similar.
    • Cache entries are located in directory typo3temp/Cache/Code/cache_code. The full directory and any file in this directory can be safely removed and will be re-created upon next request. This is especially useful during development
    • group: system
  • cache_classes
    • Maps class names (and potentially one or more aliases) to the location of the class files in the filesystem. This cache is used by the class loader in order to require the correct class file when that class needs to be instantiated.
    • group: system
  • cache_hash
    • Stores several key-value based cache entries, mostly used during frontend rendering.
    • groups: all, pages
  • cache_pages
    • The frontend page cache. Stores full frontend pages.
    • Content is compressed by default to reduce database memory and storage overhead.
    • groups: all, pages
  • cache_pagesection
    • Used to store "parts of a page", for example used to store Typoscript snippets and compiled frontend templates.
    • Content is compressed by default to reduce database memory and storage overhead.
    • groups: all, pages
  • cache_phpcode
    • Code cache with PhpFrontend and FileBackend.
    • Unused by core since TYPO3 CMS 6.0.
    • group: system
  • cache_runtime
    • Runtime cache to store data specific for current request.
    • Used by several core parts during rendering to re-use already calculated data.
    • Valid for one request only.
    • Can be re-used by extensions that have similar caching needs.
  • cache_rootline
    • Cache for rootline calculations.
    • Quick and simple cache dedicated for core usage, Should not be re-used by extensions.
    • groups: all, pages
  • l10n
    • Cache for the localized labels.
    • group: system
  • extbase_object
    • Contains general information about classes, name, interfaces implemented, etc..
    • group: system
  • extbase_reflection
    • Contains detailed information about a class' member variables and methods.
    • group: system

Tip

In rare cases, for example when classes that are required during the bootstrap process are introduced (usually when working on the TYPO3 core), cache clearings requests themselves might throw fatal errors. The solution here is to manually remove the cache files from typo3temp/Cache/Code/.

Garbage collection task

The core system provides a Scheduler task to collect the garbage of all cache backends. This is important for backends like the database backend that do not remove old cache entries and tags internally. It is highly recommended to add this Scheduler task and run it once in a while (maybe once a day at night) for all used backends that do not delete entries which exceeded their lifetime on their own to free up memory or hard disk space.

Cache API

The caching framework architecture is based on the following classes:

  • \TYPO3\CMS\Core\Cache\Cache: Adapter class between TYPO3 FLOW cache logic and TYPO3 CMS core implementation. Used by core and extensions to initialize the framework. Creates singleton instances of \TYPO3\CMS\Core\Cache\CacheFactory and \TYPO3\CMS\Core\Cache\CacheManager.
  • \TYPO3\CMS\Core\Cache\CacheManager: Returns the cache frontend of a specific cache. This is the main class used by core and extensions to access the instance of a specific cache. Handles configuration settings and default configuration.
  • \TYPO3\CMS\Core\Cache\CacheFactory: Factory class to instantiate cache manager and caches. Extensions usually do not need to fiddle with this class.
  • \TYPO3\CMS\Core\Cache\Frontend\FrontendInterface: Main interface to handle cache entries of a specific cache. Different frontends and further interfaces exist to handle different data types.
  • \TYPO3\CMS\Core\Cache\Backend\BackendInterface: Main interface that every valid storage backend must implement. Several backends and further interfaces exist to specify specific backend capabilities. Some frontends require backends to implement additional interfaces.

Cache frontends

Frontend API

All frontends must implement the API defined in interface TYPO3\CMS\Core\Cache\Frontend\FrontendInterface. All operations on a specific cache must be done with these methods. The frontend object of a cache is the main object any cache manipulation is done with, usually the assigned backend object should not be used directly.

Method Description
getIdentifier Returns the cache identifier.
getBackend Returns the backend instance of this cache. It is seldom needed in usual code.
set Sets/overwrites an entry in the cache.
get Returns the cache entry for the given identifier.
has Checks for existence of a cache entry. Do no use this prior to get() since get() returns NULL if an entry does not exist.
remove Removes the entry for the given identifier from the cache.
flushByTag Flushes all cache entries which are tagged with the given tag.
collectGarbage Calls the garbage collection method of the backend. This is important for backends which are unable to do this internally (like the DB backend).
isValidEntryIdentifier Checks if a given identifier is valid.
isValidTag Checks if a given tag is valid.
requireOnce PhpFrontend only Requires a cached PHP file directly.
Available frontends

Currenly three different frontends are implemented. The main difference is the data types which can be stored using a specific frontend.

String Frontend

The string frontend accepts strings as data to be cached.

Variable Frontend

Strings, arrays and objects are accepted by this frontend. Data is serialized before it is passed to the backend.

Note

Since version 4.5, the igbinary serializer is used transparently (if available in the system), which speeds up both serialization and unserialization while also reducing data size.

Tip

The variable frontend is the most frequently used frontend and handles the widest range of data types. While it can also handle string data, the string frontend should be used if the cache needs to store strings, if only to avoid the additional serialization done by the variable frontend.

PHP Frontend

This is a special frontend to cache PHP files. It extends the string frontend with the method requireOnce() which allows PHP files to be require()'d if a cache entry exists. This can be used by extensions to cache and speed up loading of calculated PHP code and becomes handy if a lot of reflection and dynamic PHP class construction is done.

A backend to be used in combination with the PHP frontend must implement the interface TYPO3\CMS\Core\Cache\Backend\PhpCapableBackendInterface. Currently the file backend and the simple file backend fulfill this requirement.

Note

The PHP frontend can only be used to cache PHP files. It does not work with strings, arrays or objects. It is not intended as a page content cache.

Cache backends

A variety of storage backends exists. They have different characteristics and can be used for different caching needs. The best backend depends on a given server setup and hardware, as well as cache type and usage. A backend should be chosen wisely, as a wrong decision could end up actually slowing down a TYPO3 installation.

Backend API

All backends must implement at least implement interface TYPO3\CMS\Core\Cache\Backend\BackendInterface. All operations on a specific cache must be done with these methods. There are several further interfaces that can be implemented by backends to declare additional capabilities. Usually, extension code should not handle cache backend operations directly, but should use the frontend object instead.

Method Description
setCache Reference to the frontend which uses the backend. This method is mostly used internally.
set Save data in the cache.
get Load data from the cache.
has Checks if a cache entry with the specified identifier exists.
remove Remove a cache entry with the specified identifier.
flush Remove all cache entries.
collectGarbage Does garabage collection.
flushByTag TaggableBackendInterface only Removes all cache entries which are tagged by the specified tag.
findIdentifiersByTag TaggableBackendInterface only Finds and returns all cache entry identifiers which are tagged by the specified tag.
requireOnce PhpCapableBackendInterface only Loads PHP code from the cache and require_onces it right away.
freeze FreezableBackendInterface only Freezes this cache backend.
isFrozen FreezableBackendInterface only Tells if this backend is frozen.
Common options
Option Description Mandatory Type Default
defaultLifetime Default lifetime in seconds of a cache entry if it is not specified for a specific entry on set() No integer 3600
Database Backend

This is the main backend suitable for most storage needs. It does not require additional server daemons nor server configuration.

The database backend does not automatically perform garbage collection. Instead the Scheduler garbage collection task should be used.

It stores data in the configured database (usually MySQL) and can handle large amounts of data with reasonable performance. Data and tags are stored in two different tables, every cache needs its own set of tables. In terms of performance the database backend is already pretty well optimized and should be used as default backend if in doubt. This backend is the default backend if no backend is specifically set in the configuration.

The core takes care of creating and updating required database tables.

For caches with a lot of read and write operations, it is important to tune the MySQL setup. The most important setting is innodb_buffer_pool_size. A generic goal is to give MySQL as much RAM as needed to have the main table space loaded completely in memory.

The database backend tends to slow down if there are many write operations and big caches which do not fit into memory because of slow harddrive seek and write performance. If the data table grows too big to fit into memory, it is possible to compress given data transparently with this backend, which often shrinks the amount of needed space to 1/4 or less. The overhead of the compress/uncrompress operation is usually not high. A good candidate for a cache with enabled compression is the core pages cache: it is only read or written once per request and the data size is pretty large. The compression should not be enabled for caches which are read or written multiple times during one request.

InnoDB issues

The database backend for MySQL uses InnoDB tables. Due to the nature of InnoDB, deleting records does not reclaim the actual disk space. E.g. if the cache uses 10GB, cleaning it will still keep 10GB allocated on the disk even though phpMyAdmin will show 0 as the cache table size. To reclaim the space, turn on the MySQL option file_per_table, drop the cache tables and re-create them using the Install tool. This does not by any mean that you should skip the scheduler task. Deleting records still improves performance.

Options
Option Description Mandatory Type Default
compression Whether or not data should be compressed with gzip. This can reduce size of the cache data table, but incurs CPU overhead for compression and decompression. No boolean false
compressionLevel Gzip compression level (if the compression option is set to true). The default compression level is usually sufficient. - -1: Default gzip compression (recommended) - 0: No compression - 9: Maximum compression (costs a lot of CPU) No integer from -1 to 9 -1
Memcached Backend

Memcached is a simple, distributed key/value RAM database. To use this backend, at least one memcached daemon must be reachable, and the PECL module "memcache" must be loaded. There are two PHP memcached implementations: "memcache" and "memcached". Currently, only memcache is supported by this backend.

Warning and design constraints

Memcached is by design a simple key-value store. Values must be strings and there is no relation between keys. Since the caching framework needs to structure it to store the identifier-data-tags relations, for each cache entry it stores an identifier->data, identifier->tags and a tag->identifiers entry.

This leads to structural problems:

  • If memcache runs out of memory but must store new entries, it will toss some other entry out of the cache (this is called an eviction in memcached speak).
  • If data is shared over multiple memcache servers and some server fails, key/value pairs on this system will just vanish from cache.

Both cases lead to corrupted caches. If, for example, a tags->identifier entry is lost, dropByTag() will not be able to find the corresponding identifier->data entries which should be removed and they will not be deleted. This results in old data delivered by the cache. Additionally, there is currently no implementation of the garbage collection that could rebuild cache integrity.

It is important to monitor a memcached system for evictions and server outages and to clear clear caches if that happens.

Furthermore memcache has no sort of namespacing. To distinguish entries of multiple caches from each other, every entry is prefixed with the cache name. This can lead to very long runtimes if a big cache needs to be flushed, because every entry has to be handled separately and it is not possible to just truncate the whole cache with one call as this would clear the whole memcached data which might even hold non TYPO3 related entries.

Because of the mentioned drawbacks, the memcached backend should be used with care or in situations where cache integrity is not important or if a cache has no need to use tags at all. Currently, the memcache backend implements the TaggableBackendInterface, so the implementation does allow tagging, even if it is not advised to used this backend together with heavy tagging.

Warning

Since memcached has no sort of namespacing and access control, this backend should not be used if other third party systems have access to the same memcached daemon for security reasons. This is a typical problem in cloud deployments where access to memcache is cheap (but could be read by third parties) and access to databases is expensive.

Options
Option Description Mandatory Type Default
servers

Array of used memcached servers. At least one server must be defined. Each server definition is a string, allowed syntaxes:

  • hostname or IP: TCP connect to host on memcached default port (usually 11211, defined by PHP ini variable memcache.default_port)
  • hostname:port: TCP connect to host on port
  • tcp://hostname:port: Same as above
  • unix:///path/to/memcached.sock: Connect to memcached server using unix sockets
Yes array  
compression Enable memcached internal data compression. Can be used to reduce memcached memory consumption, but adds additional compression / decompression CPU overhead on the related memcached servers. No boolean false
Redis Backend

Redis is a key-value storage/database. In contrast to memcached, it allows structured values. Data is stored in RAM but it allows persistence to disk and doesn't suffer from the design problems of the memcached backend implementation. The redis backend can be used as an alternative to the database backend for big cache tables and helps to reduce load on database servers this way. The implementation can handle millions of cache entries each with hundreds of tags if the underlying server has enough memory.

Redis is known to be extremely fast but very memory hungry. The implementation is an option for big caches with lots of data because most important operations perform O(1) in proportion to the number of (redis) keys. This basically means that the access to an entry in a cache with a million entries is not slower than to a cache with only 10 entries, at least if there is enough memory available to hold the complete set in memory. At the moment only one redis server can be used at a time per cache, but one redis instance can handle multiple caches without performance loss when flushing a single cache.

The garbage collection task should be run every once in a while to find and delete old tags.

The implementation is based on the PHP phpredis module, which must be available on the system.

Note

It is important to monitor the redis server and tune its settings to the specific caching needs and hardware capabilities. There are several articles on the net and the redis configuration file contains some important hints on how to speed up the system if it reaches bounds. A full documentation of available options is far beyond this documentation.

Options
Option Description Mandatory Type Default
hostname IP address or name of redis server to connect to. No string 127.0.0.1
port Port of the redis daemon. No integer 6379
persistentConnection Activate a persistent connection to redis server. This could be a benefit under high load cloud setups. No boolean false
database Number of the database to store entries. Each cache should use its own database, otherwise all caches sharing a database are flushed if the flush operation is issued to one of them. Database numbers 0 and 1 are used and flushed by the core unit tests and should not be used if possible. No integer 0
password

Password used to connect to the redis instance if the redis server needs authentication.

Warning

The password is sent to the redis server in plain text.

No string  
compression Whether or not data compression with gzip should be enabled. This can reduce cache size, but adds some CPU overhead for the compression and decompression operations in PHP. No boolean false
compressionLevel

Set gzip compression level to a specific value. The default compression level is usually sufficient.

  • -1: Default gzip compression (recommended)
  • 0: No compression
  • 9: Maximum compression (but more CPU overhead)
No integer from -1 to 9 -1
APC Backend

APC is mostly known as an opcode cache for PHP source files but can be used to store user data in shared memory as well. Its main advantage is that data can be shared between different PHP processes and requests. All calls directly access shared memory. This makes this backend lightning fast for get() and set() operations. It can be an option for relatively small caches (few dozens of megabytes) which are read and written very often and becomes handy if APC is used as opcode cache anyway.

The implementation is very similar to the memcached backend implementation and suffers from the same problems if APC runs out of memory. Garbage collection is currently not implemented. In its latest version, APC will fail to store data with a PHP warning if it runs out of memory. This may change in the future. Even without using the cache backend, it is advisable to increase the memory cache size of APC to at least 64MB when working with TYPO3, simply due to the large number of PHP files to be cached. A minimum of 128MB is recommended when using the additional content cache. Cache TTL for file and user data should be set to zero (disabled) to avoid heavy memory fragmentation.

Note

It is not advisable to use the APC backend in shared hosting environments for security reasons. The user cache in APC is not aware of different virtual hosts. Basically every PHP script which is executed on the system can read and write any data to this shared cache, given data is not encapsulated or namespaced in any way. Only use the APC backend in environments which are completely under your control and where no third party can read or tamper your data.

Xcache Backend

Xcache is a PHP opcode cache similar to APC. It can also store in-memory key/value user data.

The cache backend implementation is nearly identical to the implementation of APC backend and has the same design constraints.

Important

Xcache does not work in command-line context. The Xcache backend implementation is constructed to silently discard any cache operation if in CLI context. That means if Xcache backend is used, it is of no effect in CLI.

Furthermore, it is important to set the PHP ini value xcache.var_size to a value (eg. 100M) that is big enough to store the needed data. The usage of this capacity should be monitored.

(Available since TYPO3 CMS 6.1)

Wincache backend

Wincache is a PHP opcode cache similar to APC, but dedicated to the Windows OS platform. Similar to APC, the cache can also be used as in-memory key/value cache.

The cache backend implementation is nearly identical to the implementation of APC backend and has the same design constrains.

File Backend

The file backend stores every cache entry as a single file to the file system. The lifetime and tags are added after the data part in the same file.

This backend is the big brother of the Simple file backend and implements additional interfaces. Like the simple file backend it also implements the PhpCapableInterface, so it can be used with PhpFrontend. In contrast to the simple file backend it furthermore implements TaggableInterface and FreezableInterface.

A frozen cache does no lifetime check and has a list of all existing cache entries that is reconstituted during initialization. As a result, a frozen cache needs less file system look ups and calculation time if accessing cache entries. On the other hand, a frozen cache can not manipulate (remove, set) cache entries anymore. A frozen cache must flush the complete cache again to make cache entries writable again. Freezing caches is currently not used in TYPO3 CMS core. It can be an option for code logic that is able to calculate and set all possible cache entries during some initialization phase, to then freeze the cache and use those entries until the whole thing is flushed again. This can be useful especially if caching PHP code.

In general, the backend was specifically optimized to cache PHP code, the get and set operations have low overhead. The file backend is not very good with tagging and does not scale well with the number of tags. Do not use this backend if cached data has many tags.

Warning

The performance of flushByTag() is bad and scales just O(n).

On the contrary performance of get() and set() operations. is good and scales well. Of course if many entries have to be handled, this might still slow down after a while and a different storage strategy should be used (e.g. RAM disks, battery backed up RAID systems or SSD hard disks).

Options
Option Description Mandatory Type Default
cacheDirectory The directory where the cache files are stored. By default it is assumed that the directory is below TYPO3_DOCUMENT_ROOT. However, an absolute path can be selected, too. Every cache should be assigned its own directory, otherwise flushing of one cache would flush all other caches within the same directory as well. No string typo3temp/cache/
Simple File Backend

The simple file backend is the small brother of the file backend. In contrast to most other backends, it does not implement the TaggableInterface, so cache entries can not be tagged and flushed by tag. This improves the performance if cache entries do not need such tagging. TYPO3 CMS core uses this backend for its central core cache (that hold autoloader cache entries and other important cache entries). The core cache is usually flushed completly and does not need specific cache entry eviction.

PDO Backend

The PDO backend can be used as a native PDO interface to databases which are connected to PHP via PDO. It is an alternative to the database backend if a cache should be stored in a database which is otherwise only supported by TYPO3 dbal to reduce the parser overhead.

The garbage collection is implemented for this backend and should be called to clean up hard disk space or memory.

Note

There is currently very little production experience with this backend, especially not with a capable database like Oracle. Any feedback for real life use cases of this cache is appreciated.

Options
Option Description Mandatory Type Default
dataSourceName

Data source name for connecting to the database. Examples:

  • mysql:host=localhost;dbname=test
  • sqlite:/path/to/sqlite.db
  • sqlite::memory
Yes string  
username Username for the database connection. No string  
password Password to use for the database connection. No string  
Transient Memory Backend

The transient memory backend stores data in a PHP array. It is only valid for one request. This becomes handy if code logic needs to do expensive calculations or must look up identical information from a database over and over again during its execution. In this case it is useful to store the data in an array once and just lookup the entry from the cache for consecutive calls to get rid of the otherwise additional overhead. Since caches are available system wide and shared between core and extensions they can profit from each other if they need the same information.

Since the data is stored directly in memory, this backend is the quickest backend available. The stored data adds to the memory consumed by the PHP process and can hit the memory_limit PHP setting.

Null Backend

The null backend is a dummy backend which doesn't store any data and always returns false on get(). This backend becomes handy in development context to practically "switch off" a cache.

Developer information

This chapter is targeted at extension authors who want to use the caching framework for their needs. It is about how to use the framework properly. For details about its inner working, please refer to the section about architecture.

Example usages can be found throughout the TYPO3 CMS Core, in particular in system extension "core" and "extbase".

Cache registration and usage

Registration of a new cache should be done in ext_localconf.php. The example below just defines an empty sub-array in cacheConfigurations. Neither frontend nor backend are defined, meaning that the cache manager will choose the default variable frontend and the database backend by default.

if (!is_array($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'] = array();
}

Tip

The is_array() check is done to enable administrators to overwrite configuration of caches in LocalConfiguration.php. During bootstrap, any ext_localconf.php is loaded after DefaultConfiguration.php and AdditionalConfiguration.php are loaded, so it is important to make sure that the administrator did not already set any configuration of the extensions cache.

If special settings are needed, for example a specific backend (like the transient memory backend), it can be defined with an additional line below the cache array declaration. The extension documentation should hint an integrator about specific caching needs or setups in this case.

Tip

Extensions should not force specific settings, therefore the selection is again encapsulated in a if (!isset()) check to allow administrators to overwrite those settings. It is recommended to set up a cache configuration with sane defaults, but administrators should always be able to overwrite them for whatever reason.

if (!is_array($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache'] = array();
}
if (!isset($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache']['backend'])) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['myext_mycache']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\TransientMemoryBackend';
}

To get an instance of a cache, GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Cache\\CacheManager')->getCache('myext_mycache') should be used. The cache manager will return the fully initialized cache instance:

$myCacheInstance = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Cache\\CacheManager')->getCache('myext_mycache');
Cache access logic

Cache usage patterns are usually wrappers around the main code sections. Here is some example code:

protected function getCachedMagic() {
    $cacheIdentifier = $this->calculateCacheIdentifier();
    $cache = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Cache\\CacheManager')->getCache('myext_mycache');

    // If $entry is null, it hasn't been cached. Calculate the value and store it in the cache:
    if (($entry = $cache->get($cacheIdentifier)) === FALSE) {
        $entry = $this->calculateMagic();

        // [calculate lifetime and assigned tags]

        // Save value in cache
        $cache->set($cacheIdentifier, $entry, $tags, $lifetime);
    }
    return $entry;
}

Tip

It isn't needed to call has() before accessing cache entries with get() as the latter returns FALSE if no entry exists.

Using the system log

Note

A new logging API was introduced in TYPO3 CMS 6.0. It is far more flexible than the old one described here, but is not yet in use in the Core. Thus this section remains valid if you want to write to the "sys_log" table. Otherwise please consider using the new API.

Writing to the system log is done using the backend user object, which writes to the "sys_log" table:

$this->BE_USER->writelog($type, $action, $error, $details_nr, $details, $data, $table, $recuid, $recpid,$event_pid, $NEWid);

Here is a description of the arguments to this function call, and corresponding database fields in table "sys_log":

Field Type Var Description
type tinyint $type

Value telling which module in TYPO3 set the log entry. The type values are paired with an action-integer which is telling in more detail what the event was. Here type and action values are arranged hierarchically (type on first level, action on second level):


  • 1 : \TYPO3\CMS\Core\DataHandling\DataHandler ("TYPO3 Core Engine" where database records are manipulated)
    • Action values are:
      • 0 = no category
      • 1 = new record
      • 2 = update record
      • 3 = delete record
      • 4 = move record
      • 5 = check/evaluate
  • 2 : "tce_file" (File handling in fileadmin/ and absolute filemounts)
    • Action values are for various file handling types like upload, rename, edit etc.
  • 3 : System (e.g. sys_history save)
  • 4 : Modules: This is the mode you may use for extensions having backend module functionality. Probably you would like to use BE_USER->simplelog() for your extensions.
  • 254 : Personal settings changed
  • 255 : Login or Logout action
    • 1 = login
    • 2 = logout
    • 3 = failed login (+ errorcode 3)
    • 4 = failure_warning_email sent
action tinyint $action

See "type" above

When not available, use value "0"

error tinyint $error

Error level:

  • 0 = message, a notice of an action that happened.
  • 1 = error, typically a permission problem for the user
  • 2 = System Error, something which should not happen for technical reasons.
  • 3 = Security notice, like login failures
details_nr tinyint $details_nr

Number of "detail" message. This number should be unique for the combination of type/action

-1 is a temporary detail number you can use while developing and error messages are not fixed yet.

0 is a value that means the message is not supposed to be translated

>= 1 means the message is fixed and ready for translation.

details tinytext $details

The log message text (in english). By identification through type/action/details_nr this can be translated through the localization system.

If you insert "%s" markers in the details message and set $data to an array the first 5 entries (keys 0-4) from $data will substitute the markers sequentially (using sprintf).

log_data tinyblob $data Data that follows the log entry. Can be an array. See "details" for more info.
tablename varchar(40) $table Table name. Special field used by tce_main.php.
recuid int $recuid Record UID. Special field used by tce_main.php.
recpid int $recpid Record PID. Special field used by tce_main.php. [OBSOLETE; not used anymore.]
event_pid int $event_pid The page ID (pid) where the event occurred. Used to select log-content for specific pages.
NEWid varchar(20) $NEWid Special field used by tce_main.php. NEWid string of newly created records.
tstamp int - EXEC_TIME of event, UNIX time in seconds.
uid int - Unique ID for log entry, automatically inserted
userid int - User ID of backend user, automatically set for you
IP varchar(39) - REMOTE_ADDR of client
workspace int - Workspace ID

Making logging simple

While it is nice to have log message categorized and numbered during development and sometimes beyond that point a simpler logging API is necessary. Therefore you can also call this function:

BE_USER->simplelog($message, $extKey='', $error=0);

All you need is to set $message to store a log message. If you call it from an extension it is good practice to also supply the extension key. Finally you can add the error number (according to the table above) if you need to signal an error.

Logging with TYPO3

TYPO3 Logging consists of the following components:

  • A Logger that receives the log message and related details, like a severity
  • A LogRecord model which encapsulates the data
  • Configuration of the logging system
  • Writers which write the log records to different targets (like file, database, rsyslog server, etc.)
  • Processors which add more detailed information to the log record.

Contents:

Quickstart

Instantiate a logger for the current class:

/** @var $logger \TYPO3\CMS\Core\Log\Logger */
$logger = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\CMS\Core\Log\LogManager')->getLogger(__CLASS__);

Log a simple message:

$logger->info('Everything went fine.');
$logger->warning('Something went awry, check your configuration!');

Provide additional information with the log message:

$logger->error(
  'This was not a good idea',
  array(
    'foo' => $bar,
    'bar' => $foo,
  )
);

$logger->warning() etc. are only shorthands - you can also call $logger->log() directly and pass the severity level:

$logger->log(
   \TYPO3\CMS\Core\Log\LogLevel::CRITICAL,
   'This is an utter failure!'
);

TYPO3 has the FileWriter enabled by default, so all log entries are written to a file. If the filename is not set, then the file will contain a hash like typo3temp/logs/typo3_<hash>.log, for example typo3temp/logs/typo3_7ac500bce5.log.

A sample output looks like this:

Fri, 08 Mar 2013 09:45:00 +0100 [INFO] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": Everything went fine.
Fri, 08 Mar 2013 09:45:00 +0100 [WARNING] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": Something went awry, check your configuration!
Fri, 08 Mar 2013 09:45:00 +0100 [ERROR] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": This was not a good idea - {"foo":"bar","bar":{}}
Fri, 08 Mar 2013 09:45:00 +0100 [CRITICAL] request="5139a50bee3a1" component="TYPO3.Examples.Controller.DefaultController": This is an utter failure!

Logger

Instantiation

The LogManager enables an auto-configured usage of loggers in your PHP code by reading the logging configuration and setting the minimum severity level of the Logger accordingly.

/** @var $logger \TYPO3\CMS\Core\Log\Logger */
$logger = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\CMS\Core\Log\LogManager')->getLogger(__CLASS__);

Using __CLASS__ as name for the logger is recommended to enable logging configuration based on the class hierarchy.

Log() method

\TYPO3\CMS\Core\Log\Logger provides a central point for submitting log messages, the log() method:

$logger->log($level, $message, $data);

which takes three parameters:

Parameter Type Description
$level Type integer

One of either:

  • \TYPO3\CMS\Core\Log\LogLevel::EMERGENCY
  • \TYPO3\CMS\Core\Log\LogLevel::ALERT
  • \TYPO3\CMS\Core\Log\LogLevel::CRITICAL
  • \TYPO3\CMS\Core\Log\LogLevel::ERROR
  • \TYPO3\CMS\Core\Log\LogLevel::WARNING
  • \TYPO3\CMS\Core\Log\LogLevel::NOTICE
  • \TYPO3\CMS\Core\Log\LogLevel::INFO
  • \TYPO3\CMS\Core\Log\LogLevel::DEBUG
$message Type string The log message itself.
$data Type array Optional parameter, can contain additional data, which is added to the log record in the form of an array.

An early return in the log() method prevents unneeded computation work to be done. So you are safe to call $logger->debug() frequently without slowing down your code too much. The Logger will know by its configuration, what the most explicit severity level is.

As next step, all registered Processors are notified. They can modify the log records or add extra information.

The Logger then forwards the log records to all of its configured Writers, which will then persist the log record.

Shorthand methods

For each of the severity levels mentioned above, a shorthand method exists in \TYPO3\CMS\Core\Log\Logger, like

  • $logger->debug($message, array $data = array());
  • $logger->info($message, array $data = array());
  • $logger->notice($message, array $data = array());
  • etc.

Configuration of the Logging system

Instantiation of Loggers is configuration-free, as the LogManager automatically applies its configuration.

The Logger configuration is read from $GLOBALS['TYPO3_CONF_VARS']['LOG'], which contains an array reflecting the namespace and class hierarchy of your TYPO3 project.

Example:

To apply a configuration for all Loggers within the \TYPO3\CMS\Core\Cache namespace, the configuration is read from $GLOBALS['TYPO3_CONF_VARS']['LOG']['TYPO3']['CMS']['Core']['Cache']. So every logger requested for classes like \TYPO3\CMS\Core\Cache\CacheFactory, \TYPO3\CMS\Core\Cache\Backend\NullBackend, etc. will get this configuration applied. The same holds for the old pseudo-namespaces with underscore separator which are still common in extensions.

Configuring Logging for extensions works the same. If an extension uses namespaces, the syntax for the configuration is as above.

For older extensions, configuration is searched for in $GLOBALS['TYPO3_CONF_VARS']['LOG']['tx'] or $GLOBALS['TYPO3_CONF_VARS']['LOG']['Tx'] to differentiate extension classes from Core classes (as extension class names start with tx or Tx).

Writer configuration

The Log Writer configuration is read from the subkey writerConfiguration of the configuration array:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'] = array(
    // configuration for ERROR level log entries
  \TYPO3\CMS\Core\Log\LogLevel::ERROR => array(
      // add a FileWriter
    'TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter' => array(
        // configuration for the writer
      'logFile' => 'typo3temp/logs/typo3_7ac500bce5.log'
    )
  )
);

The above configuration applies to all log entries of level "ERROR" or above.

To apply a special configuration for the controllers of the examples extension, use the following configuration:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['Documentation']['Examples']['Controller']['writerConfiguration'] = array(
   // configuration for WARNING severity, including all
   // levels with higher severity (ERROR, CRITICAL, EMERGENCY)
    \TYPO3\CMS\Core\Log\LogLevel::WARNING => array(
     // add a SyslogWriter
        'TYPO3\\CMS\\Core\\Log\\Writer\\SyslogWriter' => array(),
    ),
);

This overwrites the default configuration shown in the first example for classes located in the namespace \Documentation\Examples\Controller.

For extension "foo" with key "tx_foo" (not using namespaces), the configuration would be located at:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['Tx']['Foo']['writerConfiguration'] = array(
   // ...
);

An arbitrary number of writers can be added for every severity level (INFO, WARNING, ERROR, ...). The configuration based on severity levels is applied to log entries of the particular severity level plus all levels with a higher severity. Thus, a log messages created with $logger->warning() will be affected by a writerConfiguration for \TYPO3\CMS\Core\Log\LogLevel::DEBUG, \TYPO3\CMS\Core\Log\LogLevel::INFO, \TYPO3\CMS\Core\Log\LogLevel::NOTICE and \TYPO3\CMS\Core\Log\LogLevel::WARNING. For the above example code that means:

  • Calling $logger->warning($msg); will result in $msg being written to the computer's syslog on top of the default configuration.
  • Calling $logger->debug($msg); will result in $msg being written only to the default log file (typo3temp/logs/typo3.log).

For a list of writers shipped with the TYPO3 Core see the section about Log Writers.

Processor configuration

Similar to the writer configuration, log record processors can be configured on a per-class and per-namespace basis from the subkey processorConfiguration

$GLOBALS['TYPO3_CONF_VARS']['LOG']['Documentation']['Examples']['Controller']['processorConfiguration'] = array(
    // configuration for ERROR level log entries
  \TYPO3\CMS\Core\Log\LogLevel::ERROR => array(
      // add a MemoyUsageProcessor
    'TYPO3\\CMS\\Core\\Log\\Processor\\MemoryUsageProcessor' => array(
      'formatSize' => TRUE
    )
  )
);

For a list of processors shipped with the TYPO3 Core, see the section about Log Processors.

The LogRecord model

All logging data is modeled using \TYPO3\CMS\Core\Log\LogRecord.

This model has the following properties:

requestId
A unique identifier for each request which is created by the TYPO3 bootstrap.
created
The micro-timestamp when the record is created.
component
The name of the logger which created the LogRecord, usually the fully qualified class name where the Logger has been instanciated.
level
An integer severity level from \TYPO3\CMS\Core\Log\LogLevel.
message
The log message string.
data
Any additional data, encapsulated within an array.

The API to create a new instance of LogRecord is \TYPO3\CMS\Core\Log\Logger:log() or one of the shorthand methods.

LogRecord implements the ArrayAccess interface so that the properties can be accessed like a native array, for example: $logRecord['requestId']. It also implements a __toString() method for your convenience, which returns the log records as a simplified string.

A LogRecord can be processed using LogProcessors or LogWriters. LogProcessors are meant to add values to the data property of LogRecord. For example, if you would like to add a stack trace, use \TYPO3\CMS\Core\Log\Processor\IntrospectionProcessor.

LogWriters are used to write a LogRecord to a particular target, for example a log file.

Log Writers

The purpose of a log writer is (usually) to save all log records into a persistent storage, like a log file, a database table, or to a remote syslog server.

Different log writers offer possibilities to log into different targets. Custom log writers can extend the functionality shipped with TYPO3 core.

Built-in Log Writers

This section describes the log writers shipped with the TYPO3 core. Some writers have options to allow customization of the particular writer. See the Configuration section for how to use these options.

DatabaseWriter

The database writer logs into a database table. This table has to reside in the database used by TYPO3 and is not automatically created.

Option Mandatory Description Default
logTable no Database table sys_log

Warning

The Admin Tools > Log module is not adapted to the records written by the DatabaseWriter into the sys_log table. If you write such records there, you will not be able to see them using that module.

Tip: There's a tool for viewing such records in the TYPO3 backend at github.com/vertexvaar/logs.

Example of a CREATE TABLE statement for logTable:

#
# Table structure for table 'tx_myextname_log'
#
# The KEY on request_id is optional
#
CREATE TABLE tx_myextname_log (
        request_id varchar(13) DEFAULT '' NOT NULL,
        time_micro double(16,4) NOT NULL default '0.0000',
        component varchar(255) DEFAULT '' NOT NULL,
        level tinyint(1) unsigned DEFAULT '0' NOT NULL,
        message text,
        data text,

        KEY request (request_id)
);

Warning

If you are using a MariaDB Galera Cluster you should definitely add a primary key field to the database definition, since it is required by Galera (this can be a normal uid field as known from other tables): MariaDB Galera Cluster - Known Limitations.

FileWriter

The file writer logs into a log file, one log record per line. If the log file does not exist, it will be created (including parent directories, if needed). Please make sure that your web server has write-permissions to that path and it is below the root directory of your web site (defined by PATH_site). The filename is appended with a hash, that depends on the encryption key. If $GLOBALS['TYPO3_CONF_VARS']['SYS']['generateApacheHtaccess'] is set, an .htaccess file is added to the directory. It protects your log files from being accessed from the web. If the log file is not set, then TYPO3 will use a filename containing a random hash, like typo3temp/logs/typo3_7ac500bce5.log.

Option Mandatory Description Default
logFile no Path to log file typo3temp/logs/typo3_<hash>.log like for example typo3temp/logs/typo3_7ac500bce5.log
PhpErrorLogWriter

Logs into the PHP error log using error_log()

SyslogWriter

Logs into the syslog (Unix only).

Option Mandatory Description Default
facility no Syslog Facility to log into. USER
Custom Log Writers

Custom log writers can be added through extensions. Every log writer has to implement the interface \TYPO3\CMS\Core\Log\Writer\WriterInterface. It is suggested to extend the abstract class \TYPO3\CMS\Core\Log\Writer\AbstractWriter which allows you use configuration options by adding the corresponding properties and setter methods.

Please keep in mind that TYPO3 will silently continue operating, in case a log writer is throwing an exception while executing the writeLog() method. Only in the case that all registered writers fail, the log entry plus additional information will be added to the configured fallback logger (which defaults to the PhpErrorLog writer).

Log Processors

The purpose of a log processor is (usually) to modify a log record or add more detailed information to it.

Log processors allow to manipulate log records without changing the code where the log method actually is called (inversion of control). This enables you to add any information from outside the scope of the actual calling function, for example webserver environment variables. The TYPO3 core ships some basic log processors, but more can be added with extensions.

Built-in Log Processors

This section describes the log processors shipped with the TYPO3 core. Some processors have options to allow customization of the particular processor. See the Configuration section for how to use these options.

IntrospectionProcessor

The introspection processor adds backtrace data about where the log event was triggered.

By default the following parameters from the original function call are added:

file
absolute path to the file.
line
line number.
class
class name.
function
function name.

If appendFullBackTrace is set, the full backstrace stack is added instead.

Option Mandatory Description Default
appendFullBackTrace no Adds a full backtrace stack to the log. TRUE
shiftBackTraceLevel no Removes the given number of entries from the top of the backtrace stack. 0
MemoryUsageProcessor

The memory usage processor adds the amount of used memory to the log record (result from memory_get_usage()).

Option Mandatory Description Default
realMemoryUsage no Use real size of memory allocated from system instead of emalloc() value. TRUE
formatSize no Whether the size is formatted with GeneralUtility::formatSize() TRUE
MemoryPeakUsageProcessor

The memory peak usage processor adds the peak amount of used memory to the log record (result from memory_get_peak_usage()).

Option Mandatory Description Default
realMemoryUsage no Use real size of memory allocated from system instead of emalloc() value. TRUE
formatSize no Whether the size is formatted with GeneralUtility::formatSize() TRUE
WebProcessor

The web processor adds selected webserver environment variables to the log record, i.e. all possible values from \TYPO3\CMS\Core\Utility\GeneralUtility::getIndpEnv('_ARRAY').

Custom Log Processors

Custom log processors can be added through extensions. Every log processor has to implement the interface \TYPO3\CMS\Core\Log\Processor\ProcessorInterface. It is suggested to extend the abstract class \TYPO3\CMS\Core\Log\Processor\AbstractProcessor which allows you use configuration options by adding the corresponding properties and setter methods.

Please keep in mind that TYPO3 will silently continue operating, in case a log processor is throwing an exception while executing the processLogRecord() method.

Error and Exception Handling

Since version 4.3.0 TYPO3 comes with an build-in error and exception handling system. Admins can configure how errors and exceptions should be displayed in Backend and Frontend. Errors and exception can be logged to all available logging systems in TYPO3 including \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() which is – among other features – able to send error messages by mail (see example setups below).

Configuration

All configuration options related to error and exception handling are found in $TYPO3_CONF_VARS[SYS]:

Key Data type Description
displayErrors integer

Configures whether PHP errors should be displayed.

  • 0 = Do not display any PHP error messages. Overrides the value of "exceptionalErrors" and sets it to 0 (= no errors are turned into exceptions),the configured productionExceptionHandler is used as exception handler
  • 1 = Display error messages with the registered error handler,the configured debugExceptionHandler is used as exception handler
  • 2 = Display errors only if client matches $TYPO3_CONF_VARS[SYS][devIPmask]. If devIPmask matches the users IP address the configured "debugExceptionHandler" is used for exceptions, if not "productionExceptionHandler" will be used.
  • -1 = Default setting. With this option, you can override the PHP setting display_errors. If devIPmask matches the users IP address the configured debugExceptionHandler is used for exceptions, if not productionExceptionHandler will be used.
errorHandler string

Classname to handle PHP errors. Leave empty to disable error handling.

Default: \TYPO3\CMS\Core\Error\ErrorHandler. This class will register itself as error handler. It is able to write error messages to all available logging systems in TYPO3 (\TYPO3\CMS\Core\Utility\GeneralUtility::syslog, \TYPO3\CMS\Core\Utility\GeneralUtility::devlog() and to the "sys_log" table).

Additionally the errors can be displayed as flash messages in the Backend or in the adminpanel in Frontend. The flash messages in Backend are only displayed if the error and exception handling is in "debug-mode", which is the case when the configured "debugExceptionHandler" is registered as exception handler (see: $TYPO3_CONF_VARS[SYS][displayErrors]).

Errors which are registered as "exceptionalErrors" will be turned into exceptions (to be handled by the configured exceptionHandler).

errorHandlerErrors integer

The E_* constant that will be handled by the error handler

Default: E_ALL ^ E_NOTICE

exceptionalErrors integer

The E_* constant that will be handled as an exception by the error handler.

Default: E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER\_ERROR ^ E_USER\_NOTICE ^ E_USER\_WARNING (4341) and "0" if $TYPO3_CONF_VARS[SYS][displayErrors] = 0.

Refer to the PHP documentation for more details on this value.

productionExceptionHandler string

Classname to handle exceptions that might happen in the TYPO3-code.

Leave empty to disable exception handling.

Default: \TYPO3\CMS\Core\Error\ProductionExceptionHandler. This exception handler displays a nice error message when something went wrong. The error message is logged to the configured logs.

Note

The configured productionExceptionHandler is used if $TYPO3_CONF_VARS[SYS][displayErrors] is set to "0" or to "-1" and $TYPO3_CONF_VARS[SYS][devIPmask] doesn't match.

debugExceptionHandler string

Classname to handle exceptions that might happen in the TYPO3 code.

Leave empty to disable exception handling.

Default: \TYPO3\CMS\Core\Error\DebugExceptionHandler. This exception handler displays the complete stack trace of any encountered exception. The error message and the stack trace is logged to the configured logs.

Note

The configured debugExceptionHandler is used if $TYPO3_CONF_VARS[SYS][displayErrors] is set to "1" or if $TYPO3_CONF_VARS[SYS][displayErrors] is "-1" or "2" and the $TYPO3_CONF_VARS[SYS][devIPmask] matches.

enable_errorDLOG boolean Whether errors should be written to the Developer's Log (requires an installed *devlog extension).
enable_exceptions boolean Whether exceptions should be written to the Developer's Log (requires an installed *devlog extension).
syslogErrorReporting integer

Configures which PHP errors should be logged to the configured syslogs (see: [SYS][systemLog]). If set to "0" no PHP errors are logged to the syslog.

Default: E_ALL ^ E_NOTICE (6135).

belogErrorReporting integer

Configures which PHP errors should be logged to the "sys_log" table (extension: belog). If set to "0" no PHP errors are logged to the "sys_log" table.

Default: E_ALL ^ E_NOTICE (6135).

systemLog string

Configures which mechanism is used to log errors being logged using GeneralUtility::sysLog() calls. Multiple settings are possible by separating them with a semicolon.

Every distinct option can contain up to three values separated by comma. The first option defines the mechanism which is used for logging. The second option defines the destination/target of the log message. The third option defines a severity level. Messages will get logged to the mechanism/target only if the severity of the message is higher than the specified level. Giving "0" as level or not giving a level at all will log all messages. Possible options are:

  • file,<abs-path-to-file>[,<level>] = Writes the error messages to the specified file.
  • mail,<to>[/<from>][,<level>] = Sends an email for every error message to the configured email "<to>". Additionally sets the e-mail From header.
  • syslog,<facility>[,<level>] = Uses the PHP method "syslog" to send the log message. Depending on the operating system the message will get logged in different ways. On Linux (Debian, Ubuntu) the message will usually appear in /var/log/syslog if not configured otherwise. The <facility> option is not used currently.
  • error_log[,,<level>] = This setting will use the PHP method "error_log" to log the message. The message will get handled according to the settings of the variable "error_log" in the php.ini file.

Using the method "file" is very reliable but could also pose a security risk if you write error logs into the webroot of your site. The option "error_log" allows to use the same logging facility which is by default used by PHP scripts on the server. Be aware that on production machines error logging for PHP scripts could be disabled.

The table below shows which values can be set by the user and which are set by TYPO3.

Values in plain text can be changed in LocalConfiguration.php.

Values in bold are set by TYPO3.

displayErrors errorHandlerErrors exceptionalErrors errorHandler devIPmask exceptionHandler
display_errors
(php_ini)
-1 E_ALL ^ E_NOTICE E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE TYPO3CMS CoreError ErrorHandler Matters

If devIPmask matches: debugExceptionHandler


If devIPmask doesn't match: productionExceptionHandler

Not changed
0 E_ALL ^ E_NOTICE 0 (no errors are turned into exceptions) TYPO3CMS CoreError ErrorHandler Doesn't matter production ExceptionHandler 0 (Off)
1 E_ALL ^ E_NOTICE E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE ^ E_USER_WARNING TYPO3CMS CoreError ErrorHandler Doesn't matter debugException Handler 1 (On)
2 E_ALL ^ E_NOTICE E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE ^ E_USER_WARNING TYPO3CMS CoreError ErrorHandler Matters

If devIPmask matches: debugExceptionHandler


If devIPmask doesn't match: productionExceptionHandler

1 (On)



0 (Off)

The following sections highlight the roles and goals of the various classes related to error and exception handling. Examples and custom handlers are also discussed.

Error Handler

Class \TYPO3\CMS\Core\Error\ErrorHandler is the default error handler in TYPO3.

Functions:

  • Can be registered for all, or for only a subset of the PHP errors which can be handled by an error handler
  • Displays error messages as flash messages in the Backend (if exceptionHandler is set to \TYPO3\CMS\Core\Error\DebugExceptionHandler). Since flash messages are integrated in the Backend template, PHP messages will not destroy the Backend layout.
  • Displays errors as TsLog messages in the adminpanel.
  • Logs error messages to \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() which is able to write error messages to a file, to the web server's error_log, the system's log and it can send you errors and exceptions in an email. \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() offers a hook and can be extended by user-defined logging methods.
  • Logs error messages to \TYPO3\CMS\Core\Utility\GeneralUtility::devLog() if $TYPO3_CONF_VARS[SYS][enable_errorDLOG] is enabled (depending on the devlog extension used, this might require an existing DB connection).
  • Logs error messages to the sys_log table. Logged errors are displayed in the belog extension (Admin Tools > Log) (works only if there is an existing DB connection).
Production Exception Handler

Functions of \TYPO3\CMS\Core\Error\ProductionExceptionHandler:

  • Shows brief exception message ("Oops, an error occurred!") using \TYPO3\CMS\Core\Messaging\ErrorpageMessage and its attendant template.
  • Logs exception messages to \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() which is able to write exception messages to a file, to the web server's error_log, the system's log and it can send you errors and exceptions in an email. \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() offers a hook an can be extended by user-defined logging methods.
  • Logs exception messages to \TYPO3\CMS\Core\Utility\GeneralUtility::devLog() if $TYPO3_CONF_VARS[SYS][enable_errorDLOG] is enabled (depending on the devlog extension used, this might require an existing DB connection).
  • Logs exception messages to the sys_log table. Logged errors are displayed in the belog extension (Admin Tools > Log) (works only if there is an existing DB connection).
Debug Exception Handler

Functions of \TYPO3\CMS\Core\Error\DebugExceptionHandler:

  • Shows detailed exception messages and full trace of an exception.
  • Logs exception messages to \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() which is able to write exception messages to a file, to the web server's error_log, the system's log and it can send you errors and exceptions in an email. \TYPO3\CMS\Core\Utility\GeneralUtility::syslog() offers a hook an can be extended by user-defined logging methods.
  • Logs exception messages to \TYPO3\CMS\Core\Utility\GeneralUtility::devLog() if $TYPO3_CONF_VARS[SYS][enable_errorDLOG] is enabled (depending on the devlog extension used, this might require an existing DB connection).
  • Logs exception messages to the sys_log table. Logged errors are displayed in the belog extension (Admin Tools > Log) (works only if there is an existing DB connection).
Examples
Debugging and development setup

Very verbose configuration which logs and displays all errors and exceptions.

In LocalConfiguration.php:

'SYS' => array(
   'displayErrors' => '1',
   'devIPmask' => '*',
   'errorHandler' => 'TYPO3\\CMS\\Core\\Error\\ErrorHandler',
   'errorHandlerErrors' => E_ALL ^ E_NOTICE,
   'exceptionalErrors' => E_ALL ^ E_NOTICE ^ E_WARNING ^ E_USER_ERROR ^ E_USER_NOTICE ^ E_USER_WARNING,
   'debugExceptionHandler' => 'TYPO3\\CMS\\Core\\Error\\DebugExceptionHandler',
   'productionExceptionHandler' => 'TYPO3\\CMS\\Core\\Error\\DebugExceptionHandler',
   'systemLogLevel' => '0',
   'systemLog' => 'mail,test@localhost.local,4;error_log,,2;syslog,LOCAL0,,3;file,/abs/path/to/logfile.log',
   'enable_errorDLOG' => '1',
   'enable_exceptionDLOG' => '1',
),

In .htaccess:

php_flag display_errors on
php_flag log_errors on
php_value error_log /path/to/php_error.log
Production setup

Example for a production configuration which displays only errors and exceptions if the devIPmask matches. Errors and exceptions are only logged if their level is at least 2 (=Warning).

In LocalConfiguration.php:

'SYS' => array(
   'displayErrors' => '2',
   'devIPmask' => '[your.IP.address]',
   'errorHandler' => 'TYPO3\\CMS\\Core\\Error\\ErrorHandler',
   'systemLogLevel' => '2',
   'systemLog' => 'mail,test@localhost.local,4;error_log,,2;syslog,LOCAL0,,3',
   'enable_errorDLOG' => '0',
   'enable_exceptionDLOG' => '0',
   'syslogErrorReporting' => E_ALL ^ E_NOTICE ^ E_WARNING,
   'belogErrorReporting' => '0',
),

In .htaccess:

php_flag display_errors off
php_flag log_errors on
php_value error_log /path/to/php_error.log
Performance setup

Since the error and exception handling and also the logging need some performance, here's an example how to disable error and exception handling completely.

In LocalConfiguration.php:

'SYS' => array(
   'displayErrors' => '0',
   'devIPmask' => '',
   'errorHandler' => '',
   'debugExceptionHandler' => '',
   'productionExceptionHandler' => '',
   'systemLog' => '',
   'enable_errorDLOG' => '0',
   'enable_exceptionDLOG' => '0',
   'syslogErrorReporting' => '0',
   'belogErrorReporting' => '0',
),

In .htaccess:

php_flag display_errors off
php_flag log_errors off
Extending the error and exception handling

If you want to register your own error or exception handler, simply include the class and insert its name into "productionExceptionHandler", "debugExceptionHandler" or "errorHandler":

$TYPO3_CONF_VARS['SYS']['errorHandler'] = 'myOwnErrorHandler';
$TYPO3_CONF_VARS['SYS']['debugExceptionHandler'] = 'myOwnDebugExceptionHandler';
$TYPO3_CONF_VARS['SYS']['productionExceptionHandler'] = 'myOwnProductionExceptionHandler';

An error or exception handler class must register an error (exception) handler in its constructor. Have a look at the files in EXT:core/Classes/Error/ to see how this should be done.

If you want to use the built-in error and exception handling but extend it with your own functionality, simply derive your class from the error and exception handling classes shipped with TYPO3 and register this class as error (exception) handler:

class tx_postExceptionsOnTwitter extends \TYPO3\CMS\Core\Error\DebugExceptionHandler {
    function echoExceptionWeb(Exception $exception) {
        $this->postExceptionsOnTwitter($exception);
    }
    function postExceptionsOnTwitter($exception) {
        // do it ;-)
    }
}
$TYPO3_CONF_VARS['SYS']['debugExceptionHandler'] = 'tx_postExceptionsOnTwitter';
$TYPO3_CONF_VARS['SYS']['productionExceptionHandler'] = 'tx_postExceptionsOnTwitter';

Using the system registry

The purpose of the registry (introduced in TYPO3 4.3) is to hold key- value pairs of information. You can actually think of it being an equivalent to the Windows registry (just not as complicated).

You might use the registry to store information that your script needs to store across sessions or request.

An example would be a setting that needs to be altered by a PHP script, which currently is not possible with TypoScript.

Another example: The scheduler system extension stores when it ran the last time. The reports system extension then checks that value, in case it determines that the scheduler hasn't run for a while it issues a warning. While this might not be of great use to anyone with an actual cron job set up for the scheduler, it is of use for users that have to run the scheduler tasks by hand due to missing access to a cron job.

The registry is not meant to store things that are supposed to go into a session or a cache, use the appropriate API for these instead.

The registry table (sys_registry)

Here's a description of the fields found in the sys_registry table:

Field Type Description
uid int Primary key, needed for replication and also useful as an index.
entry_namespace varchar (128)

Represents an entry's namespace. In general the namespace is an extension key starting with "tx_", a user script's prefix "user_", or "core" for entries that belong to the core.

The point of namespaces is that entries with the same key can exist inside different namespaces.

entry_key varchar (255) The entry's key. Together with the namespace the key is unique for the whole table. The key can be any string to identify the entry. It's recommended to use dots as dividers if necessary. This way the naming is similar to the already known syntax in TypoScript.
entry_value blob The entry's actual value. The value is stored as a serialized string, thus you can even store arrays or objects in a registry entry – it's not recommended though. Using phpMyAdmin's Show BLOB option you can check the value in that field although being stored as a binary.

The registry API

To use the registry, there's an easy to use API. Simply use the code below to retrieve an instance of the registry. The instance returned will always be the same as the registry is a singleton:

$registry = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Registry');

After retrieving an instance of the registry you can access the registry values through its get() method. The get() method offers an interesting third parameter to specify a default value, that value is returned in case the requested entry was not found in the registry. That happens when accessing an entry for the first time for example. Setting a value is easy as well using the set() method.

Method Parameters Description
set

$namespace : namespace in which to set the value

$key : the key of the value to set

$value : the value to store

Represents an entry's namespace. In general the namespace is an extension key starting with "tx_", a user script's prefix "user_", or "core" for entries that belong to the core.
get

$namespace : namespace to get the value from

$key : the key of the value to retrieve

$defaultValue : a default value if the key was not found in the given namespace

Used to get a value from the registry.
remove

$namespace : namespace to remove the value from

$key : the key of the value to remove

Remove an entry from a given namespace.
removeAllByNamespace $namespace : namespace to empty Deletes all value for a given namespace.

Note that you should not store binary data into the registry, it's not designed to do that. Use the filesystem instead, if you have such needs.

Examples

Here's an example taken from the Scheduler system extension:

$registry = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Registry');
$runInformation = array('start' => $GLOBALS['EXEC_TIME'], 'end' => time(), 'type' => $type);
$registry->set('tx_scheduler', 'lastRun', $runInformation);

It is retrieved later using:

$registry = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Registry');
$lastRun = $registry->get('tx_scheduler', 'lastRun');

Mail API

Since version 4.5 TYPO3 CMS provides a RFC compliant mailing solution, based on SwiftMailer.

Configuration

Several settings are available in the Install Tool ("All Configuration") affecting the sending process. The most important one is $GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'], which can take the following values:

mail
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = 'mail';
Default and backwards compatible setting. This is the most unreliable option. If you are serious about sending mails, consider using "smtp" or "sendmail".
smtp
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = 'smtp';
Sends messages over SMTP. It can deal with encryption and authentication. Works exactly the same on Windows, Unix and MacOS. Requires a mail server and the following additional settings:
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_server'] = '<server:port>';
Mailserver name and port to connect to. Port defaults to "25".
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_encrypt'] = '<transport protocol>';
Connect to the server using the specified transport protocol. Requires openssl library. Usually available: ssl, sslv2, sslv3, tls. Check stream_get_transports().
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_username] = '<username>';
If your SMTP server requires authentication, the username.
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_password] = '<password>';
If your SMTP server requires authentication, the password.

Example:

$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = 'smtp';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_server'] = 'localhost';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_encrypt'] = 'ssl'; // ssl, sslv3, tls
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_username'] = 'johndoe';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_password'] = 'cooLSecret';
sendmail
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = 'sendmail';
Sends messages by communicating with a locally installed MTA - such as sendmail. This may require setting the additional option:
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_sendmail_command'] = '<command>';

The command to call to send a mail locally. The default works on most modern UNIX based mail servers (sendmail, postfix, exim).

Example:

$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = 'sendmail';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_sendmail_command'] = '/usr/sbin/sendmail -bs';
mbox
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = 'mbox';
This doesn't send any mail out, but instead will write every outgoing mail to a file adhering to the RFC 4155 mbox format, which is a simple text file where the mails are concatenated. Useful for debugging the mail sending process and on development machines which cannot send mails to the outside. The file to write to is defined by:
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_mbox_file'] = '<abs/path/to/mbox/file>';
The file where to write the mails into. Path must be absolute.
<classname>
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = '<classname>';
Custom class which implements Swift_Transport. The constructor receives all settings from the MAIL section to make it possible to add custom settings.

How to create and send mails

This shows how to generate and send a mail in TYPO3:

// Create the message
$mail = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Mail\\MailMessage');

// Prepare and send the message
$mail

   // Give the message a subject
   ->setSubject('Your subject')

   // Set the From address with an associative array
   ->setFrom(array('john@doe.com' => 'John Doe'))

   // Set the To addresses with an associative array
   ->setTo(array('receiver@domain.org', 'other@domain.org' => 'A name'))

   // Give it a body
   ->setBody('Here is the message itself')

   // And optionally an alternative body
   ->addPart('<q>Here is the message itself</q>', 'text/html')

   // Optionally add any attachments
   ->attach(Swift_Attachment::fromPath('my-document.pdf'))

   // And finally do send it
   ->send()
 ;

Or if you prefer, don't concatenate the calls:

$mail = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Mail\\MailMessage');
$mail->setSubject('Your subject');
$mail->setFrom(array('john@doe.com' => 'John Doe'));
$mail->setTo(array('receiver@domain.org', 'other@domain.org' => 'A name'));
$mail->setBody('Here is the message itself');
$mail->addPart('<q>Here is the message itself</q>', 'text/html');
$mail->attach(Swift_Attachment::fromPath('my-document.pdf'));
$mail->send();

How to add attachments

Here is a code sample for attaching a file to mail:

// Create the attachment, the content-type parameter is optional
$attachment = \Swift_Attachment::fromPath('</path/to/image.jpg>', 'image/jpeg');

// Set the filename (optional)
$attachment->setFilename('<cool.jpg>');

// Attach attachment to message
$mail->attach($attachment);

How to add inline media

Here is how to add some inline media like images in a mail:

// Attach the message with a "cid"
$cid = $mail->embed(\Swift_Image::fromPath('<path/to/image.png>'));

// Create HTML body refering to it
$mail->setBody(
   '<html><head></head><body>' .
   '  Here is an image <img src="' . $cid . '" alt="Image" />' .
   '  Rest of message' .
   ' </body></html>',
   'text/html' //Mark the content-type as HTML
);

How to set and use a default sender

It is possible to define a default email sender ("From:") in the Install Tool:

$GLOBALS['TYPO3_CONF_VARS']['MAIL']['defaultMailFromAddress'] = 'john@doe.com';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['defaultMailFromName'] = 'John Doe';

This is how you can use these defaults:

$from = \TYPO3\CMS\Core\Utility\MailUtility::getSystemFrom();
$mail = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(
   'TYPO3\\CMS\\Core\\Mail\\MailMessage');
$mail->setFrom($from);
// ...
$mail->send();

SwiftMailer documentation

Please refer to the SwiftMailer documentation for more information about available methods,

Form protection tool

Since TYPO3 4.5, the TYPO3 Core provides a generic way of protecting forms against Cross-Site Request Forgery (CSRF).

Usage in the backend

For each form in the BE (or link that changes some data), create a token and insert is as a hidden form element. The name of the form element does not matter; you only need it to get the form token for verifying it.

$formToken = TYPO3\CMS\Core\FormProtection\FormProtectionFactory::get()
   ->generateToken('BE user setup', 'edit')
);
$this->content .= '<input type="hidden" name="formToken" value="' . $formToken . '" />';

The three parameters $formName, $action and $formInstanceName can be arbitrary strings, but they should make the form token as specific as possible. For different forms (e.g. BE user setup and editing a tt_content record) or different records (with different UIDs) from the same table, those values should be different.

For editing a tt_content record, the call could look like this:

$formToken = TYPO3\CMS\Core\FormProtection\FormProtectionFactory::get()->generateToken('tt_content', 'edit', $uid);

At the end of the form, you need to persist the tokens. This makes sure that generated tokens get saved, and also that removed tokens stay removed:

TYPO3\CMS\Core\FormProtection\FormProtectionFactory::get()->persistTokens();

In BE lists, it might be necessary to generate hundreds of tokens. So the tokens do not get automatically persisted after creation for performance reasons.

When processing the data that has been submitted by the form, you can check that the form token is valid like this:

  if ($dataHasBeenSubmitted &&
     TYPO3\CMS\Core\FormProtection\FormProtectionFactory::get()->validateToken(
        (string) \TYPO3\CMS\Core\Utility\GeneralUtility::_POST('formToken'),
        'BE user setup', 'edit'
     ) ) {
     // processes the data
  } else {
     // no need to do anything here as the BE form protection will create a
     // flash message for an invalid token
}

Note that validateToken invalidates the token with the token ID. So calling the validation with the same parameters twice in a row will always return FALSE for the second call.

It is important that the tokens get validated before the tokens are persisted. This makes sure that the tokens that get invalidated by validateToken cannot be used again.

Usage in the Install Tool

For each form in the install tool (or link that changes some data), create a token and insert is as a hidden form element. The name of the form element does not matter; you only need it to get the form token for verifying it.

$formToken = $this->formProtection->generateToken('installToolPassword', 'change');
// then puts the generated form token in a hidden field in the template

The three parameters $formName, $action and $formInstanceName can be arbitrary strings, but they should make the form token as specific as possible. For different forms (e.g. the password change and editing a the configuration), those values should be different.

At the end of the form, you need to persist the tokens. This makes sure that generated tokens get saved, and also that removed tokens stay removed:

$this->formProtection()->persistTokens();

When processing the data that has been submitted by the form, you can check that the form token is valid like this:

if ($dataHasBeenSubmitted &&
   $this->formProtection()->validateToken(
      (string) $_POST['formToken'],
      'installToolPassword',
      'change')
) {
   // processes the data
} else {
   // no need to do anything here as the install tool form protection will
   // create an error message for an invalid token
}

Note that validateToken invalidates the token with the token ID. So calling the validation with the same parameters twice in a row will always return FALSE for the second call.

It is important that the tokens get validated before the tokens are persisted. This makes sure that the tokens that get invalidated by validateToken cannot be used again.

Flash messages

There exists a generic system to show users that an action was performed successfully, or more importantly, failed. This system is known as "flash messages". The screenshot below shows the various severity levels of messages that can be emitted.

All levels of flash messages

The "examples" BE module shows one of each type of flash message

The different severity levels are described below:

  • Notifications are used to show very low severity information. Such information usually is so unimportant that it can be left out, unless running in some kind of debug mode.
  • Information messages are to give the user some information that might be good to know.
  • OK messages are to signal a user about a successfully executed action.
  • Warning messages show a user that some action might be dangerous, cause trouble or might have partially failed.
  • Error messages are to signal failed actions, security issues, errors and the like.

Flash messages API

Creating a flash message is achieved by simply instantiating an object of class \TYPO3\CMS\Core\Messaging\FlashMessage:

$message = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\Messaging\FlashMessage::class,
   'My message text',
   'Message Header', // [optional] the header
   \TYPO3\CMS\Core\Messaging\FlashMessage::WARNING, // [optional] the severity defaults to \TYPO3\CMS\Core\Messaging\FlashMessage::OK
   true // [optional] whether the message should be stored in the session or only in the \TYPO3\CMS\Core\Messaging\FlashMessageQueue object (default is false)
);
Flash messages severities

The severity is defined by using class constants provided by \TYPO3\CMS\Core\Messaging\FlashMessage:

  • \TYPO3\CMS\Core\Messaging\FlashMessage::NOTICE for notifications
  • \TYPO3\CMS\Core\Messaging\FlashMessage::INFO for information messages
  • \TYPO3\CMS\Core\Messaging\FlashMessage::OK for success messages
  • \TYPO3\CMS\Core\Messaging\FlashMessage::WARNING for warnings
  • \TYPO3\CMS\Core\Messaging\FlashMessage::ERROR for errors

The fourth parameter passed to the constructor is a flag that indicates whether the message should be stored in the session or not (the default is not). Storage in the session should be used if you need the message to be still present after a redirection.

In backend modules you can then make that message appear on top of the module after a page refresh or the rendering of the next page request or render it on your own where ever you want.

This example adds the flash message at the top of modules when rendering the next request:

$flashMessageService = $this->objectManager->get(\TYPO3\CMS\Core\Messaging\FlashMessageService::class);
$messageQueue = $flashMessageService->getMessageQueueByIdentifier();
$messageQueue->addMessage($message);

The message is added to the queue and then the template class calls \TYPO3\CMS\Core\Messaging\FlashMessageQueue::renderFlashMessages() which renders all messages from the queue. Here's how such a message looks like in a module:

A flash message in action

A typical (success) message shown at the top of a module

By default flash messages are shown atop the content of a module. However, if needed, you can change where the messages are shown by manipulating a module's template and inserting the ###FLASHMESSAGES### marker. Messages will then replace that marker instead of appearing at the top of the module.

The recommend way is to use the fluid ViewHelper <f:flashMessages />. This ViewHelper works in any context because it use the FlashMessageRendererResolver class to find the correct renderer for the current context.

Flash messages renderer

The implementation of rendering FlashMessages in the core has been optimized.

A new class called FlashMessageRendererResolver has been introduced. This class detects the context and renders the given FlashMessages in the correct output format. It can handle any kind of output format. The core ships with the following FlashMessageRenderer classes:

  • TYPO3\CMS\Core\Messaging\Renderer\BootstrapRenderer This renderer is used by default in the TYPO3 backend. The output is based on Bootstrap markup
  • TYPO3\CMS\Core\Messaging\Renderer\ListRenderer This renderer is used by default in the TYPO3 frontend. The output is a simple <ul> list
  • TYPO3\CMS\Core\Messaging\Renderer\PlaintextRenderer This renderer is used by default in the CLI context. The output is plain text

All new rendering classes have to implement the TYPO3\CMS\Core\Messaging\Renderer\FlashMessageRendererInterface interface. If you need a special output format, you can implement your own renderer class and use it:

$out = GeneralUtility::makeInstance(MySpecialRenderer::class)
   ->render($flashMessages);

The core has been modified to use the new FlashMessageRendererResolver. Any third party extension should use the provided FlashMessageViewHelper or the new FlashMessageRendererResolver class:

$out = GeneralUtility::makeInstance(FlashMessageRendererResolver::class)
   ->resolve()
   ->render($flashMessages);

Flash messages in Extbase

In Extbase the standard way of issuing flash messages is to add them in the controller. Code from the "examples" extension:

$this->addFlashMessage('This is a simple success message');

The full API of this function is:

$this->addFlashMessage(
   $messageBody,
   $messageTitle = '',
   $severity = \TYPO3\CMS\Core\Messaging\AbstractMessage::OK,
   $storeInSession = TRUE
);

The messages are then displayed by Fluid with the relevant ViewHelper as shown in this excerpt of EXT:examples/Resources/Private/Layouts/Module.html:

<div id="typo3-docbody">
   <div id="typo3-inner-docbody">
      <f:flashMessages />
      <f:render section="main" />
   </div>
</div>

Where to display the flash messages in an Extbase-based BE module is as simple as moving the View Helper around.

JavaScript-based flash messages (Notifications API)

Flash messages can also be fired up from JavaScript, coming up as small pop up windows. This works only in the TYPO3 backend. Here is sample code:

top.TYPO3.Notification.notice('Test-Notice', 'Lorem ipsum dolor sit amet, consetetur sadipscing elitr');
top.TYPO3.Notification.info('Test-Info', 'Lorem ipsum dolor sit amet, consetetur sadipscing elitr');
top.TYPO3.Notification.success('Test-OK', 'Lorem ipsum dolor sit amet, consetetur sadipscing elitr');
top.TYPO3.Notification.warning('Test-Warning', 'Lorem ipsum dolor sit amet, consetetur sadipscing elitr');
top.TYPO3.Notification.error('Test-Error', 'Lorem ipsum dolor sit amet, consetetur sadipscing elitr');

The last parameter is the duration (in seconds) after which the message should fade out. 0 = sticky, means it will only disappears on user interaction.

System categories

Since version 6.0, TYPO3 CMS provides a generic categorization system. Categories can be created in the backend like any other type of record. Any table can be made categorizable and thus be attached to system categories.

Since version 6.2, pages, content elements and files are categorizable by default.

Using categories

Managing categories

System categories are defined just like any other record. Each category can have a parent, making for a tree-like structure.

Editing a category

A category with a parent defined

The "items" tab shows all related records, i.e. all records that have been marked as belonging to this category.

Making a table categorizable

There are two ways to activate categories on a given table. The first one is to use the global setting $GLOBALS['TYPO3_CONF_VARS']['SYS']['defaultCategorizedTables']. It is a comma-separated list of tables for which categories should be activated. The default value is pages,tt_content,sys_file_metadata.

Important

It is recommended to avoid changing this setting. You should rather use the API described just below so as to avoid overriding a default which may change in future versions of TYPO3 CMS. The API is also more powerful.

The second way is to call \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::makeCategorizable(). This method adds a new entry into the registry managed by \TYPO3\CMS\Core\Category\CategoryRegistry. The registry will take care of adding the relevant $TCA definition to create a field for making relations to the system categories.

The call to makeCategorizable() must be located in an extension's Configuration/TCA/Overrides folder (or ext_tables.php file before TYPO3 CMS 6.2.1).

The default $GLOBALS['TCA'] structure provided by the registry can be overridden by an array options passed to makeCategorizable(). The example below illustrates how this is done:

// Add an extra categories selection field to the pages table
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::makeCategorizable(
   'examples',
   'pages',
   // Do not use the default field name ("categories") for pages, tt_content, sys_file_metadata, which is already used
   'tx_examples_cats',
   array(
      // Set a custom label
      'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang.xlf:additional_categories',
      // This field should not be an exclude-field
      'exclude' => FALSE,
      // Override generic configuration, e.g. sort by title rather than by sorting
      'fieldConfiguration' => array(
         'foreign_table_where' => ' AND sys_category.sys_language_uid IN (-1, 0) ORDER BY sys_category.title ASC',
      ),
      // string (keyword), see TCA reference for details
      'l10n_mode' => 'exclude',
      // list of keywords, see TCA reference for details
      'l10n_display' => 'hideDiff',
   )
);

The above code will add a categories field to the "pages" table, which will be called tx_examples_cats. The fieldConfiguration part of the options array is the one which overrides the base $TCA structure. In this case we would like categories to be listed alphabetically instead of using the "sorting" field.

If no label part is set in the options array, the field will be labelled "Categories".

By default, the field will be an exclude-field. The exclude part can be used to override this.

This is the result of the above code:

The new categories-relation field

The newly added field to define relations to categories (on top of the default one)

Using categories in flexforms

It is possible to create relations to categories also in Flexforms, although this has to be done manually since no API exists for this.

The code will look something like:

<settings.categoriesList>
   <TCEforms>
   <exclude>1</exclude>
   <label>Categories:</label>
   <config>
      <type>select</type>
      <autoSizeMax>50</autoSizeMax>
      <foreign_table>sys_category</foreign_table>
      <foreign_table_where> AND sys_category.sys_language_uid IN (-1, 0) ORDER BY sys_category.sorting ASC</foreign_table_where>
      <MM>sys_category_record_mm</MM>
      <MM_opposite_field>items</MM_opposite_field>
      <MM_match_fields>
         <tablenames>tt_content</tablenames>
         <fieldname>tx_myextension_categories</fieldname>
      </MM_match_fields>
      <maxitems>9999</maxitems>
      <renderMode>tree</renderMode>
      <size>10</size>
      <treeConfig>
         <appearance>
            <expandAll>1</expandAll>
            <showHeader>1</showHeader>
         </appearance>
         <parentField>parent</parentField>
      </treeConfig>
   </config>
   </TCEforms>
</settings.categoriesList>

Properties tablenames and fieldname would need to be adjusted.

System categories API

Beyond makeCategorizable(), class \TYPO3\CMS\Core\Category\CategoryRegistry has many other methods related to the management of categorized table. The best way to discover is to follow the link above and explore the methods provided by this class. They are all quite specialized and should not be needed most of the time.

Category collections

The \TYPO3\CMS\Core\Category\Collection\CategoryCollection classe provides the API for retrieving records related to a given category. Since TYPO3 CMS 6.2, it is extended by class \TYPO3\CMS\Frontend\Category\Collection\CategoryCollection which does the same job but in the frontend, i.e. respecting all enable fields and performing version and language overlays.

The main method is load() which will return a traversable list of items related to the given category. Here is an example usage, taken from the RECORDS content object:

$collection = \TYPO3\CMS\Frontend\Category\Collection\CategoryCollection::load(
   $aCategory,
   TRUE,
   $table,
   $relationField
);
if ($collection->count() > 0) {
   // Add items to the collection of records for the current table
   foreach ($collection as $item) {
      $tableRecords[$item['uid']] = $item;
      // Keep track of all categories a given item belongs to
      if (!isset($categoriesPerRecord[$item['uid']])) {
         $categoriesPerRecord[$item['uid']] = array();
      }
      $categoriesPerRecord[$item['uid']][] = $aCategory;
   }
}

As all collection classes in the TYPO3 CMS Core implement the Iterator interface, it is also possible to use expected methods like next(), rewind(), etc. Note that methods such as add() will only add items to the collection temporarily. The relations are not persisted in the database.

Usage with TypoScript

(since TYPO3 CMS 6.2)

In the frontend, it is possible to get collections of categorized records loaded into a RECORDS content object for rendering. Check out the categories property.

The HMENU object also has a "categories" special type, to display a menu based on categorized pages.

System collections

Since TYPO3 CMS 4.7 there exists a generic interface for managing collections of records. The File Abstraction Layer itself extends this mechanism for managing file collections.

Record collections

Record collections are managed via the TYPO3 backend. They can only be created in Folders.

A record collection

A collection of blog posts

Note

In the above image, the third record in the collection is actually a page. This will be ignored in all process using the Collections API, because the chosen table prevails.

Record collections are stored in the "sys_collection" table. The selected records are stored in the "sys_collection_entries" table.

Record collections can also have a type. Only the "static" type exits so far. Others might be added in the future.

File collections

File collections are similar in principle, but reference files. They are used by the "File links" (download) content element.

A file links content element

A "File links" content element referencing a file collection

File collections are stored in the "sys_file_collection" table. The selected files are stored in the "sys_file_reference" table.

Note that a file collection may also reference a folder, in which case all files inside the folder will be returned when calling that collection.

A folder collection

A file collection referencing a folder

Collections API

The TYPO3 CMS Core provides an API to enable usage of collections inside extensions. The most important classes are:

\TYPO3\CMS\Core\Collection\RecordCollectionRepository
Used to retrieve collections. It's not exactly an Extbase repository but functions in a similar way. The default "find" methods refer to the "sys_collection" table and will fetch "static"-type collections. Method findByTypeAndTableName() makes it possible to fetch collections from other tables and of a different type.
\TYPO3\CMS\Core\Collection\StaticRecordCollection
This class models the static record collection. It is important to note that collections returned by the repository (described above) are "empty". If you need to access their records, you need to load them first, using method loadContents(). On top of some specific API methods, this class includes all setters and getters that you may need to access the collection's data. For accessing the selected records, just loop on the collection (see example).
\TYPO3\CMS\Core\Resource\FileCollectionRepository
Similar to the record collection repository, but for file collections.
\TYPO3\CMS\Core\Resource\Collection\StaticFileCollection
Similar to the static record collection, but for files.
\TYPO3\CMS\Core\Resource\Collection\FolderBasedFileCollection
Again similar, but for file collections based on a folder.

Example

The "examples" extension provides a simple frontend plugin to demonstrate usage of collections. Here is what happens in the controller:

/**
 * Renders the list of all existing collections and their content
 *
 * @return void
 */
public function indexAction() {
   // Get all existing collections
   /** @var \TYPO3\CMS\Core\Collection\AbstractRecordCollection $collections */
   $collections = $this->collectionRepository->findAll();

   // Load the records in each collection
   /** @var \TYPO3\CMS\Core\Collection\StaticRecordCollection $aCollection */
   foreach ($collections as $aCollection) {
      $aCollection->loadContents();
   }

   // Assign the "loaded" collections to the view
   $this->view->assign('collections', $collections);
}

The base is code is quite simple: all collections are fetched and passed to the view. The one specific step is the loop over all collections to load their referenced records. Remember that a collection is otherwise "empty".

In the view we can then either use collection member variables as usual (like their title) or put them directly in a loop to iterate over the record selection:

<f:section name="main">
   <ul>
      <f:for each="{collections}" as="collection">
         <li>
            {collection.title} (Records from <code>{collection.itemTableName}</code>)
            <ul>
               <f:for each="{collection}" as="record">
                  <li>{record.title}</li>
               </f:for>
            </ul>
         </li>
      </f:for>
   </ul>
</f:section>

Note that the above code is a bit rough as it assumes that each record has a "title" property. This may of course not be the case and proper checks should be made.

Here is what the result may look like (the exact result will obviously depend on the content of the selection):

Collections plugin output

Typical output from the "Collections" plugin of extension "examples"

Enumerations

Since version 6.2 the TYPO3 CMS core includes an enumeration implementation. An enumeration should be used if you have a fixed list of values and it should always be prefered to just pure constants. That is because constants cannot be deprecated but values of an enumeration can.

How To Use Enumerations

Create an Enumeration

To create a new enumeration you have to extend the class TYPO3\CMS\Core\Type\Enumeration. Values are defined as constants in your implementation. The names of the constants must be given in uppercase.

A special, optional constant __default represents the default value of your enumeration if it is present. In that case the enumeration can be instantiated without a value and will be set to the default.

Example:

class LikeWildcard extends \TYPO3\CMS\Core\Type\Enumeration
{
   const __default = self::BOTH;

   /** @var int Do not use any wildcard */
   const NONE = 0;

   /** @var int Use wildcard on left side */
   const LEFT = 1;

   /** @var int Use wildcard on right side */
   const RIGHT = 2;

   /** @var int Use wildcard on both sides */
   const BOTH = 3;
}
Use an Enumeration

You always have to use the Enumeration::cast() method for instantiation.

((The ::cast() itself calls the constructor of the enumeration if it's not always an instance of the enum.))((make sentence clearer!))

That allows to deprecate enumeration values or do special value casts before finding a suitable value in the enumeration.

Example:

$likeWildcardLeft = LikeWildcard::cast(LikeWildcard::LEFT);

$valueFromDatabase = 1;

// will cast the value automatically to an enumeration.
// Result is true.
$likeWildcardLeft->equals($valueFromDatabase);

$enumerationWithValueFromDb = LikeWildcard::cast($valueFromDatabase);

// Remember to always use ::cast and never use the constant directly
$enumerationWithValueFromDb->equals(LikeWildcard::cast(LikeWildcard::RIGHT));
Exceptions

If the enumeration is instantiated with an invalid value an TYPO3\CMS\Core\Type\Exception\InvalidEnumerationValueException is thrown. This exception must be catched and you have to decide what the appropriate behavior should be.

Important

Always be prepared to handle exceptions when instantiating enumerations from user defined values!

Example:

try {
   $foo = LikeWildcard::cast($valueFromPageTs);
} catch (\TYPO3\CMS\Core\Type\Exception\InvalidEnumerationValueException) {
   $foo = LikeWildcard::cast(LikeWildcard::NONE);
}
Implement custom logic

Sometimes it not only makes sense to validate a value but to also have custom logic as well..

For example, the TYPO3\CMS\Core\Versioning\VersionState enumeration contains values of version states. Some of the values indicate that the state is a "placeholder". This logic can be implemented by a custom method:

class VersionState extends \TYPO3\CMS\Core\Type\Enumeration
{
   const __default = self::DEFAULT_STATE;
   const NEW_PLACEHOLDER_VERSION = -1;
   const DEFAULT_STATE = 0;
   const NEW_PLACEHOLDER = 1;
   const DELETE_PLACEHOLDER = 2;
   const MOVE_PLACEHOLDER = 3;
   const MOVE_POINTER = 4;

   /**
    * @return bool
    */
   public function indicatesPlaceholder()
   {
      return (int)$this->__toString() > self::DEFAULT_STATE;
   }
}

$myVersionState = VersionState::cast($versionStateValue);
if ($myVersionState->indicatesPlaceholder()) {
   echo 'The state indicates that this is a placeholder';
}

HTTP request library / Guzzle / PSR-7

Since TYPO3 CMS 8.1 the PHP library Guzzle has been added via composer dependency to work as a feature rich solution for creating HTTP requests based on the PSR-7 interfaces already used within TYPO3.

Guzzle auto-detects available underlying adapters available on the system, like cURL or stream wrappers and chooses the best solution for the system.

A TYPO3-specific PHP class called TYPO3\CMS\Core\Http\RequestFactory has been added as a simplified wrapper to access Guzzle clients.

All options available under $TYPO3_CONF_VARS[HTTP] are automatically applied to the Guzzle clients when using the RequestFactory class. The options are a subset to the available options on Guzzle (http://docs.guzzlephp.org/en/latest/request-options.html) but can further be extended.

Existing $TYPO3_CONF_VARS[HTTP] options have been removed and/or migrated to the new Guzzle-compliant options.

A full documentation for Guzzle can be found at http://docs.guzzlephp.org/en/latest/.

Although Guzzle can handle Promises/A+ and asynchronous requests, it currently acts as a drop-in replacement for the previous mixed options and implementations within GeneralUtility::getUrl() and a PSR-7-based API for HTTP requests.

The existing TYPO3-specific wrapper GeneralUtility::getUrl() now uses Guzzle under the hood automatically for remote files, removing the need to configure settings based on certain implementations like stream wrappers or cURL directly.

Basic usage

The RequestFactory class can be used like this:

// Initiate the Request Factory, which allows to run multiple requests
$requestFactory = GeneralUtility::makeInstance(\TYPO3\CMS\Core\Http\RequestFactory\RequestFactory::class);
$url = 'https://typo3.com';
$additionalOptions = [
   // Additional headers for this specific request
   'headers' => ['Cache-Control' => 'no-cache'],
   // Additional options, see http://docs.guzzlephp.org/en/latest/request-options.html
   'allow_redirects' => false,
   'cookies' => true
];
// Return a PSR-7 compliant response object
$response = $requestFactory->request($url, 'GET', $additionalOptions);
// Get the content as a string on a successful request
if ($response->getStatusCode() === 200) {
   if ($response->getHeader('Content-Type') === 'text/html') {
      $content = $response->getBody()->getContents();
   }
}

Extension authors are advised to use the RequestFactory class instead of using the Guzzle API directly in order to ensure a clear upgrade path when updates to the underlying API need to be done.

Icon API

Since version 7.5 TYPO3 CMS provides an Icon API for all icons in the TYPO3 backend.

Registration

All icons must be registered in the IconRegistry. To register icons for your own extension use the following code in your ext_localconf.php file:

$iconRegistry = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(
   \TYPO3\CMS\Core\Imaging\IconRegistry::class
);
$iconRegistry->registerIcon(
   $identifier, // Icon-Identifier, z.B. tx-myext-action-preview
   \TYPO3\CMS\Core\Imaging\IconProvider\SvgIconProvider::class,
   ['source' => 'EXT:myext/Resources/Public/Icons/action-preview.svg']
);
IconProvider

The TYPO3 core ships three IconProvider which can be used:

  • BitmapIconProvider – For all kind of bitmap icons (format like: gif, png, jpeg, etc)
  • SvgIconProvider – For all SVG icons
  • FontawesomeIconProvider – For all icons which can be found in fontawesome.io

In case your need special IconProvider you can implement your own, your class has to implement the IconProviderInterface.

Use icons in your code

You can use the Icon API to receive icons with in your PHP code or directly in fluid.

The PHP way

Your can use the IconFactory to request an icon:

$iconFactory = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(
   \TYPO3\CMS\Core\Imaging\IconFactory::class
);
$icon = $iconFactory->getIcon(
   'tx-myext-action-preview',
   \TYPO3\CMS\Core\Imaging\Icon::SIZE_SMALL,
   'overlay-identifier'
);
$this->view->assign('icon', $icon);
The Fluid ViewHelper

You can also simple use the Fluid ViewHelper to render an icon in your view:

{namespace core = TYPO3\CMS\Core\ViewHelpers}
<core:icon identifier="my-icon-identifier" size="small" overlay="overlay-identifier" />

Hooks

Contents:

The concept of "hooks"

Hooks are basically places in the source code where a user function will be called for processing if such has been configured. Hooks provide an easy way to extend the functionality of TYPO3 and its extensions without blocking others to do the same.

Hooks vs. XCLASS extensions

Hooks are the recommended way of extending TYPO3 compared to extending PHP classes with a child class (see "XCLASS extensions"). Because only one extension of a PHP class can exist at a time while hooks may allow many different user-designed processor functions to be executed. However, hooks have to be implemented in the TYPO3 core before you can use them, while extending a PHP class via the XCLASS method allows you to extend any class you like.

Proposing hooks

If you need to extend something which has no hook yet, then you should suggest implementing a hook. Normally that is rather easily done by the author of the source you want to extend.

Using hooks

The two lines of code below are an example of how a hook is used for clear-cache post-processing. The objective of this could be to perform additional actions whenever the cache is cleared for a specific page.

$TYPO3_CONF_VARS['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['clearCachePostProc'][] = 'myext_cacheProc->proc';

This registers the class/method name to a hook inside of \TYPO3\CMS\Core\DataHandling\DataHandler. The hook will call the user function after the clear-cache command has been executed. The user function will receive parameters which allows it to see what clear-cache action was performed and typically also an object reference to the parent object. Then the user function can take additional actions as needed.

The class has to be declared with the TYPO3 autoloader.

If we take a look inside of \TYPO3\CMS\Core\DataHandling\DataHandler we find the hook to be activated like this:

1
2
3
4
5
6
7
    // Call post processing function for clear-cache:
if (is_array($TYPO3_CONF_VARS['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['clearCachePostProc'])) {
    $_params = array('cacheCmd' => $cacheCmd);
    foreach($TYPO3_CONF_VARS['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['clearCachePostProc'] as $_funcRef) {
        \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction($_funcRef, $_params, $this);
    }
}

This is how hooks are typically constructed. The main action happens in line 5 where the function \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction() is called. The user function is called with two arguments, an array with variable parameters and the parent object.

In line 3 the contents of the parameter array is prepared. This is of high interest to you because this is where you see what data is passed to you and what data might possibly be passed by reference and thereby possible to manipulate from your hook function.

Finally, notice how the array $TYPO3_CONF_VARS['SC_OPTIONS']['t3lib/class.t3lib\_tcemain.php']['clearCachePostProc'] is traversed and for each entry the value is expected to be a function reference which will be called. This allows many hooks to be called at the same place. The hooks can even rearrange the calling order if they dare.

The syntax of a function reference (or object reference if \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj is used in the hook instead) can be seen in the API documentation of \TYPO3\CMS\Core\Utility\GeneralUtility.

Note

The example hook shown above refers to old class names. All these old class names were left in hooks, for obvious reasons of backwards-compatibility.

Hook configuration

There is no complete index of hooks in the core. But they are easy to search for and find. And typically it comes quite naturally since you will find the hooks in the code you want to extend - if they exist.

This index will list the main variable spaces for configuration of hooks. By the names of these you can easily scan the source code to find which hooks are available or might be interesting for you.

The index below also includes some variable spaces which do not only carry hook configuration but might be used for other purposes as well.

$TYPO3_CONF_VARS['EXTCONF']

Configuration space for extensions.

This will contain all kinds of configuration options for specific extensions including possible hooks in them! What options are available to you will depend on a search in the documentation for that particular extension.

$TYPO3_CONF_VARS['EXTCONF'][ extension_key ][ sub_key ] = value
  • extension_key : The unique extension key
  • sub_key : Whatever the script defines. Typically it identifies the context of the hook
  • value : It is up to the extension what the values mean, if they are mere configuration options or hooks or whatever and how deep the arrays go. Read the source code where the options are implemented to see. Or the documentation of the extension, if available.

Note

$TYPO3_CONF_VARS['EXTCONF'] is the recommended place to put hook configuration that are available inside your extensions!

This example shows hooks used in the "linkvalidator" system extension. The code looks inside the $TYPO3_CONF_VARS['EXTCONF'] array for items listed under the "checkLinks" key of the "linkvalidator" extension itself. All found classes are stored in an array, to be instantiated and used at a later point.

/**
 * Fill hookObjectsArr with different link types and possible XClasses.
 */
public function __construct() {
      // Hook to handle own checks
   if (is_array($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['linkvalidator']['checkLinks'])) {
      foreach ($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['linkvalidator']['checkLinks'] as $key => $classRef) {
         $this->hookObjectsArr[$key] = \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj($classRef);
      }
   }
}
$TYPO3_CONF_VARS['SC_OPTIONS']

Configuration space for core scripts.

This array is created as an ad hoc space for creating hooks from any script. This will typically be used from the core scripts of TYPO3 which do not have a natural identifier like extensions have their extension keys.

$TYPO3_CONF_VARS['SC_OPTIONS'][ main_key ][ sub_key ][ index ] = function_reference
  • main_key : The relative path of a script (for output scripts it should be the "script ID" as found in a comment in the HTML header )
  • sub_key : Whatever the script defines. Typically it identifies the context of the hook.
  • index : Integer index typically. Can be unique string if you have a reason to use that. Normally it has no greater significance since the value of the key is not used. The hooks normally traverse over the array and uses only the value (function reference)
  • function_reference : A function reference using the syntax of \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction() or \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj() depending on implementation of the hook.

The above syntax is how a hook is typically defined but it might differ and it might not be a hook at all, but just configuration. Depends on implementation in any case.

The following example shows a hook from \TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController. In this case the function \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj() is used for the hook. The function_reference is referring to the class name only since the function returns an object instance of that class. The method name to call is predefined by the hook, in this case sendFormmail_preProcessVariables(). This method allows to pass any number of variables along instead of the limited $params and $pObj variables from \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction().

    // Hook for preprocessing of the content for formmails:
if (is_array($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['sendFormmail-PreProcClass'])) {
    foreach($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['sendFormmail-PreProcClass'] as $_classRef) {
        $_procObj = \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj($_classRef);
        $EMAIL_VARS = $_procObj->sendFormmail_preProcessVariables($EMAIL_VARS, $this);
    }
}

In this example we are looking at a special hook, namely the one for RTE transformations. It is not a "hook" in the strict sense, but the same principles are used. In this case the "index" key is defined to be the transformation key name, not a random integer since we do not iterate over the array as usual. \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj() is also used.

if ($_classRef = $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_parsehtml_proc.php']['transformation'][$cmd]) {
    $_procObj = \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj($_classRef);
    $_procObj->pObj = &$this;
    $_procObj->transformationKey = $cmd;
    $value = $_procObj->transform_db($value, $this);
}

A classic hook also from \TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController. This one is based on \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction() and it passes a reference to $this along to the function via $_params. In the user-defined function $_params['pObj']->content is meant to be manipulated in some way. The return value is insignificant - everything works by the reference to the parent object.

    // Hook for post-processing of page content cached/non-cached:
if (is_array($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['contentPostProc-all'])) {
    $_params = array('pObj' => &$this);
    foreach($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['contentPostProc-all'] as $_funcRef) {
        \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction($_funcRef, $_params, $this);
    }
}
$TYPO3_CONF_VARS['TBE_MODULES_EXT']

** Configuration space for backend modules.**

Among these configuration options you might find entry points for hooks in the backend. This somehow overlaps the intention of SC_OPTIONS above but this array is an older invention and slightly outdated.

$TBE_MODULES_EXT[ backend_module_key ][ sub_key ] = value
  • backend_module_key : The backend module key for which the configuration is used.
  • sub_key : Whatever the backend module defines.
  • value : Whatever the backend module defines.

The following example shows TBE_MODULES_EXT being used for adding items to the Context Sensitive Menus (Clickmenu) in the backend. The hook value is an array with a key pointing to a file reference to class file to include. Later each class is instantiated and a fixed method inside is called to do processing on the array of menu items. This kind of hook is non-standard in the way it is made.

Warning

The API for registering context-sensitive menus was changed completely in TYPO3 4.5.

    // Setting internal array of classes for extending the clickmenu:
$this->extClassArray = $GLOBALS['TBE_MODULES_EXT']['xMOD_alt_clickmenu']['extendCMclasses'];

    // Traversing that array and setting files for inclusion:
if (is_array($this->extClassArray)) {
    foreach($this->extClassArray as $extClassConf) {
        if ($extClassConf['path'])    $this->include_once[]=$extClassConf['path'];
    }
}

The following code listings works in the same way. First, a list of class files to include is registered. Then in the second code listing the same array is traversed and each class is instantiated and a fixed function name is called for processing.

    // Setting class files to include:
if (is_array($TBE_MODULES_EXT['xMOD_db_new_content_el']['addElClasses'])) {
    $this->include_once = array_merge($this->include_once,$TBE_MODULES_EXT['xMOD_db_new_content_el']['addElClasses']);
}

    // PLUG-INS:
if (is_array($TBE_MODULES_EXT['xMOD_db_new_content_el']['addElClasses'])) {
    reset($TBE_MODULES_EXT['xMOD_db_new_content_el']['addElClasses']);
    while(list($class,$path)=each($TBE_MODULES_EXT['xMOD_db_new_content_el']['addElClasses'])) {
        $modObj = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance($class);
        $wizardItems = $modObj->proc($wizardItems);
    }
}

Creating hooks

You are encouraged to create hooks in your extensions if they seem meaningful. Typically someone would request a hook somewhere. Before you implement it, consider if it is the right place to put it etc. On the one hand we want to have many hooks but not more than needed. Redundant hooks or hooks which are implemented in the wrong context is just confusing. So put a little thought into it first, but be generous.

There are two main methods of calling a user defined function in TYPO3.

  • \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction() - The classic way. Takes a file/class/method reference as value and calls that function. The argument list is fixed to a parameter array and a parent object. So this is the limitation. The freedom is that the reference defines the function name to call. This method is mostly useful for small-scale hooks in the sources.
  • \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObject() - Create an object from a user defined file/class. The method called in the object is fixed by the hook, so this is the non-flexible part. But it is cleaner in other ways, in particular that you can even call many methods in the object and you can pass an arbitrary argument list which makes the API more beautiful. You can also define the objects to be singletons, instantiated only once in the global scope.

Here follows some examples.

Using TYPO3CMSCoreUtilityGeneralUtility::getUserObj()
    // Hook for pre-processing of the content for formmails:
if (is_array($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['sendFormmail-PreProcClass'])) {
    foreach($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['sendFormmail-PreProcClass'] as $_classRef) {
        $_procObj = \TYPO3\CMS\Core\Utility\GeneralUtility::getUserObj($_classRef);
        $EMAIL_VARS = $_procObj->sendFormmail_preProcessVariables($EMAIL_VARS, $this);
    }
}
Using with TYPO3CMSCoreUtilityGeneralUtility::callUserFunction()
    // Call post-processing function for constructor:
if (is_array($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['tslib_fe-PostProc'])) {
    $_params = array('pObj' => &$this);
    foreach($this->TYPO3_CONF_VARS['SC_OPTIONS']['tslib/class.tslib_fe.php']['tslib_fe-PostProc'] as $_funcRef) {
        \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction($_funcRef,$_params, $this);
    }
}

Extending classes (XCLASSes)

Introduction

XCLASSing is a mechanism in TYPO3 CMS to extend classes or overwrite methods from the Core or extensions with one's own code. This enables a developer to easily change a given functionality, if other options like hooks, signals or the extbase dependency injection mechanisms do not work or do not exist.

However there are several limitations.

Note

If you need a hook or a signal that does not exist, feel free to submit a feature request and - even better - a patch. Consult the TYPO3 Contribution Guide about how to do this.

How does it work?

In general every class instance in the Core and in extensions that stick to the recommended TYPO3 coding guidelines is created with the API call \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(). The methods takes care of singletons and also searches for existing XCLASSes. If there is an XCLASS registered for the specific class that should be instantiated, an instance of that XCLASS is returned instead of an instance of the original class.

Limitations

  • Using XCLASSes is risky: neither the core, nor extensions authors can guarantee that XCLASSes will not break if the underlying code changes (for example during upgrades). Be aware that your XCLASS can easily break and has to be maintained and fixed if the underlying code changes. If possible, you should use a hook instead of an XCLASS.
  • XCLASSes do not work for static classes, static methods or final classes.
  • There can be only one XCLASS per base class, but an XCLASS can be XCLASSed again. Be aware that such a construct is even more risky and definitely not advisable.
  • A small number of Core classes are required very early during bootstrap before configuration and other things are loaded. XCLASSing those classes will fail if they are singletons or might have unexpected side-effects.

Declaration

The $GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'] global array acts as a registry of overloaded (XCLASSed) classes.

The syntax is as follows and is commonly located in an extension's ext_localconf.php file:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects']['TYPO3\\CMS\\Backend\\Controller\\NewRecordController'] = array(
   'className' => 'Documentation\\Examples\\Xclass\\NewRecordController'
);

In this example, we declare that the \TYPO3\CMS\Backend\Controller\NewRecordController class will be overridden by the \Documentation\Examples\Xclass\NewRecordController class, the latter being part of the "examples" extension.

Note

In the above declaration, namespaced classes are entered without the leading backslash.

When XCLASSing a class that does not use namespaces, simply use that class' name in the declaration.

Note

To be compatible with older versions of TYPO3 CMS, you need to also add old-style XCLASS declarations. Please refer to older versions of this document for more information.

Coding practices

The recommended way of writing an XCLASS is to extend the original class and overwrite only the methods where a change is needed. This lowers the chances of the XCLASS breaking after a code update.

Tip

You're even safer if you can do your changes before or after the parent method and just call the latter with parent::.

The example below extends the new record wizard screen. It first calls the original method and then adds its own content:

class NewRecordController extends \TYPO3\CMS\Backend\Controller\NewRecordController {
   function regularNew() {
      parent::regularNew();
      $this->code .= $this->doc->section(
         $GLOBALS['LANG']->sL('LLL:EXT:examples/locallang.xml:help'),
         $GLOBALS['LANG']->sL('LLL:EXT:examples/locallang.xml:make_choice'),
         0,
         1
      );
   }
}

The result can be seen here:

Adding an element to the new record wizard

A help section is added at the bottom of the new record wizard.

JavaScript in TYPO3

Some thid-party JavaScript libraries are packaged with the TYPO3 source code. The TYPO3 backend itself relies on quite a lot of JavaScript to do its job. The topic of this chapter is to present how to use JavaScript properly with TYPO3, in particular in the backend. It presents the most important APIs in that regard.

Attention

Since TYPO3 7, the TYPO3 backend relies primarily on bootstrap and jQuery. Since TYPO3 7.4, Prototype and Scriptaculous are not packed with the Core anymore. If you need them for your projects, you need to take care of shipping them yourself, preferable by usage of RequireJS. Since TYPO3 8, ExtJS will be removed step by step, the most parts are now ExtJS free and replaces with mostly pure JavaScript components.

Contents:

AJAX in the TYPO3 Backend

In TYPO3 4.2 a new model for writing AJAX code in the TYPO3 Backend was introduced. Although there were some parts in the TYPO3 Backend that used AJAX already, they are now unified into a single interface that handles errors and dispatches the different calls to their final locations. This way it is ensured that e.g. a BE user is logged in and all TYPO3 variables are loaded.

The whole architecture builds on top of successful techniques developers already know. It's a mixture between the eID concept from the TYPO3 Frontend, the hooking idea we know from other places in TYPO3, piped through the single entrypoint file index.php?ajaxID=foobar that creates a PHP AJAX object to see if an error occurred or not. If something went wrong, the X-JSON header is set to false and the client-side AjaxRequestHandler will know that there is an error.

In-depth presentation
Client-Side programming

Note

This paragraph describes obsolete processes using Prototype and Scriptaculous

On the client-side we are using the Prototype JS library (located in typo3/contrib/prototype/prototype.js). If you have used it already, you know that you can make AJAX calls with AJAX.Request, AJAX.Updater and AJAX.PeriodicalUpdater. We extended the library and hooked in these objects, or better: in the callbacks users can define. If an AJAX request is made to our server-side component (typo3/ajax.php), everything developers need to do is to call this URL and add a unique, already registered parameter for their ajaxID. Their defined "onComplete" and "onSuccess" are only rendered if the X-JSON header is set to true by the server-side script. If the X-JSON header is set to false, the Responder checks if there is a callback function named "onT3Error" and executes it instead of the "onComplete" method. If the "onT3Error" method is not defined, the default TYPO3 error handler will be displaying the error in the TYPO3 backend. If the X-JSON header is set to false, the "onSuccess" callback will not be executed as well as but an error message will be shown in the notification area. This behaviour is done automatically with every AJAX call to "ajax.php" made through Prototype's AJAX classes. This responder is also only active if "typo3/js/common.js" is added to the base script.

Since TYPO3 4.4, ExtJS is used instead for AJAX calls. TYPO3 even supports usage of Ext.Direct.

Server-side programming

If you look into "typo3/ajax.php", it is only a small dispatcher script. It checks for an ajaxID in the $TYPO3_CONF_VARS['BE']['AJAX'] array and tries to execute the function pointer. The function has two parameters, where the first (an array) is not used yet. The second parameter is the TYPO3 AJAX Object (located in typo3/classes/typo3ajax.php) that is used to add the content that should be returned as the server-response to the Javascript part, or the error message that should be displayed. The X-JSON header will be set depending on whether setError() was called on this AJAX object. You can also specify if the object should return the result in a valid XML object tree, as text/html (default) or as a JSON object, see below.

The "ajaxID" is a unique identifier and can be used to override the existing AJAX calls. Therefore you can extend existing AJAX calls that already exist in the backend by redirecting it to your function. But be aware of the side-effects of this feature: Other extensions could overwrite this function as well (similar problem as with XCLASSing or single inheritance in OOP).

Also, for every TYPO3 request, you will now have a TYPO3_REQUESTTYPE variable that can be used for bitwise comparison. You can now check if you're in Backend or Frontend or in an valid AJAX request with

if (TYPO3_REQUESTTYPE && TYPO3_REQUESTTYPE_AJAX)

to see if you're calling through the new AJAX interface.

Different Content Formats

As with every AJAX response you can send it in different response formats.

  • text/html - plain text
  • text/xml - strict XML formatting
  • application/json - JSON notation

You can also specify the contentFormat in the AJAX object like this:

$ajaxObj->setContentFormat('json');

For the keyword you can choose between "plain" (default), "xml" and "json", "jsonbody" and "jsonhead".

Here are the specifics for each format.

Plain Text

The content array in the backend will be concatenated and returned uninterpreted.

The result will be available in the transport object as a string through "xhr.responseText".

XML

The content needs to be valid XML and will be available in javascript as "xhr.responseXML".

JSON

The content is transformed to JSON using PHP's built-in functions and is then available in JSON notation through the second parameter in the onComplete / onSuccess methods, and additionally in the "responseText" part of the transport object ("xhr.responseText"). If it is set to "jsonbody", only the latter variable is filled, if "jsonhead" is set, it is only in the second parameter. This is useful to save traffic and you can use it with whatever format you like.

Developing with AJAX in the TYPO3 Backend

This section describes how to correctly make AJAX calls in the TYPO3 CMS BE.

Note

This section was fully updated for TYPO3 CMS 6.2. For older versions, please refer to the related version of this manual.

How to choose the right ajaxID

An AJAX call is represented by a system-wide identifier which is used to register the handler that will receive the call. The ajaxID consists of two parts, the class name and the action name, delimited by "::" (<class>::<action>).

Although it looks like a static function call, it is really just a key. Developers must stick to a common naming scheme as described above, to avoid using identical names in different extensions.

Some good examples for an ajaxID:

  • SC_alt_db_navframe::expandCollapse
  • BackendLogin::refreshLogin
  • tx_myext_module1::executeSomething

Some bad examples for an ajaxID (the first part is too generic or the identifier contains a single part):

  • search::findRecordByTitle
  • core::reloadReferences
  • inline::processAjaxRequest
  • updateRecordList
Server-Side

Since TYPO3 CMS 6.2, the registration is done via an API, which provides CSRF protection on the AJAX call and an automatic registration of the AJAX call URL (in typo3conf/opendocs/ext_tables.php):

\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::registerAjaxHandler (
   'OpendocsController::renderMenu',
   'TYPO3\\CMS\\Opendocs\\Controller\\OpendocsController->renderAjax'
);

This is how the "opendocs" system extension registers the AJAX call to render the open documents menu in the top toolbar. The first argument is the ajaxID (as described above) and the second argument is a pointer to a class and method. This code must be located in an extension's ext_tables.php file.

The target method receives an array of parameters (depending on the call context) and a backreference to the general AJAX handler (\TYPO3\CMS\Core\Http\AjaxRequestHandler). The API of this object is used to set the content to output or to write an error message if something went wrong.

In the above example, here's how the handling method looks like (in typo3conf/opendocs/Classes/Controller/OpendocsController.php):

/**
 * Renders the menu so that it can be returned as response to an AJAX call
 *
 * @param array $params Array of parameters from the AJAX interface, currently unused
 * @param \TYPO3\CMS\Core\Http\AjaxRequestHandler $ajaxObj Object of type AjaxRequestHandler
 * @return void
 */
public function renderAjax($params = array(), \TYPO3\CMS\Core\Http\AjaxRequestHandler &$ajaxObj = NULL) {
   $menuContent = $this->renderMenu();

   // addContent('key', 'content to add')
   // 'key' = the new content key where the content should be added in the content array
   $ajaxObj->addContent('opendocsMenu', $menuContent);

   // the new content, "$menuContent" can now be referenced like this:
   // $ajaxObj->getContent('opendocsMenu');
}
Client-part

The API mentioned above registers a corresponding AJAX URL in the global TYPO3.settings.ajaxUrls JavaScript array.

Whatever library you use, this URL can easily be accessed by using the registration key.

var ajaxUrl = TYPO3.settings.ajaxUrls['<registration key>'];

Here is the client-side part corresponding to the above example (an extract of typo3/sysext/backend/Resources/Public/JavaScript/shortcutmenu.js):

var del = new Ajax.Request(TYPO3.settings.ajaxUrls['ShortcutMenu::delete'], {
   parameters : '&shortcutId=' + shortcutId,
   onComplete : this.reRenderMenu.bind(this)
});

RequireJS in the TYPO3 Backend

Since TYPO3 CMS 7 it is possible to use RequireJS in the backend. A short explanation: RequireJS enables developers to have some kind of dependency handling for JavaScript. The JavaScript is written as so-called "Asynchronous Module Definition" (AMD). Some libraries delivered with TYPO3 are written as modules.

Credits

The complete documentation about RequireJS was inspired by the blog post of Andreas Fernandez.

Use RequireJS in your own extension

To be able to use RequireJS at all, some prerequisites must be fulfilled:

  • Your extension must have a Resources/Public/JavaScript directory. That directory is used for autoloading the modules stored in your extension.
  • Think about what's the purpose of the module. You can only write one module per file (anything else is bad practice anyway)
  • Each module has a namespace and a module name. The namespace is TYPO3/CMS/<EXTKEY>, <EXTKEY> is your extension key in UpperCamelCase, e.g. foo_bar = FooBar
  • The namespace maps automatic to your Resources/Public/JavaScript directory
  • The filename is the modulename + .js
  • complete example: TYPO3/CMS/FooBar/MyMagicModule is resided in EXT:foo_bar/Resources/Public/JavaScript/MyMagicModule.js

Every AMD is wrapped in the same construct:

define([], function() {
   // your module logic here
});

This is the "container" of the module. It holds the module logic and takes care of dependencies.

Core-wise, TYPO3 itself defines in its own modules an object holding the module logic, namely parameters and methods. The object has the same name as the module, in our case "MyMagicModule":

define([], function() {
   var MyMagicModule = {
      foo: 'bar'
   };

   MyMagicModule.init = function() {
     // do init stuff
   };

   // To let the module be a dependency of another module, we return our object
   return MyMagicModule;
});
Dependency Handling

Let us try to explain the dependency handling with the most used JS lib: jQuery

To prevent the "$ is undefined" error, you should use the dependency handling of RequireJS. To get jQuery working in your code, use the following line:

define(['jquery'], function($) {
   // in this callback $ can be used
});

The code above is very easy to understand:

  1. every dependency in the array of the first argument
  2. will be injected in the callback function at the same position

Let us combine jQuery with out own module from the Extension example

define(['jquery', 'TYPO3/CMS/FooBar/MyMagicModule'], function($, MyMagicModule) {
   // $ is our jQuery object
   // MyMagicModule is the object, which is returned from our own module
});
Loading your own or other RequireJS modules

In case you use the ready event, you may wonder how to use the module. Answer: it depends! If you use Fluid's f:be.pageRenderer view helper, simply add the argument includeRequireJsModules:

<f:be.pageRenderer includeRequireJsModules="{
   0:'TYPO3/CMS/FooBar/Wisdom'
}" />

However, if you don't use Fluid you may use PageRenderer:

$pageRenderer = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\Page\PageRenderer::class);
$pageRenderer->loadRequireJsModule('TYPO3/CMS/FooBar/MyMagicModule');

Bonus: loadRequireJsModule takes a second argument $callBackFunction which is executed right after the module was loaded. The callback function must be wrapped within function() {}:

$pageRenderer->loadRequireJsModule(
   'TYPO3/CMS/FooBar/MyMagicModule',
   'function() { console.log("Loaded own module."); }'
);
Shim library to use it as own RequireJS modules

Not all javascript libraries are compatible with RequireJS. In the rarest cases, you can adjust the library code to be AMD or UMD compatible. So you need to configure RequireJS to accept the library.

In RequireJS you can use requirejs.config({}) to shim a library. In TYPO3 the RequireJS config will be defined in the PageRenderer:

$pageRenderer = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\Page\PageRenderer::class);
$pageRenderer->addRequireJsConfiguration(
   [
      'paths' => [
         'jquery' => 'sysext/core/Resources/Public/JavaScript/Contrib/jquery/',
         'plupload' => '../typo3conf/ext/your_extension/node_modules/plupload/js/plupload.full.min',
      ],
      'shim' => [
         'deps' => ['jquery'],
         'plupload' => ['exports' => 'plupload'],
      ],
   ]
);

In this example we configure RequireJS to use plupload. The only dependency is jquery. We already have jquery in the TYPO3 core extension.

After the shim and export of plupload it is usable in the dependency handling:

define([
   'jquery',
   'plupload'
], function($, plupload) {
   'use strict';
});

Soft References

"Soft References" are references to database elements, files, email addresses, URLs etc. which are found inside text fields. The <link[page_id]> tag found in bodytext fields is an example of this.

The Soft Reference parsers are used by the system to find these references and process them accordingly in import/export actions and copy operations. Also, the soft references are used by integrity checking functions.

Default soft reference parsers

The TYPO3\CMS\Core\Database\SoftReferenceIndex class contains generic parsers for the most well-known types which are default for most TYPO3 installations. This is the list of the possible keys:

substitute

softref key

substitute

Description

A full field value targeted for manual substitution (for import /export features)

notify

softref key

notify

Description

Just report if a value is found, nothing more.

images

softref key

images

Description

HTML <img> tags for RTE images / images from upload/.

ext_fileref

softref key

ext_fileref

Description

Relative file reference, prefixed EXT:[extkey]/ - for finding extension dependencies.

email

softref key

email

Description

Email highlight.

url

softref key

url

Description

URL highlights (with a scheme).

The default set up is found in typo3/sysext/core/Configuration/DefaultConfiguration.php:

'SC_OPTIONS' => array(
        'GLOBAL' => array(
                'softRefParser' => array(
                        'substitute' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                        'notify' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                        'images' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                        'typolink' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                        'typolink_tag' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                        'ext_fileref' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                        'email' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                        'url' => 'TYPO3\\CMS\\Core\\Database\\SoftReferenceIndex',
                ),
                ...
        ),
        ...
),

User-defined soft reference parsers

Soft References can also be user-defined. It is easy to set them up by simply adding new keys in $TYPO3_CONF_VARS['SC_OPTIONS']['GLOBAL']['softRefParser']. Use key names based on the extension you put it in, e.g. tx_myextensionkey.

The class containing the soft reference parser must have a function named findRef. Please refer to class TYPO3\CMS\Core\Database\SoftReferenceIndex for API usage and expected return values.

Session Storage Framework

As of version 8.6, TYPO3 comes with the option to choose between different storages for both frontend end backend user sessions (called session backends). Previously, all sessions were stored in the database in the tables fe_sessions, fe_session_data and be_sessions respectively.

The Core ships two session backends by default: - Database storage - Redis storage

By default user sessions are still stored in the database using the database storage backend, but the former table fe_session_data is obsolete and has therefore been removed.

Database storage backend

The database storage backend only requires two configuration options: The table name (table option) and whether anonymous sessions (has_anonymous option) may be stored.

The default configuration used for sessions by the Core is:

'SYS' => [
    'session' => [
        'BE' => [
            'backend' => \TYPO3\CMS\Core\Session\Backend\DatabaseSessionBackend::class,
            'options' => [
                'table' => 'be_sessions'
            ]
        ],
        'FE' => [
            'backend' => \TYPO3\CMS\Core\Session\Backend\DatabaseSessionBackend::class,
            'options' => [
                'table' => 'fe_sessions',
                'has_anonymous' => true,
            ]
        ]
    ],
],

Using Redis to store sessions

TYPO3 also comes with the possibility to store sessions in a Redis key-value database.

Note

This requires a running Redis instance (refer to the Redis documentation for help on this) and the PHP extension "redis" to be installed.

The Redis session storage can be configured with LocalConfiguration.php in the SYS entry:

A sample configuration will look like this:

'SYS' => [
    'session' => [
        'BE' => [
            'backend' => \TYPO3\CMS\Core\Session\Backend\RedisSessionBackend::class,
            'options' => [
                'hostname' => 'redis.myhost.example',
                'password' => 'passw0rd',
                'database' => 0,
                'port' => 6379
            ]
        ],
        'FE' => [
            'backend' => \TYPO3\CMS\Core\Session\Backend\RedisSessionBackend::class,
            'options' => [
                'hostname' => 'redis.myhost.example',
                'password' => 'passw0rd',
                'database' => 0,
                'port' => 6379
            ]
        ],
    ],
],

The available options are:

  • hostname: Name of the server the redis database service is running on. Default: 127.0.0.1
  • port: Port number the redis database service is listening to. Default: 6379
  • database: The redis database number to use. Default: 0
  • password: The password to use when connecting to the specified database. Optional.

Tip

If a Redis instance is running on the same machi