TYPO3 Explained

Previous Key:doc_core_api
Version:latest (9-dev)
Language:en
Description:Main TYPO3 core documentation
Keywords:forEditors, forBeginners, forDevelopers, forAdmins, forAdvanced, security
Copyright:2000-2018
Author:Documentation Team
Email:documentation@typo3.org
License:Open Publication License available from www.opencontent.org/openpub/
Rendered:2018-10-17 18:34

The content of this document is related to TYPO3 CMS, a GNU/GPL CMS/Framework available from www.typo3.org

Official Documentation

This document is included as part of the official TYPO3 documentation. It has been approved by the TYPO3 Documentation Team following a peer- review process. The reader should expect the information in this document to be accurate - please report discrepancies to the Documentation Team (documentation@typo3.org). Official documents are kept up-to-date to the best of the Documentation Team's abilities.

Core Manual

This document is a Core Manual. Core Manuals address the built in functionality of TYPO3 CMS and are designed to provide the reader with in- depth information. Each Core Manual addresses a particular process or function and how it is implemented within the TYPO3 source code. These may include information on available APIs, specific configuration options, etc.

Core Manuals are written as reference manuals. The reader should rely on the Table of Contents to identify what particular section will best address the task at hand.

Sitemap

Introduction

About this document

TYPO3 is known for its extensibility. To really benefit from this power, a complete documentation is needed: "TYPO3 Explained" aims to provide such information to everyone. Not all areas are covered with the same amount of detail, but at least some pointers are provided.

The document does not contain any significant information about the frontend of TYPO3. Creating templates, setting up TypoScript objects etc. is not the scope of the document, it addresses the backend and management part of the core only.

The TYPO3 Documentation Team hopes that this document will form a complete picture of the TYPO3 Core architecture. It will hopefully be the knowledge base of choice in your work with TYPO3.

Intended audience

This document is intended to be a reference for TYPO3 CMS developers and partially for integrators. The document explains all major parts of TYPO3 and the concepts. Some chapters presumes knowledge in the technical end: PHP, MySQL, Unix etc, depending on the specific chapter.

The goal is to take you "under the hood" of TYPO3 CMS. To make the principles and opportunities clear and less mysterious. To educate you to help continue the development of TYPO3 along the already established lines so we will have a consistent CMS application in a future as well. And hopefully this teaching on the deep technical level will enable you to educate others higher up in the "hierarchy". Please consider that as well!

Code examples

Many of the code examples found in this document come from the TYPO3 Core itself.

Quite a few others come from the "examples" and the "styleguide" extension. You can install them if you want to try out these examples yourself and use them as a basis for your own stuff.

Feedback and Fixing

If you find a bug in this manual, please be so kind as to check the online version. From there you can hit the "Edit me on GitHub" button in the top right corner and submit a pull request via GitHub. Alternatively you can just file an issue using the bug tracker.

Maintaining high quality documentation requires time and effort and the TYPO3 Documentation Team always appreciates support.

If you want to support us, please join the slack channel #typo3-documentation on Slack. Visit forger to gain access to Slack.

And finally, as a last resort, you can get in touch with the documentation team by mail.

Credits

This manual was originally written by Kasper Skårhøj. It was further maintained, refreshed and expanded by François Suter.

A first version of the security chapter has been done Ekkehard Guembel and Michael Hirdes and we would like to thank them for this. Further thanks to the TYPO3 Security Team for their work for the TYPO3 project. A special thank goes to Stefan Esser for his books and articles on PHP security, Jochen Weiland for an initial foundation and Michael Schams for compiling the content of the security chapter and coordinating the collaboration between several teams. He managed the whole process of getting the Security Guide to a high quality.

Dedication

I want to dedicate this document to the people in the TYPO3 community who have the discipline to do the boring job of writing documentation for their extensions or contribute to the TYPO3 documentation in general. It's great to have good coders, but it's even more important to have coders with character to carry their work through till the end - even when it means spending days writing good documents. Go for completeness!

- kasper

Further Documentation

This manual covers many different APIs of the TYPO3 CMS Core, but some other documents exist which cover more specific aspects.

TCA Reference

TCA is the backbone of database tables displayed in the backend, it configures how data is stored if editing records in the backend, how fields are displayed, relations to other tables and much more. It is a huge array loaded in almost all access contexts.

A detailed insight on TCA is documented in the TCA Reference. Next to a small introduction, the document forms a complete reference of all different TCA options, with bells and whistles. The document is a must-read for Developers, partially for Integrators, and is often used as a reference book on a daily basis.

TypoScript Reference

TypoScript - or more precisely Frontend TypoScript - is used in TYPO3 to steer the frontend rendering (the actual website) of a TYPO3 instance. It is based on the TypoScript syntax which is outlined in detail here in this document.

Frontend TypoScript is very powerful and has been the backbone of frontend rendering ever since. However, with the rise of the Fluid templating engine, many parts of Frontend TypoScript are much less often used. Nowadays, TypoScript in real life projects is often not much more than a way to set a series of options for plugins, to set some global config options, and to act as a simple pre processor between database data and Fluid templates.

Still, the TypoScript Reference reference document that goes deep into the incredible power of Frontent TypoScript is daily bread for Integrators.

TSconfig Reference

While Frontend TypoScript is used to steer the rendering of the frontend, TSconfig is used to configure backend details for backend users. Using TSconfig it is possible to enable or disable certain views, change the editing interfaces, and much more. All that without coding a single line of PHP. TSconfig can be set on a page (Page TSconfig), as well as a user / group (User TSconfig) basis.

TSconfig uses the same syntax as Frontend TypoScript, the syntax is outlined in detail here in this document. Other than that, TSconfig and Frontend TypoScript don't have much more in common - they consist of entirely different properties.

A full reference of properties as well as an introduction to explain details configuration usage, API and load orders can be found in the TSconfig Reference document. While Developers should have an eye on this document, it is mostly used as a reference for Integrators who make life as easy as possible for backend users.

System Overview

For most people TYPO3 is equivalent to a CMS providing a backend for management of the content and a frontend engine for website display. However the core of TYPO3 is natively designed to be a general purpose framework for management of database content. The core of TYPO3 CMS delivers a set of principles for storage of this content, user access management, editing of the content, uploading and managing files, etc. These principles are expressed as an API (Application Programming Interface) for use in extensions which ultimately add most of the real functionality.

Main TYPO3 CMS core architecture

So the core is the skeleton and extensions are the muscles, fibers and skin making a full bodied CMS. In this document I cut to the bone and provide a detailed look at the core of TYPO3 CMS including the API available to the outside. This is supposed to be the final technical reference apart from source code itself which is - of course - the ultimate documentation.

A basic installation

To follow this document, it might help to have a totally trimmed down installation of TYPO3 CMS with only the core and the required system extensions at hand.

The installation process is covered in the Installation and Upgrade Guide. You should perform the basic installation steps and not install any distribution. This will give you the "lightest" possible version of TYPO3 CMS.

Log into your basic installation and move to the ADMIN TOOLS > Extensions module. You will see all extensions which are loaded by default. Required extensions are not only loaded by default, they have no "Activate/Deactivate" button, too.

The Extension Manager with a bare bones installation

The most important thing to note for now is that everything is an extension in TYPO3 CMS. Even the most basic functions are packaged in a system extension called "core".

Extension Architecture

Introduction

TYPO3 CMS is entirely built around the concept of extensions. The Core itself is entirely comprised of extensions, called "system extensions". Some are required and will always be activated. Others can be activated or deactivated at will.

Many more extensions - developed by the community - are available in the TYPO3 Extension Repository (TER).

Yet more extensions are not officially published and are available straight from source code repositories like GitHub.

It is also possible to set up TYPO3 CMS using Composer. This opens the possibility of including any library published on Packagist.

TYPO3 can be extended in nearly any direction without loosing backwards compatibility. The Extension API provides a powerful framework for easily adding, removing, installing and developing such extensions to TYPO3.

"Extensions" is a general term in TYPO3 which covers many kinds of additions to TYPO3. The main types are:

  • Plugins which play a role on the website itself, e.g. a discussion board, guestbook, shop, etc. Therefore plugins are content elements, that can be placed on a page like a text element or an image.
  • Modules are backend applications which have their own entry in the main menu. They require a backend login and work inside the framework of the backend. We might also call something a module if it exploits any connectivity of an existing module, that is if it simply adds itself to the function menu of existing modules. A module is an extension in the backend.
  • Distributions are fully packaged TYPO3 CMS web installations, complete with files, templates, extensions, etc. Distributions are covered in their own chapter.

Extensions and the Core

Extensions are designed in a way so that extensions can supplement the core seamlessly. This means that a TYPO3 system will appear as "a whole" while actually being composed of the core application and a set of extensions providing various features. This philosophy allows TYPO3 to be developed by many individuals without loosing fine control since each developer will have a special area (typically a system extension) of responsibility which is effectively encapsulated.

So, at one end of the spectrum system extensions make up what is known as "TYPO3" to the outside world. At the other end, extensions can be entirely specific to a given project and contain only files and functionality related to a single implementation.

Notable system extensions

This section describes the main system extensions, their use and what main resources and libraries they contain. The system extensions are located in directory typo3/sysext.

core
As its name implies, this extension is crucial to the working of TYPO3 CMS. It defines the main database tables (BE users, BE groups, pages and all the "sys_*" tables. It also contains the default global configuration (in typo3/sysext/core/Configuration/DefaultConfiguration.php). Last but not least, it delivers a huge number of base PHP classes, far too many to describe here.
backend
This system extension provides all that is necessary to run the TYPO3 CMS backend. This means quite a few PHP classes, a lot of controllers and Fluid templates.
frontend
This system extension contains all the tools for performing rendering in the frontend, i.e. the actual web site. It is mostly comprised of PHP classes, in particular those in typo3/sysext/frontend/Classes/ContentObject, which are used for rendering the various content objects (one class per object type, plus a number of base and utility classes).
extbase
Extbase is a MVC framework, with the "View" part being actually system extension "fluid". Not all of the TYPO3 CMS backend is written in Extbase, but some modules are.
fluid
Fluid is a templating engine. It forms the "View" part of the MVC framework. The templating engine itself is provided as "fluid standalone" which can be used in other frameworks or as a standalone templating engine. This system extension provides a number of classes and many View Helpers (in typo3/sysext/fluid/Classes/ViewHelpers), which extend the basic templating features of standalone Fluid. Fluid can be used in conjunction with Extbase (where it is the default template engine), but also in non-extbase extensions.
install
This system extension is the package containing the TYPO3 CMS Install Tool.

Extension Management

Extensions are managed from the Extension Manager inside TYPO3 by "admin" users. The module is located at ADMIN TOOLS > Extensions and offers a menu with options to see loaded extensions (those that are installed or activated), available extensions on the server and the possibility to import extensions from online resources, typically the TER (TYPO3 Extension Repository) located at typo3.org.

The Extension Manager

Interface of the Extension Manager showing all available extensions.

The interface is really easy to use. You just click the +/- icon to the left of an extension in order to install it and follow the instructions.

Installing extensions

There are only two (possibly three) steps involved in using extensions with TYPO3:

  1. You must import it.

    This simply means to copy the extensions files into the correct directory into. More commonly you import an extension directly from the online TYPO3 Extension Repository (TER) using the Extension Manager. When an extension is found located in one of the extension locations, it is available to the system.

    The Extension Manager (EM) should take care of this process, including updates to newer versions if needed.

    Another convenient way to install extensions is offered by using composer (https://getcomposer.org/). Besides TYPO3 CMS itself the TYPO3 composer repository includes all TYPO3 Extensions that are uploaded to TER. Read more on https://composer.typo3.org/ .

  2. You must load it.

    An extension is loaded only if it is listed in the PackageStates.php file. Extensions are loaded in the order they appear in this list.

    An enabled extension is always global to the TYPO3 Installation - you cannot disable an extension from being loaded in a particular branch of the page tree. The EM takes care of enabling extensions. It's highly recommended that the EM is doing this, because the EM will make sure the priorities, dependencies and conflicts are managed according to the extension characteristics, including clearing of the cache-files if any.

  3. You might be able to configure it.

    Certain extensions may allow you to configure some settings. ADMIN TOOLS > Settings > Extension configuration provides an interface to configure extensions that provide configuration settings. Any settings - if present - configured for an extension are available as an array in the variable $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS'][extensionKey] and thus reside in typo3conf/LocalConfiguration.php.

Loaded extensions are registered in a global variable, $GLOBALS['TYPO3_LOADED_EXT'], available in both frontend and backend of TYPO3.

This is how the data structure for an extension in this array looks:

$GLOBALS['TYPO3_LOADED_EXT'][extension key] = array(
        "type" =>                S, G, L for system, global or local type of availability.
        "siteRelPath" => Path of extension dir relative to :php:`\TYPO3\CMS\Core\Core\Environment::getPublicPath()`
                                e.g. "typo3/ext/my_ext/" or "typo3conf/ext/my_ext/"
        "typo3RelPath" => Path of extension dir relative to the "typo3/" admin folder
                                e.g. "ext/my_ext/" or "../typo3conf/ext/my_ext/"
        "ext_localconf" => Contains absolute path to 'ext_localconf.php' file if present
        "ext_tables" => [same]
        "ext_tables_sql" => [same]
        "ext_tables_static+adt.sql" => [same]
        "ext_typoscript_constants.txt" => [same]
        "ext_typoscript_setup.txt" => [same]
        "ext_typoscript_editorcfg.txt" => [same]
)

The order of the registered extensions in this array corresponds to the order they were listed in PackageStates.php.

Package Manager

On a low level, the list of loaded extensions is written to the file typo3conf/PackageStates.php. PHP class \TYPO3\CMS\Core\Package\PackageManager manages this file as part of the "Package management", it is part of the core extension, but mostly used by the Extension Manager as the low level work horse.

The typo3conf/PackageStates.php file contains a list of all active packages, example:

<?php
# PackageStates.php

# This file is maintained by TYPO3's package management. Although you can edit it
# manually, you should rather use the Extension Manager for maintaining packages.
# This file will be regenerated automatically if it doesn't exist. Deleting this file
# should, however, never become necessary if you use the package commands.

return [
    'packages' => [
        'core' => [
            'packagePath' => 'typo3/sysext/core/',
        ],
        'extbase' => [
            'packagePath' => 'typo3/sysext/extbase/',
        ],
        'fluid' => [
            'packagePath' => 'typo3/sysext/fluid/',
        ],
        'install' => [
            'packagePath' => 'typo3/sysext/install/',
        ],
        'frontend' => [
            'packagePath' => 'typo3/sysext/frontend/',
        ],
        // ...
    ],
    'version' => 5,
];

Warning

You should not edit this file manually, unless you know exactly what you are doing. It is rather easy to get this wrong, for instance the order of the list is important and should be handled with care.

Files and locations

Files

An extension consists of:

  1. A directory named by the extension key (which is a worldwide unique identification string for the extension), usually located in typo3conf/ext for local extensions, or typo3/sysext for system extensions.
  2. Standard files with reserved names for configuration related to TYPO3 (of which most are optional, see list below)
  3. Any number of additional files for the extension functionality itself.

Reserved file names

This lists special files within an extension that have a special meaning by convention. If put at the according places, TYPO3 will find them and use for specific functionality. For example, if a svg logo of your extension is placed at Resources/Public/Icons/Extension.svg, the Extension Manager will show that image.

Nearly none of these are required, but for example you can not have a TYPO3 extension recognized by TYPO3 without the ext_emconf.php file, etc. You can read more details like that in the table below.

In general, do not introduce your own files in the root directory of extensions with the name prefix ext_.

Filename Description
ext_emconf.php

Definition of extension properties. This is the only mandatory file in the extension. It describes the extension for the rest of TYPO3.

Name, category, status etc. used by the Extension Manager. The content of this file is described in more details below. Note that it is auto-written by Extension Manager when extensions are imported from the repository.

Note

If this file is not present, the Extension Manager will not find the extension.

ext_localconf.php

Addition to LocalConfiguration.php which is included if found. Should contain additional configuration of $GLOBALS['TYPO3_CONF_VARS'].

This file contains hook definitions and plugin configuration. It must not contain a PHP encoding declaration.

All ext_localconf.php files of loaded extensions are included right after the files typo3conf/LocalConfiguration.php and typo3conf/AdditionalConfiguration.php during TYPO3 bootstrap.

Pay attention to the rules for the contents of these files. For more details, see the section below.

ext_tables.php

Included if found. Contains extensions of existing tables, declaration of backend modules, etc. All code in such files is included after all the default definitions provided by the Core and loaded after ext_localconf.php files during TYPO3 bootstrap.

Pay attention to the rules for the contents of these files. For more details, see the section below.

Note

In old TYPO3 core versions, this file contained additions to the global $GLOBALS['TCA'] array. This changed since core version 6.2 to allow effective caching:

TCA definition of new database tables must be done entirely in Configuration/TCA/<table name>.php. These files are expected to contain the full TCA of the given table (as an array) and simply return it (with a return statement).

Customizations of existing tables must be done entirely in Configuration/TCA/Overrides/<table name>.php.

ext_tables.sql

SQL definition of database tables.

This file should contain a table-structure dump of the tables used by the extension. It is used for evaluation of the database structure and is therefore important to check and update the database when an extension is enabled.

If you add additional fields (or depend on certain fields) to existing tables you can also put them here. In that case insert a CREATE TABLE structure for that table, but remove all lines except the ones defining the fields you need, here is an example adding a column to the pages table:

CREATE TABLE pages (
    tx_myext_field int(11) DEFAULT '0' NOT NULL,
);

TYPO3 will merge this table definition to the existing table definition when comparing expected and actual table definitions. Partial definitions can also contain indexes and other directives. They can also change existing table fields though that is not recommended, because it may create problems with the TYPO3 core and/or other extensions.

The ext_tables.sql file may not necessarily be "dumpable" directly to MySQL (because of the semi-complete table definitions allowed defining only required fields). But the Extension Manager or Install Tool can handle this. The only very important thing is that the syntax of the content is exactly like MySQL made it so that the parsing and analysis of the file is done correctly by the Extension Manager.

TYPO3 parses ext_tables.sql files. TYPO3 expects that all table definitions in this file look like the ones produced by the mysqldump utility. Incorrect definitions may not be recognized by the TYPO3 SQL parser or may lead to MySQL errors, when TYPO3 tries to apply them. If TYPO3 is not running on MySQL or directly compatible other DBMS like MariaDB, the system will parse the file towards the target DBMS like PostgreSQL.

ext_tables_static+adt.sql

Static SQL tables and their data.

If the extension requires static data you can dump it into a sql-file by this name. Example for dumping mysql data from bash (being in the extension directory):

mysqldump --add-drop-table \
          --password=[password] [database name] \
          [tablename]  > ./ext_tables_static.sql

--add-drop-table will make sure to include a DROP TABLE statement so any data is inserted in a fresh table.

You can also drop the table content using the Extension Manager in the backend.

Note

The table structure of static tables needs to be in the ext_tables.sql file as well - otherwise an installed static table will be reported as being in excess in the Install Tool.

Warning

Static data is not meant to be extended by other extensions. On re-import all extended fields and data is lost due to DROP TABLE statements.

ext_typoscript_constants.typoscript

Preset TypoScript constants. Will be included in the constants section of all TypoScript templates.

Warning

Use such a file if you absolutely need to load some TS (because you would get serious errors without it). Otherwise static templates or usage of the Extension Management API of class TYPO3\CMS\Core\Utility\ExtensionManagementUtility are preferred.

ext_typoscript_setup.typoscript

Preset TypoScript setup. Will be included in the setup section of all TypoScript templates.

Warning

Use such a file if you absolutely need to load some TS (because you would get serious errors without it). Otherwise static templates or usage of the Extension Management API of class TYPO3\CMS\Core\Utility\ExtensionManagementUtility are preferred.

ext_conf_template.txt

Extension Configuration template.

Configuration code in TypoScript syntax setting up a series of values which can be configured for the extension in the Install Tool. Read more about the file format here.

If this file is present 'Settings' of the Install Tool provides you with an interface for editing the configuration values defined in the file. The result is written as an array to LocalConfiguration.php in the variable $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS'][*extension_key* ]

Configuration/Backend/Routes.php and Configuration/Backend/AjaxRoutes.php Registry of backend routes. Extensions that add backend modules must register their routes here to be correctly linkable in the backend. The file must return an array with routing details. See core extensions like backend for examples.
Resources/Public/Icons/Extension.svg

Extension icon. If exists, this icon is displayed in the Extension Manager. Preferred is using an SVG file, Extension icon will look nicer when provided as vector graphics (SVG) rather than bitmaps (GIF or PNG).

18x16 GIF, PNG or SVG icon for the extension.

class.ext_update.php See section class.ext_update.php in chapter Update wizards.

Reserved folders

In the early days, every extension baked it own bread when it came to file locations of PHP classes, public web resources and templates.

With the rise of Extbase, a generally accepted structure for file locations inside extensions has been established. If extension authors stick to this, the system helps in various ways. For instance, if putting PHP classes into the Classes/ folder and naming classes accordingly, the system will be able to autoload these without further action from the developer.

Extension kickstarters like the Extension Builder extension will create the correct structure for you.

It is described below:

Classes
Contains all PHP classes. One class per file. Should have sub folders like Controller/, Domain/, Service/ or View/. For more details on class file namings an PHP namespaces, see chapter namespaces.
Classes/Controller
Contains MVC Controller classes.
Classes/Domain/Model
Contains MVC Domain model classes.
Classes/Domain/Repository
Contains data repository classes.
Classes/ViewHelpers
Helper classes used in (Fluid) views.
Configuration
General configuration folder. Some of the sub directories in here like TCA and Backend have special meaning and files in there are automatically included during TYPO3 bootstrap.
Configuration/Backend/
Contains backend routing configurations. See files description of Routes.php and AjaxRoutes.php above.
Configuration/TCA
One file per database table, using the name of the table for the file, plus ".php". Only for new tables.
Configuration/TCA/Overrides
For extending existing tables, one file per database table, using the name of the table for the file, plus ".php".
Configuration/TsConfig/Page
Page TSconfig, see chapter 'Page TSconfig' in the TSconfig Reference. Files should have the file extension .tsconfig.
Configuration/TsConfig/User
User TSconfig, see chapter 'User TSconfig' in the TSconfig Reference. Files should have the file extension .tsconfig.
Configuration/TypoScript
TypoScript static setup (setup.typoscript) and constants (constants.typoscript). Use subfolders if you have several static templates.
Documentation
Contains the extension documentation in ReStructuredText (ReST, .rst) format. Read more on the topic in chapter extension documentation. Documentation/ and its subfolders may contain several ReST files, images and other resources.
Documentation/Index.rst
This file contains the cover page of the extension manual in ReST format. The name or format of the file may not be changed. You may include other ReST files as you like. See the "Extension Template" on docs.typo3.org for more information about structure and syntax of extension manuals.
Resources
Contains the subfolders Public/ and Private/, which contain resources, possibly in further subfolders, e.g. Templates/, Css/, Language/, Images/ or JavaScript/. This is also the directory for non–TYPO3 files supplied with the extension. TYPO3 is licensed under GPL version 2 or any later version. Any non–TYPO3 code must be compatible with GPL version 2 or any later version.
Resources/Private/Language
XLIFF files for localized labels.
Resources/Private/Layouts
Main layouts for (Fluid) views.
Resources/Private/Partials
Partial templates for repetitive use.
Resources/Private/Templates
One template per action, stored in a folder named after each Controller.
Resources/Public/Css
Any CSS file used by the extension.
Resources/Public/Images
Any images used by the extension.
Resources/Public/JavaScript
Any JS file used by the extension.
Tests/Unit
Contains unit tests and fixtures.
Tests/Functional
Contains functional tests and fixtures.

System and Local extensions

The files for an extension are located in a folder named by the extension key . The location of this folder can be either inside typo3/sysext/ or typo3conf/ext/.

The extension must be programmed so that it does automatically detect where it is located and can work from all two locations.

Local extensions

Local extensions are located in the typo3conf/ext/ directory.

This is where to put extensions which are local for a particular TYPO3 installation. The typo3conf directory is always local, containing local configuration (e.g. LocalConfiguration.php), local modules etc. If you put an extension here it will be available for a single TYPO3 installation only. This is a "per-database" way to install an extension.

System extensions

System extensions are located in the typo3/sysext/ directory.

This is system default extensions which cannot and should not be updated by the EM. They are distributed with TYPO3 core source code and generally understood to be a part of the core system.

Loading precedence

Local extensions take precedence which means that if an extension exists both in typo3conf/ext/ and typo3/sysext/ the one in typo3conf/ext/ is loaded. This means that extensions are loaded in the order of priority local-system.

Choosing an extension key

The "extension key" is a string uniquely identifying the extension. The folder where the extension resides is named by this string. The string can contain characters a-z0-9 and underscore. No uppercase characters should be used (keeps folder-,file- and table/field-names in lowercase). Furthermore the name must not start with an "tx" or "u" (this is prefixes used for modules) and because backend modules related to the extension should be named by the extension name without underscores, the extension name must still be unique even if underscores are removed (underscores are allowed to make the extension key easily readable).

The naming conventions of extension keys are automatically validated by the registration at the repository, so you have nothing to worry about here.

There are two ways to name an extension:

  • Project specific extensions (not generally usable or shareable): Select any name you like and prepend it "user_" (which is the only allowed use of a key starting with "u"). This prefix denotes that this extension is a local one which does not come from the central TYPO3 Extension Repository or is ever intended to be shared. Probably this is an "adhoc" extension you have made for some special occasion.
  • General extensions: Register an extension name online at the TYPO3 Extension Repository. Your extension name will automatically be validated and you are sure to have a unique name returned which nobody else in the world uses. This makes it very easy to share your extension later on with every one else, because it ensures that no conflicts with other extension will happen. But by default a new extension you make is defined "private" which means nobody else but you have access to it until you permit it to be public. It's free of charge to register an extension name. By definition all code in the TYPO3 Extension Repository is covered by the GPL license because it interfaces with TYPO3. You should really consider making general extensions!

Tip

It is far easier to settle for the right extension key from the beginning. Changing it later involves a cascade of name changes to tables, modules, configuration files, etc. Think carefully.

About GPL and extensions

Remember that TYPO3 is GPL software and at the same moment you extend TYPO3 your extensions are legally covered by GPL. This does not force you to share your extension, but it should inspire you to do so and legally you cannot prevent anyone who gets hold of your extension code from using it and further develop it. The TYPO3 Extension API is designed to make sharing of your work easy as well as using others' work easy. Remember TYPO3 is Open Source Software and we rely on each other in the community to develop it further.

Important

It's also your responsibility to make sure that all content of your extensions is legally covered by GPL. The webmaster of TYPO3.org reserves the right to kick out any extension without notice that is reported to contain non-GPL material.

Security

You are responsible for security issues in your extensions. People may report security issues either directly to you or to the TYPO3 Security Team. Whatever the case you should get in touch with the Security Team which will validate the security fixes. They will also include information about your (fixed) extension in their next Security bulletin. If you don't respond to requests from the Security Team, your extension will be forcibly removed from the TYPO3 Extension Repository.

More details on the security team's policy on handling security issues can be found at http://typo3.org/teams/security/extension-security-policy/.

Registering an extension key

Before starting a new extension you should register an extension key on typo3.org (unless you plan to make an implementation-specific extension – of course – which it does not make sense to share).

Go to typo3.org, log in with your (pre-created) username / password and go to Extensions > Extension Keys and click on the "Register keys" tab. On that page you can enter the key name you want to register.

The extension registration form

The extension registration form on typo3.org.

Naming conventions

Based on the extension key of an extension these naming conventions should be followed:

Attention

((The following table is unreadable and has been translated to the following normal text. The table will be dropped soon.))

  General Example User-specific Example

Extension key

(Lowercase "alnum" + underscores. )

Assigned by the TYPO3 Extension Repository. cool_shop Determined by yourself, but prefixed "user_" user_my_shop
Database tables and fields Prefix with "tx_[ key ]_" where key is without underscores!

Prefix: tx_coolshop_

Examples:

tx_coolshop_products

tx_coolshop_categories

Prefix with "[ key ]_"

Prefix: user_my_shop_

Examples:

user_my_shop_products

user_my_shop_categories

Backend module

(Names are always without underscores!)

Name: The extension key name without underscores, prefixed "tx" txcoolshop Name: No underscores, prefixed "u" uMyShop or umyshop or ...
Abbreviations
TER = TYPO3 extension repository
extkey = extension key
modkey = backend module key
Public extensions
  1. Public extensions are available from the TER or via Packagist. Private extensions are not published to the TER or Packagist.

  2. The extkey is made up of alphanumeric characters and underscores only and should start with a letter.

    Example: cool_shop

  3. The extkey is valid if the TER accepts it. This makes sure that the name follows the rules and is unique.

  4. Database tablenames look like tx_ + extkey (without underscores) + _specification.

    Examples: tx_coolshop_products, tx_coolshop_categories, tx_coolshop_more_categories, tx_coolshop_domain_model_tag.

Backend modules
  1. The modkey is made up of alphanumeric characters only. It does not contain underscores and starts with a letter.

    Example: coolshop

Frontend PHP classes
For frontend PHP classes, follow the same conventions as for database tables and fields.

You may also want to refer to the TYPO3 Core Coding Guidelines for more on general naming conventions in TYPO3.

Tip

If you study the naming conventions above closely you will find that they are complicated due to varying rules for underscores in key names. Sometimes the underscores are stripped off, sometimes not.

The best practice you can follow is to avoid using underscores in your extensions keys at all! That will make the rules simpler. This is highly encouraged.

Note on "old" extensions:

Some the "classic" extensions from before the extension structure came about do not comply with these naming conventions. That is an exception made for backwards compatibility. The assignment of new keys from the TYPO3 Extension Repository will make sure that any of these old names are not accidentally reassigned to new extensions.

Further, some of the classic plugins (tt_board, tt_guest etc) use the "user_" prefix for their classes as well.

Extending "extensions classes"

As a standard procedure you should include the "class extension code" even in your own extensions. This is placed at the bottom of every class file:

if (defined('TYPO3_MODE') && isset($GLOBALS['TYPO3_CONF_VARS'][TYPO3_MODE]['XCLASS']['ext/myext/pi1/class.tx_myext_pi1.php'])) {
        include_once($GLOBALS['TYPO3_CONF_VARS'][TYPO3_MODE]['XCLASS']['ext/myext/pi1/class.tx_myext_pi1.php']);
}

Normally the key used as example here ("ext/myext/pi1/class.tx_myext_pi1.php") would be the full path to the script relative to \TYPO3\CMS\Core\Core\Environment::getPublicPath(). However because modules are required to work from both typo3/sysext/ and typo3conf/ext/ it is a policy that any path before "ext/" is omitted.

Declaration file

The ext_emconf.php is the single most important file in an extension. Without it, the Extension Manager (EM) will not detect the extension, much less be able to install it. This file contains a declaration of what the extension is or does for the EM. The only thing included is an associative array, $EM_CONF[extension key]. The keys are described in the table below.

This file is overwritten, when extensions are imported from the online repository. So don't write your custom code in this file - only change values in the $EM_CONF array if needed.

Key Data type Description
title string, required The name of the extension in English.
description string, required Short and precise description in English of what the extension does and for whom it might be useful.
version string Version of the extension. Automatically managed by EM / TER. Format is [int].[int].[int]
category string

Which category the extension belongs to:

  • be

    Backend (Generally backend-oriented, but not a module)

  • module

    Backend modules (When something is a module or connects with one)

  • fe

    Frontend (Generally frontend oriented, but not a "true" plugin)

  • plugin

    Frontend plugins (Plugins inserted as a "Insert Plugin" content element)

  • misc

    Miscellaneous stuff (Where not easily placed elsewhere)

  • services

    Contains TYPO3 services

  • templates

    Contains website templates

  • example

    Example extension (Which serves as examples etc.)

  • doc

    Documentation (e.g. tutorials, FAQ's etc.)

  • distribution

    Distribution, an extension kickstarting a full site

constraints array

List of requirements, suggestions or conflicts with other extensions or TYPO3 or PHP version. Here's how a typical setup might look:

'constraints' => array(
    'depends' => array(
        'typo3' => '4.5.0-6.1.99',
        'php' => '5.3.0-5.5.99'
    ),
    'conflicts' => array(
        'dam' => ''
    ),
    'suggests' => array(
        'tt_news' => '2.5.0-0.0.0'
    )
)
depends
List of extensions that this extension depends on. Extensions defined here will be loaded before the current extension.
conflicts
List of extensions which will not work with this extension.
suggests

List of suggestions of extensions that work together or enhance this extension. Extensions defined here will be loaded before the current extension. Dependencies take precedence over suggestions.

Note: If a "suggested" extension depends on the current extension (directly or indirectly), the suggestion is not taken into account for loading order calculation. Read more at Forge #57825.

The above example indicated that the extension depends on a version of TYPO3 between 4.5 and 6.1 (as only bug and security fixes are integrated into TYPO3 when the last digit of the version changes, it is safe to assume it will be compatible with any upcoming version of the corresponding branch, thus .99). Also the extension has been tested and is known to work properly with PHP 5.3, 5.4 and 5.5. It will conflict with the DAM (any version) and it is suggested that it might be worth installing "tt_news" (version at least 2.5.0).

state string

Which state is the extension in

  • alpha

    Alpha state is used for very initial work, basically the state is has during the very process of creating its foundation.

  • beta

    Under current development. Beta extensions are functional but not complete in functionality. Most likely beta-extensions will not be reviewed.

  • stable

    Stable extensions are complete, mature and ready for production environment. You will be approached for a review. Authors of stable extensions carry a responsibility to maintain and improve them.

  • experimental

    Experimental state is useful for anything experimental - of course. Nobody knows if this is going anywhere yet... Maybe still just an idea.

  • test

    Test extension, demonstrates concepts, etc.

  • obsolete

    The extension is obsolete or deprecated. This can be due to other extensions solving the same problem but in a better way or if the extension is not being maintained anymore.

  • excludeFromUpdates

    This state makes it impossible to update the extension through the Extension Manager (neither by the Update mechanism, nor by uploading a newer version to the installation). This is very useful if you made local changes to an extension for a specific installation and don't want any admin to overwrite them.

    New since TYPO3 4.3.

uploadfolder boolean If set, then the folder named "uploads/tx_[extKey-with-no- underscore]" should be present!
createDirs list of strings Comma list of directories to create upon extension installation.
clearCacheOnLoad boolean If set, the EM will request the cache to be cleared when this extension is loaded.
author string Author name
author_email email address Author email address
author_company string Author company
autoload array

To get better class loading support for websites in non-composer mode+ the following information can be provided.

Extensions having one folder with classes or single files

Considering you have an Extbase extension (or an extension where all classes and interfaces reside in a Classes folder) or single classes you can simply add the following to your ext_emconf.php file:

'autoload' => [
   'classmap' => [
      'Classes',
      'a-class.php',
   ]
],

Extensions using namespaces

If the extension has namespaced classes following the PSR-4 standard, then you can add the following to your ext_emconf.php file:

'autoload' => [
   'psr-4' => [
      'Vendor\\ExtName\\' => 'Classes'
   ]
],

// Important: The prefix must end with a backslash.

autoload-dev array Same as the configuration "autoload" but it is only used if the ApplicationContext is set to Testing.

Deprecated configuration

The following fields are deprecated and should not be used anymore:

  • dependencies
  • conflicts
  • suggests
  • docPath
  • CGLcompliance
  • CGLcompliance_note
  • private
  • download_password
  • shy
  • loadOrder
  • priority
  • internal
  • modify_tables
  • module
  • lockType
  • TYPO3_version
  • PHP_version

Configuration files

Files ext_tables.php and ext_localconf.php are the two most important files for the execution of extensions within TYPO3. They contain configuration used by the system on almost every request. They should therefore be optimized for speed.

ext_localconf.php

ext_localconf.php is always included in global scope of the script, either frontend or backend.

Should not be used for

While you can put functions and classes into the script, it is a really bad practice because such classes and functions would always be loaded. It is better to have them included only as needed.

Should be used for

These are the typical functions that extension authors should place within ext_localconf.php

  • Registering hooks or any simple array assignments to $GLOBALS['TYPO3_CONF_VARS'] options
  • Registering additional Request Handlers within the Bootstrap
  • Adding any PageTSconfig or Default TypoScript via ExtensionManagementUtility APIs
  • Registering Extbase Command Controllers
  • Registering Scheduler Tasks
  • Adding reports to the reports module
  • Adding slots to signals via Extbase's SignalSlotDispatcher
  • Registering Icons to the IconRegistry
  • Registering Services via the Service API

ext_tables.php

ext_tables.php is not always included in the global scope of the frontend context.

This file is only included when

  • a TYPO3 Backend or CLI request is happening
  • or the TYPO3 Frontend is called and a valid Backend User is authenticated

This file usually gets included later within the request and after TCA information is loaded, and a Backend User is authenticated as well.

Should be used for

These are the typical functions that should be placed inside ext_tables.php

  • Registering of Backend modules or Backend module functions
  • Adding Context-Sensitive-Help docs via ExtensionManagementUtility API
  • Adding TCA descriptions (via ExtensionManagementUtility::addLLrefForTCAdescr())
  • Adding table options via ExtensionManagementUtility::allowTableOnStandardPages
  • Assignments to the global configuration arrays $TBE_STYLES and $PAGES_TYPES
  • Adding new fields to User Settings ("Setup" Extension)
Best practices

Additionally, it is possible to extend TYPO3 in a lot of different ways (adding TCA, Backend Routes, Symfony Console Commands etc) which do not need to touch these files.

It is recommended to AVOID checks for values on TYPO3_MODE or TYPO3_REQUESTTYPE constants (e.g. if (TYPO3_MODE === 'BE')) within these files as it limits the functionality to cache the whole systems' configuration. Any extension author should remove the checks if not explicitly necessary, and re-evaluate if these context-depending checks could go inside the hooks / caller function directly.

It is recommended to check for the existence of the constants defined('TYPO3_MODE') or die(); at the top of ext_tables.php and ext_localconf.php files to make sure the file is executed only indirectly within TYPO3 context. This is a security measure since this code in global scope should not be executed through the web server directly as entry point.

Additionally, it is recommended to use the extension name (e.g. "tt_address") instead of $_EXTKEY within the two configuration files as this variable will be removed in the future. This also applies to $_EXTCONF.

However, due to limitations to TER, the $_EXTKEY option should be kept within an extension's ext_emconf.php.

See any system extension for best practice on this behaviour.

  • $GLOBALS['TYPO3_LOADED_EXT'][extensionKey] contains information about whether the module is loaded as local or system type, including the proper paths you might use, absolute and relative.
  • Your ext_tables.php and ext_localconf.php files must be designed so that they can safely be read and subsequently imploded into one single file with all the other configuration scripts!
  • You must never use a "return" statement in the files global scope - that would make the cached script concept break.
  • You must never use a "use" statement in the files global scope - that would make the cached script concept break and could conflict with other extensions.
  • You should not rely on the PHP constant __FILE__ for detection of include path of the script - the configuration might be executed from a cached script and therefore such information should be derived from e.g. \TYPO3\CMS\Core\Utility\GeneralUtility::getFileAbsFileName() or ExtensionManagementUtility::extPath().

Best practices for ext_tables.php and ext_localconf.php

It is a good practice to use a directly called closure function to encapsulate all locally defined variables and thus keep them out of the surrounding scope. This avoids unexpected side-effects with files of other extensions.

The following example contains the complete code:

<?php
defined('TYPO3_MODE') or die();

(function () {
    // Add your code here
})();

In many cases, the file ext_tables.php is no longer needed, since TCA definitions must be placed in Configuration/TCA/*.php files nowadays.

Configuration options

In the ext_conf_template.txt file configuration options for an extension can be defined. They will be accessible in the TYPO3 backend from the Extension Manager.

There's a specific syntax to declare these options properly, which is similar to the one used for TypoScript constants (see "Declaring constants for the Constant editor" in Constants section in TypoScript Reference. This syntax applies to the comment line that should be placed just before the constant. Consider the following example (taken from system extension "rsaauth"):

# cat=basic/enable; type=string; label=Path to the temporary directory:This directory will contain...
temporaryDirectory =

First a category (cat) is defined ("basic") with the subcategory "enable". Then a type is given ("string") and finally a label, which is itself split (on the colon ":") into a title and a description (this should actually be a localized string). The above example will be rendered like this in the EM:

Configuration screen for the rsaauth extension

The configuration tab displays all options from a single category. A selector is available to switch between categories. Inside an option screen, options are grouped by subcategory. At the bottom of the screenshot, the label – split between header and description – is visible. Then comes the field itself, in this case an input, because the option's type is "string".

Available option types:

Option type Description
boolean checkbox
color colorpicker
int integer value
int+ positive integer value
integer integer value
offset offset
options option select
small small text field
string text field
user user function
wrap wrap field

Once you saved the configuration in the Extension Manager, it will be stored in $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS']['your_extension_key'] as an array.

To retrieve the configuration use the API provided by the \TYPO3\CMS\Core\Configuration\ExtensionConfiguration class:

$backendConfiguration = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(ExtensionConfiguration::class)
   ->get('your_extension_key');

This will return the whole configuration as an array.

To directly fetch specific values like temporaryDirectory from the example above:

$backendConfiguration = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(ExtensionConfiguration::class)
   ->get('your_extension_key', 'temporaryDirectory');

You can also define nested options using the TypoScript notation:

directories {
   # cat=basic/enable; type=string; label=Path to the temporary directory
   tmp =
   # cat=basic/enable; type=string; label=Path to the cache directory
   cache =
}

This will result in a multidimensional array:

$extensionConfiguration = $GLOBALS['TYPO3_CONF_VARS']['EXTENSIONS']['your_extension_key'];
$extensionConfiguration['directories.']['tmp']
$extensionConfiguration['directories.']['cache']

Important

Notice the dot at the end of the directories key. This notation must be used for every grouping key and is a convention of the TypoScript parser.

Extending the $TCA array

Being a PHP array, the Table Configuration Array can be easily extended. It can be accessed as the global variable $GLOBALS['TCA']. TYPO3 also provides APIs for making this simpler.

Storing the changes

There are various ways to store changes to $GLOBALS['TCA']. They depend - partly - on what you are trying to achieve and - a lot - on the version of TYPO3 CMS which you are targeting.

There are two main ways to store your changes to the TCA: inside an extension or straight in the typo3conf folder. Both are described below in more details.

Storing in extensions

The advantage of putting your changes inside an extension is that they are nicely packaged in a self-contained entity which can be easily deployed on multiple servers.

The drawback is that the extension loading order must be finely controlled. Indeed if your extension modifies another extension, your extension must be loaded after the extension you are modifying. This can be achieved by registering that other extension as a dependency of yours. See the description of constraints in Core APIs.

For more information about an extension's structure, please refer to the extension architecture chapter in Core APIs.

Storing in the Overrides folder

Since TYPO3 CMS 6.2 (6.2.1 to be precise) changes to $GLOBALS['TCA'] must be stored inside a folder called Configuration/TCA/Overrides with one file per modified table. These files are named along the pattern <tablename>.php.

Thus if you want to customize the TCA of tx_foo_domain_model_bar, you'd create the file Configuration/TCA/Overrides/tx_foo_domain_model_bar.php.

The advantage of this method is that all such changes are incorporated into $GLOBALS['TCA'] before it is cached. This is thus far more efficient.

Important

Be aware that you cannot extend the TCA of extensions if it was configured within its ext_tables.php file, usually containing the "ctrl" section referencing a "dynamicConfigFile". Please ask the extension author to switch to the Configuration/TCA/<tablename>.php setup.

Important

Only TCA-related changes should go into Configuration/TCA/Overrides files. Some API calls may be okay as long as they also manipulate only $GLOBALS['TCA']. For example, it is fine to register a plugin with \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addPlugin() in Configuration/TCA/Overrides/tt_content.php because that API call only modifies $GLOBALS['TCA'] for table "tt_content".

Storing in ext_tables.php files

Until TYPO3 CMS 6.1 (still supported for 6.2) changes to $GLOBALS['TCA'] are packaged into an extension's ext_tables.php file. This is strongly discouraged in more recent versions of TYPO3 CMS.

Nowadays the only usecase for TCA changes in ext_tables.php is to override TCA definitions done in the ext_tables.php of a legacy extension. TCA overrides cannot be used in this case until the author of the legacy extension migrates his code.

Changing the TCA "on the fly"

It is also possible to perform some special manipulations on $GLOBALS['TCA'] right before it is stored into cache, thanks to the tcaIsBeingBuilt signal. This signal was introduced in TYPO3 CMS 6.2.1.

Customization examples

Many extracts can be found throughout the manual, but this section provides more complete examples.

Example 1: extending the fe_users table

The "examples" extension adds two fields to the "fe_users" table. Here's the complete code, taken from file Configuration/TCA/Overrides/fe_users.php:

<?php
defined('TYPO3_MODE') or die();

// Add some fields to FE Users table to show TCA fields definitions
// USAGE: TCA Reference > $GLOBALS['TCA'] array reference > ['columns'][fieldname]['config'] / TYPE: "select"
$temporaryColumns = array (
        'tx_examples_options' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options',
                'config' => array (
                        'type' => 'select',
                        'items' => array (
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.0', '1'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.1', '2'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.2', '--div--'),
                                array('LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_options.I.3', '3'),
                        ),
                        'size' => 1,
                        'maxitems' => 1,
                )
        ),
        'tx_examples_special' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:fe_users.tx_examples_special',
                'config' => array (
                        'type' => 'user',
                        'size' => '30',
                        'userFunc' => 'Documentation\\Examples\\Userfuncs\\Tca->specialField',
                        'parameters' => array(
                                'color' => 'blue'
                        )
                )
        ),
);

\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns(
        'fe_users',
        $temporaryColumns
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addToAllTCAtypes(
        'fe_users',
        'tx_examples_options, tx_examples_special'
);

First of all, the fields that we want to add are detailed according to the $GLOBALS['TCA'] syntax for columns. This configuration is stored in the $temporaryColumns array.

Then two essential steps are performed:

  • first the columns are actually added to the table by using \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns().
  • then the fields are added to the "types" definition of the "fe_users" table by using \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addToAllTCAtypes(). It is possible to be more fine-grained.

This does not create the corresponding fields in the database. The new fields must also be defined in the ext_tables.sql file of the extension:

CREATE TABLE fe_users (
        tx_examples_options int(11) DEFAULT '0' NOT NULL,
        tx_examples_special varchar(255) DEFAULT '' NOT NULL
);

Warning

The above statement uses the SQL CREATE TABLE statement. This is the way TYPO3 expects it to be. The Extension Manager will automatically transform this into a ALTER TABLE statement when it detects that the table already exists.

By default new fields are added at the bottom of the form when editing a record from that table. If the table uses tabs, new fields are added at the bottom of the "Extended" tab (this tab is created if it does not exist). The following screenshot shows the placement of the two new fields when editing a "fe_users" record:

New fields for fe\_users table

The new fields added at the bottom of the "Extended" tab

The next example shows how to place a field more precisely.

Example 2: extending the tt_content table

In this second example, we will add a "No print" field to all content element types. First of all, we add its SQL definition in ext_tables.sql:

CREATE TABLE tt_content (
        tx_examples_noprint tinyint(4) DEFAULT '0' NOT NULL
);

Then we add it to the $GLOBALS['TCA'] in Configuration/TCA/Overrides/tt_content.php:

$temporaryColumn = array(
        'tx_examples_noprint' => array (
                'exclude' => 0,
                'label' => 'LLL:EXT:examples/Resources/Private/Language/locallang_db.xlf:tt_content.tx_examples_noprint',
                'config' => array (
                        'type' => 'check',
                )
        )
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addTCAcolumns(
        'tt_content',
        $temporaryColumn
);
\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::addFieldsToPalette(
        'tt_content',
        'visibility',
        'tx_examples_noprint',
        'after:linkToTop'
);

The code is mostly the same as in the first example, but the last line is very different and requires an explanation. The "pages" and "tt_content" use palettes extensively for all fields and not just for secondary options, for increased flexibility. So in this case we use addFieldsToPalette() instead of addToAllTCAtypes(). We need to specify the palette's key as the second argument (visibility). Precise placement of the new field is achieved with the fourth parameter (after:linkToTop). This will place the "no print" field right after the "link to top" field, instead of putting it in the "Extended" tab.

The result is the following:

New fields for tt\_content table

The new field added next to an existing one

Note

Obviously this new field will now magically exclude a content element from being printed. For it to have any effect, it must be used during the rendering by modifying the TypoScript used to render the "tt_content" table. Although this is outside the scope of this manual, here is an example of what you could do, for the sake of showing a complete process.

Assuming you are using "css_styled_content" (which is installed by default), you could add the following TypoScript to your template:

tt_content.stdWrap.outerWrap = <div class="noprint">|</div>
tt_content.stdWrap.outerWrap.if.isTrue.field = tx_examples_noprint

This will wrap a "div" tag with a "noprint" class around any content element that has its "No print" checkbox checked. The final step would be to declare the appropriate selector in the print-media CSS file so that "noprint" elements don't get displayed.

This is just an example of how the effect of the "No print" checkbox can be ultimately implemented. It is meant to show that just adding the field to the $GLOBALS['TCA'] is not enough.

Verifying the $TCA

You may find it necessary – at some point – to verify the full structure of the $GLOBALS['TCA'] in your TYPO3 installation. The SYSTEM > Configuration module makes it possible to have an overview of the complete $GLOBALS['TCA'], with all customizations taken into account.

The Configuration module

Checking the existence of the new field via the Configuration module

If you cannot find your new field, it probably means that you have made some mistake.

This view is also useful when trying to find out where to insert a new field, to explore the combination of types and palettes that may be used for the table that we want to extend.

Creating a new extension

This chapter is not a tutorial about how to create an Extension. It only aims to be a list of steps to perform and key information to remember.

First you have to register an extension key. This is the unique identifier for your extension.

Kickstarting the extension

Although it is possible to write every single line of an extension from scratch, there is a tool which makes it easier to start. It is called "Extension builder" (key: "extension_builder") and can be installed from TER.

The Extension Builder comes with its own BE module:

A view from the Extension Builder

The Domain Modeller screen of the Extension Builder. The comfort of building your model with drag and drop.

Note that this tool is not a complete editor. It helps you creating the scaffolding of your extension, generating the necessary files. It's then up to you to fill these with the relevant code.

Warning

The Extension Builder has some possibility to preserve code, but it should still be used with care.

After the extension is written to your computer's disk you will be able to install it locally and start using it.

Please refer to the Extension Builder's manual for more information.

Creating a new distribution

This chapter describes the main steps in creating a new distribution. It should not be considered as a full fledge tutorial.

Concept of distributions

Distributions are full TYPO3 CMS websites ready to be unpacked. They provide an easy quick start for using TYPO3 CMS. The most well known distribution is "The official Introduction Package". Distributions can most easily be installed in the backend Extension Manager in "Get preconfigured distribution", it lists all available distributions for the given core version.

A distribution is just an extension enriched with some further data that is loaded or executed upon installing that extension. A distribution takes care of the following parts:

  • Deliver initial database data
  • Deliver fileadmin files
  • Deliver configuration for a package
  • Hook into the process after saving configuration to trigger actions dependent on configuration values
  • Deliver dependent extensions if needed (e.g., customized versions or extensions not available through TER)

Kickstarting the distribution

A distribution is a special kind of extension. The first step is thus to create a new extension. Start by registering an extension key, which will be the unique identifier of your distribution.

Next create the Extension declaration file as usual, except for the "category" property which must be set to distribution.

Configuring the distribution display in the EM

You should provide two preview images for your distribution. Provide a small 220x150 pixels for the list in the Extension Manager as Resources/Public/Images/Distribution.png and a larger 300x400 pixels welcome image as Resources/Public/Images/DistributionWelcome.png. The welcome image is displayed in the distribution detail view inside the Extension Manager.

Fileadmin files

Create the following folder structure inside your extension:

  • Initialisation
  • Initialisation/Files

All the files inside that second folder will be copied to fileadmin/<extkey> during installation, where "extkey" is the extension key of your distribution.

A good strategy on files (as followed by ext:introduction) is to construct the distribution in a way that it can be unloaded after initial import and removed from the file system.

To achieve that, when creating content for your distribution, all your content related files (assets) should be located within fileadmin/<extkey> in the first place, and content elements or other records should reference these files via FAL. A good export preset will then contain the content related assets within your dump.

If there are files not directly referenced in tables selected for export (for example ext:form .yml form configurations), you can locate them within fileadmin/<extkey>, too. Only those need to be copied to Initialization/Files - all other files referenced in database rows will be within your export dump.

Note you should not end up with having all your site configuration (TypoScript files, logos, css and so on) within fileadmin. This is considered bad practice. The main site setup should be an extension, keep in mind that fileadmin is for editors. In case of the introduction distribution, the main site setup (templates, content elements, ...) is included in the extension bootstrap_package, and ext:introduction has a dependency to this. This way, ext:introduction only provides the database dump and the asset files, while ext:bootstrap_package is the real site setup. This ends up with only content related stuff being located in fileadmin, delivered by ext:introduction.

Database data

The database data is delivered as TYPO3 CMS export data.xml. Generate this file by exporting your whole installation from the tree root with the import/export module.

Warning

Do NOT include backend users in the dump! If you do, you end up having your user on other systems who loaded your distribution. Give the export a special check in this area. Having your backend user in the dump is most likely a security vulnerability of your distribution if that distribution is uploaded to the public.

The file has to be named data.xml (or data.t3d, where the .t3d format is harder to maintain). The dump file must be located in the Initialisation folder.

It is also possible to have referenced files (images / media) in an own folder called Initialisation/data.xml.files/ - a good export preset should prepare that.

Note

Due to core bugs, importing extracted files from standalone file folder only works since core version 8.7.10 and 9.1.0. For older target core versions, files must not be extracted (tab Advanced options), but directly included in data.xml.

Another core issue prevents loading data.xml if it is bigger than 10MB. In this case the only option left is going with data.t3d

Exporting the correct data can be a bit tricky to get right. It is a good idea to create an "Export preset" within the Export module for that and deliver an sql dump of that preset within the distribution. The introduction distribution comes with a maintained sql dump that could be useful as kick start. Just load that row into table tx_impexp_presets and adapt to the needs of your distribution. The ext:introduction preset is configured as:

  • Export db data as data.xml
  • Export only referenced FAL file relations into data.xml.files directory, do not just export all files from fileadmin
  • Do not export be_users (!)
  • Do not export some other tables like sys_log and friends
Distribution configuration

A distribution is technically handled as an extension. Therefore you can make use of all configuration options as needed.

After installing the extension, the signal hasInstalledExtensions is dispatched. You may use this to alter your website configuration (e.g. color scheme) on the fly.

Delivering custom dependencies

Normally extension dependencies are setup in the Extension declaration file.

However sometimes, extensions are not available in the TYPO3 Extension Repository (TER), or you need to deliver a modified version. Therefore, a distribution can act as its own extension repository. Add unpacked extensions to Initialisation/Extensions/ to provide dependencies. Your main extension has to be dependent on these extensions as normal dependencies in ext_emconf.php.

Extensions delivered inside an extension have the highest priority when extensions need to be fetched.

Caution

This will not overwrite extensions already present in the system.

Test your distribution

To test your distribution, simply copy your extension to an empty TYPO3 CMS installation and try to install it from the Extension Manager.

To test a distribution locally without uploading to TER, just install a blank TYPO3 (last step in installer "Just get me to the Backend"), then go to Extension Manager, select "Get extensions" once to let the Extension Manager initialize the extension list (this is needed if your distribution has dependencies to other extensions, for instance ext:introduction depends on ext:bootstrap_package). Next, copy or move the distribution extension to typo3conf/ext, it will then show up in Extension Manager default tab "Installed Extensions".

Install the distribution extension from there. The Extension Manager will then resolve TER dependencies, loads the database dump and will handle the file operations. Under the hood, this does the same as later installing the distribution via "Get preconfigured distribution", when it has been uploaded or updated in TER, with the only difference that you can provide and test the distribution locally without uploading to TER first.

Warning

It is not enough to clean all files and the page tree if you want to try again to install your distribution. Indeed, TYPO3 CMS remembers that it previously imported your distribution and will skip any known files and the database import. Make sure to clean the table "sys_registry" if you want to work around that, or, even better, install a new blank TYPO3 to test again. Tip: Optimize creating the empty TYPO3 instance with a script, you probably end up testing the import a couple of times until you are satisfied with the result.

More information

The introduction extension is a good starting point to see how distributions are handled in practice. It also comes with an impexp preset to easily export database data with correct settings and dependencies.

Some additional backgrounds can be retrieved from the blueprint for this feature.

Adding documentation

If you plan to upload your extension to the TYPO3 Extension Repository (TER), you should first consider adding a documentation to your extension. A documentation will help users and administrators to quickly install and configure your extension and give it more weight.

The documentation platform https://docs.typo3.org centralizes documentation for every project. It supports two different kind of documentation:

  1. (recommended) A Sphinx project, stored within EXT:extkey/Documentation/
  2. A simple README file stored as EXT:extkey/README.rst as seen on Github

Sphinx project

Sphinx is the official format for official TYPO3 documentation. A Sphinx-based documentation is a set of plain text files making up the chapters or sections of the documentation. It uses a markup language called "reStructuredText" (reST).

Advantages of this new documentation format are numerous:

  • Output formats: Sphinx projects may be automatically rendered as HTML or TYPO3-branded PDF.
  • Cross-references: It is easy to cross-reference other chapters and sections of other manuals (either TYPO3 references or extension manuals).
  • Multilingual: Unlike OpenOffice, Sphinx projects may be easily localized and automatically presented in the most appropriate language to TYPO3 users.
  • Collaboration: As the documentation is plain text, it is easy to work as a team on the same manual or quickly review changes using any versioning system.

Although it is possible to write every single line of a Sphinx-based documentation from scratch, the TYPO3 community provides tools that help write and manage Sphinx projects:

  • The extension "Sphinx" (Sphinx Python Documentation Generator and Viewer) installs a local Sphinx environment to view, edit and compile documentation in the backend of your TYPO3 website. It can be installed from the TYPO3 Extension Repository (TER) like any other extension.
  • The Sphinx extension is able to convert existing OpenOffice manuals (manual.sxw) into Sphinx projects with just one click.
  • An example manual is available on the TYPO3 Documentation Github repository.
  • The Extension Builder provides a skeleton documentation based on the above-mentioned Git repository.
  • A good primer to get started using the reStructuredText markup.

README.rst

A "README.rst" is a simple text file stored at the root of your extension directory and briefly describing the purpose of your extension. It is best suited when installing or using your extension is straightforward. The format of this file is reStructuredText, as for chapters of a Sphinx project.

Tip

In TYPO3 6.2, the system extension "documentation" is using such a simple manual.

Other resources

Beyond the general overview given in this chapter, other sections in this manual will be of particular interest to extension developers:

API overview

The TYPO3 APIs are first and foremost documented inside of the source scripts. It would be impossible to maintain documentation at more than one location given the fact that things change and sometimes fast. This chapter describes the most important elements of the API.

Note

The source is the documentation! (General wisdom)

Directory structure

By default a TYPO3 installation consists of a structure of main directories within the web server document root. You will find this structure to be almost always like that. Depending on the installation variant you choose however, this may be slightly different. For instance, it is possible to have all PHP files except the entry points index.php within the composer managed vendor/ directory, outside of the document root. This setup however did not fully settle yet, and is not documented here in detail. So, if you look at "casual" TYPO3 installations, you will almost always find the directory structure as outlined below.

Also see Environment for further information, especially how to retrieve the paths within PHP code.

Directory Description
fileadmin/

This is a directory in which editors store files. Typically images, PDFs or video files appear in this directory and/or its subdirectories.

Note this is only the default editor's file storage. This directory is handled via the FAL API internally, there may be further storage locations configured outside of fileadmin/, even pointing to different servers or using 3rd party digital asset management systems.

Note

Note this directory is meant for editors! Integrators should not locate frontend website layout related files in here: Storing HTML templates, logos, Css and similar files used to build the website layout in here is considered bad practice. Integrators should locate and ship these files within a project specific extension.

typo3/ TYPO3 Backend directory. This directory contains most of the files coming with the TYPO3 Core. The files are arranged logically in the different system extensions in the sysext/ directory, according to the application area of the particular file. For example, the "frontend" extension amongst other things contains the "TypoScript library", the code for generating the Frontend website. In each system extension the PHP files are located in the folder Classes/. See extension files locations for more information on how single extensions are structured.
getConfigPath() either typo3conf/ or config/

TYPO3 configuration directory. This directory contains installation wide configuration.

The most important file within this folder is LocalConfiguration.php. This one contains local settings of the main global PHP array $GLOBALS['TYPO3_CONF_VARS], crucial settings like database connect credentials are in here. The file is managed by the Install Tool and the Extension Manager and the content should not be managed manually since Extension Manager or Install Tool may override manually changed settings again.

The file LocalConfiguration.php can be enriched by AdditionalConfiguration.php which is never touched by TYPO3 internal management tools. Be aware that having settings within AdditionalConfiguration.php may prevent the system from doing automatic upgrades and should be used with care and only if you know what you are doing.

typo3conf/ext/ Directory for local TYPO3 extensions. Each subdirectory contains one extension.
getLabelsPath() either typo3conf/l10n or var/labels Directory for extension localisations. Contains all downloaded translation files.
typo3temp/ Directory for temporary files. It contains subdirectories for temporary files of extensions and TYPO3 components.

PHP Namespaces

Since version 6.0, TYPO3 CMS uses PHP namespaces for all classes in the Core.

The general structure of namespaces is the following:

\{VendorName}\{PackageName}\({CategoryName}\)*{ClassName}

For the Core, the vendor name is TYPO3\CMS and the package name corresponds to a system extension.

All classes must be located inside the Classes folder at the root of the (system) extension. The category name may contain several segments that correspond to the path inside the Classes folder.

Finally the class name is the same as the corresponding file name, without the .php extension.

"UpperCamelCase" is used for all segments.

Tip

See the chapter about 'ClassAliasMap.php' in the 6.2 documentation.. It may help you with migrating code from old to new conventions.

Core example

The good old t3lib_div class has been renamed to:

\TYPO3\CMS\Core\Utility\GeneralUtility

This means that the class is now found in the "core" system extension, in folder Classes/Utility, in a file named GeneralUtility.php.

Usage in extensions

Extension developers are free to use their own vendor name. Important: It may consist of one segment only. Vendor names must start with an uppercase character and are usually written in UpperCamelCase style. In order to avoid problems with different filesystems, only the characters a-z, A-Z, 0-9 and the dash sign "-" are allowed for package names – don't use special characters:

// good vendor name:
\Webcompany

// wrong vendor name:
\Web\Company

Attention

The vendor name TYPO3\CMS is reserved and may not be used by extensions!

The package name corresponds to the extension key. Underscores in the extension key are removed in the namespace and replaced by upper camel-case. So extension key:

weird-name_examples

would become:

Weird-nameExamples

in the namespace.

As mentioned above, all classes must be located in the Classes folder inside your extension. All sub-folders translate to a segment of the category name and the class name is the file name without the .php extension.

Looking at the "examples" extension, class:

examples/Classes/Controller/DefaultController.php

corresponds to namespace:

\Documentation\Examples\Controller\DefaultController

Inside the class, the namespace is declared as:

<?php
namespace Documentation\Examples\Controller;

Namespaces in Extbase

When registering components in Extbase, the vendor name must be used on top of the extension key.

For a backend module:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::registerModule(
    '<vendorName>.<ExtensionName>',
    // ...
);

For a frontend module:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::configurePlugin(
    '<vendorName>.<ExtensionName>',
    // ...
);

Important

  • Do not forget the dot after the vendor name.
  • Do not use dots inside the vendor name.

Namespaces for test classes

As for ordinary classes, namespaces for test classes start with a vendor name followed by the extension key.

All test classes reside in a Tests folder and thus the third segment of the namespace must be "Tests". Unit tests are located in a Unit folder which is the fourth segment of the namespace. Any further subfolders will be subsequent segments.

So a test class in EXT:foo_bar_baz/Tests/Unit/Bla/ will have as namespace \Vendor\FooBarBaz\Tests\Unit\Bla.

Creating instances

The following example shows how you can create instances by means of GeneralUtility::makeInstance():

$contentObject = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(
   \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer::class);

Or, use use to make the code more readable:

use TYPO3\CMS\Core\Utility\GeneralUtility;
use TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer;

$contentObject = GeneralUtility::makeInstance(ContentObjectRenderer::class);

include and required

There is no need for require() or include() statements. All classes adhering to namespace conventions will automatically be located and included by the autoloader.

References

For more information about PHP namespaces in general, you may want to refer to the PHP documentation and in particular the Namespaces FAQ.

Autoloading

The autoloader takes care of finding classes in TYPO3. It is closely related to \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance() which takes care of singleton and XCLASS handling.

As a developer you should always instantiate classes either through \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance() or with the Extbase \TYPO3\CMS\Extbase\Object\ObjectManager> (which internally uses makeInstance() again).

Autoloading classes since TYPO3 7.x

TYPO3 6.2 was still delivered with a couple of different autoloaders, that all had different approaches and rules to find a class. This led to the naming conventions in and outside Extbase and the optional ext_autoload.php file to load classes that didn't follow the conventions. Since TYPO3 7.0 all this is gone and there is only a single autoloader left, the one of composer. No matter if you run TYPO3 in composer mode or not, TYPO3 uses the composer autoloader to resolve all class file locations. However, the autoloader is a little bit more sophisticated in composer mode as it then supports PSR-4 autoloading.

Loading classes without composer mode

This means, you did not install TYPO3 via a require-statement inside your composer.json. It's a regular old-school install where the TYPO3 source and the symlinks (typo3/index.php) are setup manually. In this case, every time you install an extension, the autoloader scans the whole extension directory for classes. No matter if they follow any convention at all. There is just one rule. Put each class into its own file. The generated classmap is a huge array with a mapping of classnames to their location on the disk.

Example:

<?php

// autoload_classmap.php @generated by TYPO3

$typo3InstallDir = \TYPO3\CMS\Core\Core\Environment::getPublicPath();

return array(
   'Schnitzler\\Templavoila\\Clipboard\\Clipboard' => $typo3InstallDir . 'typo3conf/ext/templavoila/Classes/Clipboard/Clipboard.php',
   'tx_templavoila_pi1' => $typo3InstallDir . 'typo3conf/ext/templavoila/Compatibility/class.tx_templavoila_pi1.php',
   ...
);

This method is failsafe unless the autoload information cannot be written. In this case, check the Install Tool for warnings and make sure that typo3temp is writable.

Troubleshooting:

If your classes cannot be found, try the following approaches.

  • Dump the class loading information manually with the following command: php typo3/cli_dispatch.phpsh extbase extension:dumpclassloadinginformation
  • If that command itself fails, please (manually) uninstall the extension and simply try reinstalling it (via the Extension Manager).
  • If you are still not lucky, the issue is definitely on your side and you should double check the write permissions on typo3temp.

Loading classes with composer mode

In composer mode, the autoloader checks for (classmap and PSR-4) autoloading information inside your extensions' composer.json. If you do not provide any information, the autoloader falls back to the classmap autoloading like in non composer mode.

Troubleshooting:

  • Dump the class loading information manually via composer dumpautoload and check that the autoload information is updated. Typically you would check vendor/composer to hold files like autoload_classmap.php and autoload_psr4.php etc.

Example:

$ tree vendor/composer
.
├── ClassLoader.php
├── LICENSE
├── autoload_classmap.php
├── autoload_files.php
├── autoload_namespaces.php
├── autoload_psr4.php
├── autoload_real.php
├── autoload_static.php
├── include_paths.php
└── installed.json

Best practices

  • If you didn't do so before, have a look at the PSR-4 standard. It defines very good rules for naming classes and the files they reside in. Really, read the specs and start using PSR-4 in your projects. It's unlikely that there will be any other more advanced standard in the near future in the PHP world. PSR-4 is the way to go and you should embrace it.
  • Even if you do not use composer mode and the class mapping of the autoloader allows you to use whatever you want, stick to PSR-4. It's not only a very good standard to find classes, but it will also help organizing your code.
  • PSR-4 is all about namespaces. No matter if you like namespaces or not, use them. Namespaces exist since PHP 5.3, so you will be able to use them in any modern TYPO3 project due to the minimum PHP requirements of TYPO3 itself.

Tip

PSR-4 is a standard that has been developed by the PHP Framework Interop Group (FIG). PSR-4 is an advanced standard for autoloading php classes and replaces PSR-0. If you want to know more about the PHP FIG in general and PSR-4 in specific, please visit http://www.php-fig.org/psr/psr-4/.

Bootstrapping

TYPO3 CMS has a clean bootstrapping process driven mostly by class \TYPO3\CMS\Core\Core\Bootstrap. This class contains a host of methods each responsible for a little step along the initialization of a full TYPO3 process, be it the backend or other contexts.

Some contexts add their own bootstrap class (like the command line, which additionally requires \TYPO3\CMS\Core\Core\CliBootstrap).

Note

The frontend's bootstrapping process is not yet fully encapsulated in a bootstrap class.

Warning

This bootstrapping API is internal and may change at any time in the near future even in minor updates. It is thus discouraged to use it in third party code. Use this class only if other extensibility possibilities such as Hooks, Signals or XCLASS are not enough to reach your goals.

One can see the bootstrapping process in action in file typo3/sysext/backend/Classes/Http/Application.php:

use TYPO3\CMS\Core\Core\Bootstrap;

###

$this->bootstrap = Bootstrap::getInstance()
   ->initializeClassLoader($classLoader)
   ->setRequestType(TYPO3_REQUESTTYPE_BE | (!empty($_GET['ajaxID']) ? TYPO3_REQUESTTYPE_AJAX : 0))
   ->baseSetup($this->entryPointLevel);

// Redirect to Install Tool if base configuration is not found
if (!$this->bootstrap->checkIfEssentialConfigurationExists()) {
   $this->bootstrap->redirectToInstallTool($this->entryPointLevel);
}

foreach ($this->availableRequestHandlers as $requestHandler) {
   $this->bootstrap->registerRequestHandlerImplementation($requestHandler);
}

$this->bootstrap->configure();

###

Note that most methods of the Bootstrap class must be called in a precise order. It is perfectly possible to define one's own bootstrapping process, but care should be taken about the call order.

Also note that all bootstrapping methods return the instance of the Bootstrap class itself, allowing calls to be chained.

Initialization

Whenever a call to TYPO3 CMS is made, the application goes through a bootstrapping process managed by a dedicated API. This process is also used in the frontend, but only the backend process is described here.

Note

This chapter is outdated and should probably be merged with the "HTTP request library / Guzzle / PSR-7" chapter below. The chapter should include an overview of single bootstrap steps, PSR-15 and routing.

Classes involved in the backend bootstrapping process are \TYPO3\CMS\Core\Core\Bootstrap and TYPO3\CMS\Backend\Http\Application.

The following steps are performed during bootstrapping.

1. Define legacy constants

In Application::defineLegacyConstants some constants are defined, which will eventually be dropped, but are still initialized for now.

2. Initialize class loader

This defines which autoloader to use.

3. Set request type

The request type is set - this defines whether the current request is a frontend, backend, cli, ajax or Install Tool request. (see defineTypo3RequestTypes).

4. Perform base setup

An instance of \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder is created. This class in turn defines a large number of constants and global variables. If you want to have an overview of these base values, it is worth taking a look into the following methods:

  • SystemEnvironmentBuilder::defineBaseConstants() defines constants containing values such as the current version number, blank character codes and error codes related to services.
  • SystemEnvironmentBuilder::definePaths() defines constants containing paths to various parts of the TYPO3 installation like the absolute path to the typo3 directory or the absolute path to the installation root.
  • SystemEnvironmentBuilder::checkMainPathsExist() checks if expected paths like typo3 or index.php exist. If that is not the case, the process will quit immediately.
  • SystemEnvironmentBuilder::initializeGlobalVariables() sets some global variables as empty arrays.
  • SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables() defines special variables which contain, for example, the current time or a simulated time as may be set using the Admin Panel.
  • SystemEnvironmentBuilder::initializeBasicErrorReporting() sets up default error reporting level during the bootstrapping process.
5. Define class loading information

This part of the bootstrap processes all the information available to be able to determine where to load classes from, including class alias maps which are used to map legacy class names to new class names.

6. Check essential configuration

In this step we check if crucial configuration elements have been set. If that is not the case, the installation is deemed incomplete and the user is redirected to the Install Tool.

7. Register request handlers

The backend recognizes various request handlers, one to handle general requests, one for backend module requests, one for cli requests and one for AJAX requests.

8. More configuration

Next Bootstrap::configure() is called which in turn triggers a whole new series of configuration. This is actually a major step, with too many actions to detail efficiently here. However here is the list of the most important stuff happening at this point:

  • the main configuration ("TYPO3_CONF_VARS") is loaded
  • the Caching Framework and the Package Management are set up
  • all configuration items from extensions are loaded
  • the database connection is established
9. Dispatch

After all that the Application::run() method is called, which basically dispatches the request to the right handler.

10. Initialization of the TYPO3 Backend

The backend request handler has its own boot() method, which performs yet more initialization and set up as needed. A general request to the backend will typically go through such important steps like:

  • checking backend access: Is it locked? Does it have proper SSL setup?
  • loading the full TCA
  • verifying and initializing the backend user

Application Context

Each request, no matter if it runs from the command line or through HTTP, runs in a specific application context. TYPO3 CMS provides exactly three built-in contexts:

  • Production (default) - should be used for a live site
  • Development - used for development
  • Testing - is only used internally when executing TYPO3 core tests. It must not be used otherwise.

The context TYPO3 runs in is specified through the environment variable TYPO3_CONTEXT. It can be set on the command line:

# run the TYPO3 CMS CLI commands in development context
TYPO3_CONTEXT=Development ./typo3/cli_dispatch.phpsh

or be part of the web server configuration:

# In your Apache configuration, you usually use:
SetEnv TYPO3_CONTEXT Development

# Set context with mod_rewrite
# Rules to set ApplicationContext based on hostname
RewriteCond %{HTTP_HOST} ^dev\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Development]

RewriteCond %{HTTP_HOST} ^staging\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Production/Staging]

RewriteCond %{HTTP_HOST} ^www\.example\.com$
RewriteRule .? - [E=TYPO3_CONTEXT:Production]
# In your Nginx configuration, you can pass the context as a fastcgi parameter
location ~ \.php$ {
   include         fastcgi_params;
   fastcgi_index   index.php;
   fastcgi_param   TYPO3_CONTEXT  Development/Dev;
   fastcgi_param   SCRIPT_FILENAME  $document_root$fastcgi_script_name;
}
Custom Contexts

In certain situations, more specific contexts are desirable:

  • a staging system may run in a Production context, but requires a different set of credentials than the production server.
  • developers working on a project may need different application specific settings but prefer to maintain all configuration files in a common Git repository.

By defining custom contexts which inherit from one of the three base contexts, more specific configuration sets can be realized.

While it is not possible to add new "top-level" contexts at the same level like Production and Testing, you can create arbitrary sub-contexts, just by specifying them like <MainContext>/<SubContext>.

For a staging environment a custom context Production/Staging may provide the necessary settings while the Production/Live context is used on the live instance.

Note

This even works recursively, so if you have a multiple-server staging setup, you could use the context Production/Staging/Server1 and Production/Staging/Server2 if both staging servers needed different configuration.

Attention

Testing Is reserved for internal use when executing TYPO3 core functional and unit tests It must not be used otherwise. Instead sub-contexts must be used: Production/Testing or Development/Testing

Usage Example

The current Application Context is set very early in the bootstrap process and can be accessed through public API for example in the AdditionalConfiguration.php file to automatically set different configuration for different contexts.

In file typo3conf/AdditionalConfiguration.php:

switch (\TYPO3\CMS\Core\Utility\GeneralUtility::getApplicationContext()) {
   case 'Development':
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 1;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '*';
      break;
   case 'Production/Staging':
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 0;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '192.168.1.*';
      break;
   default:
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['displayErrors'] = 0;
      $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] = '127.0.0.1';
}

Variables and Constants

After TYPO3's bootstrap sequence has completed, a number of global variables, constants and classes are available to any script.

Constants

Constants normally define paths and database information. These values are global and cannot be changed when they are first defined. This is why constants are used for such vital information.

These constants are defined at various points during the bootstrap sequence.

The column "Avail. in FE" is an indicator that tells you if the constant, variable or class mentioned is also available to scripts running under the frontend of the "cms" extension.

Note

To make the table below a bit more compact, namespaces were left out. Here are the fully qualified class names referred to below:

  • "SystemEnvironmentBuilder" = \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder
  • "Bootstrap" = \TYPO3\CMS\Core\Core\Bootstrap
Table 1: Traditional List
Constant Defined in Description Avail. in FE
TYPO3_MODE \TYPO3\CMS\Backend\Http\Application::defineLegacyConstants() \TYPO3\CMS\Core\Console\CommandApplication::defineLegacyConstants() \TYPO3\CMS\Frontend\Http\Application::defineLegacyConstants() \TYPO3\CMS\Install\Http\Application::defineLegacyConstants() Mode of TYPO3: Set to either "FE" or "BE" depending on frontend or backend execution and context.

Yes

value = "FE"

TYPO3_OS SystemEnvironmentBuilder::defineBaseConstants()

Note

this constant has been marked as deprecated and will be removed with TYPO3 v10. Use \TYPO3\CMS\Core\Core\Environment::getPublicPath() to retrieve the information. Use Environment::isWindows() and Environment::isUnix() instead.

Operating system; Windows = "WIN", other = "" (presumed to be some sort of Unix)

Yes
PATH_thisScript SystemEnvironmentBuilder::definePaths()

Note

this constant has been marked as deprecated and will be removed with TYPO3 v10. Use \TYPO3\CMS\Core\Core\Environment::getCurrentScript() to retrieve the information.

Abs. path to current script.

Yes
TYPO3_mainDir SystemEnvironmentBuilder::definePaths() This is the directory of the backend administration for the sites of this TYPO3 installation. Hardcoded to typo3/. Must be a subdirectory to the website. See elsewhere for descriptions on how to change the default admin directory, typo3/, to something else. Yes
PATH_typo3 SystemEnvironmentBuilder::definePaths()

Note

this constant has been marked as deprecated and will be removed with TYPO3 v10. Use \TYPO3\CMS\Core\Core\Environment::getPublicPath() . '/typo3' to retrieve the information.

Abs. path of the TYPO3 admin dir.

No
PATH_site SystemEnvironmentBuilder::definePaths()

Note

this constant has been marked as deprecated and will be removed with TYPO3 v10. Use \TYPO3\CMS\Core\Core\Environment::getPublicPath() to retrieve the information.

Absolute path to directory with the frontend (one directory above \TYPO3\CMS\Core\Core\Environment::getPublicPath() . '/typo3/)

Yes
PATH_typo3conf SystemEnvironmentBuilder::definePaths()

Note

this constant has been marked as deprecated and will be removed with TYPO3 v10. Use \TYPO3\CMS\Core\Core\Environment::getPublicPath() . '/typo3conf' to retrieve the information.

Absolute TYPO3 configuration path (local, not part of source).

Yes
TYPO3_version SystemEnvironmentBuilder::defineBaseConstants() The TYPO3 version, as a "x.y.z" number. Development versions will be either "x.y.z-dev" for stable versions or "x.y-dev" for the current master. Yes
TYPO3_branch SystemEnvironmentBuilder::defineBaseConstants() The TYPO3 version Branch, as a "x.y" number. Without the patch level. Yes
Table 2: Base Constants

Check \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::defineBaseConstants() for updates.

String constants
Operating system identifier
Service error constants
Constant Value Description
T3_ERR_SV_GENERAL -1 General error - something went wrong
T3_ERR_SV_NOT_AVAIL -2 During execution it showed that the service is not available and should be ignored. The service itself should call $this->setNonAvailable()
T3_ERR_SV_WRONG_SUBTYPE -3 Passed subtype is not possible with this service
T3_ERR_SV_NO_INPUT -4 Passed subtype is not possible with this service
T3_ERR_SV_FILE_NOT_FOUND -20 File not found which the service should process
T3_ERR_SV_FILE_READ -21 File not readable
T3_ERR_SV_FILE_WRITE -22 File not writable
T3_ERR_SV_PROG_NOT_FOUND -40 Passed subtype is not possible with this service
T3_ERR_SV_PROG_FAILED -41 Passed subtype is not possible with this service
Filetypes

Different types of files constants are defined in TYPO3\CMS\Core\Resource\AbstractFile. These constants are available for different groups of files as documented in https://www.iana.org/assignments/media-types/media-types.xhtml

These file types are assigned to all FAL resources. They can, for example, be used in Fluid to decide how to render different types of files.

Constant Value Description
FILETYPE_UNKNOWN 0 Unknown
FILETYPE_TEXT 1 Any kind of text
FILETYPE_IMAGE 2 Any kind of image
FILETYPE_AUDIO 3 Any kind of audio
FILETYPE_VIDEO 4 Any kind of video
FILETYPE_APPLICATION 5 Any kind of application
HTTP status codes

The different status codes available are defined in TYPO3\CMS\Core\Utility\HttpUtility. These constants are defined as documented in https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml

Global variables

Note

Variables in italics may be set in a script prior to the bootstrap process so they are optional.

Note

To make the table below a bit more compact, namespaces were left out. Here are the fully qualified class names referred to below:

  • "SystemEnvironmentBuilder" = \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder
  • "Bootstrap" = \TYPO3\CMS\Core\Core\Bootstrap
  • "PackageManager" = \TYPO3\CMS\Core\Package\PackageManager
Global variable Defined in Description Avail. in FE
$GLOBALS['TYPO3_CONF_VARS'] typo3/sysext/core/Configuration/DefaultConfiguration.php TYPO3 configuration array. Please refer to file typo3/sysext/core/Configuration/DefaultConfigurationDescription.php where each option is described in detail in the comments. The same comments are also available in the Install Tool when you choose "All Configuration". Yes
$TYPO3_LOADED_EXT PackageManager::loadPackageManagerStatesFromCache() PackageManager::initializeCompatibilityLoadedExtArray() Array with all loaded extensions listed with a set of paths. You can check if an extension is loaded by the function \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::isLoaded($key) where $key is the extension key. Yes
$EXEC_TIME SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables()

Is set to time() so that the rest of the script has a common value for the script execution time.

Note

Should not be used anymore, rather use the DateTime Aspect.

YES
$SIM_EXEC_TIME SystemEnvironmentBuilder::initializeGlobalTimeTrackingVariables()

Is set to $EXEC_TIME but can be altered later in the script if we want to simulate another execution-time when selecting from e.g. a database (used in the frontend for preview of future and past dates)

Note

Should not be used anymore, rather use the DateTime Aspect.

Yes
$PAGES_TYPES typo3/sysext/core/ext_tables.php See Page types (occasionally)
$TCA Bootstrap::loadExtensionTables() See TCA Reference Yes, partly
$TBE_MODULES typo3/sysext/core/ext_tables.php The backend main/sub-module structure. See section elsewhere plus source code of class \TYPO3\CMS\Backend\Module\ModuleLoader which also includes some examples. (occasionally)
$TBE_STYLES typo3/sysext/core/ext_tables.php Contains information related to BE skinning. (will be removed on CMS 9) (occasionally)
$T3_SERVICES SystemEnvironmentBuilder::initializeGlobalVariables() Global registration of services. Yes
$T3_VAR SystemEnvironmentBuilder::initializeGlobalVariables()

Space for various internal global data storage in TYPO3. Each key in this array is a data space for an application. Keys currently defined for use is:

['callUserFunction'] + ['callUserFunction_classPool']: Used by \TYPO3\CMS\Core\Utility\GeneralUtility::callUserFunction to store singleton objects.

['RTEobj'] : Used to hold the current RTE object if any. See \TYPO3\CMS\Backend\Utility\BackendUtility.

['ext'][ extension-key ] : Free space for extensions.

Yes
$BE_USER Bootstrap::initializeBackendUser() Backend user object. See Backend User Object. (depends)
$TBE_MODULES_EXT [In ext_tables.php files of extensions] Used to store information about modules from extensions that should be included in "function menus" of real modules. See the Extension API for details. (occasionally)
$TCA_DESCR [tables.php files] Can be set to contain file references to local lang files containing TCA_DESCR labels. See section about Context Sensitive Help. No
Exploring global variables

Many of the global variables described above can be inspected using the ADMIN TOOLS > Configuration module.

Warning

This module is always viewed in the BE context. Variables defined only in the FE context will not be visible there.

Note

This module is purely a browser. It does not let you change values.

It also lets you browse a number of other global arrays. Just be curious and investigate!

The Configuration module in **ADMIN TOOLS**

Viewing the $GLOBALS['TYPO3_CONF_VARS] array using the ADMIN TOOLS > Configuration module

TYPO3_CONF_VARS

However the main configuration is achieved via a set of global settings stored in a global array called $GLOBALS['TYPO3_CONF_VARS'].

This chapter describes this global configuration in more details and hints at other configuration possibilities.

File LocalConfiguration.php

The global configuration is stored in file typo3conf/LocalConfiguration.php. This file overrides default settings from typo3/sysext/core/Configuration/DefaultConfiguration.php.

Important

Since configuration settings can be manipulated from within the TYPO3 CMS backend, the typo3conf/LocalConfiguration.php must be writable by the web server user.

The local configuration file is basically a long array which is simply returned when the file is included. It represents the global TYPO3 CMS configuration. This configuration can be modified/extended/overridden by extensions, by setting configuration options inside an extension's ext_localconf.php file. See extension files and locations for more details about extension structure.

A typical content of typo3conf/LocalConfiguration.php looks like this:

<?php
return [
   'BE' => [
      'debug' => true,
      'explicitADmode' => 'explicitAllow',
      'installToolPassword' => '$P$Cbp90UttdtIKELNrDGjy4tDxh3uu9D/',
      'loginSecurityLevel' => 'normal',
   ],
   'DB' => [
      'Connections' => [
         'Default' => [
            'charset' => 'utf8',
            'dbname' => 'empty_typo3',
            'driver' => 'mysqli',
            'host' => '127.0.0.1',
            'password' => 'foo',
            'port' => 3306,
            'user' => 'bar',
         ],
      ],
   ],
   'EXTCONF' => [
       'lang' => [
           'availableLanguages' => [
               'de',
               'eo',
           ],
       ],
   ],
   'EXTENSIONS' => [
       'backend' => [
           'backendFavicon' => '',
           'backendLogo' => '',
           'loginBackgroundImage' => '',
           'loginFootnote' => '',
           'loginHighlightColor' => '',
           'loginLogo' => '',
       ],
       'extensionmanager' => [
           'automaticInstallation' => '1',
           'offlineMode' => '0',
       ],
       'scheduler' => [
           'maxLifetime' => '1440',
           'showSampleTasks' => '1',
       ],
   ],
   'FE' => [
      'debug' => true,
      'loginSecurityLevel' => 'normal',
   ],
   'GFX' => [
      'jpg_quality' => '80',
   ],
   'MAIL' => [
      'transport_sendmail_command' => '/usr/sbin/sendmail -t -i ',
   ],
   'SYS' => [
      'devIPmask' => '*',
      'displayErrors' => 1,
      'encryptionKey' => '0396e1b6b53bf48b0bfed9e97a62744158452dfb9b9909fe32d4b7a709816c9b4e94dcd69c011f989d322cb22309f2f2',
      'exceptionalErrors' => 28674,
      'sitename' => 'New TYPO3 site',
      'systemLogLevel' => 0,
   ],
];

As you can see, the array is structured on two main levels. The first level corresponds roughly to a category, the second one being properties, which may themselves be arrays.

The configuration categories are:

BE
Options related to the TYPO3 CMS backend
DB
Database connection configuration
EXTCONF
Backend related language pack configuration resides here.
EXTENSIONS
Extension specific settings
FE
Frontend-related options.
GFX
Options related to image manipulation.
MAIL
Options related to the sending of emails (transport, server, etc.).
SYS
General options which may affect both the frontend and the backend.

Details on the various configuration options can be found in the Install Tool as well as the TYPO3 source at typo3/sysext/core/Configuration/DefaultConfigurationDescription.yaml. The documentation shown in the Install Tool is automatically extracted from those values of DefaultConfigurationDescription.yaml.

The Install Tool provides various dedicated modules that change parts of LocalConfiguration.php, those can be found in ADMIN TOOLS > Settings, most importantly section Configure installation-wide options:

Configure installation-wide options in Install Tool with an active search
File AdditionalConfiguration.php

Although you can manually edit the typo3conf/LocalConfiguration.php file, it is limited in scope because the file is expected to return a PHP array. Also the file is rewritten every time an option is changed in the Install Tool or some other operation (like changing an extension configuration in the Extension Manager). Thus custom code cannot reside in that file.

Such code should be placed in the typo3conf/AdditionalConfiguration.php file. This file is never touched by TYPO3, so any code will be left alone.

Furthermore this file is loaded after typo3conf/LocalConfiguration.php, which means it represents an opportunity to change global configuration values programmatically if needed.

typo3conf/AdditionalConfiguration.php is a plain PHP file. There are no specific rules about what it may contain. However since the code it contains is included on every request to TYPO3 CMS - whether frontend or backend - you should avoid inserting code which requires heavy duty processing.

File DefaultConfiguration.php

TYPO3 CMS comes with some default settings, which are defined in file typo3/sysext/core/Configuration/DefaultConfiguration.php.

This is the base configuration, the other files like LocalConfiguration.php just overlay it.

Here is an extract of that file:

return [
        'GFX' => [
                'thumbnails' => true,
                'thumbnails_png' => true,
                'gif_compress' => true,
                'imagefile_ext' => 'gif,jpg,jpeg,tif,tiff,bmp,pcx,tga,png,pdf,ai,svg',
                // ...
        ],
        // ...
];

You will probably find it interesting to take a look at that file, which also contains values not displayed in the Install Tool and thus not easily available for modification.

Backend Modules

TYPO3 CMS offers a number of ways to attach custom functionality to the backend. They are described in this chapter.

Backend interface

The backend interface is essentially driven by the "backend" system extension and extended by many other system extensions.

It is divided into the following main areas:

An overview of the visual structure of the backend
Top bar

The top bar is always present. It is itself divided into two areas: the logo and top bar tools.

The logo can be changed using the $GLOBALS['TBE_STYLES']['logo'] setting. Additional top bar tools can be registered using $GLOBALS['TYPO3_CONF_VARS']['BE']['toolbarItems'].

Module menu

This is the main navigation. All modules are structured in main modules (which can be collapsed) and submodules which is where the action really happens.

The module menu can be opened or closed by using the icon on the top left.

New main or submodules are registered using the \TYPO3\CMS\Extbase\Utility\ExtensionUtility::registerModule() API.

Note

In the TYPO3 CMS world, "module" is typically used for the backend. Extension components which add features in the frontend are referred to as "plugins".

Navigation frame

Any backend module may have a navigation frame or not. This frame will typically display the page tree or the folder tree, but custom navigation frames are possible.

The current location (i.e. page or frame) is carried over between navigation frames when changing modules. This means, for example, that when you move from the WEB > Page module to the WEB > List module, the same page stays selected in the page tree.

Docheader
This part is always located above the Content area. It will generally contain a drop-down menu called the "Function menu", which allows to navigate into the various functions offered by the module. When editing it will also contain all the buttons necessary for saving, closing or reverting. It may contain additional buttons for shortcuts or any specific feature needed by the module.
Content area
This is the actual work area. Any information to show or content to edit will be displayed here.
Contextual menus

(Right) clicking on record icons will often reveal a contextual menu. New functions can be added to the contextual menus, but the mechanisms vary: the page tree behaves differently than the rest of the backend.

A typical contextual menu appear when clicking on a record icon

The backend template view

Warning

Templating in the backend has been redesigned since a couple of major releases ago. This chapter describes the current new way of doing things. It may yet change. Please refer to older versions of this manual if you need a reference to the old way of programming backend modules.

Modern backend modules are written using the Extbase/Fluid combination. Thus, templates are Fluid-based. On top of that the "backend" system extension provides a general view class TYPO3\CMS\Backend\View\BackendTemplateView which provides common features for all backend modules, like the management of the action menu or the registration of docheader buttons.

This view class gives access to the \TYPO3\CMS\Backend\Template\ModuleTemplate class which is - more or less - the old backend module template, cleaned up and refreshed. This class performs a number of basic operations for backend modules, like loading base JS libraries, loading stylesheets, managing a flash message queue and - in general - performing all kind of necessary setups.

To access these resources, the trick is to force your backend module controller to use the TYPO3\CMS\Backend\View\BackendTemplateView class by changing the value of the $defaultViewObjectName member variable in the controller. Here is an example taken from system extension "beuser":

/**
 * Backend module user/group action controller
 */
class BackendUserActionController extends \TYPO3\CMS\Extbase\Mvc\Controller\ActionController
{
        /**
         * Backend Template Container
         *
         * @var string
         */
        protected $defaultViewObjectName = \TYPO3\CMS\Backend\View\BackendTemplateView::class;

        // ...
}

After that, you can use the initializeView() method to build the general elements of your backend module. Again looking at the "beuser" extension:

/**
 * Set up the doc header properly here
 *
 * @param ViewInterface $view
 * @return void
 */
protected function initializeView(ViewInterface $view)
{
    /** @var BackendTemplateView $view */
    parent::initializeView($view);
    if ($this->actionMethodName == 'indexAction'
        || $this->actionMethodName == 'onlineAction'
        || $this->actionMethodName == 'compareAction') {
        $this->generateMenu();
        $this->registerDocheaderButtons();
        $this->view->getModuleTemplate()->setFlashMessageQueue($this->controllerContext->getFlashMessageQueue());
    }
    if ($view instanceof BackendTemplateView) {
        $view->getModuleTemplate()->getPageRenderer()->loadRequireJsModule('TYPO3/CMS/Backend/Modal');
    }
}

The main actions performed here are the generation of the action menu, the generation of buttons for the Docheader, the initialization of the Flash message queue and the registration of a JS library to be loaded using RequireJS.

Using this BackendTemplateView class, the Fluid templates for your module need only take care of the actual content of your module. As such, the Layout may be as simple as (again from "beuser"):

<f:render section="headline" />
<f:render section="content" />

and the actual Template needs to render the title and the content only. For example, here is an extract of the "Index" action template of the "beuser" extension:

{namespace be = TYPO3\CMS\Backend\ViewHelpers}
{namespace bu = TYPO3\CMS\Beuser\ViewHelpers}
{namespace core = TYPO3\CMS\Core\ViewHelpers}

<f:layout name="Default" />

<f:section name="headline">
        <h1><f:translate key="backendUserListing" /></h1>
</f:section>

<f:section name="content">
        ...
</f:section>

The best resources for learning is to look at existing modules from TYPO3 CMS. With the information given here, you should be able to find your way around the code.

Backend Module API

Registering new modules

Modules added by extensions are registered in the ext_tables.php using the following API:

// Module System > Backend Users
\TYPO3\CMS\Extbase\Utility\ExtensionUtility::registerModule(
    'TYPO3.CMS.Beuser',
    'system',
    'tx_Beuser',
    'top',
    [
        'BackendUser' => 'index, addToCompareList, removeFromCompareList, compare, online, terminateBackendUserSession',
        'BackendUserGroup' => 'index'
    ],
    [
        'access' => 'admin',
        'icon' => 'EXT:beuser/Resources/Public/Icons/module-beuser.svg',
        'labels' => 'LLL:EXT:beuser/Resources/Private/Language/locallang_mod.xlf',
    ]
);

Here the module tx_Beuser is declared as being a submodule of main module system. It should be placed at the top of that main module, if possible (if several modules are declared at the same position, the last one wins). The following positions are possible:

  • top: the module is prepended to the top of the submodule list
  • bottom or empty string: the module is appended to the end of the submodule list
  • before:<submodulekey>: the module is inserted before the submodule identified by <submodulekey>
  • after:<submodulekey>: the module is inserted after the submodule identified by <submodulekey>

The last array is the module configuration and contains important information: the module is accessible only to admin users. The following options are available and should be defined as comma-separated string:

  • admin: the module is accessible to admins only
  • user: the module can be made accessible per user
  • group: the module can be made accessible per usergroup

The configuration also contains pointers to the module icon and the language file containing labels like the module title and description, for building the module menu and for the display of information in the About Modules module (found in the main help menu in the top bar). The LLL: prefix is mandatory here and is there for historical reasons.

Registering a toplevel module

Toplevel modules like "Web" or "File" are registered with the same API:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::registerModule(
    'Vendor.MyExtension',
    'mysection',
    '',
    '',
    [],
    [
        'access' => '...',
        'icon' => '...',
        'labels' => '...',
    ]
);

This adds a new toplevel module mysection. This identifier can now be used to add submodules to this new toplevel module:

\TYPO3\CMS\Extbase\Utility\ExtensionUtility::registerModule(
    'Vendor.MyExtension',
    'mymodule1',
    'mysection',
    '',
    [],
    [
        'access' => '...',
        'labels' => '...'
    ]
);
$TBE_MODULES

When modules are registered, they get added to a global array called $GLOBALS['TBE_MODULES']. It contains the list of all registered modules, their configuration and the configuration of any existing navigation component (the components which may be loaded into the navigation frame).

$GLOBALS['TBE_MODULES'] can be explored using the SYSTEM > Configuration module.

Exploring the TBE_MODULES array using the Configuration module

The list of modules is parsed by the class \TYPO3\CMS\Backend\Module\ModuleLoader.

TYPO3 CMS shell scripts (CLI mode)

Besides the backend, it is also possible to run some TYPO3 CMS scripts from the command line. This makes it possible - for example - to set up cronjobs. There are two ways to register CLI scripts:

  • using the TYPO3 command-line dispatcher based on Symfony Commands.
  • creating an Extbase command controller.
The command-line dispatcher

TYPO3 uses Symfony commands to provide an easy to use, well-documented API for writing CLI commands.

Creating a new Symfony command in your extension

Symfony commands should extend the class Symfony\Component\Console\Command\Command.

TYPO3 looks in a file Commands.php in the Configuration folder of extensions for configured commands. The Commands.php file returns a simple array with the command name and class.

For example to add a command which can be called via bin/typo3 yourext:dothings add the following:

return [
    'yourext:dothings' => [
        'class' => \Vendor\Extension\Command\DoThingsCommand::class
    ],
];

The command should implement at least a configure and an execute method.

configure as the name would suggest allows to configure the command. Via configure a description or a help text can be added, or mandatory and optional arguments and parameters defined.

A simple example can be found in the ListSysLogCommand:

/**
 * Configure the command by defining the name, options and arguments
 */
public function configure()
{
    $this->setDescription('Show entries from the sys_log database table of the last 24 hours.');
    $this->setHelp('Prints a list of recent sys_log entries.' . LF . 'If you want to get more detailed information, use the --verbose option.');
}

The execute method contains the logic you want to execute when executing the command.

A detailed description and an example can be found at the Symfony Command Documentation.

Extbase command controllers

Note

If you do not need Extbase in your command it is recommended to directly use a Symfony command (see above).

First of all, the command controller must be registered in an extension's ext_localconf.php file (example taken from the "lang" system extension):

// Register language update command controller
$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['extbase']['commandControllers'][] = \TYPO3\CMS\Lang\Command\LanguageCommandController::class;

The class itself must extend the \TYPO3\CMS\Extbase\Mvc\Controller\CommandController class. Each action that should be available from the command line must be named following the pattern "[action name]Command". The PHPdoc information is directly used as help text (description of the action, what arguments it takes).

Here's an extract from the command controller class of the "lang" extension:

/**
 * Language command controller updates translation packages
 */
class LanguageCommandController extends \TYPO3\CMS\Extbase\Mvc\Controller\CommandController
{
    // ...

    /**
     * Update language file for each extension
     *
     * @param string $localesToUpdate Comma separated list of locales that needs to be updated
     * @return void
     */
    public function updateCommand($localesToUpdate = '')
    {
        // ...
    }
}

This command would be called by using:

$ /path/to/php bin/typo3 extbase language:update fr

which would update translation packages for the French language.

Backend User Object

The backend user of a session is always available to the backend scripts as the global variable $BE_USER. The object is created in \TYPO3\CMS\Core\Core\Bootstrap::initializeBackendUser() and is an instance of the class \TYPO3\CMS\Core\Authentication\BackendUserAuthentication (which extends \TYPO3\CMS\Core\Authentication\AbstractUserAuthentication).

In addition to $BE_USER one other global variables is of interest - $FILEMOUNTS, holding an array with the File mounts of the $BE_USER.

Checking user access

The $BE_USER object is mostly used to check user access right, but contains other helpful information. This is presented here by a few examples:

Checking access to current backend module

$MCONF is module configuration and the key $MCONF['access'] determines the access scope for the module. This function call will check if the $BE_USER is allowed to access the module and if not, the function will exit with an error message.

$BE_USER->modAccess($MCONF, 1);
Checking access to any backend module

If you know the module key you can check if the module is included in the access list by this function call:

$BE_USER->check('modules', 'web_list');

Here access to the module WEB > List is checked.

Access to tables and fields?

The same function ->check() can actually check all the ->groupLists inside $BE_USER. For instance:

Checking modify access to the table "pages":

$BE_USER->check('tables_modify', 'pages');

Checking read access to the table "tt_content":

$BE_USER->check('tables_select', 'tt_content');

Checking if a table/field pair is allowed explicitly through the "Allowed Excludefields":

$BE_USER->check('non_exclude_fields', $table . ':' . $field);
Is "admin"?

If you want to know if a user is an "admin" user (has complete access), just call this method:

$BE_USER->isAdmin();
Read access to a page?

This function call will return true if the user has read access to a page (represented by its database record, $pageRec):

$BE_USER->doesUserHaveAccess($pageRec, 1);

Changing the "1" for other values will check other permissions:

  • use "2" for checking if the user may edit the page
  • use "4" for checking if the user may delete the page.
Is a page inside a DB mount?

Access to a page should not be checked only based on page permissions but also if a page is found within a DB mount for ther user. This can be checked by this function call ($id is the page uid):

$BE_USER->isInWebMount($id)
Selecting readable pages from database?

If you wish to make a SQL statement which selects pages from the database and you want it to be only pages that the user has read access to, you can have a proper WHERE clause returned by this function call:

$BE_USER->getPagePermsClause(1);

Again the number "1" represents the "read" permission; "2" is "edit" and "4" is "delete" permission. The result from the above query could be this string:

((pages.perms_everybody & 1 = 1)OR(pages.perms_userid = 2 AND pages.perms_user & 1 = 1)OR(pages.perms_groupid in (1) AND pages.perms_group & 1 = 1))
Saving module data

This stores the input variable $compareFlags (an array!) with the key "tools_beuser/index.php/compare"

$compareFlags = \TYPO3\CMS\Core\Utility\GeneralUtility::_GP('compareFlags');
$BE_USER->pushModuleData('tools_beuser/index.php/compare', $compareFlags);
Getting module data

This gets the module data with the key "tools_beuser/index.php/compare" (lasting only for the session)

$compareFlags = $BE_USER->getModuleData('tools_beuser/index.php/compare', 'ses');
Getting TSconfig

This function can return a value from the "User TSconfig" structure of the user. In this case the value for "options.clipboardNumberPads":

$tsconfig = $BE_USER->getTSConfig('');
$clipboardNumberPads = $tsconfig['options.clipboardNumberPads'] ?? '';
Getting the username

The full "be_users" record of a authenticated user is available in $BE_USER->user as an array. This will return the "username":

$BE_USER->user['username']
Get User Configuration value

The internal ->uc array contains options which are managed by the User Tools > User Settings module (extension "setup"). These values are accessible in the $BE_USER->uc array. This will return the current state of "Notify me by email, when somebody logs in from my account" for the user:

$BE_USER->uc['emailMeAtLogin']

Context API and Aspects

Introduction

Context API encapsulates various information for data retrieval (e.g. inside the database) and analysis of current permissions and caching information.

Previously, various information was distributed inside globally accessible objects ($TSFE or $BE_USER) like the current workspace ID or if a frontend or backend user is authenticated. Having a global object available was also dependent on the current request type (frontend or backend), instead of having one consistent place where all this data is located.

The context is instantiated at the very beginning of each TYPO3 entry point, keeping track of the current time (formally known as $GLOBALS['EXEC_TIME'], if a user is logged in, and which workspace is currently accessed.

This information is separated in so-called "Aspects", each being responsible for a certain area:

DateTime Aspect

Contains time, date and timezone information for the current request.

In comparison to known behaviour until TYPO3 v9, DateTimeAspect replaces for example $GLOBALS['SIM_EXEC_TIME'] and $GLOBALS['EXEC_TIME'].

The DateTime Aspect accepts following properties:

Property Call Result
timestamp $context->getPropertyFromAspect('date', 'timestamp'); unix timestamp as integer value
timezone $context->getPropertyFromAspect('date', 'timezone'); timezone name, e.g. Germany/Berlin
iso $context->getPropertyFromAspect('date', 'iso'); datetime as string in ISO 8601 format, e.g. 2004-02-12T15:19:21+00:00
full $context->getPropertyFromAspect('date', 'full'); the complete DateTimeImmutable object
Example
$context = GeneralUtility::makeInstance(Context::class);

// Reading the current data instead of $GLOBALS['EXEC_TIME']
$currentTimestamp = $context->getPropertyFromAspect('date', 'timestamp');
Language Aspect

Contains information about language settings for the current request, including fallback and overlay logic.

In comparison to known behaviour until TYPO3 v9, LanguageAspect replaces various properties related to language Id, overlay and fallback logic, mostly within Frontend.

The Language Aspect accepts following properties:

Property Call Result
id $context->getPropertyFromAspect('language', 'id'); the requested language of the current page as integer (uid)
contentId $context->getPropertyFromAspect('language', 'contentId'); the language id of records to be fetched in translation scenarios as integer (uid)
fallbackChain $context->getPropertyFromAspect('language', 'fallbackChain'); the fallback steps as array
overlayType $context->getPropertyFromAspect('language', 'overlayType'); one of on, mixed, off or includeFloating
legacyLanguageMode $context->getPropertyFromAspect('language', 'legacyLanguageMode'); one of strict, ignore or content_fallback, kept for compatibility reasons. Don't use if not really necessary, the option will be removed rather sooner than later.
legacyOverlayType $context->getPropertyFromAspect('language', 'legacyLanguageMode'); one of hideNonTranslated, 0 or 1, kept for compatibility reasons. Don't use if not really necessary, the option will be removed rather sooner than later.

Replaced calls:

  • $TSFE->sys_language_uid -> id
  • $TSFE->sys_language_content -> contentId
  • $TSFE->sys_language_mode -> fallbackChain
  • $TSFE->sys_language_mode -> legacyLanguageMode
  • $TSFE->sys_language_contentOL -> legacyOverlayType
Example
$context = GeneralUtility::makeInstance(Context::class);

// Reading the current fallback chain instead $TSFE->sys_language_mode
$fallbackChain = $context->getPropertyFromAspect('language', 'fallbackChain');
User Aspect

Contains information about authenticated users in the current request. Can be used for frontend and backend users.

In comparison to known behaviour until TYPO3 v9, UserAspect replaces various calls and checks on $GLOBALS['BE_USER'] and $GLOBALS['TSFE']->fe_user options when only some information is needed.

The User Aspect accepts following properties:

Property Call Result
id $context->getPropertyFromAspect('backend.user', 'id'); uid of the currently logged in user, 0 if no user
username $context->getPropertyFromAspect('backend.user', 'username'); the username of the currently authenticated user. Empty string if no user.
isLoggedIn $context->getPropertyFromAspect('frontend.user', 'isLoggedIn'); whether a user is logged in, as boolean.
isAdmin $context->getPropertyFromAspect('backend.user', 'isAdmin'); whether the user is admin, as boolean. Only useful for BEuser.
groupIds $context->getPropertyFromAspect('backend.user', 'groupIds'); the groups the user is a member of, as array
groupNames $context->getPropertyFromAspect('frontend.user', 'groupNames'); the names of all groups the user belongs to, as array
Example
$context = GeneralUtility::makeInstance(Context::class);

// Checking if a user is logged in
$userIsLoggedIn = $context->getPropertyFromAspect('frontend.user', 'isLoggedIn');
Visibility Aspect

The aspect contains whether to show hidden pages, records (content) or even deleted records.

In comparison to known behaviour until TYPO3 v9, VisibilityAspect replaces for example $GLOBALS['TSFE']->showHiddenPages and $GLOBALS['TSFE']->showHiddenRecords.

The Visibility Aspect accepts following properties:

Property Call Result
includeHiddenPages $context->getPropertyFromAspect('visibility', 'includeHiddenPages'); whether hidden pages should be displayed, as boolean
includeHiddenContent $context->getPropertyFromAspect('visibility', 'includeHiddenContent'); whether hidden content should be displayed, as boolean
includeDeletedRecords $context->getPropertyFromAspect('visibility', 'includeDeletedRecords'); whether deleted records should be displayed, as boolean.
Example
$context = GeneralUtility::makeInstance(Context::class);

// Checking if hidden pages should be displayed
$showHiddenPages = $context->getPropertyFromAspect('visibility', 'includeHiddenPages');
Workspace Aspect

The aspect contains information about the currently accessed workspace

In comparison to known behaviour until TYPO3 v9, WorkspaceAspect replaces e.g. $GLOBALS['BE_USER']->workspace.

The Workspace Aspect accepts following properties:

Property Call Result
id $context->getPropertyFromAspect('workspace', 'id'); UID of the currently accessed workspace as integer
isLive $context->getPropertyFromAspect('workspace', 'isLive'); whether the current workspace is live, or a custom offline WS, as boolean
isOffline $context->getPropertyFromAspect('workspace', 'isOffline'); whether the current workspace is offline, as boolean.
Example
$context = GeneralUtility::makeInstance(Context::class);

// Retrieving the uid of currently accessed workspace
$workspaceId = $context->getPropertyFromAspect('workspace', 'id');

TYPO3 Core Engine (TCE)

Introduction

Database

The TYPO3 Core Engine is the class that handles all *data* writing to database tables configured in $TCA. In addition the class handles commands such as copy, move, delete. It will handle undo/history and versioning of records and everything will be logged to the sys_log. And it will make sure that write permissions are evaluated correctly for the user trying to write to the database. Generally, any processing specific option in the $TCA array is handled by TCE.

Using TCE for manipulation of the database content in the $TCA-configured tables guarantees that the data integrity of TYPO3 is respected. This cannot be safely guaranteed if you write to $TCA-configured database tables directly. It will also manage the relations to files and other records.

TCE requires a backend login to work. This is due to the fact that permissions are observed (of course) and thus TCE needs a backend user to evaluate against. This means you cannot use DataHandler from the frontend scope. Thus writing to tables (such as a guestbook) will have to be done from the frontend without DataHandler.

The features of the $TCA are described in the TCA Reference.

Files

TCE also has a part for handling files. The file operations are normally performed in the FILE > Filelist module where you can manage a directory on the server by copying, moving, deleting and editing files and directories. The file operations are managed by two core classes, \TYPO3\CMS\Core\Utility\File\BasicFileUtility and \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility.

Database: DataHandler basics (formerly known as TCEmain)

When you are using TCE from your backend applications you need to prepare two arrays of information which contain the instructions to DataHandler (\TYPO3\CMS\Core\DataHandling\DataHandler) of what actions to perform. They fall into two categories: data and commands.

"Data" is when you want to write information to a database table or create a new record.

"Commands" is when you want to move, copy or delete a record in the system.

The data and commands are created as multidimensional arrays and to understand the API of DataHandler you simply need to understand the hierarchy of these two arrays.

Caution

The DataHandler needs a properly configured TCA. If your field is not configured in the TCA the DataHandler will not be able to interact with it. This also is the case if you configured "type"="none" (which is in fact a valid type) or if an invalid type is specified. In that case the DataHandler is not able to determine the correct value of the field.

Commands Array

Syntax:

$cmd[ tablename ][ uid ][ command ] = value

Description of keywords in syntax:

Key Data type Description
tablename string Name of the database table. Must be configured in $GLOBALS['TCA'] array, otherwise it cannot be processed.
uid integer The UID of the record that is manipulated. This is always an integer.
command string (command keyword)

The command type you want to execute.

Note

Only one command can be executed at a time for each record! The first command in the array will be taken.

See table below for command keywords and values

value mixed

The value for the command

See table below for command keywords and values

Command keywords and values
Command Data type Value
copy integer

The significance of the value depends on whether it is positive or negative:

  • Positive value: The value points to a page UID. A copy of the record (and possibly child elements/tree below) will be inserted inside that page as the first element.

  • Negative value: The (absolute) value points to another record from the same table as the record being copied. The new record will be inserted on the same page as that record and if $GLOBALS['TCA'][...]['ctrl']['sortby'] is set, then it will be positioned after.

  • Zero value: Record is inserted on tree root level.

  • array: The array has to contain the integer value as in examples above and may contain field => value pairs for updates. The array is structured like:

    [
       'action' => 'paste', // 'paste' is used for both move and copy commands
       'target' => $pUid, // Defines the page to insert the record, or record uid to copy after
       'update' => $update, // Array with field => value to be updated.
    ]
    
move integer Works like "copy" but moves the record instead of making a copy.
delete 1

Value should always be "1"

This action will delete the record (or mark the record "deleted" if configured in $GLOBALS['TCA']).

undelete 1

Value should always be "1".

This action will set the deleted-flag back to 0.

localize integer

Value is an uid of the sys_language to localize the record into. Basically a localization of a record is making a copy of the record (possibly excluding certain fields defined with l10n_mode) but changing relevant fields to point to the right sys_language / original language record.

Requirements for a successful localization is this:

  • [ctrl] options "languageField" and "transOrigPointerField" must be defined for the table
  • A sys_language record with the given sys_language_uid must exist.
  • The record to be localized by currently be set to "Default" language and not have any value set for the TCA transOrigPointerField either.
  • There cannot exist another localization to the given language for the record (looking in the original record PID).

Apart from this, ordinary permissions apply as if the user wants to make a copy of the record on the same page.

The localize DataHandler command should be used when translating records in "Connected Mode" (strict translation of records from the default language). This command is used when selecting the "Translate" strategy in the content elements translation wizard.

copyToLanguage integer

It behaves like localize command (both record and child records are copied to given language), but does not set transOrigPointerField fields (e.g. l10n_parent).

The copyToLanguage command should be used when localizing records in the "Free Mode". This command is used when localizing content elements using translation wizard's "Copy" strategy.

inlineLocalizeSynchronize array

Performs localization or synchronization of child records. The command structure is like:

$cmd['tt_content'][13]['inlineLocalizeSynchronize'] = [ // 13 is a parent record uid
  'field' => 'tx_myfieldname', // field we want to synchronize
  'language' => 2, // uid of the target language
  // either the key 'action' or 'ids' must be set
  'action' => 'localize' // or 'synchronize'
  'ids' =>  [1, 2, 3] // array of child-ids to be localized
]
version array

Versioning action.

Keys:

  • [action] : Keyword determining the versioning action. Options are:
    • "new": Indicates that a new version of the record should be created.Additional keys, specific for "new" action:
      • [treeLevels]: (Only pages) Integer, -1 to 4, indicating the number of levels of the page tree to version together with a page. This is also referred to as the versioning type:-1 ("element") means only the page record gets versioned (default)0 ("page") means the page + content tables (defined by ctrl-flag versioning_followPages )>0 ("branch") means the the whole branch is versioned ( full copy of all tables), down to the level indicated by the value (1= 1 level down, 2= 2 levels down, etc.). The treeLevel is recorded in the field t3ver_swapmode and will be observed when the record is swapped during publishing.
      • [label]: Indicates the version label to apply. If not given, a standard label including version number and date is added.
    • "swap": Indicates that the current online version should be swapped with another.Additional keys, specific for "swap" action:
      • [swapWith]: Indicates the uid of the record to swap current version with!
      • [swapIntoWS]: Boolean, indicates that when a version is published it should be swapped into the workspace of the offline record.
    • "clearWSID": Indicates that the workspace of the record should be set to zero (0). This removes versions out of workspaces without publishing them.
    • "flush": Completely deletes a version without publishing it.
    • "setStage": Sets the stage of an element. Special feature: The id- key in the array can be a comma list of ids in order to perform the stageChange over a number of records. Also, the internal variable ->generalComment (also available through `/record/commit` route as `&generalComment`) can be used to set a default comment for all stage changes of an instance of the data handler. Additional keys for this action are:
      • [stageId]: Values are: -1 (rejected), 0 (editing, default), 1 (review), 10 (publish)
      • [comment]: Comment string that goes into the log.
Examples of commands:
$cmd['tt_content'][54]['delete'] = 1;    // Deletes tt_content record with uid=54
$cmd['tt_content'][1203]['copy'] = -303; // Copies tt_content uid=1203 to the position after tt_content uid=303 (new record will have the same pid as tt_content uid=1203)
$cmd['tt_content'][1203]['copy'] = 400;  // Copies tt_content uid=1203 to first position in page uid=400
$cmd['tt_content'][1203]['move'] = 400;  // Moves tt_content uid=1203 to the first position in page uid=400
Data Array

Syntax:

$data[tablename][uid][fieldname] = value

Description of keywords in syntax:

Key Data type Description
tablename string Name of the database table. Must be configured in $GLOBALS['TCA'] array, otherwise it cannot be processed.
uid mixed The UID of the record that is modified. If the record already exists, this is an integer. If you're creating new records, use a random string prefixed with "NEW", e.g. "NEW7342abc5e6d".
fieldname string Name of the database field you want to set a value for. Must be configured in $GLOBALS['TCA'][*tablename*]['columns'].
value string Value for "fieldname".

Note

For FlexForms the data array of the FlexForm field is deeper than three levels. The number of possible levels for FlexForms is infinite and defined by the data structure of the FlexForm. But FlexForm fields always end with a "regular value" of course.

Examples of Data submission

This creates a new page titled "The page title" as the first page inside page id 45:

$data['pages']['NEW9823be87'] = array(
   'title' => 'The page title',
   'subtitle' => 'Other title stuff',
   'pid' => '45'
);

This creates a new page titled "The page title" right after page id 45 in the tree:

$data['pages']['NEW9823be87'] = array(
   'title' => 'The page title',
   'subtitle' => 'Other title stuff',
   'pid' => '-45'
);

This creates two new pages right after each other, located right after the page id 45:

$data['pages']['NEW9823be87'] = array(
   'title' => 'Page 1',
   'pid' => '-45'
);
$data['pages']['NEWbe68s587'] = array(
   'title' => 'Page 2',
   'pid' => '-NEW9823be87'
);

Notice how the second "pid" value points to the "NEW..." id placeholder of the first record. This works because the new id of the first record can be accessed by the second record. However it works only when the order in the array is as above since the processing happens in that order!

This creates a new content record with references to existing and one new system category:

$data['sys_category']['NEW9823be87'] = array(
    'title' => 'New category',
    'pid' => 1,
);
$data['tt_content']['NEWbe68s587'] = array(
    'header' => 'Look ma, categories!',
    'pid' => 45,
    'categories' => array(
        1,
        2,
        'NEW9823be87', // You can also use placeholders here
    ),
);

Note

To get real uid of the record you have just created use DataHandler's substNEWwithIDs property like: $uid = $dataHandler->substNEWwithIDs['NEW9823be87'];

This updates the page with uid=9834 to a new title, "New title for this page", and no_cache checked:

$data['pages'][9834] = array(
    'title' => 'New title for this page',
    'no_cache' => '1'
);
Clear cache

TCE also has an API for clearing the cache tables of TYPO3:

Syntax:

$tce->clear_cacheCmd($cacheCmd);
$cacheCmd values Description
[integer] Clear the cache for the page id given.
"all"

Clears all cache tables (cache_pages, cache_pagesection, cache_hash).

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.all".

"pages"

Clears all pages from cache_pages.

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.pages".

"temp_cached" or "system"

Clears all cache entries cache group system.

Only available for admin-users unless explicitly allowed by User TSconfig "options.clearCache.system".

Hook for cache post-processing

You can configure cache post-processing with a user defined PHP function. Configuration of the hook can be done from ext_localconf.php. An example might look like:

$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['clearCachePostProc'][] = \Vendor\Package\Hook\DataHandlerHook::class . '->postProcessClearCache';
Flags in DataHandler

There are a few internal variables you can set prior to executing commands or data submission. These are the most significant:

Internal variable Data type Description
->deleteTree Boolean

Sets whether a page tree branch can be recursively deleted.

If this is set, then a page is deleted by deleting the whole branch under it (user must have delete permissions to it all). If not set, then the page is deleted only if it has no branch.

Default is false.

->copyTree Integer

Sets the number of branches on a page tree to copy.

If 0 then branch is not copied. If 1 then pages on the 1st level is copied. If 2 then pages on the second level is copied, and so on.

Default is zero.

->reverseOrder Boolean

If set, the data array is reversed in the order, which is a nice thing if you're creating a whole bunch of new records.

Default is zero.

->copyWhichTables list of strings (tables)

This list of tables decides which tables will be copied. If empty then none will. If "*" then all will (that the user has permission to of course).

Default is "*".

Using DataHandler in scripts

It's really easy to use the class \TYPO3\CMS\Core\DataHandling\DataHandler in your own scripts. All you need to do is include the class, build a $data/$cmd array you want to pass to the class and call a few methods.

Important

Mind that these scripts have to be run in the backend scope! There must be a global $BE_USER object.

In your script you simply insert this line to include the class:

What follows are a few code listings with comments which will provide you with enough knowledge to get started. It is assumed that you have populated the $data and $cmd arrays correctly prior to these chunks of code. The syntax for these two arrays is explained in the previous chapter.

DataHandler examples
Submitting data

This is the most basic example of how to submit data into the database. It is four lines. Line 1 instantiates the class, line 2 defines that values will be provided without escaped characters (recommended!), line 3 registers the $data array inside the class and initializes the class internally! Finally line 4 will execute the data submission.

1
2
3
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\DataHandling\DataHandler::class);
$tce->start($data, array());
$tce->process_datamap();
Executing commands

The most basic way of executing commands. Line 1 creates the object, line 2 defines that values will be provided without escaped characters (recommended), line 3 registers the $cmd array inside the class and initializes the class internally! Finally line 4 will execute the commands.

1
2
3
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\DataHandling\DataHandler::class);
$tce->start(array(), $cmd);
$tce->process_cmdmap();
Clearing cache

In this example the cache clearing API is used. No data is submitted, no commands executed. Still you will have to initialize the class by calling the start() method (which will initialize internal variables).

Note

Clearing a given cache is possible only for users that are "admin" or have specific permissions to do so.

1
2
3
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\DataHandling\DataHandler::class);
$tce->start(array(), array());
$tce->clear_cacheCmd('all');

Since TYPO3 CMS 6.2, caches are organized in groups. Clearing "all" caches will actually clear caches from the "all" group and not really all caches. Check the caching framework architecture section for more details about available caches and groups.

Complex data submission

Imagine the $data array something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$data = array(
    'pages' => array(
        'NEW_1' => array(
            'pid' => 456,
            'title' => 'Title for page 1',
        ),
        'NEW_2' => array(
            'pid' => 456,
            'title' => 'Title for page 2',
        ),
    )
);

This aims to create two new pages in the page with uid "456". In the follow code this is submitted to the database. Notice how line 3 reverses the order of the array. This is done because otherwise "page 1" is created first, then "page 2" in the same PID meaning that "page 2" will end up above "page 1" in the order. Reversing the array will create "page 2" first and then "page 1" so the "expected order" is preserved.

To insert a record after a given record, set the other record's negative uid as pid in the new record you're setting as data.

Apart from this line 6 will send a "signal" that the page tree should be updated at the earliest occasion possible. Finally, the cache for all pages is cleared in line 7.

1
2
3
4
5
6
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\DataHandling\DataHandler::class);
$tce->reverseOrder = 1;
$tce->start($data, array());
$tce->process_datamap();
\TYPO3\CMS\Backend\Utility\BackendUtility::setUpdateSignal('updatePageTree');
$tce->clear_cacheCmd('pages');
Both data and commands executed with alternative user object

In this case it is shown how you can use the same object instance to submit both data and execute commands if you like. The order will depend on the order of line 4 and 5.

In line 3 the start() method is called, but this time with the third possible argument which is an alternative $BE_USER object. This allows you to force another backend user account to create stuff in the database. This may be useful in certain special cases. Normally you should not set this argument since you want TCE to use the global $BE_USER.

1
2
3
4
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Core\DataHandling\DataHandler::class);
$tce->start($data, $cmd, $alternative_BE_USER);
$tce->process_datamap();
$tce->process_cmdmap();

The "/record/commit" route

This route is a gateway for posting form data to the \TYPO3\CMS\Backend\Controller\SimpleDataHandlerController.

You can send data to this file either as GET or POST vars where POST takes precedence. The variable names you can use are:

GP var name Data type Description
data array

Data array on the form [tablename][uid][fieldname] = value.

Typically it comes from a POST form which submits a form field like <input name="data[tt_content][123][header]" value="This is the headline" />.

cmd array

Command array on the form [tablename][uid][command] = value. This array may get additional data set internally based on clipboard commands send in CB var!

Typically this comes from GET vars passed to the script like &cmd[tt\_content][123][delete]=1 which will delete Content Element with UID 123.

cacheCmd string Cache command sent to ->clear_cacheCmd
redirect string Redirect URL. Script will redirect to this location after performing operations (unless errors has occurred)
flags array Accepts options to be set in TCE object. Currently it supports "reverseOrder" (boolean).
mirror array Example: [mirror][table][11] = '22,33' will look for content in [data][table][11] and copy it to [data][table][22] and [data][table][33].
CB array Clipboard command array. May trigger changes in "cmd".
vC string Verification code

File functions basics

File operations in the TCE are handled by the class \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility which extends \TYPO3\CMS\Core\Utility\File\BasicFileUtility. The instructions for file manipulation are passed to this class as a multidimensional array.

Files Array

Syntax:

$file[ command ][ index ][ key ] = value

Description of keywords in syntax:

Key Data type Description
command string (command keyword)

The command type you want to execute.

See table below for command keywords, keys and values

index integer Integer index in the array which separates multiple commands of the same type.
key string

Depending on the command type. The keys will carry the information needed to perform the action. Typically a "target" key is used to point to the target directory or file while a "data" key carries the data.

See table below for command keywords, keys and values

value string

The value for the command

See table below for command keywords, keys and values

Command keywords and values
Command Keys Value
delete "data" "data" = Absolute path to the file/folder to delete
copy

"data"

"target"

"altName"

"data" = Absolute path to the file/folder to copy

"target" = Absolute path to the folder to copy to (destination)

"altName" = (boolean): If set, a new filename is made by appending numbers/unique-string in case the target already exists.

move

"data"

"target"

"altName"

(Exactly like copy, just replace the word "copy" with "move")
rename

"data"

"target"

"data" = New name, max 30 characters alphanumeric

"target" = Absolute path to the target file/folder

newfolder

"data"

"target"

"data" = Folder name, max 30 characters alphanumeric

"target" = Absolute path to the folder where to create it

newfile

"data"

"target"

"data" = New filename

"target" = Absolute path to the folder where to create it

editfile

"data"

"target"

"data" = The new content

"target" = Absolute path to the target file

upload

"data"

"target"

upload_$id

"data" = ID-number (points to the global var that holds the filename- ref ($_FILES["upload_" . $id]["name"]).

"target" = Absolute path to the target folder (destination)

upload_$id = File reference. $id must equal value of file[upload][...][data]!

See \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility::func_upload().

unzip

"data"

"target"

"data" = Absolute path to the zip-file. (file extension must be "zip")

"target" = The absolute path to the target folder (destination) (if not set, default is the same as the zip-file)

It is unlikely that you will need to use this internally in your scripts like you will need \TYPO3\CMS\Core\DataHandling\DataHandler. It is fairly uncommon to need the file manipulations in own scripts unless you make a special application. Therefore the most typical usage of this API is from TYPO3CMSBackendControllerFileFileController and the core scripts that are activated by the "File > List" module.

However, if needed, this is an example of how to initialize usage. It is taken from ImportExportController.php:

1
2
3
4
5
6
   // Initializing:
$this->fileProcessor = GeneralUtility::makeInstance(ExtendedFileUtility::class);
$this->fileProcessor->setActionPermissions();

$this->fileProcessor->start($this->file);
$this->fileProcessor->processData();

Explanation: Line 2 creates an instance of the class. Then the file operation permissions are loaded from the user object in line 3. Finally, the file command array is loaded in line 5 and internally additional configuration takes place according to $GLOBALS['TYPO3_CONF_VARS']!. In line 6 the command map is executed.

The "tce_file.php" API

This script serves as the file administration part of the TYPO3 Core Engine. It's a gateway for TCE (TYPO3 Core Engine) file-handling through POST forms. It uses \TYPO3\CMS\Core\Utility\File\ExtendedFileUtility for the manipulation of the files.

This script is used from the File > List module where you can rename, create, delete etc. files and directories on the server.

You can send data to this file either as GET or POST vars where POST takes precedence. The variable names you can use are:

GP var name Data type Description
file array

Array of file operations. See previous information about basic file functions.

This could typically be a GET var like &file[delete][0][data]=[absolute file path] or a POST form field like:

"<input type="text" name="file[newfolder][0][data]" value=""/>
<input type="hidden" name="file[newfolder][0][target]"
value="[absolute path to folder to create in]"/>"
redirect string Redirect URL. Script will redirect to this location after performing operations.
CB array Clipboard command array. May trigger changes in "file"
vC string Verification code
overwriteExistingFiles boolean If existing files should be overridden.

FormEngine

FormEngine renders records in the backend. This chapter explains the main code logics behind and how the rendering can be influenced and extended on a PHP developer level. Record editing can also be configured and fine tuned by integrators using Page TSconfig, see the according section of the Page TSconfig reference for details.

Introduction

Looking at TYPO3's main constructs from an abstract position, the system splits into three most important pillars:

DataHandler
TYPO3\CMS\Core\DataHandling\...: Construct taking care of persisting data into the database. The DataHandler takes an array representing one or more records, inserts, deletes or updates them in the database and takes care of relations between multiple records. If editing content in the backend, this construct does all main database munging. DataHandler is fed by some controller that most often gets GET or POST data from FormEngine.
FormEngine
TYPO3\CMS\Backend\Form\...: FormEngine renders records, usually in the backend. It creates all the HTML needed to edit complex data and data relations. Its GET or POST data is then fed to the DataHandler by some controller.
Frontend rendering
TYPO3\CMS\Frontend\...: Renders the website frontend. The frontend rendering, usually based on TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController uses TypoScript and / or Fluid to process and render database content into the frontend.

The glue between these three pillars is TCA (Table Configuration Array): It defines how database tables are constructed, which localization or workspace facilities exist, how it should be displayed in the backend, how it should be written to the database, and - next to TypoScript - which behaviour it has in the frontend.

This chapter is about FormEngine. It is important to understand this construct is based on TCA and is usually used in combination with the DataHandler. However, FormEngine is constructed in a way that it can work without DataHandler: A controller could use the FormEngine result and process it differently. Furthermore, all dependencies of FormEngine are abstracted and may come from "elsewhere", still leading to the form output known for casual records.

This makes FormEngine an incredible flexible construct. The basic idea is "feed something that looks like TCA and render forms that have the full power of TCA but look like all other parts of the backend".

The FormEngine code base has been significantly refactored in TYPO3 CMS version 7 and version 8 to be much more flexible, more easy to use and extend, and much more powerful than before. This is an ongoing process and some areas still need a major overhaul. The current state of the documentation aims to explain the main constructs of FormEngine and gives an insight on how to re-use, adapt and extend it with extensions. The core team expects to see more usages of FormEngine within core itself and within extensions in the future, and encourages developers to solve feature needs based on FormEngine. With the ongoing changes, those areas that may need code adaptions in the foreseeable future have notes within the documentation and developers should be available to adapt with younger cores. Watch out for breaking changes if using FormEngine and updating core.

Main rendering workflow

This is done by example. The details to steer and how to use only sub-parts of the rendering chain are explained in more detail in the following sections.

Editing a record in the backend - often from within the Page or List module - triggers the EditDocumentController by routing definitions using UriBuilder->buildUriFromRoute($moduleIdentifier) and handing over which record of which table should be edited. This can be an existing record, or it could be a command to create the form for a new record. The EditDocumentController is the main logic triggered whenever an editor changes a record!

The EditDocumentController has two main jobs: Trigger rendering of one or multiple records via FormEngine, and hand over any given data by a FormEngine POST result over to the DataHandler to persist stuff in the database.

The rendering part of the EditDocumentController job splits into these parts:

  • Initialize main FormEngine data array using POST or GET data to specify which specific record(s) should be edited.
  • Select which group of DataProviders should be used.
  • Trigger FormEngine DataCompiler to enrich the initialized data array with further data by calling all data providers specified by selected data provider group.
  • Hand over DataCompiler result to an entry "render container" of FormEngine and receive a result array.
  • Take result array containing HTML, CSS and JavaScript details and put them into FormResultCompiler which hands them over to the PageRenderer.
  • Let the PageRenderer output its compiled result.
Main FormEngine workflow

The controller does two distinct things here: First, it initializes a data array and lets it get enriched by data providers of FormEngine which add all information needed for the rendering part. Then feed this data array to the rendering part of FormEngine to end up with a result array containing all HTML, CSS and JavaScript.

In code, this basic workflow looks like this:

$formDataGroup = GeneralUtility::makeInstance(TcaDatabaseRecord::class);
$formDataCompiler = GeneralUtility::makeInstance(FormDataCompiler::class, $formDataGroup);
$nodeFactory = GeneralUtility::makeInstance(NodeFactory::class);
$formResultCompiler = GeneralUtility::makeInstance(FormResultCompiler::class);
$formDataCompilerInput = [
    'tableName' => $table,
    'vanillaUid' => (int)$theUid,
    'command' => $command,
];
$formData = $formDataCompiler->compile($formDataCompilerInput);
$formData['renderType'] = 'outerWrapContainer';
$formResult = $nodeFactory->create($formData)->render();
$formResultCompiler->mergeResult($formResult);

This basically means the main FormEngine concept is a two-fold process: First create an array to gather all render-relevant information, then call the render engine using this array to come up with output.

This two-fold process has a number of advantages:

  • The data compiler step can be regulated by a controller to only enrich with stuff that is needed in any given context. This part is supported by encapsulating single data providers in data groups, single data providers can be omitted if not relevant in given scope.
  • Data providing and rendering is split: Controllers could re-use the rendering part of FormEngine while all or parts of the data providers are omitted, or their data comes from "elsewhere". Furthermore, controllers can re-use the data providing part of FormEngine and output the result in an entirely different way than HTML. The latter is for instance used when FormEngine is triggered for a TCA tree by an ajax call and thus outputs a JSON array.
  • The code constructs behind "data providing" and "rendering" can be different to allow higher re-use and more flexibility with having the "data array" as main communication base in between. This will become more obvious in the next sections where it is shown that data providers are a linked list, while rendering is a tree.

Data compiling

This is the first step of FormEngine. The data compiling creates an array containing all data the rendering needs to come up with a result.

A basic call looks like this:

$formDataGroup = GeneralUtility::makeInstance(TcaDatabaseRecord::class);
$formDataCompiler = GeneralUtility::makeInstance(FormDataCompiler::class, $formDataGroup);
$formDataCompilerInput = [
   'tableName' => $table,
   'vanillaUid' => (int)$theUid,
   'command' => $command,
];
$formData = $formDataCompiler->compile($formDataCompilerInput);

The above code is a simplified version of the relevant part of the EditDocumentController. This controller knows by its GET or POST parameters which record ("vanillaUid") of which specific table ("tableName") should be edited (command="edit") or created (command="new"), and sets this as init data to the DataCompiler. The controller also knows that it should render a full database record and not only parts of it, so it uses the TcaDatabaseRecord data provider group to trigger all data providers relevant for this case. By calling ->compile() on this data group, all providers configured for this group are called after each other, and formData ends up with a huge array of data record details.

So, what happens here in detail?

  • Variable $formDataCompilerInput maps input values to keys specified by FormDataCompiler as "init" data.
  • FormDataCompiler returns a unified array of data. This array is enriched by single data providers.
  • A data provider group is a list of single data providers for a specific scope and enriches the array with information.
  • Each data provider is called by the DataGroup to add or change data in the array.

The variable $formData roughly consists of this data after calling $formDataCompiler->compile():

  • A validated and initialized list of current database row field variables.
  • A processed version of $TCA['givenTable'] containing only those column fields a current user has access to.
  • A processed list of items for single fields like select and group types.
  • A list of relevant localizations.
  • Information of expanded inline record details if needed.
  • Resolved flex form data structures and data.
  • A lot more

Basic goal of this step is to create an array in a specified format with all data needed by the render-part of FormEngine. A controller initializes this with init data, and then lets single data providers fetch additional data and write it to the main array. The deal is here that the data within that array is not structured in an arbitrary way, and each single data provider only adds data the render part of FormEngine understands and needs later. This is why the main array keys are restricted: The main array is initialized by FormDataCompiler, and each DataProvider can only add data to sub-parts of that array.

Note

The main data array is prepared by FormDataCompiler, each key is well documented in this class. To find out which data is expected to reside in this array, those comments are worth a look.

Note

It may happen in future versions of FormEngine (core version 9+) that the responsibility for the main structure and integrity of the data array will be moved away from FormDataCompiler into the single FormDataGroup class. This may even make the FormDataCompiler obsolete in total.

Data groups and providers

So we have this empty data array, pre-set with data by a controller and then initialized by FormDataCompiler, which in turn hands over the data array to a specific FormDataGroup. What are these data providers now? Data providers are single classes that add or change data within the data array. They are called in a chain after each other. A FormDataGroup has the responsibility to find out, which specific single data providers should be used, and calls them in a specific order.

Data compiling by multiple providers
Why do we need this?
  • Which data providers are relevant depends on the specific scope: For instance, if editing a full database based record, one provider fetches the according row from the database and initializes $data['databaseRow'] . But if flex form data is calculated, the flex form values are fetched from table fields directly. So, while the DatabaseEditRow data provider is needed in the first case, it's not needed or even counter productive in the second case. The FormDataGroup's are used to manage providers for specific scopes.
  • FormDataGroups know which providers should be used in a specific scope. They usually fetch a list of providers from some global configuration array. Extensions can add own providers to this configuration array for further data munging.
  • Single data providers have dependencies to each other and must be executed in a specific order. For Instance, the PageTsConfig of a record can only be determined, if the rootline of a record has been determined, which can only happen after the pid of a given record has been consolidated, which relies on the record being fetched from the database. This makes data providers a linked list and it is the task of a FormDataGroup to manage the correct order.

Main data groups:

TcaDatabaseRecord
List of providers used if rendering a database based record.
FlexFormSegment
List of data providers used to prepare flex form data and flex form section container data.
TcaInputPlaceholderRecord
List of data providers used to prepare placeholder values for type=input and type=text fields.
InlineParentRecord
List of data providers used to prepare data needed if an inline record is opened from within an ajax call.
OnTheFly
A special data group that can be initialized with a list of to-execute data providers directly. In contrast to the others, it does not resort the data provider list by its dependencies and does not fetch the list of data providers from a global config. Used in the core at a couple of places, where a small number of data providers should be called right away without being extensible.

Note

It is a good idea to set a breakpoint at the form data result returned by the DataCompiler and to have a look at the data array to get an idea of what this array contains after compiling.

Let's have a closer look at the data providers. The main TcaDatabaseRecord group consists mostly of three parts:

Main record data and dependencies
  • Fetch record from DB or initialize a new row depending on $data['command'] being "new" or "edit", set row as $data['databaseRow']
  • Add userTs and pageTsConfig to data array
  • Add table TCA as $data['processedTca']
  • Determine record type value
  • Fetch record translations and other details and add to data array
Single field processing
  • Process values and items of simple types like type=input, type=radio, type=check and so on. Validate their databaseRow values and validate and sanitize their processedTca settings.
  • Process more complex types that may have relations to other tables like type=group and type=select, set possible selectable items in $data['processedTca'] of the according fields, sanitize their TCA settings.
  • Process type=inline and type=flex fields and prepare their child fields by using new instances of FormDataCompiler and adding their results to $data['processedTca'].
Post process after single field values are prepared
  • Execute display conditions and remove fields from $data['processedTca'] that shouldn't be shown.
  • Determine main record title and set as $data['recordTitle']
Extending data groups with own providers

The base set of DataProviders for all DataGroups is defined within typo3/sysext/core/Configuration/DefaultConfiguration.php in section ['SYS']['formEngine']['formDataGroup'], and ends up in variable $GLOBALS['TYPO3_CONF_VARS'] after core bootstrap. The provider list can be read top-down, so the DependencyOrderingService typically does not resort this list to a different order.

Adding an own provider to this list means adding an array key to that array having a specification where the new data provider should be added in the list. This is done by the arrays depends and before.

As an example, the extension "news" uses an own data provider to do additional flex form data structure preparation. The core internal flex preparation is already split into two providers: TcaFlexPrepare determines the data structure and parses it, TcaFlexProcess uses the prepared data structure, processes values and applies defaults if needed. The data provider from the extension "news" hooks in between these two to add some own preparation stuff. The registration happens with this code in ext_localconf.php:

// Modify flexform fields since core 8.5 via formEngine: Inject a data provider
// between TcaFlexPrepare and TcaFlexProcess
if (\TYPO3\CMS\Core\Utility\VersionNumberUtility::convertVersionNumberToInteger(TYPO3_version) >= 8005000) {
    $GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['formDataGroup']['tcaDatabaseRecord']
    [\GeorgRinger\News\Backend\FormDataProvider\NewsFlexFormManipulation::class] = [
        'depends' => [
            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexPrepare::class,
        ],
        'before' => [
            \TYPO3\CMS\Backend\Form\FormDataProvider\TcaFlexProcess::class,
        ],
    ];
}

This is pretty powerful since it allows extensions to hook in additional stuff at any point of the processing chain, and it does not depend on the load order of extensions.

Limitations:
  • It is not easily possible to "kick out" an existing provider if other providers have dependencies to them - which is usually the case.
  • It is not easily possible to substitute an existing provider with an own one.

Note

It may happen that the core splits or deletes the one or the other DataProvider in the future. If then an extension has a dependency to a removed provider, the DependencyOrderingService, which takes care of the sorting, throws an exception. There is currently no good solution in the core on how to mitigate this issue.

Note

Data providers in general should not know about renderType, but only about type. Their goal is to prepare and sanitize data independent of a specific renderType. At the moment, the core data provider just has one or two places, where specific renderType's are taken into account to process data, and those show that these areas are a technical dept that should be changed.

Adding data to data array

Most custom data providers change or add existing data within the main data array. A typical use case is an additional record initialization for specific fields in $data['databaseRow'] or additional items somewhere within $data['processedTca']. The main data array is documented in FormDataCompiler->initializeResultArray().

Sometimes, own DataProviders need to add additional data that does not fit into existing places. In those cases they can add stuff to $data['customData']. This key is not filled with data by core DataProviders and serves as a place for extensions to add things. Those data components can be used in own code parts of the rendering later. It is advisable to prefix own data in $data['customData'] with some unique key (for instance the extension name) to not collide with other data a different extension may add.

Rendering

This is the second step of the processing chain: The rendering part gets the data array prepared by FormDataCompiler and creates a result array containing HTML, CSS and JavaScript. This is then post-processed by a controller to feed it to the PageRenderer or to create an ajax response.

The rendering is a tree: The controller initializes this by setting one container as renderType entry point within the data array, then hands over the full data array to the NodeFactory which looks up a class responsible for this renderType, and calls render() on it. A container class creates only a fraction of the full result, and delegates details to another container. The second one does another detail and calls a third one. This continues to happen until a single field should be rendered, at which point an element class is called taking care of one element.

Render tree example

Each container creates some "outer" part of the result, calls some sub-container or element, merges the sub-result with its own content and returns the merged array up again. The data array is given to each sub class along the way, and containers can add further render relevant data to it before giving it "down". The data array can not be given "up" in a changed way again. Inheritance of a data array is always top-bottom. Only HTML, CSS or JavaScript created by a sub-class is returned by the sub-class "up" again in a "result" array of a specified format.

class SomeContainer extends AbstractContainer
{
    public function render()
    {
        $result = $this->initializeResultArray();
        $data = $this->data;
        $data['renderType'] = 'subContainer';
        $childArray = $this->nodeFactory->create($data)->render();
        $resultArray = $this->mergeChildReturnIntoExistingResult($result, $childArray, false);
        $result['html'] = '<h1>A headline</h1>' . $childArray['html'];
        return $result;
    }
}

Above example lets NodeFactory find and compile some data from "subContainer", and merges the child result with its own. The helper methods initializeResultArray() and mergeChildReturnIntoExistingResult() help with combining CSS and JavaScript.

An upper container does not directly create an instance of a sub node (element or container) and never calls it directly. Instead, a node that wants to call a sub node only refers to it by a name, sets this name into the data array as $data['renderType'] and then gives the data array to the NodeFactory which determines an appropriate class name, instantiates and initializes the class, gives it the data array, and calls render() on it.

Note

The SingleFieldContainer and FlexFormElementContainer will probably vanish with core version 9.

Note

Data set by containers and given down to children will likely change in core version 9: All fields not registered in the main data array of FormDataCompiler and only added within containers will move into section renderData. Furthermore, it is planned to remove parameterArray and substitute it with something better. This will affect most elements and will probably break a lot of these elements.

Class inheritance
Main render class inheritance

All classes must implement NodeInterface to be routed through the NodeFactory. The AbstractNode implements some basic helpers for nodes, the two classes AbstractContainer and AbstractFormElement implement helpers for containers and elements respectively.

The call concept is simple: A first container is called, which either calls a container below or a single element. A single element never calls a container again.

NodeFactory

The NodeFactory plays an important abstraction role within the render chain: Creation of child nodes is always routed through it, and the NodeFactory takes care of finding and validating the according class that should be called for a specific renderType. This is supported by an API that allows registering new renderTypes and overriding existing renderTypes with own implementations. This is true for all classes, including containers, elements, fieldInformation, fieldWizards and fieldControls. This means the child routing can be fully adapted and extended if needed. It is possible to transparently "kick-out" a core container and to substitute it with an own implementation.

As example, the TemplaVoila implementation needs to add additional render capabilities of the flex form rendering to add for instance an own multi-language rendering of flex fields. It does that by overriding the default flex container with own implementation:

// Default registration of "flex" in NodeFactory:
// 'flex' => \TYPO3\CMS\Backend\Form\Container\FlexFormEntryContainer::class,

// Register language aware flex form handling in FormEngine
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1443361297] = [
    'nodeName' => 'flex',
    'priority' => 40,
    'class' => \TYPO3\CMS\Compatibility6\Form\Container\FlexFormEntryContainer::class,
];

This re-routes the renderType "flex" to an own class. If multiple registrations for a single renderType exist, the one with highest priority wins.

Note

The NodeFactory uses $data['renderType']. This has been introduced with core version 7 in TCA, and a couple of TCA fields actively use this renderType. However, it is important to understand the renderType is only used within the FormEngine and type is still a must-have setting for columns fields in TCA. Additionally, type can not be overridden in columnsOverrides. Basically, type specifies how the DataHandler should put data into the database, while renderType specifies how a single field is rendered. This additionally means there can exist multiple different renderTypes for a single type, and it means it is possible to invent a new renderType to render a single field differently, but still let the DataHandler persist it the usual way.

Adding a new renderType in ext_localconf.php

// Add new field type to NodeFactory
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1487112284] = [
    'nodeName' => 'selectTagCloud',
    'priority' => '70',
    'class' => \MyVendor\CoolTagCloud\Form\Element\SelectTagCloudElement::class,
];

And use it in TCA for a specific field, keeping the full database functionality in DataHandler together with the data preparation of FormDataCompiler, but just routing the rendering of that field to the new element:

$GLOBALS['TCA']['myTable']['columns']['myField'] = [
    'label' => 'Cool Tag cloud',
    'config' => [
        'type' => 'select',
        'renderType' => 'selectTagCloud',
        'foreign_table' => 'tx_cooltagcloud_availableTags',
    ],
];

The above examples are a static list of nodes that can be changed by settings in ext_localconf.php. If that is not enough, the NodeFactory can be extended with a resolver that is called dynamically for specific renderTypes. This resolver gets the full current data array at runtime and can either return NULL saying "not my job", or return the name of a class that should handle this node.

An example of this are the core internal rich text editors. Both "ckeditor" and "rtehtmlarea" register a resolver class that are called for node name "text", and if the TCA config enables the editor, and if the user has enabled rich text editing in his user settings, then the resolvers return their own RichTextElement class names to render a given text field:

// Register FormEngine node type resolver hook to render RTE in FormEngine if enabled
$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeResolver'][1480314091] = [
    'nodeName' => 'text',
    'priority' => 50,
    'class' => \TYPO3\CMS\RteCKEditor\Form\Resolver\RichTextNodeResolver::class,
];

The trick is here that "ckeditor" registers his resolver with ah higher priority (50) than "rtehtmlarea" (40), so the "ckeditor" resolver is called first and wins if both extensions are loaded and if both return a valid class name.

Result array

Each node, no matter if it is a container, an element, or a node expansion, must return an array with specific data keys it wants to add. It is the job of the parent node that calls the sub node to merge child node results into its own result. This typically happens by merging $childResult['html'] into an appropriate position of own HTML, and then calling $this->mergeChildReturnIntoExistingResult() to add other array child demands like stylesheetFiles into its own result.

Container and element nodes should use the helper method $this->initializeResultArray() to have a result array initialized that is understood by a parent node.

Only if extending existing element via node expansion, the result array of a child can be slightly different. For instance, a FieldControl "wizards" must have a iconIdentifier result key key. Using $this->initializeResultArray() is not appropriate in these cases but depends on the specific expansion type. See below for more details on node expansion.

The result array for container and element nodes looks like this. $resultArray = $this->initializeResultArray() takes care of basic keys:

[
    'html' => '',
    'additionalInlineLanguageLabelFiles' => [],
    'stylesheetFiles' => [],
    'requireJsModules' => [],
]

CSS and language labels (which can be used in JS) are added with their file names in format EXT:extName/path/to/file. JavaScript is added only via RequireJS modules, the registration allows an init method to be called if the module is loaded by the browser.

Note

The result array handled by $this->mergeChildReturnIntoExistingResult() contains a couple of more keys, those will vanish with further FormEngine refactoring steps. If using them, be prepared to adapt extensions later.

Note

Nodes must never add JavaScript or CSS or similar stuff using the PageRenderer. This fails as soon as this container / element / wizard is called via AJAX, for instance within inline. Instead, those resources must be registered via the result array only, using stylesheetFiles and requireJsModules.

Node expansion

The "node expansion" classes FieldControl, FieldInformation and FieldWizard are called by containers and elements and allow "enriching" containers and elements. Which enrichments are called can be configured via TCA.

This API is the substitution of the old "TCA wizards array" and has been introduced with core version 8.

FieldInformation
Additional information. In elements, their output is shown between the field label and the element itself. They can not add functionality, but only simple and restricted HTML strings. No buttons, no images. An example usage could be an extension that auto-translates a field content and outputs an information like "Hey, this field was auto-filled for you by an automatic translation wizard. Maybe you want to check the content".
FieldWizard
Wizards shown below the element. "enrich" an element with additional functionality. The localization wizard and the file upload wizard of type=group fields are examples of that.
FieldControl
"Buttons", usually shown next to the element. For type=group the "list" button and the "element browser" button are examples. A field control must return an icon identifier.

Currently, all elements usually implement all three of these, except in cases where it does not make sense. This API allows adding functionality to single nodes, without overriding the whole node. Containers and elements can come with default expansions (and usually do). TCA configuration can be used to add own stuff. On container side the implementation is still basic, only OuterWrapContainer and InlineControlContainer currently implement FieldInformation and FieldWizard.

See the TCA reference ctrl section for more information on how to configure these for containers in TCA.

Example. The InputTextElement (standard input element) defines a couple of default wizards and embeds them in its main result HTML:

class InputTextElement extends AbstractFormElement
{
    protected $defaultFieldWizard = [
        'localizationStateSelector' => [
            'renderType' => 'localizationStateSelector',
        ],
        'otherLanguageContent' => [
            'renderType' => 'otherLanguageContent',
            'after' => [
                'localizationStateSelector'
            ],
        ],
        'defaultLanguageDifferences' => [
            'renderType' => 'defaultLanguageDifferences',
            'after' => [
                'otherLanguageContent',
            ],
        ],
    ];

    public function render()
        $resultArray = $this->initializeResultArray();

        $fieldWizardResult = $this->renderFieldWizard();
        $fieldWizardHtml = $fieldWizardResult['html'];
        $resultArray = $this->mergeChildReturnIntoExistingResult($resultArray, $fieldWizardResult, false);

        $mainFieldHtml = [];
        $mainFieldHtml[] = '<div class="form-control-wrap">';
        $mainFieldHtml[] =  '<div class="form-wizards-wrap">';
        $mainFieldHtml[] =      '<div class="form-wizards-element">';
        // Main HTML of element done here ...
        $mainFieldHtml[] =      '</div>';
        $mainFieldHtml[] =      '<div class="form-wizards-items-bottom">';
        $mainFieldHtml[] =          $fieldWizardHtml;
        $mainFieldHtml[] =      '</div>';
        $mainFieldHtml[] =  '</div>';
        $mainFieldHtml[] = '</div>';

        $resultArray['html'] = implode(LF, $mainFieldHtml);
        return $resultArray;
    }
}

This element defines three wizards to be called by default. The renderType concept is re-used, the values localizationStateSelector are registered within the NodeFactory and resolve to class names. They can be overridden and extended like all other nodes. The $defaultFieldWizards are merged with TCA settings by the helper method renderFieldWizards(), which uses the DependencyOrderingService again.

It is possible to:

  • Override existing expansion nodes with own ones from extensions, even using the resolver mechanics is possible.
  • It is possible to disable single wizards via TCA
  • It is possible to add own expansion nodes at any position relative to the other nodes by specifying "before" and "after" in TCA.
Add fieldControl example

To illustrate the principals discussed in this chapter see the following example which registers a fieldControl (button) next to a field in the pages table to trigger a data import via ajax.

Add a new renderType in ext_localconf.php:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['formEngine']['nodeRegistry'][1485351217] = [
   'nodeName' => 'importDataControl',
   'priority' => 30,
   'class' => \T3G\Something\FormEngine\FieldControl\ImportDataControl::class
];

Register the control in TCA/Overrides/pages.php:

'somefield' => [
   'label'   => $langFile . ':pages.somefield',
   'config'  => [
      'type' => 'input',
      'eval' => 'int, unique',
      'fieldControl' => [
         'importControl' => [
            'renderType' => 'importDataControl'
         ]
      ]
   ]
],

Add the php class for rendering the control in FormEngine/FieldControl/ImportDataControl.php:

declare(strict_types=1);

namespace T3G\Something\FormEngine\FieldControl;

use TYPO3\CMS\Backend\Form\AbstractNode;

class ImportDataControl extends AbstractNode
{
   public function render()
   {
      $result = [
         'iconIdentifier' => 'import-data',
         'title' => $GLOBALS['LANG']->sL('LLL:EXT:something/Resources/Private/Language/locallang_db.xlf:pages.importData'),
         'linkAttributes' => [
            'class' => 'importData ',
            'data-id' => $this->data['databaseRow']['somefield']
         ],
         'requireJsModules' => ['TYPO3/CMS/Something/ImportData'],
      ];
      return $result;
   }
}

Add the JavaScript for defining the behavior of the control in Resources/Public/JavaScript/ImportData.js:

/**
* Module: TYPO3/CMS/Something/ImportData
*
* JavaScript to handle data import
* @exports TYPO3/CMS/Something/ImportData
*/
define(function () {
   'use strict';

   /**
   * @exports TYPO3/CMS/Something/ImportData
   */
   var ImportData = {};

   /**
   * @param {int} id
   */
   ImportData.import = function (id) {
      $.ajax({
         type: 'POST',
         url: TYPO3.settings.ajaxUrls['something-import-data'],
         data: {
            'id': id
         }
      }).done(function (response) {
         if (response.success) {
            top.TYPO3.Notification.success('Import Done', response.output);
         } else {
            top.TYPO3.Notification.error('Import Error!');
         }
      });
   };

   /**
   * initializes events using deferred bound to document
   * so AJAX reloads are no problem
   */
   ImportData.initializeEvents = function () {

      $('.importData').on('click', function (evt) {
         evt.preventDefault();
         ImportData.import($(this).attr('data-id'));
      });
   };

   $(ImportData.initializeEvents);

   return ImportData;
});

Add an ajax route for the request in Configuration/Backend/AjaxRoutes.php:

<?php
return [
   'something-import-data' => [
      'path' => '/something/import-data',
      'target' => \T3G\Something\Controller\Ajax\ImportDataController::class . '::importDataAction'
   ],
];

Add the ajax controller class in Classes/Controller/Ajax/ImportDataController.php:

declare(strict_types=1);

namespace T3G\Something\Controller\Ajax;

use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;

class ImportDataController
{
   /**
   * @param ServerRequestInterface $request
   * @param ResponseInterface $response
   * @return ResponseInterface
   */
   public function importDataAction(ServerRequestInterface $request, ResponseInterface $response)
   {
      $queryParameters = $request->getParsedBody();
      $id = (int)$queryParameters['id'];

      if (empty($id)) {
         $response->getBody()->write(json_encode(['success' => false]));
         return $response;
      }
      $param = ' -id=' . $id;

      // trigger data import (simplified as example)
      $output = shell_exec('.' . DIRECTORY_SEPARATOR . 'import.sh' . $param);

      $response->getBody()->write(json_encode(['success' => true, 'output' => $output]));
      return $response;
   }
}

Database

Contents:

Introduction

TYPO3 CMS relies on storing its data in a Relational database management system (RDBMS). The doctrine-dbal component is used to enable connecting to different database management systems. Most used is still MySQL / MariaDB, but thanks to Doctrine others like PostgreSQL and SQLServer are also an option.

The corresponding DBMS can be selected during installation.

Note

At the time of writing the installation process does not fully work for SQL Server, the connection settings have to be manually configured in that case.

This chapter gives an overview of the basic TYPO3 database table structure, followed by some information on upgrading and maintaining table and field consistency, and then deep dives into the programming API.

Doctrine-Dbal

Database queries in TYPO3 are done with an API based on doctrine-dbal. The API is provided by the system extension core which is always loaded and thus always available.

Extension authors can use this low-level API to manage query operations directly on the configured DBMS.

Doctrine-dbal is feature rich. Drivers for various target systems enable TYPO3 to run on a long list of ANSI SQL compatible DBMS. If used properly, queries created with this API are translated to the specific database engine by doctrine without an extension developer taking care of that specifically.

The API provided by the core is basically a pretty small and lightweight facade in front of doctrine-dbal that adds some convenient methods as well as some TYPO3 CMS specific sugar. The facade additionally provides methods to retrieve specific connection objects per configured database connection based on the table that is queried. This enables instance administrators to configure different database engines for different tables while this is transparent for extension developers.

doctrine-dbal has been introduced with TYPO3 CMS version 8 and substitutes the old API based on $GLOBALS['TYPO3_DB']. Extension authors are encouraged to switch away from TYPO3_DB to the new API. A dedicated chapter helps with typical migration questions. With database abstraction being built in doctrine-dbal the old and optional extensions dbal and adodb are obsolete.

This document does not outline each and every single method the API provides. It sticks to those that are commonly used in extensions and some parts like the rewritten schema migrator are left out since they are usually of little to no interest for extensions.

Understanding Doctrine-Dbal and Doctrine-Orm

Doctrine is a two-fold project with doctrine-dbal being the low-level database abstraction and query building interface to specific database engines, while doctrine-orm is a high-level object relational mapping on top of doctrine-dbal.

The TYPO3 CMS core - only - implements the dbal part. doctrine-orm is neither required nor implemented nor used at the time of this writing.

Low-level and high-level database calls

This documentation is about low-level database calls. In many cases it is better to use higher level API's like the DataHandler or extbase repositories and to let the framework handle persistence details internally.

Tip

Always remember the high-level database calls and use them when appropriate!

Credits

Implementing the doctrine-dbal API into TYPO3 has been a huge project in 2016. Special thanks goes to awesome Mr. Morton Jonuschat for the initial design, integration and support and to more than 40 different people who actively contributed to migrate more than 1700 calls from TYPO3_DB-style to Doctrine within half a year. This was a huge community achievement, thanks everyone involved!

Configuration

Configuring doctrine-dbal for TYPO3 CMS is all about specifying the single database endpoints and handing over connection credentials. The framework supports the parallel usage of multiple database connections, a specific connection is mapped depending on its table name. The table space can be seen as a transparent layer that determines which specific connection is chosen for a query to a single or a group of tables: It allows "swapping-out" single tables from the Default connection to point them to a different database endpoint.

As with other central configuration options, the database endpoint and mapping configuration happens within typo3conf/LocalConfiguration.php and ends up in $GLOBALS['TYPO3_CONF_VARS'] after core bootstrap. The specific sub-array is $GLOBALS['TYPO3_CONF_VARS']['DB'].

A typical, basic example using only the Default connection with a single database endpoint:

// LocalConfiguration.php
// [...]
'DB' => [
   'Connections' => [
      'Default' => [
         'charset' => 'utf8',
         'dbname' => 'theDatabaseName',
         'driver' => 'mysqli',
         'host' => 'theHost',
         'password' => 'theConnectionPassword',
         'port' => 3306,
         'user' => 'theUser',
      ],
   ],
],
// [...]

Remarks:

  • The Default connection must be configured, this can not be left out or renamed.
  • For mysqli, if the host is set to localhost and if the default PHP options in this area are not changed, the connection will be socket based. This saves a little overhead. To force a TCP/IP based connection even for localhost, the IPv4 or IPv6 address 127.0.0.1 and ::1/128 respectively must be used as host value.
  • The connect options are hand over to doctrine-dbal without much manipulation from TYPO3 CMS side. Please refer to the doctrine connection docs for a full overview of settings.
  • If charset option is not specified it defaults to utf8.
  • The option wrapperClass is used by the TYPO3 CMS framework to "hang in" the extended Connection class TYPO3\CMS\Database\Connection as main facade around doctrine-dbal.

A slightly more complex example with two connections, mapping the sys_log table to a different endpoint:

// LocalConfiguration.php
// [...]
'DB' => [
   'Connections' => [
      'Default' => [
         'charset' => 'utf8',
         'dbname' => 'default_dbname',
         'driver' => 'mysqli',
         'host' => 'default_host',
         'password' => '***',
         'port' => 3306,
         'user' => 'default_user',
      ],
      'Syslog' => [
         'charset' => 'utf8',
         'dbname' => 'syslog_dbname',
         'driver' => 'mysqli',
         'host' => 'syslog_host',
         'password' => '***',
         'port' => 3306,
         'user' => 'syslog_user',
      ],
   ],
   'TableMapping' => [
      'sys_log' => 'Syslog'
   ]
],
// [...]

Remarks:

  • The array key Syslog is just a name, it can be different but it's good practice to give it a useful speaking name.
  • It is possible to map multiple tables to a different endpoint by adding further table name / connection name pairs to TableMapping.
  • Mind this "connection per table" approach is limited: If in the above example a join query that spans over different connections is fired, an exception is raised. It is up to the administrator to group affected tables to the same connection in those cases, or a developer should implement some fallback logic to suppress the join().

Attention

Connections to databases postgres, maria and mysql are actively tested. However, mssql is currently not actively tested.

Furthermore, the TYPO3 CMS installer supports only a single mysql or mariadb connection at the moment and the connection details can not be properly edited within the All configuration section of the Install Tool.

Database Structure

The database tables used by TYPO3 CMS can be divided into two rough categories:

  • Tables that are used by the system internally and are invisible to backend users (eg. be_sessions, sys_registry, cache related tables). There are often dedicated PHP API's in the core extension to manage entries of these tables, for instance the Cache framework API.
  • Tables that can be managed via the TYPO3 CMS backend, are shown in the List module and can be edited using FormEngine.

There are certain requirements for such managed tables:

  • The table must be configured in the global TCA array. This will tell TYPO3 CMS things like the table name, features you have configured, the fields of the table and how to render these in the backend, relations to other tables, etc.
  • The table must contain at least these fields:
    • "uid" - an auto-incremented integer, PRIMARY key, for the table, containing the unique ID of the record in the table.
    • "pid" - an integer pointing to the "uid" of the page (record from "pages" table) to which the record belongs.
    • other typical fields include:
      • A "title" field holding the records title as seen in the backend.
      • A "description" field holding a description displayed in WEB > List view.
      • A "crdate" field holding the creation time of the record.
      • A "tstamp" field holding the last modification time of the record.
      • A "sorting" field holding an order if records are sorted manually.
      • A "deleted" field which tells TYPO3 CMS that the record is deleted (in effect implementing a "soft delete" feature; records with a "deleted" field are not truly deleted from the database).
      • A "hidden" or "disabled" field for records which exist but should not be used (e.g. disabled backend users, content not visible in the frontend).

Note

Except for the "uid" and "pid" fields, all other fields do not fill a role automatically as soon as they exist. Their existence must be declared in the TCA configuration. This means that such fields can also be named freely, the above are the default names TYPO3 uses - for consistency it is recommended to name them that way.

The "pages" table

The pages table has a special status: It is the backbone of TYPO3 CMS, as it provides the hierarchical page structure into which all other TYPO3 CMS managed records are positioned. All other managed tables in TYPO3 have a pid field that points to a uid record in this table. So any managed table record in TYPO3 is always positioned on exactly one page in the page tree. This makes the pages table the mother of all other managed tables. It can be seen as a directory tree with all other table records as files.

Standard pages are quite literally web site pages in the frontend. But they can also be storage spaces in the backend, very much like folders on a hard disk. For any record, the pid field contains a reference to the page where that record is stored. For pages, the pid fields behaves as a reference to their parent pages.

The special "root" page has some unique properties: its pid is 0 (zero), it does not exist as a row in the pages table, only admin-users can access records on it and these records have to be explicitly configured to reside in the root page - usually table records may only be created on a real page.

Other tables

The tables which are not managed via the TYPO3 CMS backend fill various roles. Some of the most common are:

  • MM relations: when tables are related using a many-to-many relationship, another table must hold these relations. Examples are the table storing relations between categories and categorized records ("sys_category_record_mm") or the table storing relations between files and their various usages in pages, content elements, etc. ("sys_file_reference"). The latter is an interesting example, because it does actually appear in the backend, although only as part of inline records.
  • cache: when a cache is defined as using the database as a cache backend, TYPO3 CMS will automatically create and manage the relevant cache tables.
  • system information: there exist tables storing information about sessions, both frontend and backend ("fe_sessions" and "be_sessions" respectively), a table for a central registry ("sys_registry") and quite a few others.

All these tables are not subject to the uid/pid constraint mentioned above, but they may have such fields if it is convenient for whatever reason.

There is no way such tables can be managed via the TYPO3 CMS backend unless a specific module provides a form of access to it. For example, the SYSTEM > Log module provides an interface to browse records from the "sys_log" table.

Upgrade table and field definitions

Each extension in TYPO3 CMS can bring the file ext_tables.sql that defines which tables and fields the extension needs. Gathering all ext_tables.sql thus defines the full set of tables, fields and indexes of a TYPO3 instance to unfold its full feature set. Some functionality in the Install Tool can compare the defined set with the current active database schema and shows options to align those two by adding fields, removing fields and so on.

When you upgrade to newer versions of TYPO3 CMS or upgrade an extension, the data definition of tables and fields might have changed. The TYPO3 CMS Install Tool will detect such changes.

When you install a new extension, any change to the database is automatically performed. When you upgrade to a new major version of TYPO3 CMS, you should normally go through the Upgrade Wizard, whose first step is to perform all necessary database changes:

The Upgrade Wizard indicating that the database needs updates

The Upgrade Wizard indicating that the database needs updates

When performing smaller updates, after updating extensions or - in general - if you want to check the sanity of your system, you can go to ADMIN TOOLS > Maintenance > Analyze Database Structure:

Analyze Database Structure of the Install Tool

The Database analyzer is part of the Maintenance area

What this tool does is collating the information from all ext_tables.sql files of active extensions and compare it with the current database structure. It then proposes to perform the necessary changes, grouped by type: creating new tables, adding new fields to existing tables, altering existing fields, dropping unused tables and fields.

You can choose which updates you want to perform. You can even decide not to create new fields and tables, although that will very likely break your installation.

More information about the process of upgrading TYPO3 CMS can be found in the Installation and Upgrade Guide.

The ext_tables.sql files

As mentioned before, all data definition statements are stored in files called ext_tables.sql which may be present in any extension.

The peculiarity is that these files may not always contain a complete and valid SQL data definition. For example, system extension "rsaauth" defines a new table for storing RSA keys:

CREATE TABLE tx_rsaauth_keys (
   uid int(11) NOT NULL auto_increment,
   pid int(11) DEFAULT '0' NOT NULL,
   crdate int(11) DEFAULT '0' NOT NULL,
   key_value text,

   PRIMARY KEY (uid),
   KEY crdate (crdate)
);

This is a complete and valid SQL data definition. However system extension "css_styled_content" extends the "tt_content" table with additional fields. It also provides these changes in the form of a SQL CREATE TABLE statement:

CREATE TABLE tt_content (
   header_position varchar(6) DEFAULT '' NOT NULL,
   image_compression tinyint(3) unsigned DEFAULT '0' NOT NULL,
   image_effects tinyint(3) unsigned DEFAULT '0' NOT NULL,
   image_noRows tinyint(3) unsigned DEFAULT '0' NOT NULL,
   section_frame int(11) unsigned DEFAULT '0' NOT NULL,
   spaceAfter smallint(5) unsigned DEFAULT '0' NOT NULL,
   spaceBefore smallint(5) unsigned DEFAULT '0' NOT NULL,
   table_bgColor int(11) unsigned DEFAULT '0' NOT NULL,
   table_border tinyint(3) unsigned DEFAULT '0' NOT NULL,
   table_cellpadding tinyint(3) unsigned DEFAULT '0' NOT NULL,
   table_cellspacing tinyint(3) unsigned DEFAULT '0' NOT NULL
);

The classes which take care of assembling the complete SQL data definition will compile all the CREATE TABLE statements for a given table and turn it into a single CREATE TABLE statement. If the table already exists, missing fields are isolated and ALTER TABLE statements are proposed instead.

What this means is that - as an extension developer - you should always have only CREATE TABLE statements in your ext_tables.sql files, the system will handle them as needed.

Basic CRUD

A list of basic usage examples of the query API. This is just a kickstart. Details on the single methods are found in the following chapters, especially QueryBuilder and Connection.

Note

The examples use the shorthand syntax for class names. Please refer to Class overview for the full namespace.

INSERT a row

A straight insert to a table:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;

GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->insert(
        'tt_content',
        [
            'pid' => (int)42,
            'bodytext' => 'bernd',
        ]
    );
INSERT INTO `tt_content` (`pid`, `bodytext`) VALUES ('42', 'bernd')
SELECT a single row

Straight fetch of a single row from tt_content table:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;

$uid = 4;
$row = GeneralUtility::makeInstance(ConnectionPool::class)
    ->getConnectionForTable('tt_content')
    ->select(
        ['uid', 'pid', 'bodytext'], // fields to select
        'tt_content', // from
        [ 'uid' => (int)$uid ] // where
    )
    ->fetch();

Result in $row:

array(3 items)
   uid => 4 (integer)
   pid => 35 (integer)
   bodytext => 'some content' (12 chars)

The engine quotes field names, adds default TCA restrictions like "deleted=0", and prepares a query executed with this final statement:

SELECT `uid`, `pid`, `bodytext`
    FROM `tt_content`
    WHERE (`uid` = '4')
        AND ((`tt_content`.`deleted` = 0)
        AND (`tt_content`.`hidden` = 0)
        AND (`tt_content`.`starttime` <= 1473447660)
        AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1473447660)))

Note

Default restrictions deleted, hidden, startime and endtime based on TCA setting of a table are only applied to select() calls, they are not added for delete() or other query types.

SELECT multiple rows with some WHERE magic

Advanced query using the QueryBuilder and manipulating the default restrictions:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Restriction\DeletedRestriction

$uid = 4;
// Get a query builder for a query on table "tt_content"
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
// Remove all default restrictions (delete, hidden, starttime, stoptime), but add DeletedRestriction again
$queryBuilder->getRestrictions()
    ->removeAll()
    ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
// Execute a query with "bodytext=klaus OR uid=4" and proper quoting
$rows = $queryBuilder
    ->select('uid', 'pid', 'bodytext')
    ->from('tt_content')
    ->where(
        $queryBuilder->expr()->orX(
            $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
            $queryBuilder->expr()->eq('uid', $queryBuilder->createNamedParameter($uid, \PDO::PARAM_INT))
        )
    )
    ->execute()
    ->fetchAll();

Result in $rows:

array(2 items)
   0 => array(3 items)
      uid => 4 (integer)
      pid => 35 (integer)
      bodytext => 'bernd' (5 chars)
   1 => array(3 items)
      uid => 366 (integer)
      pid => 13 (integer)
      bodytext => 'klaus' (5 chars)

The executed query looks like:

SELECT `uid`, `pid`, `bodytext`
    FROM `tt_content`
    WHERE ((`bodytext` = 'klaus') OR (`uid` = 4))
        AND (`tt_content`.`deleted` = 0)
UPDATE multiple rows
// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;

GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->update(
        'tt_content',
        [ 'bodytext' => 'bernd' ], // set
        [ 'bodytext' => 'klaus' ] // where
    );
UPDATE `tt_content` SET `bodytext` = 'bernd' WHERE `bodytext` = 'klaus'

Tip

You can also use QueryBuilder for generating more complex update queries. See examples in the QueryBuilder chapter.

DELETE a row
// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;

GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content')
    ->delete(
        'tt_content', // from
        [ 'uid' => (int)4711 ] // where
    );
DELETE FROM `tt_content` WHERE `uid` = '4711'

Class overview

Doctrine provides a set of php objects to represent, create and handle SQL queries and their results. The basic class structure was slightly enriched by TYPO3 to add CMS specific features. Extension authors will typically interact with these classes and objects:

Connection
TYPO3\CMS\Core\Database\Connection: Object representing a specific connection to one connected database. Provides "shortcut" methods for simple standard queries like SELECT or UPDATE. An instance of the QueryBuilder can be retrieved to build more complex queries.
ConnectionPool
TYPO3\CMS\Core\Database\ConnectionPool: Main entry point for extensions to retrieve a specific connection a query should be executed on. Typically used to return a Connection or a QueryBuilder object.
ExpressionBuilder
TYPO3\CMS\Core\Database\Query\Expression\ExpressionBuilder: Object to model complex expressions. Mainly used for WHERE and JOIN conditions.
QueryBuilder
TYPO3\CMS\Core\Database\Query\QueryBuilder: Object to create all sort of complex queries executed on a specific connection. Provides the main CRUD methods for select(), delete() and friends.
QueryHelper
TYPO3\CMS\Core\Database\Query\QueryHelper: Set of static helper methods that can simplify the transition from old TYPO3_DB based code to the doctrine base API.
Restriction ...
TYPO3\CMS\Core\Database\Query\Restriction\...: Set of classes that add expressions like "deleted=0" to a query based on TCA settings of a table. This automatically adds TYPO3 specific restrictions like starttime and endtime, as well as deleted and hidden flags. Further restrictions for language overlays and workspaces are available. This documentation refers to these classes as the RestrictionBuilder.
Statement
Doctrine\DBAL\Driver\Statement: Result object retrieved if a SELECT or COUNT query has been executed. Single rows are returned as array by calling ->fetch() until the method returns false.

ConnectionPool

TYPO3's interface to execute queries via doctrine-dbal typically starts by asking the ConnectionPool for a QueryBuilder or a Connection object, handing over the table name to be queried:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// Get a query builder for a table
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tx_myext_comments');
// or
// Get a connection for a table
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tx_myext_comments');

The QueryBuilder is the default workhorse object used by extension authors to express complex queries, while a Connection instance can be used as shortcut to deal with some simple query cases and little written down code.

Pooling

TYPO3 can handle multiple connections to different database endpoints at the same time. This can be configured on a per-table basis in $GLOBALS['TYPO3_CONF_VARS']. It allows running tables on different databases, without an extension developer taking care of that.

The ConnectionPool implements this feature: It looks up a configured table-to-database mapping and can return a Connection or a QueryBuilder instance for that specific connection. Those objects internally know which target connection they are dealing with and will for instance quote field names accordingly.

The transparency of tables to different database endpoints is limited, though:

Executing a table JOIN between two tables that point to different connections will throw an exception. This restriction may in practice create implicit "groups" of tables that need to point to one connection at once if an extension or the TYPO3 core joins those tables.

This can turn out as a headache if multiple different extensions use for instance the core category or collection API with their mm table joins between core internal tables and their extension's counterparts.

That situation is not easy to deal with. At the time of this writing the core development will eventually implement some non-join fallbacks for typical cases that would be good to decouple, though.

Tip

In case joins cannot be decoupled but still affected tables must run on different databases, and if the code can not be easily adapted, some DBMS like PostgreSQL allow executing those queries by having own connection handlers to different other endpoints on its own.

QueryBuilder

The QueryBuilder is a rather huge class that takes care of the main query dealing.

An instance can get hold of by calling the ConnectionPool->getQueryBuilderForTable() and handing over the table. Never instantiate and initialize the QueryBuilder directly via makeInstance()!

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('aTable');

This documentation does not mention every single available method but sticks to those used in casual queries and normal code flow. There are a couple of not mentioned methods, most of them are either very seldom used or marked as internal. Extension authors typically don't have to deal with anything not mentioned here.

Warning

From security point of view, the documentation of ->createNamedParameter() and ->quoteIdentifier() are an absolute must read and follow section. Make very sure this is understood and use this for each and every query to prevent SQL injections!

The QueryBuilder comes with a happy little list of small methods:

  • Set type of query: ->select(), ->count(), ->update(), ->insert() and delete()
  • Prepare WHERE conditions
  • Manipulate default WHERE restrictions added by TYPO3 for ->select()
  • Add LIMIT, GROUP BY and other SQL stuff
  • ->execute() a query and retrieve a Statement (a query result) object

Most methods of the QueryBuilder return $this and can be chained:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Connection;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('pages')->createQueryBuilder();
$queryBuilder->select('uid')->from('pages');

Note

The QueryBuilder holds internal state and should not be re-used for different queries: Use one query builder per query. Get a fresh one by calling $connection->createQueryBuilder() if the same table is affected, or use $connectionPool->getQueryBuilderForTable() for a query on to a different table. Don't worry, creating those object instances is rather quick.

select() and addSelect()

Create a SELECT query.

Select all fields:

// SELECT *
$queryBuilder->select('*')

->select() and a number of other methods of the QueryBuilder are variadic and can handle any number of arguments. For ->select(), every argument is interpreted as a single field name to select:

// SELECT `uid`, `pid`, `aField`
$queryBuilder->select('uid', 'pid', 'aField');

Argument unpacking can be used if the list of fields is available as array already:

$fields = ['uid', 'pid', 'aField', 'anotherField'];
$queryBuilder->select(...$fields);

->select() supports AS and quotes identifiers automatically. This can become especially handy in join() operations:

// SELECT `tt_content`.`bodytext` AS `t1`.`text`
$queryBuilder->select('tt_content.bodytext AS t1.text')

->select() sets the list of fields that should be selected and ->addSelect() can add further items to an existing list.

Mind that ->select() replaces any formerly registered list instead of appending. Thus, it usually doesn't make much sense to call select() twice in a code flow, or to call it after an ->addSelect(). The methods ->where() and ->andWhere() share the same behavior: ->where() replaces all formerly registered constraints, ->andWhere() appends additional constraints.

A useful combination of ->select() and ->addSelect() can be:

$queryBuilder->select(...$defaultList);
if ($needAdditionalFields) {
   $queryBuilder->addSelect(...$additionalFields);
}

Calling ->execute() on a ->select() query returns a Statement object. To receive single rows a ->fetch() loop on that object is used, or ->fetchAll() to return a single array with all rows. A typical code flow of a SELECT query looks like:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->execute();
while ($row = $statement->fetch()) {
   // Do something with that single row
   debug($row);
}
Default restrictions

Note

->select() and ->count() queries trigger TYPO3 CMS magic that adds further default where clauses if the queried table is also registered via $GLOBALS['TCA']. See the RestrictionBuilder section for details on that topic.

count()

Create a COUNT query, a typical usage:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT COUNT(`uid`) FROM `tt_content` WHERE (`bodytext` = 'klaus')
//     AND ((`tt_content`.`deleted` = 0) AND (`tt_content`.`hidden` = 0)
//     AND (`tt_content`.`starttime` <= 1475580240)
//     AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1475580240)))
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$count = $queryBuilder
   ->count('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
    )
   ->execute()
   ->fetchColumn(0);

Remarks:

  • Similar to the ->select() query type, ->count() automatically triggers RestrictionBuilder magic that adds default deleted, hidden, starttime and endtime restrictions if that is defined in TCA.
  • Similar to ->select() query types, ->execute() with ->count() returns a Statement object. To fetch the number of rows directly, use ->fetchColumn(0).
  • First argument to ->count() is required, typically ->count(*) or ->count('uid') is used, the field name is automatically quoted.
  • There is no support for DISTINCT, a ->groupBy() has to be used instead.
  • If combining ->count() with a ->groupBy(), the result may return multiple rows. The order of those rows depends on the used DBMS. To ensure same order of result rows on multiple different databases, a ->groupBy() should thus always be combined with a ->orderBy().
delete()

Create a DELETE FROM query. The method requires the table name to drop data from. Classic usage:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// DELETE FROM `tt_content` WHERE `bodytext` = 'klaus'
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$affectedRows = $queryBuilder
   ->delete('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->execute();

Remarks:

  • For simple cases, it is often easier to write and read if using the ->delete() method of the Connection object.
  • In contrast to ->select(), ->delete() does not add WHERE restrictions like AND `deleted` = 0 automatically.
  • ->delete() does not magically transform a DELETE FROM `tt_content` WHERE `uid` = 4711 to something like UPDATE `tt_content` SET `deleted` = 1 WHERE `uid` = 4711 internally. A soft-delete must be handled on application level code with a dedicated lookup in $GLOBALS['TCA']['theTable']['ctrl']['deleted'] to check if a specific table can handle the soft-delete, together with an ->update() instead.
  • Multi-table delete is not supported: DELETE FROM `table1`, `table2` can not be created.
  • ->delete() ignores ->join()
  • ->delete() ignores setMaxResults(): DELETE with LIMIT does not work.
update() and set()

Create an UPDATE query. Typical usage:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// UPDATE `tt_content` SET `bodytext` = 'peter' WHERE `bodytext` = 'klaus'

$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->update('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->set('bodytext', 'peter')
   ->execute();

->update() requires the table to update as first argument and a table alias as optional second argument. The table alias can then be used in ->set() and ->where() expressions:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// UPDATE `tt_content` `t` SET `t`.`bodytext` = 'peter' WHERE `u`.`bodytext` = 'klaus'
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->update('tt_content', 'u')
   ->where(
      $queryBuilder->expr()->eq('u.bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->set('u.bodytext', 'peter')
   ->execute();

->set() requires a field name as first argument and automatically quotes it internally. The second mandatory argument is the value a field should be set to, the value is automatically transformed to a named parameter of a prepared statement. This way, ->set() key/value pairs are automatically SQL injection save by default.

If a field should be set to the value of another field from the row, the quoting needs to be turned off and ->quoteIdentifier() has to be used:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// UPDATE `tt_content` SET `bodytext` = `header` WHERE `bodytext` = 'klaus'
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->update('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
   )
   ->set('bodytext', $queryBuilder->quoteIdentifier('header'), false)
   ->execute();

Remarks:

  • For simple cases, it is often easier to use the ->update() method of the Connection object.
  • ->set() can be called multiple times if multiple fields should be updated.
  • ->set() requires a field name as first argument and automatically quotes it internally.
  • ->set() requires the value a field should be set to as second parameter.
  • ->update() ignores ->join() and ->setMaxResults().
  • The API does not magically add deleted = 0 or other restrictions as is currently done for example on select. (See also RestrictionBuilder).
insert() and values()

Create an INSERT query. Typical usage:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$affectedRows = $queryBuilder
   ->insert('tt_content')
   ->values([
      'bodytext' => 'klaus',
      'header' => 'peter',
   ])
   ->execute();

Remarks:

  • It is often easier to use ->insert() or ->bulkInsert() of the Connection object.
  • ->values() expects an array of key/value pairs. Both keys (field names / identifiers) and values are automatically quoted. In rare cases, quoting of values can be turned off by setting the second argument to false. In those cases the quoting has to be done manually, typically by using ->createNamedParameter() on the values, use with care ...
  • ->execute() after ->insert() returns the number of inserted rows, which is typically 1.
  • QueryBuilder does not contain a method to insert multiple rows at once, use ->bulkInsert() of Connection object instead to achieve that.
from()

->from() is a must have call for ->select() and ->count() query types. ->from() needs a table name and an optional alias name. The method is typically called once per query build and the table name is typically the same as what was given to ->getQueryBuilderForTable(). If the query joins multiple tables, the argument should be the name of the first table within the ->join() chain:

// FROM `myTable`
$queryBuilder->from('myTable');

// FROM `myTable` AS `anAlias`
$queryBuilder->from('myTable', 'anAlias');

->from() can be called multiple times and will create the cartesian product of tables if not restricted by an according ->where() or ->andWhere() expression. In general, it is a good idea to use ->from() only once per query and model multi-table selection with an explicit ->join() instead.

where(), andWhere() and orWhere()

The three methods are used to create WHERE restrictions for SELECT, COUNT, UPDATE and DELETE query types. Each argument is typically an ExpressionBuilder object that will be cast to a string on ->execute():

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT `uid`, `header`, `bodytext`
// FROM `tt_content`
// WHERE
//    (
//       ((`bodytext` = 'klaus') AND (`header` = 'a name'))
//       OR (`bodytext` = 'peter') OR (`bodytext` = 'hans')
//    )
//    AND (`pid` = 42)
//    AND ... RestrictionBuilder TCA restrictions ...
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
      $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('a name'))
   )
   ->orWhere(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('peter')),
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('hans'))
   )
   ->andWhere(
      $queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Note the parenthesis of the above example: ->andWhere() encapsulates both ->where() and ->orWhere() with an additional restriction.

Argument unpacking can become handy with these methods:

$whereExpressions = [
   $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
   $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('a name'))
];
if ($needsAdditionalExpression) {
   $whereExpressions[] = $someAdditionalExpression;
}
$queryBuilder->where(...$whereExpressions);

Remarks:

  • The three methods are variadic. They can handle any number of arguments. If for instance ->where() receives four arguments, they are handled as single expressions, all of them combined with AND.
  • ->where() should be called only once per query and it resets any previously set ->where(), ->andWhere() and ->orWhere() expression. Having a ->where() call after a previous ->where(), ->andWhere() or ->orWhere() typically indicates a bug or a rather weird code flow. Doing so is discouraged.
  • While creating complex WHERE restrictions, ->getSQL() and ->getParameters() are helpful debugging friends to verify parenthesis and single query parts.
  • If using only ->eq() expressions, it is often easier to switch to the according Connection object method to simplify quoting and increase readability.
  • It is possible to feed the methods with strings directly, but that is discouraged and typically only used in rare cases where expression strings are created at a different place that can not be resolved easily. In the core, those places are usually combined with QueryHelper::stripLogicalOperatorPrefix() to remove leading AND or OR parts. Using this gives an additional risk of missing or wrong quoting and is a potential security issue. Use with care if ever.
join(), innerJoin(), rightJoin() and leftJoin()

Joining multiple tables in a ->select() or ->count() query is done with one of these methods. Multiple joins are supported by calling the methods more than once. All methods require four arguments: The name of the left side table (or its alias), the name of the right side table, an alias for the right side table name and the join restriction as fourth argument:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT `sys_language`.`uid`, `sys_language`.`title`
// FROM `sys_language`
// INNER JOIN `pages_language_overlay` `overlay`
//     ON `overlay`.`sys_language_uid` = `sys_language`.`uid`
// WHERE
//     (`overlay`.`pid` = 42)
//     AND (
//          (`overlay`.`deleted` = 0)
//          AND (
//              (`sys_language`.`hidden` = 0) AND (`overlay`.`hidden` = 0)
//          )
//          AND (`overlay`.`starttime` <= 1475591280)
//          AND ((`overlay`.`endtime` = 0) OR (`overlay`.`endtime` > 1475591280))
//     )
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$statement = $queryBuilder
   ->select('sys_language.uid', 'sys_language.title')
   ->from('sys_language')
   ->join(
      'sys_language',
      'pages_language_overlay',
      'overlay',
      $queryBuilder->expr()->eq('overlay.sys_language_uid', $queryBuilder->quoteIdentifier('sys_language.uid'))
   )
   ->where(
      $queryBuilder->expr()->eq('overlay.pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Notes to the above example:

  • The query operates on table sys_language as main table, this table name is given to getQueryBuilderForTable().
  • The query joins table pages_language_overlay as INNER JOIN, giving it the alias overlay.
  • The join condition is `overlay`.`sys_language_uid` = `sys_language`.`uid`. It would have been identical to swap the expression arguments of the fourth ->join() argument ->eq('sys_language.uid', $queryBuilder->quoteIdentifier('overlay.sys_language_uid')).
  • The second argument of the join expression instructs the ExpressionBuilder to quote the value as a field identifier (a field name, here a table/field name combination). Using createNamedParameter() would lead to a quoting as value (' instead of ` in mysql) and the query would fail.
  • The alias overlay - the third argument of the ->join() call - does not necessarily have to be set to a different name than the table name itself here. Using pages_language_overlay as third argument and not specifying a different name would do. Aliases are mostly useful if a join to the same table is needed: SELECT `something` FROM `tt_content` JOIN `tt_content` `content2` ON .... Aliases additionally become handy to increase readability of ->where() expressions.
  • The RestrictionBuilder added additional WHERE conditions for both involved tables! Table sys_language obviously only specifies a 'disabled' => 'hidden' as enableColumns in its TCA ctrl section, while table pages_language_overlay specifies deleted, hidden, starttime and stoptime fields.

A more complex example with two joins. The first join points to the first table again using an alias to resolve a language overlay scenario. The second join uses the alias name of the first join target as left side:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT `tt_content_orig`.`sys_language_uid`
// FROM `tt_content`
// INNER JOIN `tt_content` `tt_content_orig` ON `tt_content`.`t3_origuid` = `tt_content_orig`.`uid`
// INNER JOIN `sys_language` `sys_language` ON `tt_content_orig`.`sys_language_uid` = `sys_language`.`uid`
// WHERE
//     (`tt_content`.`colPos` = 1)
//     AND (`tt_content`.`pid` = 42)
//     AND (`tt_content`.`sys_language_uid` = 2)
//     AND ... RestrictionBuilder TCA restrictions for tables tt_content and sys_language ...
// GROUP BY `tt_content_orig`.`sys_language_uid`
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$constraints = [
   $queryBuilder->expr()->eq('tt_content.colPos', $queryBuilder->createNamedParameter(1, \PDO::PARAM_INT)),
   $queryBuilder->expr()->eq('tt_content.pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)),
   $queryBuilder->expr()->eq('tt_content.sys_language_uid', $queryBuilder->createNamedParameter(2, \PDO::PARAM_INT)),
];
$queryBuilder
   ->select('tt_content_orig.sys_language_uid')
   ->from('tt_content')
   ->join(
      'tt_content',
      'tt_content',
      'tt_content_orig',
      $queryBuilder->expr()->eq(
         'tt_content.t3_origuid',
         $queryBuilder->quoteIdentifier('tt_content_orig.uid')
      )
   )
   ->join(
      'tt_content_orig',
      'sys_language',
      'sys_language',
      $queryBuilder->expr()->eq(
         'tt_content_orig.sys_language_uid',
         $queryBuilder->quoteIdentifier('sys_language.uid')
      )
   )
   ->where(...$constraints)
   ->groupBy('tt_content_orig.sys_language_uid')
   ->execute();

Further remarks:

  • ->join() and innerJoin are identical. They create an INNER JOIN query, this is identical to a JOIN query.
  • ->leftJoin() creates a LEFT JOIN query, this is identical to a LEFT OUTER JOIN query.
  • ->rightJoin() creates a RIGHT JOIN query, this is identical to a RIGT OUTER JOIN query.
  • Calls on join() methods are only considered for ->select() and ->count() type queries. ->delete(), ->insert() and update() do not support joins, those query parts are ignored and do not end up in the final statement.
  • The argument of ->getQueryBuilderForTable() should be the left most main table.
  • A join of two tables that are configured to different connections will throw an exception. This restricts which tables can be configured to different database endpoints. It is possible to test the connection objects of involved tables for equality and implement a fallback logic in PHP if they are different.
orderBy() and addOrderBy()

Add ORDER BY to a ->select() statement. Both ->orderBy() and ->addOrderBy() require a field name as first argument:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT * FROM `sys_language` ORDER BY `sorting` ASC
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->getRestrictions()->removeAll();
$languageRecords = $queryBuilder
   ->select('*')
   ->from('sys_language')
   ->orderBy('sorting')
   ->execute()
   ->fetchAll();

Remarks:

  • ->orderBy() resets any previously specified orders. It doesn't make sense to call it after a previous ->orderBy() or ->addOrderBy() again.
  • Both methods need a field name or a table.fieldName or a tableAlias.fieldName as first argument, in the above example calling ->orderBy('sys_language.sorting') would have been identical. All identifiers are quoted automatically.
  • The second, optional argument of both methods specifies the sorting order. The two allowed values are ASC and DESC where ASC is default and can be omited.
  • To create a chain of orders, use ->orderBy() and then multiple ->addOrderBy() calls. Calling ->orderBy('header')->addOrderBy('bodytext')->addOrderBy('uid', 'DESC') creates ORDER BY `header` ASC, `bodytext` ASC, `uid` DESC
  • To add more complex sorting, you can use ->add('orderBy', 'FIELD(eventtype, 0, 4, 1, 2, 3)', true), remember to quote properly
groupBy() and addGroupBy()

Add GROUP BY to a ->select() statement. Each argument to the methods is a single identifier:

// GROUP BY `pages_language_overlay`.`sys_language_uid`, `sys_language`.`uid`
->groupBy('pages_language_overlay.sys_language_uid', 'sys_language.uid');

Remarks:

  • Similar to ->select() and ->where() both methods are variadic and take any number of arguments, argument unpacking is supported: ->groupBy(...$myGroupArray)
  • Each argument is either a direct field name GROUP BY `bodytext`, a table.fieldName or a tableAlias.fieldName and will be properly quoted.
  • ->groupBy() resets any previously set group specification and should be called only once per statement.
  • For more complex statements you can use ->add('groupBy', $sql, $append), remember to quote properly.
setMaxResults() and setFirstResult()

Add LIMIT to restrict number of records and OFFSET for pagination query parts. Both methods should be called only once per statement:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT * FROM `sys_language` LIMIT 2 OFFSET 4
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder
   ->select('*')
   ->from('sys_language')
   ->setMaxResults(2)
   ->setFirstResult(4)
   ->execute();

Remarks:

  • It's allowed to call ->setMaxResults() but not to call ->setFirstResult().
  • It is possible to call ->setFirstResult() without calling setMaxResults(): This equals to "Fetch everything, but leave out the first n records". Internally, LIMIT will be added by doctrine-dbal and set to a very high value.
add()

Method ->add() appends to or replaces a single, generic query part. It can be used as a low level call if more specific calls don't give enough freedom to express parts of statments:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->select('*')->from('sys_language');
$queryBuilder->add('orderBy', 'FIELD(eventtype, 0, 4, 1, 2, 3)');

Remarks:

  • The first argument is the sql part. One of: select, from, set, where, groupBy, having or orderBy
  • Second argument is the (properly quoted!) sql segment of this part
  • Optional third boolean argument specifies if the sql fragment should be appended (true) or substitute an possibly existing sql part of this name (false, default).
getSQL()

Method ->getSQL() returns the created query statement as string. It is incredibly useful during development to verify the final statement is executed just as a developer expects it:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->select('*')->from('sys_language');
debug($queryBuilder->getSQL());
$statement = $queryBuilder->execute();

Remarks:

  • This is debugging code. Take proper actions to ensure those calls do not end up in production!
  • The method is typically called directly before ->execute() to output the final statement.
  • Casting a QueryBuilder object to (string) has the same effect as calling ->getSQL(), the explicit call using the method should be preferred to simplify a search operation for this kind of debugging statements, though.
  • The method is a simple way to see which restrictions the RestrictionBuilder added.
  • doctrine-dbal always creates prepared statements: Any value that is added via ->createNamedParameter() creates a placeholder that is later substituted when the real query is fired via ->execute(). ->getSQL() does not show those values, instead the placeholder names are displayed, usually with a string like :dcValue1. There is no simple solution to show the fully replaced query from within the framework, but you can go for ->getParameters() to see the array of parameters used to replace these placeholders within the query.
getParameters()

Method ->getParameters() returns the values for the prepared statement placeholders in an array. It is incredibly useful during development to verify the final statement is executed just as a developer expects it:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('sys_language');
$queryBuilder->select('*')->from('sys_language');
debug($queryBuilder->getParameters());
$statement = $queryBuilder->execute();

Remarks:

  • This is debugging code. Take proper actions to ensure those calls do not end up in production!
  • The method is typically called directly before ->execute() to output the final values for the statement.
  • doctrine-dbal always creates prepared statements: Any value that added via ->createNamedParameter() creates a placeholder that is later substituted when the real query is fired via ->execute(). ->getparameters() does not show the statement or those placeholders, instead the values are displayed, usually within an array using keys like :dcValue1. There is no simple solution to show the fully replaced query from within the framework, but you can go for ->getSQL() to see the string with placeholders used as a prepared statement.
execute()

Compile and fire the final query statement. This is usually the last call on a QueryBuilder object. The method has two possible return values: On success, it either returns a Statement object representing the result set of ->select() and ->count() queries, or it returns an integer representing the number of affected rows for ->insert(), ->update() and ->delete() queries.

If the query fails for whatever reason (for instance if the database connection was lost or if the query contains a syntax error), a \Doctrine\DBAL\DBALException is thrown. It is most often bad habit to catch and suppress this exception since it indicates a runtime or a program error. Both should bubble up. See the coding guidelines for more information on proper exception handling.

expr()

Return an instance of the ExpressionBuilder. This object is used to create complex WHERE query parts and JOIN expressions:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT `uid` FROM `tt_content` WHERE (`uid` > 42)
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->gt('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))
   )
   ->execute();

Remarks:

  • This object is stateless and can be called and worked on as often as needed. It however bound to the specific connection a statement is created for and is thus only available through the QueryBuilder which is specific for one connection, too.
  • Never re-use the ExpressionBuilder, especially not between multiple QueryBuilder objects, always get an instance of the ExpressionBuilder by calling ->expr().
createNamedParameter()

Create a placeholder for a prepared statement field value. Always use that when dealing with user input in expressions to make the statement SQL injection safe:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT * FROM `tt_content` WHERE (`bodytext` = 'kl\'aus')
$searchWord = "kl'aus"; // $searchWord = GeneralUtility::_GP('searchword');
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->getRestrictions()->removeAll();
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter($searchWord))
   )
   ->execute();

The above example shows the importance of using ->createNamedParameter(): The search word kl'aus is "tainted" and would break the query if not channeled through ->createNamedParameter() which quotes the backtick and makes the value SQL injection safe.

Not convinced? Suppose the code would look like this:

// NEVER EVER DO THIS!
$_POST['searchword'] = "'foo' UNION SELECT username FROM be_users";
$searchWord = GeneralUtility::_GP('searchword');
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->getRestrictions()->removeAll();
   this fails with syntax error to prevent copy and paste
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      // MASSIVE SECURITY ISSUE DEMONSTRATED HERE, USE ->createNamedParameter() ON $searchWord!
      $queryBuilder->expr()->eq('bodytext', $searchWord)
    );

Mind the missing ->createNamedParameter() in the ->eq() expression on given value! This code would happily execute the statement SELECT uid FROM `tt_content` WHERE `bodytext` = 'foo' UNION SELECT username FROM be_users; returning a list of backend user names!

Rules:

  • Always use ->createNamedParameter() around any input, no matter where it comes from.

  • The second argument of ->expr() is always either a call to ->createNamedParameter() or ->quoteIdentifier().

  • The second argument of ->createNamedParameter() specifies the type of input. For string, this can be omitted, but it is good practice to add \PDO::PARAM_INT for integers or similar for other field types. This is currently no strict rule, but following this will reduces headaches in the future, especially for DBMS that are not as relaxed as mysql when it comes to field types. The PDO constants can be used for simple types like bool, string, null, lob and integer. Additionally, the two constants Connection::PARAM_INT_ARRAY and Connection::PARAM_STR_ARRAY can be used if an array of strings or integers is handled, for instance in an IN() expression.

  • Keep the ->createNamedParameter() as close as possible to the expression. Do not structure your code in a way that it first quotes something and only later stuffs the already prepared names into the expression. Having ->createNamedParameter() directly within the created expression is much less error prone and easier to review. This is a general rule: Sanitizing input must be as close as possible to the "sink" where a value is submitted to a lower part of the framework. This paradigm should be followed for other quote operations like htmlspecialchars() or GeneralUtility::quoteJSvalue(), too. Sanitizing should be directly obvious at the very place where it is important:

    // use TYPO3\CMS\Core\Utility\GeneralUtility;
    // use TYPO3\CMS\Core\Database\ConnectionPool;
    // DO
    $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
    $queryBuilder->getRestrictions()->removeAll();
    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
           $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter($searchWord))
       )
    
    // DON'T DO, this is much harder to track:
    $queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
    $myValue = $queryBuilder->createNamedParameter($searchWord);
    // Imagine much more code here
    $queryBuilder->getRestrictions()->removeAll();
    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
           $queryBuilder->expr()->eq('bodytext', $myValue)
       )
    
quoteIdentifier() and quoteIdentifiers()

->quoteIdentifier() must be used if not a value is handled, but a field name. The quoting is different in those cases and typically ends up with backticks ` instead of ticks ':

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Connection;
// SELECT `uid` FROM `tt_content` WHERE (`header` = `bodytext`)
// Return list of rows where header and bodytext values are identical
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('header', $queryBuilder->quoteIdentifier('bodytext'))
   );

The method quotes single field names or combinations of table names or table aliases with field names:

// Single field name: `bodytext`
->quoteIdentifier('bodytext');
// Table name and field name: `tt_content`.`bodytext`
->quoteIdentifier('tt_content.bodytext')
// Table alias and field name: `foo`.`bodytext`
->from('tt_content', 'foo')->quoteIdentifier('foo.bodytext')

Remarks:

  • Similar to ->createNamedParameter() this method is crucial to prevent SQL injections. The same rules apply here.
  • Method ->set() for UPDATE statements expects their second argument to be a field value by default and quotes them using ->createNamedParameter() internally. In case a field should be set to the value of another field, this quoting can be turned off and an explicit call to ->quoteIdentifier() must be added.
  • Internally, ->quoteIdentifier() is automatically called on all method arguments that must be a field name. For instance, ->quoteIdentifier() is called on all arguments given to ->select().
  • ->quoteIdentifiers() (mind the plural) can be used to quote multiple field names at once. While that method is 'public` and thus exposed as API method, this is mostly useful internally only.
escapeLikeWildcards()

Helper method to quote % characters within a search string. This is helpful in ->like() and ->notLike() expressions:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Connection;
// SELECT `uid` FROM `tt_content` WHERE (`bodytext` LIKE '%kl\\%aus%')
$searchWord = 'kl%aus';
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->select('uid')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->like(
         'bodytext',
         $queryBuilder->createNamedParameter('%' . $queryBuilder->escapeLikeWildcards($searchWord) . '%')
      )
   );

Warning

Even with using ->escapeLikeWildcards(), the value must again be encapsulated in a ->createNamedParameter() call. Only calling ->escapeLikeWildcards() does not make the value SQL injection safe!

getRestrictions(), setRestrictions(), resetRestrictions()

API methods to deal with the RestrictionBuilder.

Connection

An instance of class TYPO3\CMS\Core\Database\Connection is retrieved from the ConnectionPool by calling ->getConnectionForTable() and handing over the table name a query should executed on.

The class extends the basic doctrine-dbal Doctrine\DBAL\Connection class and is mainly used internally within the TYPO3 CMS framework to establish, maintain and terminate connections to single database endpoints. Those internal methods are not scope of this documentation since an extension developer usually doesn't have to deal with that.

For an extension developer however, the class provides a list of "short-hand" methods that allow dealing with "simple" query cases, without the complexity of the QueryBuilder. Using those methods typically ends up in rather short and easily readable code. The methods have in common that they support only "equal" comparisons in WHERE conditions, that all fields and values are fully quoted automatically and the created queries are executed right away.

Note

The Connection object is designed to work on a single table only. If queries to multiple tables should be performed, the object must not be re-used. Instead, a single Connection instance should be retrieved via ConnectionPool per target table. However, it is allowed to use one Connection object for multiple queries to the same table.

insert()

Creates and executes an INSERT INTO statement. A (slightly simplified) example from the Registry API:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// INSERT INTO `sys_registry` (`entry_namespace`, `entry_key`, `entry_value`) VALUES ('aoeu', 'aoeu', 's:3:\"bar\";')
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_registry')
   ->insert(
      'sys_registry',
      [
         'entry_namespace' => $namespace,
         'entry_key' => $key,
         'entry_value' => serialize($value)
      ]
   );

Well, that should be rather obvious: First argument is the table name to insert a row into, second argument is an array of key/value pairs. All keys are quoted to field names and all values are quoted to string values.

It is possible to add another array as third argument to specify how single values are quoted. This is useful if date or numbers or similar should be inserted. The example below quotes the first value to an integer and the second one to a string:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Connection;
// INSERT INTO `sys_log` (`userid`, `details`) VALUES (42, 'klaus')
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_log')
   ->insert(
      'sys_log',
      [
         'userid' => (int)$userId,
         'details' => (string)$details,
      ],
      [
         Connection::PARAM_INT,
         Connection::PARAM_STR,
      ]
   );

insert() returns the number of affected rows. Guess what? That's the number 1 ... In case something goes wrong a \Doctrine\DBAL\DBALException is raised.

Note

A list of allowed field types for proper quoting can be found in the TYPO3\CMS\Core\Database\Connection class and its base class \Doctrine\DBAL\Connection

bulkInsert()

INSERT multiple rows at once. An example from the test suite:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$connection = GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('aTestTable')
$connection->bulkInsert(
   'aTestTable',
   [
      ['aField' => 'aValue'],
      ['aField' => 'anotherValue']
   ],
   [
      'aField'
   ]
);

First argument is the table to insert table into, second argument is an array of rows, third argument is the list of field names. Similar to ->insert() it is optionally possible to add another argument to specify quoting details, if omitted, everything will be quoted to strings.

Note

mysql is rather forgiving when it comes to insufficient field quoting: Inserting a string to an int field will not raise an error and mysql will adapt internally. However, other dbms are not that relaxed and may raise errors. It is good practice to specify field types for each field, especially if they are not strings. Doing so right away will reduce the number of raised bugs if people run your extension an anything else than mysql.

update()

Create and execute an UPDATE statement. The example from FAL's ResourceStorage sets a storage to offline:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Connection;
// UPDATE `sys_file_storage` SET `is_online` = 0 WHERE `uid` = '42'
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_file_storage')
   ->update(
      'sys_file_storage',
      ['is_online' => 0],
      ['uid' => (int)$this->getUid()],
      [Connection::PARAM_INT]
   );

First argument is the table an update should be executed on, the second argument is an array of key/value pairs to set, the third argument is an array of "equal" where statements that are combined with AND, the (optional) fourth argument specifies the type of values to be updated similar to ->insert() and bulkInsert().

Note the third argument WHERE `foo` = 'bar' only supports equal =. For more complex stuff the QueryBuilder has to be used.

The method returns the number of affected rows.

delete()

Execute a DELETE query using equal conditions in WHERE, example from BackendUtility to mark rows as no longer locked by a user:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Connection;
// DELETE FROM `sys_lockedrecords` WHERE `userid` = 42
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_lockedrecords')
   ->delete(
      'sys_lockedrecords',
      ['userid' => (int)42],
      [Connection::PARAM_INT]
    );

First argument is the table name, second argument is a list of AND combined WHERE conditions as array, third argument specifies the quoting of WHERE values. There is a pattern ;)

Note

TYPO3 CMS uses a "soft delete" approach for many tables. Instead of directly deleting a rows in the database, a field - often called deleted - is set from 0 to 1. Executing a DELETE query circumvents this and really removes rows from a table. For most tables, it is better to use the DataHandler API to handle deletes instead of executing such low level queries directly.

truncate()

Empty a table, removing all rows. Usually much quicker than a ->delete() of all rows. This typically resets "auto increment primary keys" to zero. Use with care:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// TRUNCATE `cache_treelist`
GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('cache_treelist')
   ->truncate('cache_treelist');
count()

A COUNT query. Again, this methods becomes handy if very simple COUNT statements are to be executed, the example returns tha number of active rows from table tt_content that have their bodytext field set to klaus:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT COUNT(*)
// FROM `tt_content`
// WHERE
//     (`bodytext` = 'klaus')
//     AND (
//         (`tt_content`.`deleted` = 0)
//         AND (`tt_content`.`hidden` = 0)
//         AND (`tt_content`.`starttime` <= 1475621940)
//         AND ((`tt_content`.`endtime` = 0) OR (`tt_content`.`endtime` > 1475621940))
//     )
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('tt_content');
$rowCount = $connection->count(
   '*',
   'tt_content',
   ['bodytext' => 'klaus']
);

First argument is the field to count on, usually * or uid. Second argument is the table name, third argument is an array of WHERE equal conditions combined with AND.

Remarks:

  • ->count() of Connection returns the number directly as integer, in contrast to the method of the QueryBuilder, there is no need to call ->fetchColumns(0) or similar.
  • The third argument expects all WHERE values to be strings, each single expression is combined with AND.
  • The RestrictionBuilder kicks in and adds additional WHERE conditions based on TCA settings.
  • Field names and values are quoted automatically.
  • If anything more complex than a simple equal condition on WHERE is needed, the QueryBuilder methods are a better choice: Next to ->select(), the ->count() query is often the least useful method of the Connection object.
select()

Creates and executes a simple SELECT query based on equal conditions. Its usage is limited, the RestrictionBuilder kicks in and key/value pairs are automatically quoted:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// SELECT `entry_key`, `entry_value` FROM `sys_registry` WHERE `entry_namespace` = 'my_extension'
$resultRows = GeneralUtility::makeInstance(ConnectionPool::class)
   ->getConnectionForTable('sys_registry')
   ->select(
      ['entry_key', 'entry_value'],
      'sys_registry',
      ['entry_namespace' => 'my_extension']
   );

Remarks:

  • In contrast to the other short-hand methods, ->select() returns a Statement object ready to ->fetch() single rows or to ->fetchAll()
  • The method accepts a series of further arguments to specify GROUP BY, ORDER BY, LIMIT and OFFSET query parts.
  • For non-trivial SELECT queries, it is often better to switch to the according method of the QueryBuilder object.
  • The RestrictionBuilder adds default WHERE restrictions. If those restrictions do not apply to the query needs, it is required to switch to the QueryBuilder->select() method for fine-grained WHERE manipulation.
lastInsertId()

Returns the uid of the last ->insert() statement. Useful if this id needs to be used afterwards directly:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Connection;
$connectionPool = GeneralUtility::makeInstance(ConnectionPool::class);
$databaseConnectionForPages = $connectionPool->getConnectionForTable('myTable');
$databaseConnectionForPages->insert(
   'myTable',
   [
      'pid' => 0,
      'title' => 'Home',
   ]
);
$pageUid = (int)$databaseConnectionForPages->lastInsertId('pages');

Remarks:

  • ->lastInsertId($tableName) needs the table name as first argument. While this is optional, you should always supply the table name for DBAL compatibility with engines like postgres.
  • If the auto increment field name is not uid, the second argument with the name of this field must be supplied. For casual TYPO3 tables, uid is ok and the argument can be left out.
createQueryBuilder()

The QueryBuilder should not be re-used for multiple different queries. However, it sometimes becomes handy to first fetch a Connection object for a specific table and to execute a simple query, and to create a QueryBuilder for a more complex query from this connection object later. The methods usefulness is limited however and no good example within the core can be found at the time of this writing.

The method can be helpful in loops to save some precious code characters, too:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable($myTable);
foreach ($someList as $aListValue) {
   $myResult = $connection->createQueryBuilder
      ->select('something')
      ->from('whatever')
      ->where(...)
      ->execute()
      ->fetchAll();
}

ExpressionBuilder

The ExpressionBuilder class is responsible to dynamically create SQL query parts for WHERE and JOIN ON conditions, functions like ->min() may also be used in SELECT parts.

It takes care of building query conditions while ensuring table and column names are quoted within the created expressions / SQL fragments. It is a facade to the actual doctrine-dbal ExpressionBuilder.

The ExpressionBuilder is used within the context of the QueryBuilder to ensure queries are being build based on the requirements of the database platform in use.

An instance of the ExpressionBuilder is retrieved from the QueryBuilder object:

$expressionBuilder = $queryBuilder->expr();

It is good practice to not assign an instance of the ExpressionBuilder to a variable but to use it within the code flow of the QueryBuilder context directly:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$rows = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content')
   ->select('uid', 'header', 'bodytext')
   ->from('tt_content')
   ->where(
      // `bodytext` = 'klaus' AND `header` = 'peter'
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus')),
      $queryBuilder->expr()->eq('header', $queryBuilder->createNamedParameter('peter'))
   )
   ->execute()
   ->fetchAll();

Warning

It is crucially important to quote values correctly to not introduce SQL injection attack vectors to your application. See the section of the QueryBuilder for details.

Junctions
  • ->andX() conjunction
  • ->orX() disjunction

Combine multiple single expressions with AND or OR. Nesting is possible, both methods are variadic and take any number of argument which are all combined. It usually doesn't make much sense to hand over zero or only one argument, though.

A core example to find a sys_domain record:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// WHERE
//     (`sys_domain`.`pid` = `pages`.`uid`)
//     AND (
//        (`sys_domain`.`domainName` = 'example.com')
//        OR
//        (`sys_domain`.`domainName` = 'example.com/')
//     )
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->where(
   $queryBuilder->expr()->eq('sys_domain.pid', $queryBuilder->createNamedParameter('pages.uid', \PDO::PARAM_INT)),
   $queryBuilder->expr()->orX(
      $queryBuilder->expr()->eq('sys_domain.domainName', $queryBuilder->createNamedParameter($domain)),
      $queryBuilder->expr()->eq('sys_domain.domainName', $queryBuilder->createNamedParameter($domain . '/'))
   )
)
Comparisons

A set of methods to create various comparison expressions or SQL functions:

  • ->eq($fieldName, $value) "equal" comparison =
  • ->neq($fieldName, $value) "not equal" comparison !=
  • ->lt($fieldName, $value) "less than" comparison <
  • ->lte($fieldName, $value) "less than or equal" comparison <=
  • ->gt($fieldName, $value) "greater than" comparison >
  • ->gte($fieldName, $value) "greater than or equal" comparison >=
  • ->isNull($fieldName) "IS NULL" comparison
  • ->isNotNull($fieldName) "IS NOT NULL" comparison
  • ->like($fieldName) "LIKE" comparison
  • ->notLike($fieldName) "NOT LIKE" comparison
  • ->in($fieldName, $valueArray) "IN ()" comparison
  • ->notIn($fieldName, $valueArray) "NOT IN ()" comparison
  • ->inSet($fieldName, $value) "FIND_IN_SET('42', aField)" Find a value in a comma separated list of values
  • ->bitAnd($fieldName, $value) A bitwise AND operation &

Remarks and warnings:

Examples:

// `bodytext` = 'foo' - string comparison
->eq('bodytext', $queryBuilder->createNamedParameter('foo'))

// `tt_content`.`bodytext` = 'foo'
->eq('tt_content.bodytext', $queryBuilder->createNamedParameter('foo'))

// `aTableAlias`.`bodytext` = 'foo'
->eq('aTableAlias.bodytext', $queryBuilder->createNamedParameter('foo'))

// `uid` = 42 - integer comparison
->eq('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))

// `uid` >= 42
->gte('uid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT))

// `bodytext` LIKE 'klaus'
->like(
   'bodytext',
   $queryBuilder->createNamedParameter($queryBuilder->escapeLikeWildcards('klaus'))
)

// `bodytext` LIKE '%klaus%'
->like(
   'bodytext',
   $queryBuilder->createNamedParameter('%' . $queryBuilder->escapeLikeWildcards('klaus') . '%')
)

// use TYPO3\CMS\Core\Database\Connection;
// `uid` IN (42, 0, 44) - properly sanitized, mind the intExplode and PARAM_INT_ARRAY
->in(
   'uid',
   $queryBuilder->createNamedParameter(
      GeneralUtility::intExplode(',', '42, karl, 44', true),
      Connection::PARAM_INT_ARRAY
   )
)

// use TYPO3\CMS\Core\Database\Connection;
// `CType` IN ('media', 'multimedia') - properly sanitized, mind the PARAM_STR_ARRAY
->in(
   'CType',
   $queryBuilder->createNamedParameter(
      ['media', 'multimedia'],
      Connection::PARAM_STR_ARRAY
   )
)
Aggregate functions

Aggregate functions used in SELECT parts, often combined with GROUP BY. First argument is the field name (or table name / alias with field name), second argument an optional alias.

  • ->min($fieldName, $alias = NULL) "MIN()" calculation
  • ->max($fieldName, $alias = NULL) "MAX()" calculation
  • ->avg($fieldName, $alias = NULL) "AVG()" calculation
  • ->sum($fieldName, $alias = NULL) "SUM()" calculation
  • ->count($fieldName, $alias = NULL) "COUNT()" calculation

Examples:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// Calculate the average creation timestamp of all rows from tt_content
// SELECT AVG(`crdate`) AS `averagecreation` FROM `tt_content`
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$result = $queryBuilder
   ->addSelectLiteral(
      $queryBuilder->expr()->avg('crdate', 'averagecreation')
   )
   ->from('tt_content')
   ->execute()
   ->fetch();

// Distinct list of all existing endtime values from tt_content
// SELECT `uid`, MAX(`endtime`) AS `maxendtime` FROM `tt_content` GROUP BY `endtime`
$statement = $queryBuilder
   ->select('uid')
   ->addSelectLiteral(
      $queryBuilder->expr()->max('endtime', 'maxendtime')
   )
   ->from('tt_content')
   ->groupBy('endtime')
   ->execute();
Various Expressions
TRIM

Using the TRIM expression makes sure fields get trimmed on database level. See the examples below to get a better idea of what can be done:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Expression\ExpressionBuilder;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->expr()->comparison(
    $queryBuilder->expr()->trim($fieldName),
    ExpressionBuilder::EQ,
    $queryBuilder->createNamedParameter('', \PDO::PARAM_STR)
);

The call to $queryBuilder->expr()-trim() can be one of the following:

  • trim('fieldName') results in TRIM("tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_LEADING, 'x') results in TRIM(LEADING "x" FROM "tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_TRAILING, 'x') results in TRIM(TRAILING "x" FROM "tableName"."fieldName")
  • trim('fieldName', AbstractPlatform::TRIM_BOTH, 'x') results in TRIM(BOTH "x" FROM "tableName"."fieldName")
LENGTH

The LENGTH string function can be used to return the length of a string in bytes, method signature is fieldName with optional alias ->length(string $fieldName, string $alias = null):

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Expression\ExpressionBuilder;
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder->expr()->comparison(
    $queryBuilder->expr()->length($fieldName),
    ExpressionBuilder::GT,
    $queryBuilder->createNamedParameter(0, \PDO::PARAM_INT)
);

RestrictionBuilder

Database tables in TYPO3 CMS that can be administrated in the backend come with TCA definitions that specify how single fields and rows of the table should be handled and displayed by the framework.

The ctrl section of a tables TCA array specifies optional framework internal handling of soft deletes and language overlays: For instance, if a row in the backend is deleted using the page or list module, many tables are configured to not entirely drop that row from the table, instead a field (often deleted) is set from zero to one for that row. Similar mechanics kick in for start- and endtime as well as language and workspace overlays. See the ['ctrl'] chapter in the TCA reference for details on this topic.

These mechanics however come with a price tag attached to it: Extension developers dealing with low-level query stuff must take care overlayed or deleted rows are not in the result set of a casual query.

This is where this "automatic restriction" stuff kicks in: The construct is created on top of native doctrine-dbal as TYPO3 CMS specific extension. It automatically adds WHERE expressions that suppress rows which are marked as deleted or exceeded their "active" life cycle. All that is based on the TCA configuration of the affected table.

Rationale

A developer may ask why she has to go through all this and why this additional stuff is added on a low-level query layer, when "just a simple query" should be fired. The construct implements some important design goals:

  • Simple: Query creation should be easy to use without forcing a developer thinking too much about those nasty TCA details.
  • Cope with developer laziness: If the framework would force a developer to always add casual restrictions for each and every query, this is easy to forget. We're all lazy, are we?
  • Security: If in doubt, it is better to show a little too less than too much. It is much better to deal with a customer who complains some records are not shown than to show too many records. The former is "just a bug" while the latter can easily escalate to a serious privilege escalation security issue.
  • Automatic query upgrades: If a table was designed without soft-delete in the first place and later a deleted flag is added and registered in TCA, queries executed on that table will automatically upgrade and add the according deleted = 0 restriction.
  • Handing over restriction details to the framework: Having the restriction expressions done by the framework gives it the opportunity to change details without breaking extension code. This may very well happen in the future and having a happy little upgrade path for such cases in place may become very handy later.
  • Flexibility: The class construct is created in a way that allows developers to extend or substitute it with own restrictions if that is useful to model the domain in question.
Main construct

The restriction builder is called whenever a SELECT or COUNT query is executed through either the QueryBuilder or Connection. The QueryBuilder allows manipulation of those restrictions while the simplified Connection class does not. If a query deals with multiple tables in a join, restrictions for all affected tables are added.

Each single restriction like a DeletedRestriction or a StartTimeRestriction is modeled as a single class implementing the QueryRestrictionInterface. Each restriction looks up in TCA if it should kick in. If so, it adds according expressions to the WHERE clause when the final statement is compiled.

Multiple restrictions can be grouped in containers which implement the QueryRestrictionContainerInterface.

The DefaultRestrictionContainer is always added by ... uuhm ... default: It adds the DeletedRestriction, the HiddenRestriction, the StartTimeRestriction and the EndTimeRestriction. Note this is true for all contexts a query is executed in: It does not matter whether a query is created from within a frontend, a backend or a cli call, they all add the DefaultRestrictionContainer if not explicitly told otherwise by an extension developer.

Note

Having this DefaultRestrictionContainer used everywhere is the second iteration of that code construct:

The first variant automatically added restrictions based on context. For instance, a query fired by a call that is executed in the backend did not add the hidden flag, while a query fired from within a frontend call did so. We quickly figured this ends up in a huge mess: The distinction between frontend, backend and cli is not that sharp in TYPO3, as example the frontend behaves much more like a backend call if the admin panel is used.

The currently active variant is much easier: It always adds sane defaults everywhere, a developer only has to deal with details if they don't fit. The core team hopes this approach is a good balance between hidden magic, security, transparency and convenience.

Restrictions
  • DeletedRestriction: (default) Evaluates ['ctrl']['delete'], adds for instance AND deleted = 0 if TCA['aTable']['ctrl']['delete'] = 'deleted' is specified.
  • HiddenRestriction: (default) Evaluates ['ctrl']['enablecolumns']['disabled'], adds AND hidden = 0 if hidden is specified as field name.
  • StartTimeRestriction: (default) Evaluates ['ctrl']['enablecolumns']['starttime'], typically adds something like AND (`tt_content`.`starttime` <= 1475580240).
  • EndTimeRestriction: (default) Evaluates ['ctrl']['enablecolumns']['endtime'].
  • FrontendGroupRestriction: Evaluates ['enablecolumns']['fe_group'].
  • RootlevelRestriction: Match records on root level, adds AND (`pid` = 0)
  • BackendWorkspaceRestriction: Determines the current workspace a backend user is working in and adds a couple of restrictions to select only records of that workspace if the table supports workspaced records.
  • FrontendWorkspaceRestriction: Restriction to filter records for fronted workspaces preview.
QueryRestrictionContainer
  • DefaultRestrictionContainer: Add DeletedRestriction, HiddenRestriction, StartTimeRestriction and EndTimeRestriction. This container is always added if not told otherwise.
  • FrontendRestrictionContainer: Adds DeletedRestriction, HiddenRestriction, StartTimeRestriction, EndTimeRestriction, FrontendWorkspaceRestriction and FrontendGroupRestriction. This container should be be added by a developer to a query if creating query statements in frontend context or if handling frontend stuff from within cli calls.
Examples

Often the default restrictions are sufficient. Nothing needs to be done in those cases.

However, many backend modules still want to show disabled records and remove the starttime and endtime restrictions to allow administration of those records for an editor. A typical setup from within a backend module:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Restriction\DeletedRestriction
// SELECT `uid`, `bodytext` FROM `tt_content` WHERE (`pid` = 42) AND (`tt_content`.`deleted` = 0)
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
// Remove all restrictions but add DeletedRestriction again
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
$result = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter($pid, \PDO::PARAM_INT)))
   ->execute()
   ->fetchAll();

The DeletedRestriction should be kept in almost all cases. Usually, the only extension that dismiss that flag is the recycler module to list and resurrect deleted records. Any object implementing the QueryRestrictionInterface can be given to ->add(). This allows extensions to deliver own restrictions.

An alternative to the recommended way of first removing all restrictions and then adding needed ones again (using ->removeAll(), then ->add()) is to kick specific restrictions with a call to ->removeByType():

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Restriction\StartTimeRestriction
// use TYPO3\CMS\Core\Database\Query\Restriction\EndTimeRestriction
// Remove starttime and endtime, but keep hidden and deleted
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$queryBuilder
   ->getRestrictions()
   ->removeByType(StartTimeRestriction::class)
   ->removeByType(EndTimeRestriction::class);

In the frontend it is often needed to swap the DefaultRestrictionContainer with the FrontendRestrictionContainer:

// use TYPO3\CMS\Core\Database\Query\Restriction\FrontendRestrictionContainer
// Kick default restrictions and add list of default frontend restrictions
$queryBuilder->setRestrictions(GeneralUtility::makeInstance(FrontendRestrictionContainer::class));

Note that ->setRestrictions() resets any previously specified restrictions. Any class instance implementing QueryRestrictionContainerInterface can be given to ->setRestrictions(). This allows extensions to deliver and use an own set of restrictions for own query statements if needed.

Tip

It can be very helpful to debug the final statements created by the RestrictionBuilder using debug($queryBuilder->getSQL()) right before the final call to $queryBuilder->execute(). Just take care these calls do not end up in production code.

Statement

A Statement object is returned by QueryBuilder->execute() for ->select() and ->count() query types and by Connection->select() and Connection->count() calls.

The object represents a query result set and comes with methods to ->fetch() single rows or to ->fetchAll() of them. Additionally, it can also be used to execute a single prepared statement with different values multiple times. This part is however not widely used within the TYPO3 CMS core yet, and thus not fully documented here.

Note

The name "Statement" instead of "Result" can be puzzling at first glance: The class represents a prepared statement that can be executed multiple times with different values and then returns multiple different result sets. From this point of view "Statement" fits much better than "Result".

Warning

The return type of single field values is NOT type safe! If selecting a value from a field that is defined as int, the Statement result may very well return that as PHP string. This is true for other database column types like FLOAT, DOUBLE and others. This is an issue with the database drivers used below, it may happen that MySQL returns an integer value for an int field, while MSSQL returns a string. In general, the application must take care of an according type cast on their own to reach maximum DBMS compatibility.

fetch()

Fetch next row from a result statement. Usually used in while() loops. Typical example:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// Fetch all records from tt_content on page 42
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute();
while ($row = $statement->fetch()) {
   // Do something useful with that single $row
}

->fetch() returns arrays with single field / values pairs until the end of the result set is reached which then returns false and thus breaks the while loop.

fetchAll()

Returns an array containing all of the result set rows by implementing the same while loop as above internally. Using that method saves some precious code characters but is more memory intensive if the result set is large with lots of rows and lot of data since big arrays are carried around in PHP:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// Fetch all records from tt_content on page 42
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$rows = $queryBuilder
   ->select('uid', 'bodytext')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute()
   ->fetchAll();
fetchColumn()

Returns a single column from the next row of a result set, other columns from that result row are discarded. This method is especially handy for QueryBuilder->count() queries. The Connection->count() implementation does exactly that to return the number of rows directly:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// Get the number of tt_content records on pid 42 into variable $numberOfRecords
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$numberOfRecords = $queryBuilder
   ->count('uid')
   ->from('tt_content')
   ->where($queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter(42, \PDO::PARAM_INT)))
   ->execute()
   ->fetchColumn(0);
rowCount()

Returns the number of rows affected by the last execution of this statement. Use that method instead of counting the number of records in a ->fetch() loop manually.

Warning

->rowCount() works well with DELETE, UPDATE and INSERT queries. However, it does NOT return a valid number for SELECT queries on some DBMS. Never use ->rowCount() on SELECT queries. This may work with MySOL, but fails with other databases like SQLite.

Re-use prepared Statement()

Doctrine usually prepares a statement first, and then executes it with given parameters. Implementing prepared statements depends on the given driver. For instance, the native mysql driver mysqli does implement prepared statements, while the pdo driver of mysql pdo_mysql does not, at least in some scenarios. A driver not properly implementing prepared statements fall back to a direct execution of given query.

There is an API to make real use of prepared statements that becomes handy if the same query is executed with different arguments over and over again. The example below prepares a statement to the pages table and executes it twice with different arguments:

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
$connection = GeneralUtility::makeInstance(ConnectionPool::class)->getConnectionForTable('pages');
$queryBuilder = $connection->createQueryBuilder();
$queryBuilder->getRestrictions()->removeAll();
$sqlStatement = $queryBuilder->select('uid')
    ->from('pages')
    ->where($queryBuilder->expr()->eq('uid', $queryBuilder->createPositionalParameter(0, \PDO::PARAM_INT)))
    ->getSQL();
$statement = $connection->executeQuery($sqlStatement, [ 24 ]);
$result1 = $statement->fetch();
$statement->bindValue(1, 25);
$statement->execute();
$result2 = $statement->fetch();

Looking at a mysql debug log:

Prepare SELECT `uid` FROM `pages` WHERE `uid` = ?
Execute SELECT `uid` FROM `pages` WHERE `uid` = '24'
Execute SELECT `uid` FROM `pages` WHERE `uid` = '25'

The log shows one statement preparation with two executions.

QueryHelper

The class contains miscellaneous helper methods to build syntactically valid SQL queries.

Most helper methods are required to deal with legacy data where the format of the input is not strict enough to reliably use the SQL parts in queries directly.

The whole class is marked as @internal, should not be used by extension authors and may - if things go wrong - change at will. The class will hopefully vanish mid-term. However, there may be situations when the class methods can become handy if extension authors migrate their own extensions away from TYPO3_DB to doctrine-dbal. In practice, the core will most likely add proper deprecations to single methods if they are target of removal later.

Extension developers may keep this class in mind for migration, but must not use methods for new code created from scratch. Apart from that, as can be seen below, using those methods often ends up in rather ugly code.

The migration benefits are the only reason the methods are documented here.

Warning

Using those methods raise the risk of SQL injections, especially for methods like ->stripLogicalOperatorPrefix() since its input string tends to come from user supplied input and is sometimes added as WHERE expression without further quoting. Keep a special eye on those scenarios!

parseOrderBy()

Some parts of the core framework allow string definitions like ORDER BY sorting for instance in TCA and TypoScript. The method rips those strings apart and prepares them to be fed to QueryBuilder->orderBy():

// 'ORDER BY aField ASC,anotherField, aThirdField DESC'
// ->
// [ ['aField', 'ASC'], ['anotherField', null], ['aThirdField', 'DESC'] ]
$uglyOrderBy = 'ORDER BY aField ASC,anotherField, aThirdField DESC'
foreach (QueryHelper::parseOrderBy((string)$uglyOrderBy) as $orderPair) {
   list($fieldName, $order) = $orderPair;
   $queryBuilder->addOrderBy($fieldName, $order);
}
parseGroupBy()

Parses GROUP BY strings ready to be added via QueryBuilder->groupBy(), similar to ->parseOrderBy():

// 'GROUP BY be_groups.title, anotherField'
// ->
// ['be_groups.title', 'anotherField']
$uglyGroupBy = 'GROUP BY be_groups.title, anotherField';
$queryBuilder->groupBy(QueryHelper::parseGroupBy($uglyGroupBy));
parseTableList()

Parse a table list, possibly prefixed with FROM, and explode it into and array of arrays where each item consists of a tableName and an optional alias name, ready to be put into QueryBuilder->from():

// 'FROM aTable a,anotherTable, aThirdTable AS c',
// ->
// [ ['aTable', 'a'], ['anotherTable', null], ['aThirdTable', 'c'] ]
$uglyTableString = 'FROM aTable a,anotherTable, aThirdTable AS c;
foreach (QueryHelper::parseTableList($uglyTableString) as $tableNameAndAlias) {
   list($tableName, $tableAlias) = $tableNameAndAlias;
   $queryBuilder->from($tableName, $tableAlias);
}
parseJoin()

Split a JOIN SQL fragment into table name, alias and join conditions:

// 'aTable AS `anAlias` ON anAlias.uid = anotherTable.uid_foreign'
// ->
// [
//     'tableName' => 'aTable',
//     'tableAlias' => 'anAlias',
//     'joinCondition' => 'anAlias.uid = anotherTable.uid_foreign'
// ],
$uglyJoinString = 'aTable AS `anAlias` ON anAlias.uid = anotherTable.uid_foreign';
$joinParts = QueryHelper::parseJoin($uglyJoinString);
$queryBuilder->join(
   $leftTable,
   $joinParts['tableName'],
   $joinParts['tableAlias'],
   $joinParts['joinCondition']
);
stripLogicalOperatorPrefix()

Removes the prefixes AND / OR from an input string.

Those prefixes are added in doctrine-dbal via QueryBuilder->where(), QueryBuilder->orWhere(), ExpressionBuilder->andX() and friends. Some parts of the TYPO3 framework however carry SQL fragments prefixed with AND or OR around and it's not always possible to easily get rid of those. The method helps by killing those prefixes before they are handed over to the doctrine API:

// 'AND 1=1'
// ->
// '1=1'
$uglyWherePart = 'AND 1=1'
$queryBuilder->where(
   // WARNING: High risk of possible SQL injection here, take additional actions!
   QueryHelper::stripLogicalOperatorPrefix($uglyWherePart)
);
getDateTimeFormats()

Just a left over method from the old TYPO3_DB DatabaseConnection class. Of little to no use for extension authors. This one is hopefully one of the first methods to vanish from the class.

quoteDatabaseIdentifiers()

This helper method is used especially in TCA and TypoScript at places where SQL fragments are specified to correctly quote table and field names for the specific database platform. It for example substitutes {#aIdentifier} to `aIdentifier` if using MySQL or to "aIdentifier" if using PostgreSQL. Let's suppose a simple TCA columns select field like this:

'aSelectFieldWithForeignTableWhere' => [
    'label' => 'some label',
    'config' => [
        'type' => 'select',
        'renderType' => 'selectSingle',
        'foreign_table' => 'tx_some_foreign_table_name',
        'foreign_table_where' => 'AND {#tx_some_foreign_table_name}.{#pid} = 42',
    ],
],

Method quoteDatabaseIdentifiers() is called for foreign_table_where, and if using MySQL, this fragment will be substituted to:

AND `tx_some_foreign_table_name`.`pid` = 42

The core had to come up with this special syntax since the core API contains various places where SQL fragments can be specified by extension developers who do not know and should not restrict on which actual platform a query is performed.

As an extension developer it is important to use this {#...} syntax in order to make extensions database platform agnostic. The TCA reference and TypoScript reference contains hints at the according properties that need this, in general the core calls this helper method whenever SQL fragments can be specified in TCA and TypoScript.

Migrating from TYPO3_DB

This chapter is for those poor souls who want to migrate old and busted $GLOBALS['TYPO3_DB'] calls to new hotness doctrine-dbal based API.

It tries to give some hints on typical pitfalls and areas a special eye should be kept on.

Migration of a single extension is finished if a search for $GLOBALS['TYPO3_DB'] does not return hits anymore. This search is the most simple entry point to see which areas need work.

Compare raw queries

The main goal during migration is usually to fire a logically identical query. One recommended and simple approach to verify this is to note down and compare the queries at the lowest possible layer. In $GLOBALS['TYPO3_DB'], the final query statement is usually retrieved by removing the exec_ part from the method name, in doctrine method QueryBuilder->getSQL() can be used:

// Initial code:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery('*', 'index_fulltext', 'phash=' . (int)$phash);

// Remove 'exec_' and debug SQL:
debug($GLOBALS['TYPO3_DB']->SELECTquery('*', 'index_fulltext', 'phash=' . (int)$phash));
// Returns:
'SELECT * FROM index_fulltext WHERE phash=42'

// Migrate to doctrine and debug SQL:
// 'SELECT * FROM index_fulltext WHERE phash=42'
$queryBuilder->select('*')
->from('index_fulltext')
->where(
   $queryBuilder->expr()->eq('phash', $queryBuilder->createNamedParameter($phash, \PDO::PARAM_INT))
);
debug($queryBuilder->getSQL());

The above example returns the exact same query as before. This is not always as trivial to see since WHERE clauses are often in a different order. This especially happens if the RestrictionBuilder is involved. Since the restrictions are crucial and can easily go wrong it is advised to keep an eye on those where parts during transition.

enableFields() and deleteClause()

BackendUtility::deleteClause() adds deleted=0 if ['ctrl']['deleted'] is specified in the table's TCA. The method call should be removed during migration. If there is no other restriction method involved in the old call like enableFields(), the migrated code typically removes all doctrine default restrictions and just adds the DeletedRestriction again:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, TSconfig',
   'pages',
   'TSconfig != \'\''
      . BackendUtility::deleteClause('pages'),
   'pages.uid'
);

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Restriction\DeletedRestriction
// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('pages');
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class));
$res = $queryBuilder->select('uid', 'TSconfig')
   ->from('pages')
   ->where($queryBuilder->expr()->neq('TSconfig', $queryBuilder->createNamedParameter('')))
   ->groupBy('uid')
   ->execute();

BackendUtility::versioningPlaceholderClause('pages') is typically substituted with the BackendWorkspaceRestriction. Example very similar to the above one:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, TSconfig',
   'pages',
   'TSconfig != \'\''
      . BackendUtility::deleteClause('pages')
      . BackendUtility::versioningPlaceholderClause('pages'),
   'pages.uid'
);

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Restriction\DeletedRestriction
// use TYPO3\CMS\Core\Database\Query\Restriction\BackendWorkspaceRestriction
// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('pages');
$queryBuilder
   ->getRestrictions()
   ->removeAll()
   ->add(GeneralUtility::makeInstance(DeletedRestriction::class))
   ->add(GeneralUtility::makeInstance(BackendWorkspaceRestriction::class));
$res = $queryBuilder->select('uid', 'TSconfig')
   ->from('pages')
   ->where($queryBuilder->expr()->neq('TSconfig', $queryBuilder->createNamedParameter('')))
   ->groupBy('uid')
   ->execute();

BackendUtility::BEenableFields() in combination with BackendUtility::deleteClause() adds the same calls as the DefaultRestrictionContainer. No further configuration needed:

// Before:
$GLOBALS['TYPO3_DB']->exec_SELECTgetRows(
   'title, content, crdate',
   'sys_news',
   '1=1'
      . BackendUtility::BEenableFields($systemNewsTable)
      . BackendUtility::deleteClause($systemNewsTable)
);

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)
   ->getQueryBuilderForTable('sys_news');
$queryBuilder
   ->select('title', 'content', 'crdate')
   ->from('sys_news')
   ->execute();

cObj->enableFields() in frontend context is typically directly substituted with FrontendRestrictionContainer:

// Before:
$GLOBALS['TYPO3_DB']->exec_SELECTquery(
   '*', $table,
   'pid=' . (int)$pid
      . $this->cObj->enableFields($table)
);

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// use TYPO3\CMS\Core\Database\Query\Restriction\FrontendRestrictionContainer
// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable($table);
$queryBuilder->setRestrictions(GeneralUtility::makeInstance(FrontendRestrictionContainer::class));
$queryBuilder->select('*')
   ->from($table)
   ->where(
      $queryBuilder->expr()->eq('pid', $queryBuilder->createNamedParameter($pid, \PDO::PARAM_INT))
   )
);
From ->exec_UDATEquery() to ->update()

Most often, the easiest way to migrate a $GLOBALS['TYPO3_DB']->exec_UDATEquery() is to use $connection->update():

// Before:
$database->exec_UPDATEquery(
    'aTable', // table
    'uid = 42', // where
    [ 'aField' => 'newValue' ] // value array
);

// After:
$connection->update(
    'aTable', // table
    [ 'aField' => 'newValue' ], // value array
    [ 'uid' => 42 ] // where
);

Warning

If switching from exec_UPDATEquery() to update, the order of arguments change, where and values are swapped!

Result set iteration

The exec_* calls return a resource object that is typically iterated over using sql_fetch_assoc(). This is typically changed to ->fetch() on the Statement object:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(...);
while ($row = $GLOBALS['TYPO3_DB']->sql_fetch_assoc($res)) {
   // Do something
}

// After:
$statement = $queryBuilder->execute();
while ($row = $statement->fetch()) {
   // Do something
}
sql_insert_id()

It is sometimes needed to fetch the new uid of a just added record to further work with that row. In TYPO3_DB this was done with a call to ->sql_insert_id() after a ->exec_INSERTquery() call on the same resource. ->lastInsertId() can be used instead:

// Before:
$GLOBALS['TYPO3_DB']->exec_INSERTquery(
   'pages',
   [
      'pid' => 0,
      'title' => 'Home',
   ]
);
$pageUid = $GLOBALS['TYPO3_DB']->sql_insert_id();

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// After:
$connectionPool = GeneralUtility::makeInstance(ConnectionPool::class);
$databaseConnectionForPages = $connectionPool->getConnectionForTable('pages');
$databaseConnectionForPages->insert(
   'pages',
   [
      'pid' => 0,
      'title' => 'Home',
    ]
);
$pageUid = (int)$databaseConnectionForPages->lastInsertId('pages');
fullQuoteStr()

->fullQuoteStr() is rather straight changed to a ->createNamedParameter(), typical case:

// Before:
$res = $GLOBALS['TYPO3_DB']->exec_SELECTquery(
   'uid, title',
   'tt_content',
   'bodytext = ' . $GLOBALS['TYPO3_DB']->fullQuoteStr('horst')
);

// use TYPO3\CMS\Core\Utility\GeneralUtility;
// use TYPO3\CMS\Core\Database\ConnectionPool;
// After:
$queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable('tt_content');
$statement = $queryBuilder
   ->select('uid', 'title')
   ->from('tt_content')
   ->where(
      $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('horst'))
   )
   ->execute();
ext_tables.sql

The schema migrator that compiles ext_tables.sql files from all loaded extensions and compares them with current schema definitions in the database has been fully rewritten. It mostly should work as before, some specific fields however tend to grow a little larger on mysql platforms than before. This usually shouldn't have negative side effects, typically no ext_tables.sql changes are needed when migrating an extension to the new query API.

TCA and TypoScript

TCA and TypoScript needs to be adapted at places where SQL fragments are specified. Table and field names are quoted differently on different platforms and extension developers should never hard code quoting for specific target platforms, but let the core quote the field according to the currently used platform. This leads to a new syntax in various places, for instance in TCA property foreign_table_where. In general it applies to all places where SQL fragments are specified:

// Before:
'foreign_table_where' => 'AND tx_some_foreign_table_name.pid = 42',

// After:
'foreign_table_where' => 'AND {#tx_some_foreign_table_name}.{#pid} = 42',

If using MySQL, this fragment will be parsed to AND `tx_some_foreign_table_name`.`pid` = 42 (note the backticks) with the help of QueryHelper::quoteDatabaseIdentifiers().

extbase QueryBuilder

The extbase internal QueryBuilder used in Repositories still exists and works a before. There is usually no manual migration needed. It is theoretically possible to use the doctrine based query builder object in Extbase which can become handy since the new one is much more feature rich, but that topic didn't yet fully settle in the core and no general recommendation can be given yet.

Various tips and tricks

  • Use Find usages of PhpStorm for examples! The source code of the core is a great way to learn how specific methods of the API are used. In PhpStorm it is extremely helpful to right click on a single method and list all method usages with Find usages. This is especially handy to quickly see usage examples of complex methods like join() from the QueryBuilder.

  • INSERT, UPDATE and DELETE statements are often easier to read and write using the Connection object instead of the QueryBuilder.

  • SELECT DISTINCT aField is not supported but can be substituted with a ->groupBy('aField').

  • getSQL() and execute() can be used after each other during development to simplify debugging:

    $queryBuilder
       ->select('uid')
       ->from('tt_content')
       ->where(
          $queryBuilder->expr()->eq('bodytext', $queryBuilder->createNamedParameter('klaus'))
       );
    debug($queryBuilder->getSql());
    $statement = $queryBuilder->execute();
    
  • In contrast to the old API based on $GLOBALS['TYPO3_DB'], doctrine-dbal will throw exceptions if something goes wrong when calling execute(). The exception type is a \Doctrine\DBAL\DBALException which can be caught and transferred to a better error message if the application has to expect query errors. Note this is not good habit and often indicates an architectural flaw of the application at a different layer.

  • count() query types using the QueryBuilder typically call ->fetchColumn(0) to receive the count value. The count() method of Connection object does that automatically and returns the count value result directly.

Internationalization and localization

Introduction

Except for some low level functions, TYPO3 CMS exclusively uses localizable strings for all labels displayed in the backend. This means that the whole user interface may be translated. The encoding is strictly UTF-8.

The default language is English, and the Core ships only with such labels (and so should extensions).

All labels are stored in XLIFF format, generally located in the Resources/Private/Language folder of an extension (old locations may still be found in some places).

The format, TYPO3 specific details and managing interfaces of XLIFF are outlined in detail in this chapter.

Label access in PHP

In PHP, a typical call in the Backend to fetch a string in the language selected by a user looks like this:

$this->getLanguageService()->sL('LLL:EXT:core/Resources/Private/Language/locallang_core.xlf:labels.minutesHoursDaysYears')

getLanguageService() is a call to a helper method that accesses $GLOBALS['LANG']. In the Backend, the bootstrap parks an initialized instance of \TYPO3\CMS\Core\Localization\LanguageService at this place. This may change in the future, but for now it the LanguageService can be reliably fetched from this global.

Note

The ->sL() API does not apply a htmlspecialchars() call to the translated string. If the string is returned in a web context, it must be added manually.

If additional placeholders are used in a translation source, they must be injected, a call then typically looks like this:

// Text string in .xlf file has a placeholder:
// <trans-unit id="message.description.fileHasBrokenReferences">
//     <source>The file has %1s broken reference(s) but it will be deleted regardless.</source>
// </trans-unit>
sprintf($this->getLanguageService()->sL('LLL:EXT:core/Resources/Private/Language/locallang_core.xlf:message.description.fileHasBrokenReferences'), count($brokenReferences));

Various classes are involved in the localization process, with \TYPO3\CMS\Core\Localization\LanguageService providing the actual methods to retrieve a localized label. sL() loads a language file if needed first, and then returns a label from it (using a string with the LLL:EXT:... syntax as argument).

Extbase class \TYPO3\CMS\Extbase\Utility\LocalizationUtility is essentially a convenience wrapper around the \TYPO3\CMS\Core\Localization\LanguageService class, whose translate() method also takes an array as argument and runs PHP's vsprintf() on the localized string. However, in the future it is expected this Extbase specific class will melt down and somehow merged into the core API classes to get rid of this duplication.

Managing translations

This sections highlights the different ways to translate and manage XLIFF files.

The TYPO3 translation server

To manage translations of extensions uploaded to the TYPO3 Extension Repository (TER), the TYPO3 community runs an official translation server, based on Pootle. Localization files of TER extensions in English are uploaded on that server and translations are packaged nightly. They can be fetched in the TYPO3 CMS backend, via the Install Tool and on the command line.

It is not the point of this manual to go into the details of the translation process. More information can be found in the TYPO3 wiki.

Fetching translations of TER extensions

The interface of the Install Tool in ADMIN TOOLS > Maintenance > Manage language packs allows to manage the list of available languages to your users and can fetch and update language packs of TER and core extensions from the official translation server. The module is rather straight forward to use and should be pretty much self explanatory. Downloaded language packs are stored in getLabelsPath().

The Languages module

The Languages module with some active languages and status of extensions language packs

Language packs can also be fetched using the command line.

/path/to/typo3/bin/typo3 language:update
Translating locally

Using Virtaal, it is possible to translate XLIFF files locally. Virtaal is an open source, cross-platform application.

Virtaal screenshot

Translating with Virtaal, with suggestions from other software

Translating files locally is useful for extensions which are not meant to be published or for creating custom translations.

Custom translations

The $GLOBALS['TYPO3_CONF_VARS']['SYS']['locallangXMLOverride'] allows to override both locallang-XML and XLIFF files. Actually this is not just about translations. Default language files can also be overridden. In the case of XLIFF files, the syntax is as follows (to be placed in an extension's ext_localconf.php file):

$GLOBALS['TYPO3_CONF_VARS']['SYS']['locallangXMLOverride']['EXT:cms/locallang_tca.xlf'][] = 'EXT:examples/Resources/Private/Language/custom.xlf';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['locallangXMLOverride']['de']['EXT:news/Resources/Private/Language/locallang_modadministration.xlf'][] = 'EXT:examples/Resources/Private/Language/Overrides/de.locallang_modadministration.xlf';

The first line shows how to override a file in the default language, the second how to override a German ("de") translation. The German language file looks like this:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<xliff version="1.0">
   <file source-language="en" datatype="plaintext" original="messages" date="2013-03-09T18:44:59Z" product-name="examples">
      <header/>
      <body>
         <trans-unit id="pages.title_formlabel" xml:space="preserve">
            <source>Most important tile</source>
            <target>Wichtigster Titel</target>
         </trans-unit>
      </body>
   </file>
</xliff>

and the result can be easily seen in the backend:

Custom label

Custom translation in the TYPO3 backend

Important

  • Please note that you do not have to copy the full reference file, but only the labels you want to translate.
  • The path to the file to override must be expressed as EXT:foo/bar/.... For the extension "xlf" or "xml" can be used interchangeably. The TYPO3 Core will try both anyway, but using "xlf" is more correct and future-proof.

Attention

The following is a bug but must be taken as a constraint for now:

  • The files containing the custom labels must be located inside an extension. Other locations will not be considered.
  • The original translation needs to exist in getLabelsPath() or next to the base translation file in extensions, for example in typo3conf/ext/myext/Resources/Private/Language/.
Custom languages

The list of supported languages is defined in \TYPO3\CMS\Core\Localization\Locales::$languages. Adding support for a new language usually starts by adding the language there and waiting for the next release.

However, it is possible to add custom languages to the TYPO3 backend and create the translations locally using XLIFF files.

First of all, the language must be declared:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['localization']['locales']['user'] = array(
    'gsw_CH' => 'Swiss German',
);

This new language does not need to be entirely translated. It can be defined as falling back to another language, so that only differing labels need be translated:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['localization']['locales']['dependencies'] = array(
   'gsw_CH' => array('de_AT', 'de'),
);

In this case we define that "gsw_CH" (which is the official code for "Schwiizertüütsch" - that is, "Swiss German") can fall back on "de_AT" (another custom translation) and then on "de".

The translations have to be stored in the appropriate labels path sub folder (getLabelsPath()), in this case /gsw_CH.

The very least you need is to translate the label containing the name of the language itself, so that it appears in the user preferences. In our example this would be in file /gsw_CH/setup/mod/gsw_CH.locallang.xlf.

<?xml version='1.0' encoding='utf-8'?>
<xliff version="1.0">
   <file source-language="en" target-language="gsw_CH" datatype="plaintext" original="messages" product-name="setup">
      <header/>
      <body>
         <trans-unit id="lang_gsw_CH" approved="yes">
            <source>Swiss German</source>
            <target state="translated">Schwiizertüütsch</target>
         </trans-unit>
      </body>
   </file>
</xliff>
User Settings screenshot

The new language appears in the user preferences

Note

Any language will always fall back on the default one (i.e. English) when a translation is not found. A custom language will fall back on its "parent" language automatically. Thus - in our second example of de_AT (German for Austria) - no fallback would have to be defined for "de_AT" if it were just falling back on "de".

Custom translation servers

With the usage of XLIFF and the freely available Pootle translation server, companies and individuals may easily set up a custom translation server for their extensions.

There is a signal that can be caught to change the translation server URL to use. The first step is to register one's code for handling the signal. Such code would be placed in an extension's ext_localconf.php file:

$signalSlotDispatcher = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Extbase\SignalSlot\Dispatcher::class);
$signalSlotDispatcher->connect(
   version_compare(TYPO3_version, '7.0', '<')
      ? 'TYPO3\\CMS\\Lang\\Service\\UpdateTranslationService'
      : 'TYPO3\\CMS\\Lang\\Service\\TranslationService',
   'postProcessMirrorUrl',
   'Company\\Extension\Slots\\CustomMirror',
   'postProcessMirrorUrl'
);

The class (slot) which receives the signal (EXT:myext/Classes/Slots/CustomMirror.php) could look something like:

<?php
namespace Company\Extensions\Slots;
class CustomMirror {
   static protected $extKey = 'myext';

   public function postProcessMirrorUrl($extensionKey, &$mirrorUrl) {
      if ($extensionKey === self::$extKey) {
         $mirrorUrl = 'http://mycompany.tld/typo3-packages/';
      }
   }
}

Note that the mirror URL is passed as a reference, so that it can be modified. In the above example, the URL is changed only for a given extension, but of course it could be changed on a more general basis.

On the custom translation server side, the structure needs to be:

https://mycompany.tld/typo3-packages/
`-- <first-letter-of-extension-key>
   `-- <second-letter-of-extension-key>
      `-- <extension-key>-l10n
         |-- <extension-key>-l10n-de.zip
         |-- <extension-key>-l10n-fr.zip
         |-- <extension-key>-l10n-it.zip
         `-- <extension-key>-l10n.xml

hence in our example:

https://mycompany.tld/typo3-packages/
`-- m
   `-- y
      `-- myext-l10n
         |-- myext-l10n-de.zip
         |-- myext-l10n-fr.zip
         |-- myext-l10n-it.zip
         `-- myext-l10n.xml

And the myext-l10n.xml file contains something like:

<?xml version="1.0" standalone="yes" ?>
<TERlanguagePackIndex>
   <meta>
      <timestamp>1374841386</timestamp>
      <date>2013-07-26 14:23:06</date>
   </meta>
   <languagePackIndex>
      <languagepack language="de">
         <md5>1cc7046c3b624ba1fb1ef565343b84a1</md5>
      </languagepack>
      <languagepack language="fr">
         <md5>f00f73ae5c43cb68392e6c508b65de7a</md5>
      </languagepack>
      <languagepack language="it">
         <md5>cd59530ce1ee0a38e6309544be6bcb3d</md5>
      </languagepack>
   </languagePackIndex>
</TERlanguagePackIndex>

XLIFF format

The XML Localisation Interchange File Format (or XLIFF) is an OASIS-blessed standard format for translations.

In a nutshell an XLIFF document contains one or more <file> elements. Each file element usually corresponds to a source (file or database table) and contains the source of the localizable data. Once translated, the corresponding localized data for one, and only one, locale is added.

Localizable data are stored in <trans-unit> elements. The <trans-unit> contains a <source> element to store the source text and a (non-mandatory) <target> element to store the translated text.

Note that having several <file> elements in the same XLIFF document is not supported by the TYPO3 CMS Core.

Keep in mind that the default language is always considered to be english, even when you have changed your typo3 backend to another language, so source-language must always be source-language="en".

Basics

Here is a sample XLIFF file:

<?xml version="1.0" encoding="UTF-8"?>
<xliff version="1.0" xmlns="urn:oasis:names:tc:xliff:document:1.1">
   <file source-language="en" datatype="plaintext" original="messages" date="2011-10-18T18:20:51Z" product-name="my-ext">
      <header/>
      <body>
         <trans-unit id="headerComment" xml:space="preserve">
            <source>The default Header Comment.</source>
         </trans-unit>
         <trans-unit id="generator" xml:space="preserve">
            <source>The "Generator" Meta Tag.</source>
         </trans-unit>
      </body>
   </file>
</xliff>

The translated file is very similar. If the original file was named locallang.xlf, the translated file for German (code "de") will be named de.locallang.xlf. Note that the original file must always be in english, so it is not allowed to create a file with the prefix "en" e.g. en.locallang.xlf. Inside the file itself, a <target-language> attribute is added in the <file> tag to indicate the translation language ("de" in our example). Then for each <source> tag there's a sibling <target> tag containing the translated string.

Here is what the translation of our sample file could look like:

<xliff version="1.0" xmlns="urn:oasis:names:tc:xliff:document:1.1">
   <file source-language="en" target-language="de" datatype="plaintext" original="messages" date="2011-10-18T18:20:51Z" product-name="my-ext">
      <header/>
      <body>
         <trans-unit id="headerComment" xml:space="preserve">
            <source>The default Header Comment.</source>
            <target>Der Standard-Header-Kommentar.</target>
         </trans-unit>
         <trans-unit id="generator" xml:space="preserve">
            <source>The "Generator" Meta Tag.</source>
            <target>Der "Generator"-Meta-Tag.</target>
         </trans-unit>
      </body>
   </file>
</xliff>

Only one language can be stored per file and each translation in a different language goes to an additional file.

File locations and naming

In the TYPO3 Core, XLIFF files are located in the various system extensions as needed and are expected to be located in Resources/Private/Language.

In Extbase, the main file (locallang.xlf) will be loaded automatically and available in the controller and Fluid views without further work needed. Other files will need to be referred to explicitly using the EXT:LLL:extkey/path/to/file:my.label syntax.

As mentioned above, the translation files follow the same naming conventions, but are prepended with the language code and a dot. They are stored alongside the default language files.

Site Handling

The site handling defines entry points to the frontend sites of a TYPO3 instance, their languages and routing details. This chapter walks through the features of the module and goes into API and programming details.

Basics

Note

Site Handling as described here is available since TYPO3 9 LTS.

TYPO3 Site Handling and Configuration is the starting point for creating new web sites. The corresponding modules are found in the TYPO3 backend in the section "Site Management".

A site configuration consists of the following parts:

  • Base URL configuration: Under which domain(s) is my site accessible
  • Language configuration: Which languages are available for my site
  • Error Handling: How should errors for this site behave (For example: configure custom 404 pages)
  • Static Routes: Add static routes to a site (For example for robots.txt on a per site base)
  • Routing Configuration: How shall routing behave for this site

Most parts of the site configuration can be edited via the graphical interface in the backend module "Site".

Site Module

The Site module in the TYPO3 backend.

Hint

While the editing mask for a site looks like a "normal" TYPO3 editing form, it is not. In contrast to other forms, site configuration is stored in the file system and not in database tables.

Site configuration storage

When creating a new site configuration, a folder in the file system is created located at <project-root>/config/sites/<identifier>/. The site configuration is stored in a file called config.yaml.

Note

If you are using a non-composer based installation, the location is typo3conf/sites/. In the future this folder can (and should) be used for more files like Fluid templates, and Backend layouts.

Hint

Add this folder to your version control.

The configuration file

The following part explains the configuration file and options:

rootPageId: 12
base: 'https://www.example.com/'
languages:
  -
    languageId: '0'
    title: English
    navigationTitle: ''
    base: /
    locale: en_US.UTF-8
    iso-639-1: en
    hreflang: en-US
    direction: ltr
    typo3Language: default
    flag: gb
  -
    languageId: '1'
    title: 'danish'
    navigationTitle: Dansk
    base: /da/
    locale: dk_DK.UTF-8
    iso-639-1: da
    hreflang: dk-DK
    direction: ltr
    typo3Language: default
    flag: dk
    fallbackType: strict
  -
    languageId: '2'
    title: Deutsch
    navigationTitle: ''
    base: 'https://www.beispiel.de/'
    locale: de_DE.UTF-8
    iso-639-1: de
    hreflang: de-DE
    direction: ltr
    typo3Language: de
    flag: de
    fallbackType: fallback
    fallbacks: '2,1,0'
errorHandling:
  -
    errorCode: '404'
    errorHandler: Page
    errorContentSource: 't3://page?uid=8'
  -
    errorCode: '403'
    errorHandler: Fluid
    errorFluidTemplate: 'EXT:my_extension/Resources/Private/Templates/ErrorPages/403.html'
    errorFluidTemplatesRootPath: 'EXT:my_extension/Resources/Private/Templates/ErrorPages'
    errorFluidLayoutsRootPath: 'EXT:my_extension/Resources/Private/Layouts/ErrorPages'
    errorFluidPartialsRootPath: 'EXT:my_extension/Resources/Private/Partials/ErrorPages'
  -
    errorCode: '0'
    errorHandler: PHP
    errorPhpClassFQCN: Vendor\ExtensionName\ErrorHandlers\GenericErrorhandler
routes:
  route: robots.txt
  type: staticText
  content: |
      Sitemap: https://example.com/sitemap.xml
      User-agent: *
      Allow: /
      Disallow: /forbidden/

Most settings can also be edited via the backend module Site Management > Configuration, exceptions being custom settings and additional routing configuration.

site identifier

The site identifier is the name of the folder within <project-root>/config/sites/ that will hold your configuration file(s). When choosing an identifier make sure to stick to ASCII but you may also use -, _ and . for convenience.

rootPageId

Root pages are identified by one of these two properties:

  • they are direct descendants of PID 0 (the root root page of TYPO3)
  • they have the "Use as Root Page" property in pages set to true.
base

The base is the base domain to run a site on. It either accepts a fully qualified URL or a relative segment "/" to react to any domain name. It is possible to set a site base prefix to just "/site1" and "/site2" or "www.mydomain.com" instead of entering a full URI.

This allows to have a Site base e.g. www.mydomain.com to be detected with http and https protocols, although it is recommended to do a HTTP to HTTPS redirect either on the webserver level, via a .htaccess rewrite rule, or by adding a redirect in TYPO3.

Note

Please note that this flexibility will introduce side-effects when having multiple sites with mixed configuration settings as Site base:

  • Site 1: /mysite/
  • Site 2: www.mydomain.com

will be unspecific when detecting a URL like www.mydomain/mysite/ and can lead to side-effects.

In this case, it is necessary by the Site Administrator to define unique Site base prefixes.

languages

Available languages for a site can be specified here. These settings determine language avaialability as well as behavior. For a detailed description see Language configuration.

errorHandling

The error handling section describes how to handle error status codes for this web site. It allows configuration of custom redirects, rendering templates and more. For a detailed description see error handling.

routes

The routes section is for adding static routes to a site, an example would be a robots.txt or humans.txt file that is dependent on the current site (as opposed to containing the same content for the whole TYPO3 installation). Read more at static routes

routeEnhancers

While page routing works out of the box with no further settings, routeEnhancers allow the configuration of routing for TYPO3 extensions.

Creating a new site

A new site can be created for every page record that either is on rootLevel (pid = 0) or has `is_siteroot` flag set. So at least one page is needed in the page tree.

To create a new site configuration, go to the Site module at Site Management.

Create a new site

The Site module without any configured sites in the TYPO3 backend.

After pressing the "big blue button" an edit form is displayed:

Create a new site

A new site creation form.

First, enter an identifier at (1).

Hint

The site identifier is the name of the folder within <project-root>/config/sites/ that will hold your configuration file(s). When choosing an identifier make sure to stick to ASCII but you may also use -, _ and . for convenience. Examples: main-site and langing-page.

Then, enter a base for your site at (2).

Tip

Be as specific as you can for your sites without losing flexibility. So, if you have a choice between using https://www.example.org, www.example.org or /, then choose https://www.example.org.

This will make resolving pages more reliable as the chance for conflicts with other sites gets minimized.

On the next tab ("Languages") you are required to configure the default language settings for your site. These will determine the default behavior - setting direction and lang tags in frontend as well as locale settings.

Set default language settings

Set default language settings

All you are required to set here is just the title (1) of the default language and the used locale (which should be available on the server)(2) - but you should also check and correct all other settings, as they will automatically be used for features like hreflang tags or displaying language flags in the backend.

That's all that is required for a new site.

Tip

Did you know that just by having a site configuration you get page based routing out of the box? Neat, isn't it?

Learn more about configuring languages, error handling and routing in the corresponding chapters.

Base Variants

In Site Handling base variants represent different bases for you web site depending on a specified condition. For example your "live" base URL might be https://example.org but on your local machine you want https://example.test as a domain - that's when you add a variant.

Variants consist of two parts:

  • a base to use for this variant
  • a condition that decides when this variant shall be active

Conditions are based on Symfony Expression Language and allow flexible conditions. For example:

applicationContext == "Development"

would define a base variant to use in Development context.

Add a base variant

A configured base variant for development context.

Hint

For those coming from earlier TYPO3 versions: With site handling, you do not need sys_domain records anymore! :)

The following variables and functions are available in addition to the default symfony functionality:

Properties
typo3.version
Datatype
string
Description
The current TYPO3 version
Example
9.5.0
typo3.branch
Datatype
string
Description
The current TYPO3 branch
Example
9.5
typo3.devIpMask
Datatype
string
Description
The configured devIpMask taken from $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask']
Example
77.176.160.*
applicationContext
Datatype
string
Description
The current application context
Example
Development
Functions

All functions from TYPO3s DefaultFunctionProvider are available:

ip
Datatype
string
Description
Match an IP address, value or regex, wildcards possible. Special value: devIp for matching devIpMask.
Example
ip("77.176.160.*")
compatVersion
Datatype
string
Description
Match a TYPO3 version
Example
compatVersion("9.5.0"), compatVersion("9.4")
like
Datatype
string
Description
Comparison function to compare two strings. The first parameter is the "haystack", the second the "needle". Wildcards are allowed.
Example
like("foobarbaz", "*bar*")
env
Datatype
string
Description
Wrapper for PHPs getenv() function. Allows accessing environment variables.
Example
env("TYPO3_BASE_URL")

Adding languages

The site module allows you to define which languages are active for your site, which languages are available in the frontend and how they should behave.

For an explanation of each of the properties, see below.

When the backend shows the list of available languages - for instance in the page module language selector, when editing records and in the list module - the list of languages is now restricted to those defined by the site module. If there are for instance five language records in the system, but a site configures only three of them for a page tree, only those three are considered when rendering language drop downs.

The language management comes with an option to hide a language in the frontend while allowing it in the backend. This allows editors to start translating pages without them directly being live.

Note

In case no site configuration has been created for a tree, all language records are shown. In this case the Page TSconfig options mod.SHARED.defaultLanguageFlag, mod.SHARED.defaultLanguageLabel and mod.SHARED.disableLanguages settings are also considered - those are obsolete if a site configuration exists.

Language fallbacks can be configured for every language but the default one. A language fallback means that if content is not available in the current language, content of the fallback language will be displayed. This may include multiple fallback levels - for example "Modern Chinese" might fall back to "Chinese (Traditional)" which may then fallback to "English". All languages can be configured separately, so you can have different fallback chains and behavior for each language.

Tip

Used to older TYPO3 versions? The following TypoScript settings will be set based on config.yaml - you don't need to have them in your TypoScript template:

  • config.language
  • config.locale_all
  • config.htmlTag_dir
  • config.htmlTag_langKey
  • config.sys_language_uid
  • config.sys_language_mode
  • config.sys_language_isocode
  • config.sys_language_isocode_default
Configuration Properties
languageId
Datatype
integer
Description
The TYPO3 sys_language_uid (the uid of the language record on pid 0)
Example
1
title
Datatype
string
Description
The internal human-readable name for this language.
Example
English
base
Datatype
string / URL
Description
Language base. Accepts either a fully qualified URL or a path segment like "/en/".
Example
/uk/
locale
Datatype
string / locale
Description
The locale to use for this language (Is set during frontend rendering for example)
Example
en_UK
iso-639-1
Datatype
string
Description
Two-letter code for the language according to ISO-639 nomenclature (see https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)
Example
en
hreflang
Datatype
string
Description
Frontend language for hreflang and lang tags.
Example
en-UK
direction
Datatype
string
Description
Text direction for content in this language (left-to-right or right-to-left)
Example
ltr
typo3Language
Datatype
string
Description
Language Identifier to use in TYPO3 locallang XLIFF files
Example
en
flagIdentifier
Datatype
string
Description
Flag identifier. The flag is for example displayed in the backend page module.
Example
en
fallbackType
Datatype
string
Description
Language fallback mode - strict = no fallbacks, fallback = fallback to another language in case no translation exists
Example
strict
fallbacks
Datatype
list of language ids
Description
List of fallback languages. If non has a matching translation, a pageNotFound is thrown.
Example
0

Error Handling

Error handling can be configured on site level and is automatically dependent on the current site and language.

The configuration consists of two parts:

  • The HTTP Error Status Code that should be handled
  • The Error Handler Configuration

You can define one error handler per HTTP error code and add a generic one that serves all error pages.

Tip

No more trouble with translated 404 error pages. With the new site handling getting translated 404 is easy!

Error Handling

Add custom error handling.

Properties
errorCode
Datatype
int
Description
The HTTP (Error) Status Code to handle. The predefined list contains the most common errors, a free definition of other error codes is also possible. Special value 0 will take care of all errors.
Example
404
errorHandler
Datatype
string / enum
Description
Define how to handle these errors. May be Fluid for rendering a fluid template, page for fetching content from a page or PHP for a custom implementation.
Example
Fluid
errorFluidTemplate
Datatype
string
Description

Only if errorHandler == `fluid`: Path to fluid template file. Path may be

  • absolute
  • relative to site root
  • starting with EXT: for files from an extension
Example
EXT:sitepackage/Resources/Private/Templates/Error.html
errorFluidTemplatesRootPath
Datatype
string [optional]
Description
Only if errorHandler == `fluid`: Pathes to Fluid Templates, Partials and Layouts in case more flexibility is needed.
Example
EXT:sitepackage/Resources/Private/Templates/
errorFluidPartialsRootPath
Datatype
string [optional]
Description
Only if errorHandler == `fluid`: Pathes to Fluid Templates, Partials and Layouts in case more flexibility is needed.
Example
EXT:sitepackage/Resources/Private/Partials/
errorFluidLayoutsRootPath
Datatype
string [optional]
Description
Only if errorHandler == `fluid`: Pathes to Fluid Templates, Partials and Layouts in case more flexibility is needed.
Example
EXT:sitepackage/Resources/Private/Layouts/
errorContentSource
Datatype
string
Description
May be either an External URL or TYPO3 Page that will be fetched with curl and displayed in case of an error.
Example
t3://page?uid=123
errorPhpClassFQCN
Datatype
string
Description
Fully qualified class name of a custom error handler implementing PageErrorHandlerInterface.
Example
My\Site\Error\Handler

Writing a custom Page Error Handler

The error handling configuration for sites allows implementing a custom error handler if the existing options of rendering a fluid template or page are not enough. An example would be an error page that uses the requested page or its parameters to search for relevant content on the web site.

A custom error handler needs to implement the PageErrorHandlerInterface (\TYPO3\CMS\Core\Error\PageErrorHandler\PageErrorHandlerInterface). The interface specifies only one method: handlePageError(ServerRequestInterface $request, string $message, array $reasons = []): ResponseInterface

Let's take a closer look:

The method handlePageError get's three parameters:

  • $request: the current HTTP request - we can for example access query parameters and the request path via this object
  • $message: an error message string - for example Cannot connect to the configured database. or Page not found
  • $reasons: an arbitrary array of failure reasons - see for example \TYPO3\CMS\Frontend\Controller\TypoScriptFrontendController::getPageAccessFailureReasons

What you do with these variables is left to you, but you need to return a valid ResponseInterface response - most usually an HtmlResponse.

For an example implementation of the PageErrorHandlerInterface take a look at \TYPO3\CMS\Core\Error\PageErrorHandler\PageContentErrorHandler or \TYPO3\CMS\Core\Error\PageErrorHandler\FluidPageErrorHandler.

Static Routes

Static routes provide a way to create seemingly static content on a per site base. Take the following example: In a multi-site installation you want to have different robots.txt files for each site that should be reachable at /robots.txt on each site. You can now add a static route "robots.txt" to your site and define which content should be delivered.

Routes can be configured as top level files (as in the robots.txt case) but may also be configured to deeper route paths (my/deep/path/to/a/static/text for example). Matching is done on the full path but without any parameters.

Static routes can be configured via the user interface or directly in the yaml configuration. There are two options: deliver static text or resolve a TYPO3 URL.

Note

Static route resolving is implemented as a PSR-15 middleware. If the route path requested matches any one of the configured site routes, a response is directly generated and returned. This way there is minimal bootstrap code to be executed on a static route resolving request, mainly the site configuration needs to be loaded. Static routes cannot get parameters as the matching is done solely on the path level.

StaticText

The staticText option allows to deliver simple text content. The text can be added through a text field directly in the site configuration. This is suitable for files like robots.txt or humans.txt.

YAML Configuration Example:

route: robots.txt
type: staticText
content: |
  Sitemap: https://example.com/sitemap.xml
  User-agent: *
  Allow: /
  Disallow: /forbidden/
TYPO3 URL (t3://)

The type uri for TYPO3 URL provides the option to render either a file, page or url. Internally a request to the file or URL is done and its content delivered.

YAML Configuration Examples:

-
  route: sitemap.xml
  type: uri
  source: 't3://page?uid=1&type=1533906435'
-
  route: favicon.ico
  type: uri
  source: 't3://file?uid=77'

Using Environment Variables in Site Configuration

Environment Variables in site configuration allows setting placeholders for configuration options that get replaced by environment variables specific to the current environment.

The format for environment variables is %env(ENV_NAME)%. Environment variables may be used to replace complete values or parts of a value.

Example:

base: 'https://%env(BASE_DOMAIN)%/'

Note

TYPO3 does not provide a loader for .env files - you have to take care of loading them yourself. Common options include setting environment configuration via server configuration or using vlucas/phpdotenv.

Using Site Configuration in TypoScript

Site configuration can be accessed via the getText property in TypoScript.

Example:

page.10 = TEXT
page.10.data = site:base
page.10.wrap = This is your base URL: |

Where site is the keyword for accessing an aspect, and the following parts are the configuration key(s) to access.

data = site:customConfigKey.nested.value

Tip

Accessing site configuration is possible in TypoScript, which enables to store site specific configuration options in one central place (the site configuration) and allows usage of that configuration from different contexts. While this sounds similar to using TypoScript, with using site configuration this may also be used from backend or CLI context as long as the rootPageId of the site is known. To avoid duplicating configuration options, TypoScript can now access these properties, too.

Site configuration can also be used in TypoScript conditions.

Using site config in conditions

Site configuration may be used in all conditions that use Symfony Expression language Typo3ConditionFunctionsProvider - at the moment this means in EXT:form variants and TypoScript conditions.

Two objects are available: site and siteLanguage. With site you can access the properties of the top level site configuration. siteLanguage accesses the configuration of the current site language.

TypoScript Examples

The identifier of the site name is evaluated:

[site("identifier") == "someIdentifier"]
    page.30.value = foo
[global]

Property of the current site language is evaluated:

[siteLanguage("locale") == "de_CH.UTF-8"]
    page.40.value = bar
[global]
EXT:form Examples

Translate options via siteLanguage condition:

renderables:
-
    type: Page
    identifier: page-1
    label: DE
    renderingOptions:
    previousButtonLabel: 'zurück'
    nextButtonLabel: 'weiter'
    variants:
    -
        identifier: language-variant-1
        condition: 'siteLanguage("locale") == en_US.UTF-8'
        label: EN
        renderingOptions:
        previousButtonLabel: 'Previous step'
        nextButtonLabel: 'Next step'

CLI Tools for Site Handling

Two CLI commands are available:

  • site:list
  • site:show

The list command can be executed via typo3/sysext/core/bin/typo3 site:list and will list all configured sites with their configured Identifier, root page, base URL, languages, locales and a flag whether or not the site is enabled.

The show command can be executed via typo3/sysext/core/bin/typo3 site:show <identifier>. It needs an identifier of a configured site which must be provided after the command name. The command will output the complete configuration for the site in the YAML syntax.

PHP API: Accessing Site Configuration

The PHP API for Sites comes in two parts:

  • Accessing the current, resolved site object
  • Finding a site object / configuration via a page or identifier

The first case is relevant when we want to access site configuration in the current request, for example if we want to know which language is currently rendered.

The second case is about accessing site configuration options independent of the current request but based on a page id or a site identifier.

Let's look at both cases in detail:

Accessing the current site object

When rendering the frontend or backend TYPO3 builds a HTTP request object through a PSR-15 middleware stack and enriches that with information. Part of that information are the Site and SiteLanguage objects. Both objects are available as attributes on the current request object.

Depending on the context, there are two main ways to access them:

  • via the PSR-7 HTTP ServerRequest object directly - for example in a PSR-15 middleware or the admin panel
  • via $GLOBALS['TYPO3_REQUEST'] - everywhere you don't have a ServerRequest object

Hint

The first method is preferred if possible as the global access will be deprecated and removed in future versions.

Methods:

// current site
$site = $request->getAttribute('site');

// current site language
$siteLanguage = $request->getAttribute('siteLanguage');

Warning

The PSR-7 Request and the extbase request are different things. You cannot access the site configuration via the extbase request. When in extbase context use the global access - a better way will be introduced in future versions.

Finding a site object

When you need to access site configuration for a specific page ID or by identifier, you can use the SiteFinder (\TYPO3\CMS\Core\Site\SiteFinder).

The SiteFinder offers the following methods for finding a site:

  • getSiteByIdentifier(): returns site object for the specified identifier ("folder name")
  • getSiteByRootPageId(): returns site object for a specific root page (pid = 0 or is_siteroot set)
  • getSiteByPageId(): returns site object for a page (walks the root line to find the root page and returns the site configuration)
  • getAllSites(): returns all configured site objects

All methods for finding a specific site throw an exception if no site was found.

The Site Object

Now we know how to find a site object, but what can it do?

First of all, it gives us access to the site configuration options via

  • getConfiguration(): returns the complete configuration
  • getAttribute(): returns a specific configuration attribute (root level configuration only)

It additionally provides methods for accessing related objects (languages / errorHandling):

  • getErrorHandler(): returns a PageErrorHandler according to the site configuration
  • getAvailableLanguages(): returns languages available to a user (including access checks)
  • getLanguageById(): returns a site language object for a language id
  • ...

Take a look at the class to find out more: \TYPO3\CMS\Core\Site\Entity\Site

The SiteLanguage object

The SiteLanguage object is basically a simple model that represents the configuration options of the site regarding language as an object and provides getters for those properties.

See \TYPO3\CMS\Core\Site\Entity\SiteLanguage

Pages without Site Configuration

The site handling functionality has a counterpart for usages within PHP code where no site configuration can be found, which is named "Pseudo Site", a site without configuration.

For a pseudo-site it is not possible to determine all available languages (as they are only configured in TypoScript), or the proper labels for the default language (as this is done in PageTSconfig), however, a PseudoSite or Site object (both instances of "SiteInterface") is always attached to every Frontend or Backend request via a PSR-15 middleware.

Extension Developers can access a site and determine the base URL / Entry Point URL for a site, or access all available languages via the SiteInterface object, instead of querying sys_domain or sys_language respectively.

Extending Site Configuration

Adding custom / project specific options to site configuration

Site configuration is stored as yaml and provides per definition context independent configuration of a site. Especially when thinking about things like storage PIDs or general site specific settings, it makes sense to add them to the site configuration.

Note

In "the old days" these kind of options were commonly stored in TypoScript or TSConfig or LocalConfiguration, all three being in some ways a bit unfortunate - parsing TypoScript while on CLI or using TSConfig made for the backend in frontend was no fun.

Adding project configuration to site configuration is easy: The site entity will automatically provide the complete configuration via getConfiguration(), extending that means therefor "just add whatever you want to the yaml file". The GUI is built in a way that toplevel options unknown / not available in the form will be left alone and do not get overwritten when saving.

Example:

rootPageId: 1
base: https://example.com
myProject:
    recordStorage: 15

Access it via the API:

$site->getConfiguration()['myProject']['recordStorage']
Extending the form / GUI

Extending the GUI is a bit more tricky.

The backend module relies on FormEngine to render the edit interface. Since the form data is not stored in database records but in .yml files, a couple of details have been extended of the default FormEngine code.

The render configuration is stored in typo3/sysext/backend/Configuration/SiteConfiguration/ in a format syntactically identical to TCA. However, this is not loaded into $GLOBALS['TCA'] scope, and only a small subset of TCA features is supported.

Extending site configuration is experimental and may change any time.

In practice the configuration can be extended, but only with very simple fields like the basic config type input, and even for this one not all features are possible, for example the eval options are limited. The code throws exceptions or just ignores settings it does not support. While some of the limits may be relaxed a bit over time, many will be kept. The goal is to allow developers to extend the site configuration with a couple of simple things like an input field for a Google API key. However it is not possible to extend with complex TCA like inline relations, database driven select fields, Flex Form handling and similar.

The example below shows the experimental feature adding a field to site in an extensions file Configuration/SiteConfiguration/Overrides/sites.php. Note the helper methods of class TYPO3\CMS\core\Utility\ExtensionManagementUtility can not be used.

<?php
// Experimental example to add a new field to the site configuration

// Configure a new simple required input field to site
$GLOBALS['SiteConfiguration']['site']['columns']['myNewField'] = [
    'label' => 'A new custom field',
    'config' => [
        'type' => 'input',
        'eval' => 'required',
    ],
];
// And add it to showitem
$GLOBALS['SiteConfiguration']['site']['types']['0']['showitem'] = str_replace(
    'base,',
    'base, myNewField, ',
    $GLOBALS['SiteConfiguration']['site']['types']['0']['showitem']
);

The field will be shown in the edit form of the configuration module and it's value stored in the .yml file. Using the site object TYPO3\CMS\core\Site\Entity\Site, the value can be fetched using ->getConfiguration()['myNewField'].

Versioning and workspaces

TYPO3 CMS provides a feature called "workspaces", whereby changes can be made to the content of the web site without affecting the currently visible (live) version. Changes can be previewed and go through an approval process before publishing.

The technical background and a practical user guide to this feature are provided in the "workspaces" system extension manual.

All the information necessary for making any database table compatible with workspaces is described in the TCA reference (in the description of the "ctrl" section and in the description of the "versioningWS" property).

The concept of workspaces needs attention from extension programmers. The implementation of workspaces is however made so that no critical problems can appear with old extensions;

  • First of all the "Live workspace" is no different from how TYPO3 has been working for years so that will be supported out of the box (except placeholder records must be filtered out in the frontend with t3ver_state != , see below).
  • Secondly, all permission related issues are implemented in DataHandler so the worst your users can experience is an error message.

However, you probably want to update your extension so that in the backend the current workspace is reflected in the records shown and the preview of content in the frontend works as well. Therefore this chapter has been written with instructions and insight into the issues you are facing.

Frontend challenges in general

For the frontend the challenges are mostly related to creating correct previews of content in workspaces. For most extensions this will work transparently as long as they use the API functions in TYPO3 to request records from the system.

The most basic form of a preview is when a live record is selected and you lookup a future version of that record belonging to the current workspace of the logged in backend user. This is very easy as long as a record is selected based on its "uid" or "pid" fields which are not subject to versioning; You simply call sys_page->versionOL() after record selection.

However, when other fields are involved in the where clause it gets dirty. This happens all the time! For instance, all records displayed in the frontend must be selected with respect to "enableFields" configuration! What if the future version is hidden and the live version is not? Since the live version is selected first (not hidden) and then overlaid with the content of the future version (hidden) the effect of the hidden field we wanted to preview is lost unless we also check the overlaid record for its hidden field (->versionOL() actually does this). But what about the opposite; if the live record was hidden and the future version not? Since the live version is never selected the future version will never have a chance to display itself! So we must first select the live records with no regard to the hidden state, then overlay the future version and eventually check if it is hidden and if so exclude it. The same problem applies to all other "enableFields", future versions with "delete" flags and current versions which are invisible placeholders for future records. Anyway, all that is handled by the \TYPO3\CMS\Frontend\Page\PageRepository class which includes functions for "enableFields" and "deleted" so it will work out of the box for you. But as soon as you do selection based on other fields like email, username, alias etc. it will fail.

Summary

Challenge: How to preview elements which are disabled by "enableFields" in the live version but not necessarily in the offline version. Also, how to filter out new live records with t3ver_state set to 1 (placeholder for new elements) but only when not previewed.

Solution: Disable check for enableFields/where_del_hidden on live records and check for them in versionOL on input record.

Frontend implementation guidelines

  • Any place where enableFields() are not used for selecting in the frontend you must at least check that t3ver_state != 1 so placeholders for new records are not displayed.
  • Make sure never to select any record with pid = -1! (offline records - related to versioning).
  • If you need to detect preview mode for versioning and workspaces you can read this variable:
    • $GLOBALS['TSFE']->sys_page->versioningWorkspaceId: Will tell you the id of the workspace of the current backend user. Used for preview of workspaces.
  • Use these API functions for support of version previews in the frontend: