Feature: #101553 - Auto-create DB fields from TCA columns

Description

The TYPO3 v13 Core strives to auto-create database columns derived from TCA columns definitions without explicitly declaring them in ext_tables.sql.

Creating "management" fields like uid, pid automatically derived from TCA ctrl settings is available for a couple of Core versions already, the Core now extends this to single TCA columns.

As a goal, extension developers should not need to maintain a ext_tables.sql definition for casual table columns anymore, the file can vanish from extensions and the Core takes care of creating fields with sensible defaults.

Of course, it is still possible for extension authors to override single definitions in ext_tables.sql files in case they feel the Core does not define them in a way the extension author wants: Explicit definition in ext_tables.sql always take precedence over auto-magic.

New TCA config option `dbFieldLength`

For fields of type select a new TCA config option dbFieldLength has been introduced. It contains an integer value that is applied to varchar fields (not text) and defines the length of the database field. It will not be respected for fields that resolve to an integer type. Developers who wish to optimize field length can use dbFieldLength for type=select fields to increase or decrease the default length the Core comes up with.

Example:

// will result in SQL text field
'config' => [
    'itemsProcFunc => 'something',
],

// will result in SQL varchar field length for 200 characters
'config' => [
    'itemsProcFunc => 'something',
    'dbFieldLength' => 200,
],

Impact

Extension authors should start removing single column definitions from ext_tables.sql for extensions being compatible with TYPO3 v13 and up.

If all goes well, the database analyzer will not show any changes since the Core definition is identical to what has been defined in ext_tables.sql before.

In various cases though, the responsible class DefaultTcaSchema may come to different conclusions than the extension author. Those cases should be reviewed by extension authors one-by-one: Most often, the Core declares a more restricted field, which is often fine. In some cases though, the extension author may know the particular field definition better than the Core default, and may decide to keep the field definition within ext_tables.sql.

Columns are auto-created for these TCA columns types:

type = 'category' - Core v12 already
type = 'datetime' - Core v12 already
type = 'slug' - Core v12 already
type = 'json' - Core v12 already
type = 'uuid' - Core v12 already
type = 'file' - new with Core v13
type = 'email' - new with Core v13
type = 'check' - new with Core v13
type = 'folder' - new with Core v13
type = 'imageManipulation' - new with Core v13
type = 'language' - new with Core v13
type = 'group' - new with Core v13
type = 'flex' - new with Core v13
type = 'text' - new with Core v13
type = 'password' - new with Core v13
type = 'color' - new with Core v13
type = 'radio' - new with Core v13
type = 'link' - new with Core v13
type = 'inline' - new with Core v13
type = 'number' - new with Core v13
type = 'select' - new with Core v13
type = 'input' - new with Core v13

See Breaking: DateTime column definitions for a change in the datetime column definition calculation.

Also see Important: About database error "row size too large" for limits imposed by MySQL / MariaDB on table length.

Migration of `NULL` to `NOT NULL` definitions, data truncation

As mentioned, the automatic database schema migration is based on TCA configuration, and will also take the nullable TCA definition of a field into consideration.

This can lead to scenarios in which a field (from both the TYPO3 Core or third party extension table definitions) will be converted in both type and attributes, and where data conversion might lead to error message like:

MySQL/MariaDB error message

Error: Data truncated for column 'image' at row 1

This can happen if previously a field was defined via ext_tables.sql, and then the definition was removed so that the TCA automatism could take over, but the definition mismatches the TCA definition (which might have changed as well).

This can best be showcased with the following example:

Previous EXT:frontend/ext_tables.sql definition from TYPO3 v12

CREATE TABLE fe_users (
  # ...
 image tinytext,
 # ...
)

Note

Note the absence of a NOT NULL and DEFAULT definition for the column image, which previously allowed NULL values to be stored inside a field.

With the TCA definition for the column fe_users.image set to type=file, the TYPO3 schema migration will decide to set this field to:

New automatically deduced SQL definition since TYPO3 v13

CREATE TABLE fe_users (
  # ...
  image INT UNSIGNED DEFAULT 0 NOT NULL,
  # ...
)

Then, this executed SQL statement:

SQL statement as executed by the Database Compare tool on MySQL/MariaDB

ALTER TABLE `fe_users` CHANGE `image` `image` INT UNSIGNED DEFAULT 0 NOT NULL

would lead to the error mentioned above, because any row that currently contains a NULL value would no longer be allowed. The solution for this is to fix these records before the schema migration is executed, by setting all currently existing NULL values to the new schema's DEFAULT value (here: 0).

This solution is provided by the TYPO3 Core via the migration wizard Migrate NULL field values to DEFAULT values.

The wizard looks for all existing records of a table where a schema conversion of NULL to NOT NULL would take place, iterates all rows of the table, and applies the default like this:

        SQL command to fix database records NULL/NOT NULL state as executed by the upgrade wizard
    

UPDATE `fe_users` SET `image` = '0' WHERE `image` IS NULL;

Hint

Bottom line: When a definition in an extension's ext_table.sql is removed, so that the DB fields are auto-created from TCA definitions, make sure that:

the TCA definition for nullable is properly set and
existing record values must fit into the new definition.

Otherwise, executing the migration wizard or custom data migration might be needed to prevent data truncation.

This may not only affect NOT NULL/NULL definitions, but also scenarios where data types are changed from, for example, TEXT to smaller VARCHAR columns. It is fine to keep ext_tables.sql definitions in place to adjust to special needs.