Getting help 

Report a problem or ask a question in the issue tracker of the repository. An import that refuses to run says why and exits with a code of its own - quote the message and the code, and the data-factory:import <identifier> --dry-run -v output if you have it. The exit codes are listed under data-factory:import.

Contributions are welcome: the development setup, the quality gates and the commit message rules are in CONTRIBUTING.md of the source repository.

What does it do? 

The Data Factory extension rebuilds the content of a TYPO3 installation from YAML files that ship inside extensions: pages, content elements, records of any table, the relations between them, files, the references attaching those files to records, and site configurations.

That makes the starting state of an installation part of a repository instead of something that is clicked together by hand - for a project template, for a demo or reference installation, for the fixture data of a test or staging instance, and for the "empty" instance a new developer starts from.

Two console commands are all there is to it:

What a seed set is 

A seed set is a directory Configuration/DataFactory/<name>/ inside any active extension, with a config.yml as its entry file. The descriptor says what the set is and which files it is written from:

identifier: demo
title: 'Demo page tree'

scenarios:
  - Scenario.yaml

sites:
  - identifier: main
    rootPage: 1000

The records live in the scenario files it names, written in the YAML scenario format of typo3/testing-framework - the format the TYPO3 Core writes its own functional test fixtures in:

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid'}
    defaultValues: {pid: 0}
  page:
    isNode: true
    tableName: 'pages'
    parentColumnName: 'pid'
    defaultValues: {hidden: 0, doktype: 1}
  content:
    tableName: 'tt_content'
    columnNames: {title: 'header', type: 'CType'}
    defaultValues: {hidden: 0, colPos: 0}

entities:
  page:
    - self: {id: 1000, title: 'Demo', slug: '/', is_siteroot: 1}
      entities:
        content:
          - self: {id: 2000, title: 'A frontend to look at', type: 'header'}
      children:
        - self: {id: 1100, title: 'About', slug: '/about'}

An extension therefore carries the data it needs with it, and nothing has to be registered anywhere: the set is available as soon as the extension is installed and activated. The complete format is described in Configuration.

A relation between records needs nothing beyond that. Because every record has a uid before it is written, a parent names its children by writing their declared ids into its relation field, and DataHandler resolves the list like any other relation - see Relations between records. A file is the exception, and the one place where config.yml has to say something: the references of a set attach a seeded file to a seeded record, because a sys_file_reference points at its file by a uid the FAL indexer only hands out while the file is being placed.

Everything the extension writes goes through the TYPO3 DataHandler and the file storage API rather than through direct database inserts. Slugs are generated, TCA defaults and evaluations are applied, the sorting is computed, relations are resolved, the reference index is updated, the caches are flushed and a copied file is indexed - the result is what an editor entering the same content would have produced, not rows that merely look like it.

Reproducible uids 

Every record of a seed set has a uid before it is written: the id its scenario declares, or one handed out from 10000 upwards. That is what makes a seeded page tree the same page tree in every installation, so a site configuration, a TypoScript condition or a test may refer to a page by its number.

Because TYPO3 treats such a uid as a suggestion rather than as a demand, an import refuses to run when one of the suggested uids is already used in its table, and it names the records that are in the way. A deleted record occupies its uid as much as any other one.

What it does not do 

• Seeding writes, it does not synchronise. An existing page tree is never reconciled against a definition, and no import is idempotent.
• Nothing is deleted or overwritten. A uid collision and an existing site identifier are both refusals.
• A file reference reaches only the records of its own set. The record a references entry names is the one the same run writes; a file cannot be attached to a record that is already in the installation.
• A file reference is declared in config.yml, not in a scenario file. The scenario format has no concept of a file and does not gain one here, because a sys_file_reference points at its file by a uid the FAL indexer hands out while the file is being placed.
• Backend users written by a set have to declare username and password themselves, because the import mode that suppresses the automatic site configuration also suppresses the generated credentials.

The two boundaries of the format - how far a file reference reaches, and why it cannot live in a scenario file - are stated again where they are relevant, in What this version does not do.

Compatibility 

Branch 1 is the 1.x line and the one this documentation belongs to: TYPO3 v12.4 and v13.4. It carries one row per TYPO3 version, because the PHP ranges differ - PHP 8.1 is supported for TYPO3 v12 only, as typo3/cms-core 13.4 requires PHP ^8.2 and a v13 dependency set on PHP 8.1 cannot be installed at all. The lowest supported TYPO3 v12 patch level is 12.4.22.

Branch main is the 2.x line, on TYPO3 v13.4 and v14.3. One row is enough there, because both of its core versions share the same PHP range.

Both lines are released. main is the active line and receives features and fixes; branch 1 is maintained for installations still on TYPO3 v12.4 and receives fixes. TYPO3 v13.4 is served by both, so an installation on v13.4 can move between the lines without changing a seed set.

One code base serves both supported TYPO3 versions. Where an implementation has to differ, the classes are split per core version - below Core12/ and Core13/ - and the dependency injection container registers only the directory matching the running installation. Three implementations are split that way on this line: the YAML loader that reads a descriptor and a site template, the call that places a seeded file into a file storage, and the writing of a site configuration. None of it is visible in a seed set, or to anything calling the commands.

Stability 

The supported interface of this extension is the scenario format, config.yml and the two console commands with their options and exit codes. A change to it that is not backwards compatible goes into a new major version and carries a Breaking-*.rst entry in the Changelog.

Everything below Classes/, Core12/ and Core13/ is @internal. It is the implementation of that interface, it carries no compatibility promise, and it may change in any release - a seed set never touches it.

Composer mode 

Seeding demo, fixture and development data is usually a development concern, so the extension normally belongs in require-dev of the project whose data is rebuilt:

composer require --dev sbuerk/data-factory

Require it normally when a project provisions itself from a seed set — the commands then exist in the deployed installation as well:

composer require sbuerk/data-factory

Either command resolves to the line matching the installed core, because each line constrains typo3/cms-core itself - 1.x on TYPO3 v12.4, 2.x on v13.4 and v14.3. TYPO3 v13.4 is served by both, and the 2.x line wins there unless this one is pinned - which is what a project on v13.4 staying on the 1.x line writes down; the line keeps the set descriptor, the scenario format and the two commands compatible:

composer require --dev "sbuerk/data-factory:^1.0"

Classic mode 

1. Get it from the Extension Manager: Switch to the module Admin Tools > Extensions, switch to Get Extensions and search for the extension key data_factory, then import the extension from the repository.
2. Get it from typo3.org: You can always get the current version from TER by downloading the zip version. Upload the file afterwards in the Extension Manager.
3. Get it from the repository: Every tag also publishes data_factory_<version>.zip as a GitHub release, which is the same artefact the TER receives. Upload it in the Extension Manager.

After the installation 

The extension needs no configuration of its own: it has no extension configuration, no TypoScript and no backend module. Activating it adds the two console commands, and nothing else in the installation changes until one of them is run.

vendor/bin/typo3 data-factory:list

An installation whose extensions provide no seed set answers with No active extension provides a seed set. and exits successfully — that is the normal state of most installations. Writing a seed set is described in Configuration.

Requirements for an import 

Two conditions have to be met for data-factory:import, and both are refusals with an explanation rather than silent failures:

• It runs on the command line, as an administrator. The TYPO3 console application authenticates the _cli_ backend user, which is an administrator by default. TYPO3 honours a declared uid only for an administrator and ignores it silently otherwise, which would write a page tree with different uids than the set declares.
• A file storage has to exist when the set brings files. A new installation gets its fileadmin/ storage from typo3 setup; a set may also name the storage to write into.

See also 

• A complete seed set — a whole set, with what importing it writes
• Configuration — the seed definition format, key by key
• The commands — every option and every exit code
• Changelog

Where a seed set lives 

A seed set is a directory Configuration/DataFactory/<name>/ inside any active extension, with config.yml as its entry file:

packages/my_extension/Configuration/DataFactory/demo/
├── config.yml               entry file, and the only mandatory one
├── Scenario/
│   ├── Pages.yaml           the records, named by "scenarios"
│   └── Content.yaml
├── Files.yaml               optional, pulled in through "imports"
├── Files/
│   └── placeholder.svg      resources named by "files"
└── Sites/
    └── main/
        ├── config.yaml      site configuration template
        └── settings.yaml    optional site settings

Every relative path inside a set - a scenarios entry, a file source, a site template - is resolved against the directory holding the entry file, not against the file declaring it. A set can therefore be moved or renamed without touching a single path inside it. EXT: paths are accepted everywhere a path is, and an absolute path is taken as it stands.

An imports resource is the one exception: it never reaches this extension, it is resolved by the TYPO3 YAML loader relative to the file that declares it - see Splitting a set over several files.

A set is found because the extension providing it is installed and activated. There is no path to configure and nothing to register. A directory below Configuration/DataFactory/ without a config.yml is not a set and is passed over, which is what lets a set keep partials next to itself.

The set descriptor 

identifier: demo
title: 'Demo page tree'
description: 'Pages, content elements, a file and the site they are reachable through.'

imports:
  - { resource: Files.yaml }

scenarios:
  - Scenario/Pages.yaml
  - Scenario/Content.yaml

files:
  - identifier: placeholder
    source: 'Files/placeholder.svg'
    folder: 'demo'

references:
  - file: placeholder
    table: tt_content
    uid: 2000
    field: assets

sites:
  - identifier: main
    rootPage: 1000
    template: 'Sites/main'

This list is closed: any other key at this level is refused, naming the known ones. That is deliberate - scenario: instead of scenarios: would otherwise be an import that reports success and writes nothing.

The descriptor carries no entitySettings and no entities. One rule, no ambiguity: config.yml describes the set, a scenario file describes records. Nothing this extension invents is mixed into a scenario file, and nothing a scenario file declares has to be understood by the descriptor.

The three metadata keys have to be declared in config.yml itself. They cannot be pulled in through imports, because listing the sets of an installation reads them without following imports.

The scenario format 

A scenario file is a map of at most three keys. Anything else at the top level is refused, naming the file:

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid', language: 'sys_language_uid'}
    defaultValues: {pid: 0}
  page:
    isNode: true
    tableName: 'pages'
    parentColumnName: 'pid'
    languageColumnNames: ['l10n_parent', 'l10n_source']
    columnNames: {type: 'doktype', root: 'is_siteroot'}
    defaultValues: {hidden: 0, doktype: 1}
    valueInstructions:
      shortcut:
        first: {shortcut: 0, shortcut_mode: 1}
  content:
    tableName: 'tt_content'
    languageColumnNames: ['l18n_parent', 'l10n_source']
    columnNames: {title: 'header', type: 'CType'}
    defaultValues: {hidden: 0, CType: 'text', colPos: 0}
  category:
    tableName: 'sys_category'

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid'}
  category:
    tableName: 'sys_category'

page:
  valueInstructions:
    shortcut:
      first: {shortcut: 0, shortcut_mode: 1}

entities:
  page:
    - self: {id: 1000, title: 'Demo', root: true, slug: '/'}
      entities:
        content:
          - self: {id: 2000, title: 'A frontend to look at', type: 'header'}
          - self: {id: 2001, bodytext: '<p>Seeded, not clicked together.</p>'}
      children:
        - self: {id: 1100, title: 'About', slug: '/about'}
          entities:
            content:
              - self: {id: 2100, title: 'About us'}
        - self: {id: 1200, title: 'Contact', slug: '/contact'}
    - self: {id: 1900, title: 'Storage', type: 254, slug: '/storage'}
      entities:
        category:
          - self: {id: 3000, title: 'News'}

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid', language: 'sys_language_uid'}
    defaultValues: {pid: 0}
  page:
    isNode: true
    tableName: 'pages'
    parentColumnName: 'pid'
    languageColumnNames: ['l10n_parent', 'l10n_source']
    columnNames: {type: 'doktype', root: 'is_siteroot'}
    defaultValues: {hidden: 0, doktype: 1}
  content:
    tableName: 'tt_content'
    languageColumnNames: ['l18n_parent', 'l10n_source']
    columnNames: {title: 'header', type: 'CType'}
    defaultValues: {hidden: 0, CType: 'text', colPos: 0}

entities:
  page:
    - self: {id: 1000, title: 'EN: Demo', root: true, slug: '/'}
      children:
        - self: {id: 1100, title: 'EN: About', slug: '/about'}
          languageVariants:
            - self: {id: 1101, title: 'DE: Über uns', language: 1, slug: '/ueber-uns'}
            - self: {id: 1102, title: 'FR: À propos', language: 2, slug: '/a-propos'}
          entities:
            content:
              - self: {id: 2100, title: 'EN: About us'}
                languageVariants:
                  - self: {id: 2101, title: 'DE: Über uns', language: 1}

entitySettings:
  workspace:
    tableName: 'sys_workspace'

entities:
  workspace:
    - self: {id: 1, title: 'Draft'}
  page:
    - self: {id: 1000, title: 'Demo', root: true, slug: '/'}
      children:
        - version: {id: 1300, title: 'EN: Coming soon', slug: '/soon', workspace: 1}

entities:
  page:
    - self: {id: 1100, title: 'EN: About', slug: '/about'}
      versionVariants:
        - version: {title: 'EN: About us, revised', workspace: 1}

entities:
  page:
    - self: {id: 1000, title: 'Demo', root: true, slug: '/'}
      children:
        - self: {id: 1100, title: 'About', slug: '/about'}
        - self: {id: 1200, title: 'Archive', slug: '/archive'}
          versionVariants:
            # In workspace 1 the page was moved below "About".
            - version: {workspace: 1}
              actions:
                - {action: 'move', type: 'toPage', target: 1100}
        - self: {id: 1300, title: 'Draft only', slug: '/draft'}
          versionVariants:
            # In workspace 1 the page was deleted.
            - version: {workspace: 1}
              actions:
                - {action: 'delete'}

Declared uids, and what makes a seed reproducible 

Every record of a scenario has a uid before it is written. Either the item declares one as id, or the seeder hands out a dynamic one from 10000 upwards. Both are suggested to DataHandler, so a seeded page tree is the same page tree in every installation - a site configuration, a TypoScript condition or a test may refer to a page by its number.

Declare the uid of everything that is referred to from outside the set: a site root, a shortcut target, a storage folder. Let the rest be dynamic.

TYPO3 treats a suggested uid as a suggestion. If the uid is already used in its table, the record is not written elsewhere - the insert fails. The import therefore checks every suggested uid up front and refuses when one is taken, naming the records in the way:

[ERROR] The seed set "demo" suggests 2 uids this installation already uses.

 ---------- ----- -------------------
  Table      Uid   Occupied by
 ---------- ----- -------------------
  pages      1000  Company site
  pages      1100  Products
 ---------- ----- -------------------

Nothing is written in that case. --force imports anyway: every record of a table something collides in is written with a free uid instead of the suggested one, a table nothing collides in keeps its uids, and nothing that is in the way is deleted or changed.

A deleted record occupies its uid as much as any other one - the row is still there.

Suggested uids are honoured for an administrator backend user only. DataHandler ignores them silently for anybody else, so the import refuses to run as a non-admin rather than write a set whose uids are not the ones it declares.

Relations between records 

A relation needs no key of its own, in config.yml or anywhere else. Because every record has a uid before it is written, the record holding the relation can name the records on the other end of it by writing their uids into its relation field, exactly as a backend form submits them:

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid'}
    defaultValues: {pid: 0, hidden: 0}
  page:
    isNode: true
    tableName: 'pages'
    parentColumnName: 'pid'
    defaultValues: {doktype: 1}
  content:
    tableName: 'tt_content'
    columnNames: {title: 'header', type: 'CType'}
  item:
    tableName: 'tx_myext_item'

entities:
  page:
    - self: {id: 1, title: 'Root', slug: '/', is_siteroot: 1}
      entities:
        content:
          - self: {id: 21, title: 'List', type: 'my_itemlist', tx_myext_items: '32,31'}
        item:
          - self: {id: 31, title: 'One'}
          - self: {id: 32, title: 'Two'}

That is the whole declaration. The relation itself is described where it is always described, in the TCA of the field:

'tx_myext_items' => [
    'config' => [
        'type' => 'inline',
        'foreign_table' => 'tx_myext_item',
        'foreign_field' => 'parentid',
        'foreign_table_field' => 'parenttable',
        'foreign_sortby' => 'sorting_foreign',
    ],
],

The children are an entity of their own in the same scenario, and the parent declares its relation field as the comma separated list of the ids they declare. Nothing else is needed: DataHandler resolves that list like any other relation and writes the columns tying a child to its parent by itself. Which columns those are is read from the TCA of the parent field - its foreign_field, foreign_table_field and foreign_sortby, which are parentid, parenttable and sorting_foreign above. A scenario therefore never names them, and a relation whose TCA calls them something else needs nothing different here. The relation field of the parent ends up holding the number of children, the way TYPO3 stores an inline relation.

Two consequences are worth stating, because they are what an integrator will actually rely on:

• The order of the relation is the order of the declared list, not the order of the uids. The example writes '32,31', and item 32 comes first.
• It works more than one level deep. A child that is itself the parent of a relation declares its own list the same way, and the second level is resolved from the TCA of the child's field. Nothing about it is special cased.

The children live on the page the scenario declares them under, like every other record: the relation ties them to the parent record, not to its page.

Files 

files:
  - identifier: placeholder             # required, unique among the files
    source: 'Files/placeholder.svg'     # required; relative to the set, or EXT:
    folder: 'demo'                      # optional, default the storage root
    name: 'placeholder.svg'             # optional, default the source basename
    storage: 1                          # optional, default the default storage

This list is closed as well: any other key on a file is refused, naming the known ones - foldr: instead of folder: would otherwise put the file in the storage root and report success.

Files are copied before the records are written, through the file storage API, which is what indexes them - a file copied into fileadmin/ by hand exists on disk and does not exist for TYPO3. A missing target folder is created, an existing file of the same name is replaced, and the source file in the extension is left where it is.

Placing a file is one thing and attaching it to a record is another, and the second is what references below does.

File references 

references:
  - file: placeholder                 # required, an identifier declared under "files"
    table: tt_content                 # required, the table of the record it hangs on
    uid: 2000                         # required, the uid the scenario declares as "id"
    field: assets                     # required, the file relation column of that record
    values:                           # optional, the fields of the sys_file_reference row
      title: 'A placeholder'
      alternative: 'Nothing to see here'

This list is closed as well: any other key on a reference is refused, naming the known ones.

A scenario record carries no symbolic name, so a reference names its record by uid - the same rule rootPage follows, and for the same reason. That uid is resolved against what the run actually wrote rather than trusted. A reference naming a record no entity of the scenario declares is refused before anything is written:

[ERROR] The seed set "demo" declares a file reference to "placeholder" on the
        record tt_content:2001, which no entity of its scenario declares as
        its "id".

Five columns of the sys_file_reference row are structural and are written by the seeder: uid_local, uid_foreign, tablenames, fieldname and pid. Declaring one of them under values does not change it - the seeder's value wins, because a definition may not detach a reference from the record it declares it on. pid follows the convention TYPO3 uses for a file relation: a reference belongs to the page its record is on, and for a record that is a page, to that page itself.

Several references on one field come out in the order they are declared, which is what sorting_foreign is written from - so a multi image field is a gallery in the declared order rather than in whatever order the database returns. The same file may be referenced from several records, or from none.

References are written in a pass of their own, through DataHandler, after every record of the set exists. That is what puts the relation into the reference index, and it is why a mistyped uid is caught up front rather than after the whole tree has been written.

identifier: demo
title: 'A content element with two images'

scenarios:
  - Scenario.yaml

files:
  - identifier: landscape
    source: 'Files/landscape.jpg'
    folder: 'demo'
  - identifier: portrait
    source: 'Files/portrait.jpg'
    folder: 'demo'

references:
  - file: landscape
    table: tt_content
    uid: 2000
    field: assets
    values: {title: 'Landscape', alternative: 'The wide one'}
  - file: portrait
    table: tt_content
    uid: 2000
    field: assets
  - file: landscape
    table: pages
    uid: 1000
    field: media

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid'}
    defaultValues: {pid: 0, hidden: 0}
  page:
    isNode: true
    tableName: 'pages'
    parentColumnName: 'pid'
    defaultValues: {doktype: 1}
  content:
    tableName: 'tt_content'
    columnNames: {title: 'header', type: 'CType'}

entities:
  page:
    - self: {id: 1000, title: 'Demo', slug: '/', is_siteroot: 1}
      entities:
        content:
          - self: {id: 2000, title: 'Teaser', type: 'textmedia'}

Site configurations 

sites:
  - identifier: main                    # required, the directory in config/sites/
    rootPage: 1000                      # required, the uid of a seeded page
    template: 'Sites/main'              # optional, default Sites/<identifier>
    base: 'https://example.com/'        # optional, overrides the template

A site configuration is written from a template: a directory holding a config.yaml and optionally a settings.yaml, which is exactly the shape of a site below config/sites/. A template is therefore produced by copying a working site out of an installation.

This list is closed too: a site is configuration rather than a record, so nothing on it is written verbatim and an unknown key can only be a mistake.

A scenario record carries no symbolic name, so its stable handle is the uid it declares - which is why rootPage is a number here and not a name. A site naming a uid that no pages entity of the scenario declares is refused before anything is written, rather than after the whole tree exists.

Three rules apply to the result:

• The root page always wins. rootPageId is taken from the uid the declared page was actually written with, whatever the template says.
• An existing site identifier is refused. TYPO3 merges an incoming site configuration into an existing one, so seeding over it would produce neither the template nor the previous configuration but a mixture of both. Remove the site first if the seed is meant to replace it.
• A minimal template needs almost nothing. base and languages are worth declaring, because their defaults are a site on / in "Default / en_US.UTF-8". dependencies - the site sets a site pulls in - is written unchanged, and is a TYPO3 v13 key: TYPO3 v12 has no site sets, so a template declaring it seeds a configuration whose dependencies v12 simply ignores.

Placeholders such as %env(...)% inside a template are not resolved while seeding. They are written as they stand, so the installation resolves them every time it reads the file - which is what they are for.

A complete seed set 

Everything above, in one set. It is the set this extension's own functional tests import, and a test compares the three files printed here against the files on disk - so what is shown is what runs.

packages/my_extension/Configuration/DataFactory/complete/
├── config.yml
├── Scenario.yaml
├── Files/
│   └── placeholder.svg
└── Sites/
    └── main/
        └── config.yaml

identifier: import-documented
title: 'A complete seed set'
description: 'Pages with content and translations, a file, a file reference and a site.'

scenarios:
  - Scenario.yaml

files:
  - identifier: placeholder
    source: 'Files/placeholder.svg'
    folder: 'documented'

references:
  - file: placeholder
    table: pages
    uid: 1000
    field: media
    values:
      title: 'The placeholder'
      alternative: 'A grey rectangle'

sites:
  - identifier: documented-main
    rootPage: 1000
    template: Sites/main

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid', language: 'sys_language_uid'}
    defaultValues: {pid: 0, hidden: 0}
  page:
    isNode: true
    tableName: 'pages'
    parentColumnName: 'pid'
    languageColumnNames: ['l10n_parent', 'l10n_source']
    defaultValues: {doktype: 1}
  content:
    tableName: 'tt_content'
    languageColumnNames: ['l18n_parent', 'l10n_source']
    columnNames: {title: 'header', type: 'CType'}

entities:
  page:
    - self: {id: 1000, title: 'EN: Demo', slug: '/', is_siteroot: 1}
      children:
        - self: {id: 1100, title: 'EN: About', slug: '/about'}
          languageVariants:
            - self: {id: 1101, title: 'DE: Über uns', language: 1, slug: '/ueber-uns'}
          entities:
            content:
              - self: {id: 2100, title: 'EN: About us', type: 'header'}
                languageVariants:
                  - self: {id: 2101, title: 'DE: Über uns', language: 1, type: 'header'}

rootPageId: 0
base: 'https://documented.example.org/'
languages:
  -
    title: English
    enabled: true
    languageId: 0
    base: /
    locale: en_US.UTF-8
  -
    title: Deutsch
    enabled: true
    languageId: 1
    base: /de/
    locale: de_DE.UTF-8
errorHandling: []
routes: []

Importing it with vendor/bin/typo3 data-factory:import import-documented writes, in this order:

• the file placeholder.svg into fileadmin/documented/, indexed as a sys_file;
• the pages 1000, 1100 and its translation 1101, and the content elements 2100 and its translation 2101, each with the uid it declares;
• a sys_file_reference attaching the file to pages.media of page 1000, carrying the declared title and alternative;
• the site config/sites/documented-main/config.yaml, from the template, with rootPageId replaced by 1000.

Two details of the example are deliberate and worth copying. The entities declare nodeColumnName, which is what puts the translated page onto the page its original sits on rather than out of the tree, and hidden sits on the '*' entry while doktype sits on page - a key declared on both sides is merged into a list, not overridden.

Workspaces are the one construct the example leaves out. They need a sys_workspace record and have a section of their own; a set that seeds a workspace is not what a first set looks like.

Splitting a set over several files 

A set splits in two independent ways, and they are not interchangeable.

imports:
  - { resource: Files.yaml }
  - { resource: 'EXT:my_extension/Configuration/DataFactory/shared/Sites.yaml' }

scenarios:
  - Scenario/Pages.yaml
  - Scenario/Content.yaml
  - 'EXT:my_extension/Configuration/DataFactory/shared/Categories.yaml'

What this version does not do 

Two boundaries of the format, named here rather than left to be discovered:

• A file reference reaches only the records of its own set. The uid of a references entry is resolved against what the run writes, so it cannot attach a file to a record that is already in the installation. Naming one is refused as an undeclared record, not answered with a database lookup.
• A file reference cannot be declared in a scenario file. The scenario format has no concept of a file, and this extension does not give it one - that is the point of references living in config.yml. A scenario file of a seed set stays a file that could be lifted into a functional test unchanged.

The commands 

vendor/bin/typo3 data-factory:list
vendor/bin/typo3 data-factory:list -v

vendor/bin/typo3 data-factory:import demo
vendor/bin/typo3 data-factory:import demo --dry-run -v
vendor/bin/typo3 data-factory:import demo --base='https://example.com/'

What the seeder writes by itself 

Everything else in a record comes from its self or version, from the defaultValues of its entity, or from the TCA of the installation where neither declares anything. This extension adds no default of its own - in particular, a page is created hidden unless the scenario says otherwise, because pages is the one table whose TCA sets 'hidden' => ['config' => ['default' => 1]]. tt_content and the other tables of an installation default to 0. That is why every example above declares defaultValues: {hidden: 0}: it costs nothing on the tables that already default to 0, and it is what keeps a seeded page tree visible.

The same holds for doktype, l10n_parent and sys_language_uid - they come from the TCA of the installation unless the scenario declares them, so a set that wants the same records in two installations declares them.

See also 

• Introduction
• Installation
• Changelog

Breaking Changes 

Features 

Deprecation 

Important 

Description 

Two console commands make the seed sets of an installation visible and import one of them.

vendor/bin/typo3 data-factory:list
vendor/bin/typo3 data-factory:import demo --dry-run
vendor/bin/typo3 data-factory:import demo --base='https://example.com/'

data-factory:list prints identifier, title and providing extension of every seed set found in Configuration/DataFactory/*/config.yml of an active extension, in the order the installation loads its extensions in. -v adds the directory of each set. An identifier provided by more than one extension is reported with all of its providers and makes the command exit non-zero: the sets cannot be told apart, so neither listing nor importing may pick one of them.

data-factory:import writes one set. Without an identifier it asks which one; without a terminal to ask on it lists the available sets and exits non-zero rather than guessing. The options are:

Each kind of failure has its own exit code — unknown set, unresolvable set, invalid definition, uid collision, no administrator, failed write — so that a deployment script can tell "no such set" from "that would overwrite something" without parsing the output. The complete list is in the Configuration chapter.

Impact 

An installation can be provisioned from the command line, and a pipeline can act on what happened. The complete option and exit code reference is in the Configuration chapter.

Description 

A seed set brings the files its content needs, and copies them into a file storage before the first record is written:

files:
  - identifier: placeholder
    source: 'Files/placeholder.svg'
    folder: 'demo'
    name: 'placeholder.svg'
    storage: 1

identifier is unique among the files of the set, source is a path relative to the directory holding the set or an EXT: path, folder defaults to the storage root, name to the base name of the source, and storage to the default storage of the installation. Those five keys are the whole vocabulary of a file: an unknown key is refused, naming the known ones.

The copy goes through the file storage API rather than through the file system - which is what indexes the file, so the result exists for TYPO3 and not only on disk. A missing target folder is created, an existing file of the same name is replaced, and the source in the extension is left where it is.

Files are written in a pass of their own, before the records, so that a record naming a sys_file uid names one that exists.

Attaching a file to a record 

A seeded file is attached to a seeded record through references, the second key of config.yml this feature adds:

references:
  - file: placeholder
    table: tt_content
    uid: 2000
    field: assets
    values: {title: 'A placeholder', alternative: 'Nothing to see here'}

file names one of the files of the same set, table and uid name the record the reference hangs on, field is the file relation column of that record, and values are the fields of the sys_file_reference row itself - the ones an editor fills in on a file relation. Those five keys are the whole vocabulary of a reference; an unknown key is refused, naming the known ones.

A scenario record has no symbolic name, so uid is the id an entity of the scenario declares - the same rule rootPage follows - and it is resolved against what the run actually wrote. A reference naming a record no entity declares is refused before anything is written, rather than after the page tree, the content and the files are already in the database.

The structural columns of the row - uid_local, uid_foreign, tablenames, fieldname and pid - are written by the seeder and win over a value declared for them, because a definition may not detach a reference from the record it declares it on. Several references on one field come out in the order they are declared, and that order is what sorting_foreign is written from.

References are written through DataHandler in a pass of their own, after the records, so the relation reaches the reference index the way an editor's would.

Why it is a key of the descriptor rather than of a scenario 

Because the scenario format is not this extension's to extend, and because a sys_file_reference cannot be expressed in it anyway: it points at its file through uid_local, and that uid is handed out by the FAL indexer while the file is being placed. A set author cannot write it down in advance, so the reference is declared where the file is declared - in config.yml.

A relation between two records needs no key at all, and does not get one. The parent writes the declared ids of its children into its relation field, and DataHandler resolves that list like any other relation - see Relations between records.

Impact 

A seed set ships the images of the content it describes, and the content comes out with those images attached - instead of describing content that points at files someone has to upload and hook up first.

Description 

Initial skeleton of the sbuerk/data-factory extension, providing the project setup the actual implementation is built on:

• TYPO3 v12.4 and v13.4 support on PHP 8.1 up to 8.4 - PHP 8.1 for TYPO3 v12 only - with core version aware implementations below Core12/ and Core13/.
• Dependency injection wiring through Configuration/Services.php, with services configured by Symfony dependency injection attributes on the classes themselves.
• Container based tooling through Build/Scripts/runTests.sh covering linting, coding guidelines, static analysis, unit and functional tests and documentation rendering.
• GitHub Actions workflows running these gates for TYPO3 v12 and v13 on pull requests.
• A functional test setup ready to build on: strict PHPUnit configuration, an example fixture extension loaded by its composer package name, site based tests issuing frontend sub-requests in several languages, and repository tests running in a built frontend environment.
• Developer documentation below docs/, covering the architecture, the quality gates, both test suites and the release workflow.

Description 

A seed set describes the content of a TYPO3 installation in YAML. It is a directory Configuration/DataFactory/<name>/ inside any active extension, and it is made of two kinds of file.

config.yml describes the set:

identifier: demo
title: 'Demo page tree'
description: 'Pages, content elements and the site they are reachable through.'

scenarios:
  - Scenario.yaml

files:
  - identifier: placeholder
    source: 'Files/placeholder.svg'
    folder: 'demo'

references:
  - file: placeholder
    table: tt_content
    uid: 2000
    field: assets

sites:
  - identifier: main
    rootPage: 1000

Every key set of the descriptor is closed - at the top level, on a files entry, on a references entry and on a sites entry alike. An unknown key is refused, naming the known ones, because scenario: instead of scenarios: would otherwise be an import that reports success and writes nothing. identifier, title and scenarios are required.

The scenario files it names describe the records, in the YAML scenario format of typo3/testing-framework:

entitySettings:
  '*':
    nodeColumnName: 'pid'
    columnNames: {id: 'uid', language: 'sys_language_uid'}
    defaultValues: {pid: 0}
  page:
    isNode: true
    tableName: 'pages'
    parentColumnName: 'pid'
    languageColumnNames: ['l10n_parent', 'l10n_source']
    defaultValues: {hidden: 0, doktype: 1}
  content:
    tableName: 'tt_content'
    columnNames: {title: 'header', type: 'CType'}
    defaultValues: {hidden: 0, colPos: 0}

entities:
  page:
    - self: {id: 1000, title: 'Demo', slug: '/', is_siteroot: 1}
      entities:
        content:
          - self: {id: 2000, title: 'A frontend to look at', type: 'header'}
      children:
        - self: {id: 1100, title: 'About', slug: '/about'}
          languageVariants:
            - self: {id: 1101, title: 'DE: Über uns', language: 1, slug: '/ueber-uns'}

That format is not this extension's invention: it is the one the TYPO3 Core writes its own functional test fixtures in, and the scenario files shipped in the core are worked examples of it.

Properties of the format:

• entitySettings declares how a table is written - its name, its node and parent columns, its column aliases, its language columns, its default values and its value instructions. The entry '*' holds the defaults for the declared entities. An entity name is not a table name, so one table can be written under two names with different defaults.
• entities declares the records. Everything inside self: that is not id is a field and is written as it stands, so a table needs no support in this extension to be seedable.
• children: nests further records of the same entity through the parent column, and a nested entities: block puts records of other entities onto a node. Records come out in the order they are declared, whatever table they belong to.
• languageVariants: is a first-class translation construct: the columns named by languageColumnNames are filled with the uids of the ancestors of the variant. A translation of a translation is built with the full chain and reaches the database without it - a DataHandler limitation TYPO3 Core documents on its own fixtures, and one the format documentation names.
• versionVariants: writes a workspace version of a record, and version: in place of self: writes a record that only exists in a workspace.
• actions: becomes a DataHandler command - move, delete and discard - run after every record exists, so an action can name a record the same set creates.
• Every record has a uid before it is written: the declared id, or a dynamic one from 10000 upwards. Every one of them is checked against the installation up front, and an import that would collide is refused with the records that are in the way.
• scenarios may name several files. They are composed into one scenario: entitySettings merged with the later file winning, entities appended per entity name. One composed scenario rather than one per file, because the dynamic uids are handed out per scenario.
• imports splits the descriptor itself over several files, merging the imported lists into the importing one. A resource that cannot be read fails the import rather than being skipped.

Impact 

Extensions can ship the content an installation starts from, and that content is reviewable, versioned and reproducible like any other part of the repository - in a format that a TYPO3 developer has most likely already written by hand, for a functional test.

The complete reference is in the Configuration chapter.

Description 

A seed set writes the site configurations its page tree needs, from a template shipped next to the set:

sites:
  - identifier: main
    rootPage: 1000
    template: 'Sites/main'
    base: 'https://example.com/'

A template is a directory holding a config.yaml and optionally a settings.yaml - the shape of a site below config/sites/, so a template is produced by copying a working site out of an installation. Its default location is Sites/<identifier>/ next to the entry file of the set.

rootPage is the uid of the page that becomes the site root: the id an entity of the pages table declares in one of the scenario files of the set. A scenario record carries no symbolic name, so the uid it declares is its stable handle. A site whose rootPage no pages entity declares is refused before anything is written, rather than after the whole page tree exists.

The site itself is written after the records, and the uid the page was actually written with always wins over whatever the template declares.

Two refusals and one report belong to this:

• An existing site identifier is refused. TYPO3 merges an incoming site configuration into an existing one, so seeding over it would produce neither the template nor the previous configuration. Remove the site first if the seed is meant to replace it.
• The automatic site configuration is suppressed. TYPO3 writes an autogenerated-<uid> configuration for every new site root; an import switches that off for the whole run, whether the set declares sites or not.
• A seeded site root covered by no site configuration is reported, by uid. That is a page tree no frontend can render, and the suppression above is what would otherwise make it silent.

--base replaces the base of every site the run writes, which is what lets one set be imported into several installations. --no-site-config skips the declared site configurations while keeping the report.

Impact 

A seeded installation is reachable in the frontend without a manual step in the backend, and an installation whose sites are maintained elsewhere is told which of its new page trees are still uncovered.