Core testing in depth
Introduction
This chapter is about executing TYPO3 Core tests locally and is intended to give you a better understanding of testing within TYPO3's Core. A full Core git checkout comes with everything needed to run tests in TYPO3.
Core development is most likely bound to the Core main branch - backporting patches to older
branches is usually handled by Core maintainers ("Mergers") and often doesn't affect other Core contributors.
The main entry point to perform testing and build-related actions is the helper Build/Scripts/runTests.sh. It works best when executed on a Linux based host but can be run under macOS and Windows with some filesystem performance drawbacks on macOS. It utilizes Docker containers, and more details can be found in Using runTests.sh.
Additionally, it is possible to execute tests on a local system without using Docker. Depending on
which test suite is executed, developers may need to configure their environments to run the
desired test. We however learned not too many people actually do that as it can become tricky. This
chapter does not talk about test execution outside of Build/.
System dependencies
Many developers are familiar with Docker. As outlined in the history chapter, test execution needs a well defined, isolated, stable and reliable environment to run tests and also remove the need to manage niche dependencies on your local environment for tests such as "execute functional test 'X' using PostgreSQL with xdebug".
Git and docker (or podman) are required. For standalone test execution, a local installation of
PHP is not required. You can even composer install a Core by calling Build/ in a container.
If you're using a Mac, install or update Docker to the most recent version using the packaging system of your choice.
If you are using Ubuntu Linux 18.04 or higher, everything should be ok after
calling sudo apt- once. For other Linux distributions
including older releases of Ubuntu, users should have a look at the Docker homepage to see how to update
to a recent version. It usually involves adding some other package repository and updating / installing using it.
Make sure your local user is a member of the docker group, else the script will fail with something like
/var/.
Windows can rely on WSL to have a decent docker version, too.
Quick start
From now on, it is assumed that git and docker (or podman) are available with the most up-to-date release running on the host system. Executing the basic Core unit test suite boils down to:
# Initial core clone
git clone git@github.com:typo3/typo3.git && cd typo3
# Install Composer dependencies
Build/Scripts/runTests.sh -s composerInstall
# Run unit tests
Build/Scripts/runTests.shThat's it. You just executed the entire unit test suite.
Now that we have examined the initial Core clone and a Composer install process, we will look at the
different ways we can utilize the run for other scenarios.
Overview
So what just happened? We cloned a Core, Composer installed dependencies and executed Core
unit tests. Let's have a look at some more details: run is a shell script that figures out
which test suite with which options a user wants to execute, does some error handling for broken
combinations and then uses local docker commands to run specific containers with specific options. Using
these containers, the actions are performed, and the containers are stopped after execution.
A Core developer doing this for the first time may notice that several container images
will be pulled before continuing. These are the dependent images needed to execute certain jobs. For instance,
a container providing the specific PHP-version may be fetched. The same containers are used for the
TYPO3 CI GitLab Pipeline, even utilizing the same run script. What's impressive is that
you can locally run the same tests like a fully-fledged CI server..
The GitLab CI Pipeline is maintained through the Ansible infrastructure found on https://git.typo3.org/typo3/CI/testing-infrastructure/-/tree/main/ansible?ref_type=heads, and the Pipeline itself is set up through https://github.com/TYPO3/typo3/tree/main/Build/gitlab-ci/.
Compared to your local execution it's just that the combinations of tests and splitting to different jobs which is slightly different, for instance GitLab CI paralelly performs multiple tests with more complex version matrixes (PHP and Databases).
If a patch is pushed to GitLab and it complains about something being broken, it is possible to replay and fix the failing suite locally, then push an updated patch and hopefully enable the tests to pass.
A runTests.sh run
Let's pick a run example and have a closer look:
lolli@apoc /var/www/local/cms/Web $ Build/Scripts/runTests.sh -s functional typo3/sysext/core/Tests/Functional/Authentication/
PHPUnit 11.2.5 by Sebastian Bergmann and contributors.
Runtime: PHP 8.2.19
Configuration: /Users/garvin/TYPO3/typo3-core-bugreproduce-base/typo3-core/Build/phpunit/FunctionalTests.xml
................................................................. 65 / 67 ( 97%)
.. 67 / 67 (100%)
Time: 00:12.077, Memory: 103.00 MB
OK (67 tests, 176 assertions)
###########################################################################
Result of functional
Container runtime: docker
PHP: 8.2
DBMS: sqlite
SUCCESS
###########################################################################
lolli@apoc /var/www/local/cms/Web $ echo $?
0
lolli@apoc /var/www/local/cms/Web $The command asks run to execute the "functional" test suite -s functional and to not execute all
available tests but only those within typo3/. The script first
starts the containers it needs: Redis, memcached (previously also MariaDB by default, which is now using
SQLite instead, due to less dependencies). All in one network. It then starts a PHP 8.2 container and calls
phpunit from there to execute the tests. phpunit executes only one test in this case, that one is green. The containers
and networks are then removed again. Note the exit code of run (echo $?) is identical to the exit
code of the phpunit call: If phpunit reports green, run returns 0, and if phpunit is red, the exit code
would be non zero.
Examples
First and foremost, the most important call is -h - the help output. The output below is cut, but
the script returns a useful overview of options. The help output is also returned if given options
are not valid:
lolli@apoc /var/www/local/cms/Web $ Build/Scripts/runTests.sh -h
TYPO3 Core test runner. Execute acceptance, unit, functional and other test suites in
a container based test environment. Handles execution of single test files, sending
xdebug information to a local IDE and more.
...Some further examples: The most important tests suites are unit tests, functional tests and acceptance tests, but there is more:
# Execute the unit test suite with PHP 8.3
Build/Scripts/runTests.sh -s unit -p 8.3
# Execute some backend acceptance tests
Build/Scripts/runTests.sh -s acceptance typo3/sysext/core/Tests/Acceptance/Backend/Topbar/
# Execute some functional tests with PHP 8.2 and postgres DBMS
Build/Scripts/runTests.sh -s functional -p 8.2 -d postgres typo3/sysext/core/Tests/Functional/Package/
# Execute the cgl fixer
Build/Scripts/runTests.sh -s cglGit
# Verbose runTests.sh output. Shows main steps and Composer commands for debugging
Build/Scripts/runTests.sh -vAs shown there are various combinations available. Just go ahead, read the help output and play around. There are tons of further test suites to try.
Also note that you can use the -b option to switch between docker and podman container execution,
with podman being the default (when available).
One interesting detail should be mentioned: run uses several containers from
https://github.com/orgs/TYPO3/packages for PHP and JavaScript environments. Use the command
Build/ to fetch the latest versions of these containers.
Debugging
To speed up test execution, the PHP extension xdebug is not usually loaded.
However, to allow debugging tests and system under tests, it is possible to
activate xdebug and send debug output to a local IDE. We'll use PhpStorm for this example.
Let's verify our PhpStorm debug settings first. Go to File > Settings > Languages & Frameworks > PHP > Debug. Make sure "Can accept external connections" is enabled, remember the port if it is not the default port (9000) and also raise "Max. simultaneous connections" to two or three. Note remote debugging may impose a security risk since everyone on the network can send debug streams to your host.
Accept changes and enable "Start listening for PHP connections". If you changed settings, turn them off and on once to read new settings.
Now set a break point in an assignment. Note break points do not work "everywhere", for instance not on empty lines and not on array assignments. The best way is to use a straight command. We'll use a simple test file for now, add a breakpoint and then execute this test. If all goes well, PhpStorm stops at this line and opens the debug window.
Build/Scripts/runTests.sh -x -s functional -p 8.1 -d postgres typo3/sysext/core/Tests/Functional/Package/RuntimeActivatedPackagesTest.phpThe important flag here is -x! This is available for unit and functional testing. It enables xdebug
in the PHP container and sends all debug information to port 9000 of the host system. If a local PhpStorm
is listening on a non-default port, a different port can be specified with -y.
If PhpStorm does not break as expected, some adjustments in this area may be required. First, make
sure "local" debugging works. Set a breakpoint in a local project and see if it works. If it works
locally, the container based debugging should also work. Next, make sure a proper break point has been set.
Additionally, it may be useful to activate "Break at first line in PHP scripts" in your PhpStorm settings. run
mounts the local path to the same location within the container, so path mapping is not needed. PhpStorm
also comes with a guide how to set up
debugging.
Building
Luckily, run also helps us to build JavaScript and CSS assets:
Build/Scripts/runTests.sh -s buildJavascript
Build/Scripts/runTests.sh -s buildCssAgain, this utilizes all the needed containers for the proper NodeJS environment, so you have zero local dependencies on properly building.
You can also run a watch task thanks to the full integration of npm command execution:
Build/Scripts/runTests.sh -s npm -- run watch:build