Core testing¶
Introduction¶
This chapter is about executing TYPO3 Core tests locally and is intended to give you a better understanding of testing within TYPO3's Core. A full Core git checkout comes with everything needed to run tests in TYPO3.
Core development is most likely bound to the Core main branch - backporting patches to older
branches are usually handled by Core maintainers and often don't affect other Core contributors.
Note, the main script Build/Scripts/runTests.sh is relatively new. It works best when executed on a Linux based host but can be run under macOS and Windows with some performance drawbacks on macOS.
Additionally, it is possible to execute tests on a local system without using Docker. Depending on
which test suite is executed, developers may need to configure their environments to run the
desired test. We however learned not too many people actually do that as it can become tricky. This
chapter does not talk about test execution outside of Build/.
System dependencies¶
Many developers are familiar with Docker. As outlined in the history chapter, test execution needs a well defined, isolated, stable and reliable environment to run tests and also remove the need to manage niche dependencies on your local environment for tests such as "execute functional test "X" using MSSQL with xdebug".
Git, docker and docker-compose are all required. For standalone test execution, a local installation of
PHP is not required. You can even composer install a Core by calling Build/ in a container.
If you're using a Mac, install or update Docker to the most recent version using the packaging system of your choice.
If you are using Ubuntu Linux 18.04 or higher, everything should be ok after
calling sudo apt- once. For other Linux distributions
including older releases of Ubuntu, users should have a look at the Docker homepage to see how to update
to a recent version. It usually involves adding some other package repository and updating / installing using it.
Make sure your local user is a member of the docker group, else the script will fail with something like
/var/.
Windows can rely on WSL to have a decent docker version, too.
Quick start¶
From now on, it is assumed that git, docker and docker-compose are available with the most up-to-date release running on the host system. Executing the basic Core unit test suite boils down to:
# Initial core clone
git clone git@github.com:typo3/typo3.git && cd typo3
# Install composer dependencies
Build/Scripts/runTests.sh -s composerInstall
# Run unit tests
Build/Scripts/runTests.shThat's it. You just executed the entire unit test suite.
Now that we have examined the initial Core clone and a composer install process, we will then look at the
different ways we can apply the run or other scenarios.
Overview¶
So what just happened? We cloned a Core, composer installed dependencies and executed Core
unit tests. Let's have a look at some more details: run is a shell script that figures out
which test suite with which options a user wants to execute, does some error handling for broken
combinations, writes the file Build/ according to its findings and then executes a
couple of docker- commands to prepare containers, runs tests and stops containers after execution
again.
A Core developer doing this for the first time may notice docker- pulling several container images
before continuing. These are the dependent images needed to execute certain jobs. For instance the
container typo3gmbh/php72 may be fetched. Its definition
can be found at TYPO3 GmbH GitHub.
These are the exact same containers Bamboo based testing is executed in. In Bamboo, the combination of
Build/ and Build/
specifies what Bamboo executes for patches pushed to the review system. On local testing, this is the
combination of Build/, Build/ (created by
runTests.sh) and
Build/testing-docker/local/docker-compose.yml.
Whats impressive is that run can do everything locally that Bamboo executes as pre-merge tests at the same time. It's just that the combinations of tests
and splitting to different jobs is slightly different, for instance Bamboo does multiple tests in
the "integration" test at once that are single "check" suites in run. But if a patch is
pushed to Bamboo and it complains about something being broken, it is possible to replay and fix the
failing suite locally, then push an updated patch and hopefully enable the Bamboo test to pass.
A runTests.sh run¶
Let's pick a run example and have a closer look:
lolli@apoc /var/www/local/cms/Web $ Build/Scripts/runTests.sh -s functional typo3/sysext/core/Tests/Functional/Authentication/
Creating network "local_default" with the default driver
Creating local_redis4_1 ... done
Creating local_mariadb10_1 ... done
Creating local_memcached1-5_1 ... done
Waiting for database start...
Database is up
PHP 7.2.11-3+ubuntu18.04.1+deb.sury.org+1 (cli) (built: Oct 25 2018 06:44:08) ( NTS )
PHPUnit 7.1.5 by Sebastian Bergmann and contributors.
. 1 / 1 (100%)
Time: 184 ms, Memory: 16.00MB
OK (1 test, 1 assertion)
Stopping local_mariadb10_1 ... done
Stopping local_redis4_1 ... done
Stopping local_memcached1-5_1 ... done
Removing local_functional_mariadb10_run_1 ... done
Removing local_prepare_functional_mariadb10_run_1 ... done
Removing local_mariadb10_1 ... done
Removing local_redis4_1 ... done
Removing local_memcached1-5_1 ... done
Removing network local_default
lolli@apoc /var/www/local/cms/Web $ echo $?
0
lolli@apoc /var/www/local/cms/Web $The command asks run to execute the "functional" test suite -s functional and to not execute all
available tests but only those within typo3/. The script first
starts the containers it needs: Redis, memcached and a MariaDB. All in one network. It then waits until
the MariaDB container opens its database port, then starts a PHP 7.2 container and calls phpunit to execute
the tests. phpunit executes only one test in this case, that one is green. The containers and networks are then
removed again. Note the exit code of run (echo $?) is identical to the exit code of the phpunit
call: If phpunit reports green, run returns 0, and if phpunit is red, the exit code would be non zero.
Examples¶
First and foremost, the most important call is -h - the help output. The output below is cut, but
the script returns a useful overview of options. The help output is also returned if given options
are not valid:
lolli@apoc /var/www/local/cms/Web $ Build/Scripts/runTests.sh -h
TYPO3 Core test runner. Execute acceptance, unit, functional and other test suites in
a docker based test environment. Handles execution of single test files, sending
xdebug information to a local IDE and more.
...Some further examples: The most important tests suites are unit tests, functional tests and acceptance tests, but there is more:
# Execute the unit test suite with PHP 7.3
Build/Scripts/runTests.sh -s unit -p 7.3
# Execute some backend acceptance tests
Build/Scripts/runTests.sh -s acceptance typo3/sysext/core/Tests/Acceptance/Backend/Topbar/
# Execute some functional tests with PHP 7.3 and postgres DBMS
Build/Scripts/runTests.sh -s functional -p 7.3 -d postgres typo3/sysext/core/Tests/Functional/Package/
# Execute the cgl fixer
Build/Scripts/runTests.sh -s cglGit
# Verbose runTests.sh output. Shows main steps and composer commands for debugging
Build/Scripts/runTests.sh -vAs shown there are various combinations available. Just go ahead, read the help output and play around. There are tons of further test suites to try.
One interesting detail should be mentioned: run uses typo3gmbh/phpXY
as main PHP containers. Those are loosely maintained and may be updated. Use the command
Build/ to fetch the latest versions of these containers.
Debugging¶
To speed up test execution, the PHP extension xdebug is not usually loaded.
However, to allow debugging tests and system under tests, it is possible to
activate xdebug and send debug output to a local IDE. We'll use PhpStorm for this example.
Let's verify our PhpStorm debug settings first. Go to File > Settings > Languages & Frameworks > PHP > Debug. Make sure "Can accept external connections" is enabled, remember the port if it is not the default port(9000) and also raise "Max. simultaneous connections" to two or three. Note remote debugging may impose a security risk since everyone on the network can send debug streams to your host.
Accept changes and enable "Start listening for PHP connections". If you changed settings, turn them off and on once to read new settings.
Now set a break point in an assignment. Note break points do not work "everywhere", for instance not on empty lines and not on array assignments. The best way is to use a straight command. We'll use a simple test file for now, add a breakpoint and then execute this test. If all goes well, PhpStorm stops at this line and opens the debug window.
Build/Scripts/runTests.sh -x -s functional -p 7.3 -d postgres typo3/sysext/core/Tests/Functional/Package/RuntimeActivatedPackagesTest.phpThe important flag here is -x! This is available for unit and functional testing. It enables xdebug
in the PHP container and sends all debug information to port 9000 of the host system. If a local PhpStorm
is listening on a non-default port, a different port can be specified with -y.
If PhpStorm does not break as expected, some adjustments in this area may be required. First, make
sure "local" debugging works. Set a breakpoint in a local project and see if it works. If it works
locally, the container based debugging should also work. Next, make sure a proper break point has been set.
Additionally, it may be useful to activate "Break at first line in PHP scripts" in your PhpStorm settings. run
mounts the local path to the same location within the container, so path mapping is not needed. PhpStorm
also comes with a guide how to set up
debugging.