Breaking: #110966 - Remove UTF8filesystem and systemLocale settings 

See forge#110966

Description 

The configuration options $GLOBALS['TYPO3_CONF_VARS']['SYS']['UTF8filesystem'] and $GLOBALS['TYPO3_CONF_VARS']['SYS']['systemLocale'] have been removed.

Both options existed so that PHP string functions process UTF-8 file names correctly. To achieve that, TYPO3 temporarily switched LC_CTYPE to the configured system locale around calls to basename() , dirname() , pathinfo() and escapeshellarg() .

The PHP manual still describes basename() and dirname() as locale aware. The PHP source code shows that this no longer matters for TYPO3:

  • dirname() is implemented by zend_dirname(), which searches for the directory separator byte by byte. It does not evaluate the locale at all.
  • basename() is implemented by php_basename(). Since PHP 8.1, it searches for the directory separator byte by byte as well, as long as the current locale is "ASCII compatible". PHP considers a locale ASCII compatible if it uses a single-byte encoding (such as C , POSIX or ISO-8859-1) or UTF-8. Only for other multibyte encodings, such as Shift-JIS, Big5 or GBK, PHP walks the string character by character with mblen(). In these encodings, the second byte of a character can be 0x5C , which is the directory separator on Windows. In UTF-8, all bytes of a multibyte character are 0x80 or higher, so they never match a directory separator.
  • pathinfo() is based on these two functions.

This was verified with PHP 8.2, 8.4 and 8.5 on glibc: all three functions return the same, correct results for UTF-8 paths in the C , C.UTF-8 , en_US.UTF-8 and de_DE.UTF-8 locales.

escapeshellarg() is different. Its implementation php_escape_shell_arg() calls mblen() for every byte and silently skips all bytes that are invalid in the current locale. In the C locale of glibc, which is ASCII, every byte of 0x80 or higher is invalid, so äöü.txt turns into .txt. The previous code only prevented this if both options were configured, and EXT:indexed_search called escapeshellarg() directly. PHP-FPM commonly runs in the C locale, for example when started by systemd.

\TYPO3\CMS\Core\Utility\CommandUtility::escapeShellArgument() and \TYPO3\CMS\Core\Utility\CommandUtility::escapeShellArguments() now quote arguments byte by byte, independent of the current locale: the argument is wrapped in single quotes, and each single quote is replaced by '\'' . This is what escapeshellarg() does on POSIX systems, without dropping bytes. On Windows, escapeshellarg() is still used. Arguments containing null bytes are rejected with an \InvalidArgumentException . The shell calls of EXT:indexed_search and of the install tool use these methods as well.

\TYPO3\CMS\Core\Utility\PathUtility::basename() , \TYPO3\CMS\Core\Utility\PathUtility::dirname() and \TYPO3\CMS\Core\Utility\PathUtility::pathinfo() no longer change the locale and have been deprecated, see Deprecation: #110966 - PathUtility basename(), dirname(), pathinfo().

File names are always stored with their Unicode characters (normalized to NFC). The install tool checks for both options have been removed. Both options are removed from config/system/settings.php automatically when the install tool is accessed.

Impact 

The local driver no longer transliterates non-ASCII characters of uploaded or renamed files. Before this change, Übersicht.pdf was stored as Uebersicht.pdf if UTF8filesystem was disabled. It is now stored as Übersicht.pdf. Characters that are not allowed in file names are still replaced by underscores. Existing files are not renamed.

New installations have enabled UTF8filesystem by default since TYPO3 v12, so they already behave this way.

Reading one of the two options from $GLOBALS['TYPO3_CONF_VARS'] raises an "Undefined array key" warning.

Affected installations 

  • Installations that have UTF8filesystem disabled. This is usually the case for installations upgraded from TYPO3 v11 or earlier that never changed the option.
  • Installations with extensions that read one of the two options.

Migration 

Remove any code that reads the two options.

To keep transliterating file names to ASCII, register a listener for \TYPO3\CMS\Core\Resource\Event\SanitizeFileNameEvent :

EXT:my_extension/Classes/EventListener/AsciiFileName.php 
namespace MyVendor\MyExtension\EventListener;

use TYPO3\CMS\Core\Attribute\AsEventListener;
use TYPO3\CMS\Core\Charset\CharsetConverter;
use TYPO3\CMS\Core\Resource\Event\SanitizeFileNameEvent;

final readonly class AsciiFileName
{
    public function __construct(
        private CharsetConverter $charsetConverter,
    ) {}

    #[AsEventListener]
    public function __invoke(SanitizeFileNameEvent $event): void
    {
        $fileName = $this->charsetConverter->utf8_char_mapping($event->getFileName());
        $event->setFileName((string)preg_replace('/[^.\w-]/', '_', $fileName));
    }
}
Copied!