Synchronizing user-uploaded files across environments

TYPO3 sites commonly include uploaded files, such as images, videos, and documents. These files are typically stored in directories like fileadmin/ and other storage locations defined in the File storage configuration.

While database records can be synchronized through exports or dumps, synchronizing user-uploaded files across environments is different because file references and media consistency need to be kept intact.

Table of contents

Note

This chapter focuses on managing file-based content, such as uploads and media files. For database content, refer to Synchronizing database content across environments.

For an overview of environment workflows, see Multi-stage environment workflow for TYPO3.

Common file storage locations

By default, TYPO3 stores user-managed files in:

fileadmin/ — typically used for editor-uploaded media and content files.
Custom storage locations defined in File storage configuration.

These locations can contain:

Files uploaded by backend users or editors in the TYPO3 backend, such as images, PDFs, and videos used in page content.
Files uploaded by website visitors, for example through forms and application processes (uploads, form attachments).
Automatically generated files, such as scaled or transformed images, generated PDFs and cached previews.

Important

Static media assets such as CSS files, JavaScript, icons, or static images should not be placed in `fileadmin/`. These assets belong in the Resources/Public/ directory of your site package or TYPO3 extension. This ensures that versioned, developer-maintained resources are properly managed through your project’s codebase.

Example: packages/site_package/Resources/Public/Images/logo.svg

Challenges in file management

File-based content can become large or sensitive, leading to challenges such as:

Storage size and synchronization effort.
Privacy and legal considerations (uploaded personal documents, storage with restricted access).
Dependency on matching files and database references.

Best practices for file management

Mirror file structures between environments to avoid broken links.
Exclude sensitive or unnecessary files when synchronizing to non-production environments.
Automate file synchronization using deployment tools or scripts.
Use symbolic links or mounts if your infrastructure supports shared storage between environments.

File synchronization methods

Manual / rsync copying

Copy files via rsync, FTP, or SSH between servers or environments.
Filter files based on file type or directory to reduce data transfer.
Exclude temporary or processed files that can be regenerated.

Example rsync command:

rsync -avz \
    --exclude='*.bak' \
    --exclude='*/_processed_/' \
    --exclude='*/_temp_/' \
    user@source-server:/var/www/public/fileadmin/ \
    /var/www/public/fileadmin/

The exclusion parameters refer to the following:

*.bak: Skip backup or intermediate files, if present.
*/_processed_/: Skip processed files generated by TYPO3 FAL (resized or transformed images).
*/_temp_/: Skip all temporary folders, such as fileadmin/_temp_/ or fileadmin/user_upload/_temp_/, used for transient files.

Temporary and processed files can typically be regenerated by TYPO3 and do not need to be transferred between environments.

See handling processed files and metadata consistency for important details and trade-offs.

Automated deployment scripts

Integrate file copying into your CI/CD pipeline.
Use tools like Deployer, GitLab CI, or GitHub Actions with SSH or rsync tasks.

Using filefill for media synchronization and placeholders

The extension ichhabrecht/filefill provides a convenient solution for handling media files in non-production environments.

It can be configured to:

Fetch real files (such as images or videos) from a live environment, if these are publicly accessible over HTTP.
Generate placeholder files when the real files are not available or should not be copied.

This allows developers to work with realistic file structures without needing to transfer full media sets or sensitive files. It ensures that file references exist on disk, preventing broken links in the TYPO3 backend or frontend.

However, filefill only works with publicly accessible media files like images or videos. It cannot synchronize non-media files, such as:

Form configuration files
Protected private storage files
Arbitrary file types that are not publicly accessible

You typically use filefill in development environments only. It can be set using the ApplicationContext in your config/system/additional.php:

config/system/additional.php

<?php

declare(strict_types=1);

use TYPO3\CMS\Core\Core\Environment;

if (Environment::getContext()->isDevelopment()) {
    $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['filefill']['storages'][1] = [
        [
            'identifier' => 'domain',
            'configuration' => 'https://example.org',
        ],
        [
            'identifier' => 'placehold',
        ],
    ];
}

For more information and configuration options, see the filefill documentation.

Shared storage

Use NFS, cloud storage (S3), or other shared storage solutions.
Mount the same file storage across environments, if technically feasible.

Note

Sharing file storage between production and non-production environments (such as staging or development) is not recommended.

Changes made in staging or development could accidentally delete, overwrite, or expose files in production.

It is best practice to keep file storage fully isolated between environments to ensure that testing and development activities do not impact the live website.

If shared storage is unavoidable, make sure to:

Mount it read-only on non-production environments.
Provide clear backend warnings to prevent accidental changes.
Inform all users and developers about the risks.

Handling processed files and metadata consistency

TYPO3 generates processed files (such as scaled, cropped, or converted images) and stores their metadata in the sys_file_processedfile table.

This creates a dependency between the file system and the database.

If you skip synchronizing processed files (for example, by excluding _processed_ folders), TYPO3 will attempt to regenerate them when they are requested. This may result in slower initial page loads until all processed files are rebuilt.
If you synchronize the `_processed_` folder, you may transfer large amounts of data, including stale or unnecessary files that TYPO3 has not yet cleaned up.

TYPO3 uses the sys_file_processedfile table to track processed files. If you copy only the database without the matching files, TYPO3 may reference files that are no longer available.

Recommended approaches

Skip processed files and truncate the `sys_file_processedfile` table to let TYPO3 regenerate them on demand:
```
TRUNCATE TABLE sys_file_processedfile;
```
Copied!
Include processed files if you want to avoid runtime regeneration, but consider cleaning up unnecessary files first to reduce data volume.

You can use rsync or file system cleanup tools to remove outdated processed files before transferring them.

Balancing performance, storage size, and deployment speed

Based on the priorities set for your project, choose between:

Faster deployment and smaller storage, skip processed files and truncate the table.
Faster page loads on first access, copy processed files along with the table, but expect to handle larger data volumes.

Make sure your team is aware of the trade-offs and apply a consistent strategy across all environments.

Anonymization or exclusion of user managed files

Review file content for personal or sensitive data.
Provide reduced or dummy file sets in non-production environments when appropriate.
Inform clients or editors if files are excluded or replaced.

Relationship to database references

In TYPO3, user-uploaded and editor-managed files are typically referenced in the database through the File Abstraction Layer (FAL).

Common FAL-managed references include:

sys_file_reference records, which reference the UID of a `sys_file` record to link files to content elements or other database records.
sys_file records, which store metadata about the actual file in the file system (path, size, MIME type).

Best practice: Extensions and custom code should always use FAL to manage file relations. File paths should not be stored in custom database fields, except in TYPO3’s sys_file table managed by FAL.

When synchronizing environments:

Ensure file references in sys_file_reference match existing UIDs in sys_file.
Ensure sys_file records ideally describe existing files on disk, especially in production environments.
In development or staging, it is often acceptable to have incomplete file presence, provided tools like filefill or placeholders are used to simulate missing files.
Avoid broken references by making sure FAL metadata and file storage correspond as much as possible depending on the purpose of the environment.
Verify that no file paths are directly hardcoded or stored in custom tables.

Broken or inconsistent references may result in missing media in the frontend or backend and may require manual repair, re-indexing, or cleanup.

Best practices summary

Synchronize necessary files alongside database content.
Exclude or anonymize sensitive or large files when appropriate.
Automate file handling in your deployment processes.
Document your project's file synchronization strategy.