For editors: usage 

  1. Open the File list module and edit the metadata of a file (or open the file resource in the Media module).
  2. Switch to the SEO tab, enable Do not index in search engines and save.
The "Do not index in search engines" checkbox on the SEO tab

The checkbox on the SEO tab of the file metadata form.

That's it. The file's robots.txt entries appear immediately — no cache flush needed:

Generated robots.txt with disallow entries

The generated robots.txt with the disallow entries for the marked file and its processed variants.

Unticking the box removes the entries just as immediately.