DEPRECATION WARNING

This documentation is not using the current rendering mechanism and is probably outdated. The extension maintainer should switch to the new system. Details on how to use the rendering mechanism can be found here.

EXT: Import mail messages to tt_news records

Author:Kasper Skårhøj
Created:2002-11-01T00:32:00
Changed by:Loek Hilgersom
Changed:2012-06-29T11:30:13
Classification:mail2news
Description:Automatically polls an IMAP account and imports e-mail messages to tt_news and/or t3blog items, including attached images. Allows to specify news categories and storage pid. Allows you to post messages on your website directly from your mobile phone (moblogging).
Keywords:forAdmins, email, import, tt_news, t3blog, moblog, cron, scheduler
Author:Loek Hilgersom
Email:typo3extensions@netcoop.nl
Info 4:
Language:en

img-1 EXT: Import mail messages to tt_news records - mail2news

EXT: Import mail messages to tt_news records

Extension Key: mail2news

Language: en

Keywords: forAdmins, email, import, tt_news, t3blog, moblog, cron, scheduler

Copyright 2000-2012, Loek Hilgersom, <typo3extensions@netcoop.nl>

This document is published under the Open Content License

available from http://www.opencontent.org/opl.shtml

The content of this document is related to TYPO3

- a GNU/GPL CMS/Framework available from www.typo3.org

Table of Contents

EXT: Import mail messages to tt_news records 1

`Introduction 3 <#__RefHeading__1531_1304830077>`_

What does it do? 3

Screenshots 3

`Users manual 4 <#__RefHeading__1537_1304830077>`_

FAQ 4

`Administration 5 <#__RefHeading__1541_1304830077>`_

FAQ 5

`Configuration 6 <#__RefHeading__1545_1304830077>`_

Reference 9

`Tutorial 10 <#__RefHeading__1549_1304830077>`_

`Known problems 11 <#__RefHeading__1551_1304830077>`_

`To-Do list 12 <#__RefHeading__1553_1304830077>`_

`ChangeLog 13 <#__RefHeading__1555_1304830077>`_

Credits 13

Introduction

What does it do?

  • Automatically polls an IMAP or POP3 account at chosen intervals (run by cronjob) and imports e-mail messages to tt_news and/or t3blog items.
  • The contents of the email messages are stored as title, bodytext, short, image, news_files, author, author_email and datetime. You are free to configure how and where fields should be imported (see 'fieldMarkers')
  • Capable of importing multiple images in one message
  • Imports selected types of other attachments into tt_news field news_files (or a custom field when using t3blog, to be configured manually)
  • Allows to specify news categories, storage pid and backend user (cruser_id) and any other field which you need to set per imported message (see 'fieldMarkers') or needs to get a static value (see 'fieldPresetValues')
  • Allows you to post messages on your website directly from your email, e.g. from your mobile phone (moblogging)
  • Checks sender email: only mail from matching email addresses will be processed
  • Optionally hides new messages
  • Takes care of conversion to characterset as defined in TYPO3 setup (localconf.php)
  • Recognizes links, url's and emails in plain text emails and produce html-links accordingly
  • Allows for multiple instances in 1 TYPO3 installation, so it's possible to use it in a multi-domain site, or e.g. using different configurations for importing to tt_news and t3blog
  • Easily configurable through Extension Manager (general default settings) and Mail2news Importer Records (for each individual instance)
  • Works with the TYPO3 system extension Scheduler (available since 4.3), so it no longer needs a separate cronjob to run

Since version 1.9.7 you can create Mail2news Importer Records. If you do not use the importer records, mail2news will just poll 1 mailbox using the settings from the Extension Manager (and so maintaining backwards compatibility).

Please report issues at http://forge.typo3.org/projects/extension- mail2news

Screenshots

All configuration of the extension is done in the extension manager (EM). The current version of the extension has more configuration options than shown on this screenshot, see the part 'Configuration' of this manual for a description of all options.

img-2

Users manual

The extension mail2news polls a specified email account and imports the messages into tt_news records with a specified news-category of choice. Once the messages are imported you can display them where and how you want using all features of tt_news.

[ add sample email and how it's processed ]

FAQ

none so far

Administration

((generated))

Compatibility & requirements

The minimum requirements for this extension to run are:

  • TYPO3 v4 (tested on 4.2.x - 4.7.x, but should also work on 4.1.x, as long as TYPO3's cli_dispatch script is available). For TYPO3 4.5, use mail2news v1.9.9 or higher.
  • mail2news v2.0.3 has been tested with:
    • TYPO3 4.6 and 4.7 (but should also work with 4.5, please file a bug on forge.typo3.org if you find issues)
    • tt_news 2.5.2 - 3.2.1
    • t3blog 1.1.x - 1.2.0
  • PHP5
  • PHP-IMAP extensionYou can check if this php-extension is installled by typing 'php -m' on the commandline and check if it mentions 'imap'.If it's not installed, you can install it if you are the administrator of your server. On debian-type servers you can install it by typing 'apt- get install php5-imap'.
Create a Scheduler task for mail2news

Mail2news needs to run automatically (as often as you want it to) in order to see if any new email messages arrived which it should import. To do this, you either need to have TYPO3's system extension Scheduler installed and configured with a cronjob (this is the prefered way), or you need to set up a separate cronjob for mail2news.

If you do not have the Scheduler extension already running with a cronjob, see the manual of Scheduler for details.

For mail2news, just create a scheduler task 'Mail-to-news importer (mail2news)' and set it to 'recurring' and often as you want (just remember that your scheduler tasks can not run more frequent than the scheduler cronjob itself).

Set up a separate cronjob – only necessary if you are not using Scheduler

From version 1.9.2 onwards the extension uses TYPO3's cli_dispatch script to execute the cronjob in the context of TYPO3. This has several advantages, but you can no longer run the script directly. However, the procedure for setting up the cronjob has become easier because you no longer have to make the script in this extension executable (in my experience, let me know if you find anything different), and you don't need to patch the script to add the correct path to php on your server (however, cli_dispatch still needs to know that path, but that counts for your entire TYPO3 installation, not just this extension).

What you need to do to is this:

create a new backend user with the name '_cli_mail2news'.This user should NOT be an admin-user!

set up the cronjob to run the script cli_dispatch script e.g. every hour.To set up the cronjob you need to find the full path to your site:

10 * * * * /path/typo3/cli_dispatch.phpsh getmail_cli >> /path/fileadmin/cron.log 2>&1

This will run the cronjob every hour at 10 minutes past the hour and send the output to the file cron.log. If you don't want logging, you can replace the path to cron.log by /dev/null

Upgrading

As of version 1.9.5 of the mail2news extension it is no longer necessary to add the line below to typo3conf/localconf.php, the extension takes care of this itself now.$TYPO3_CONF_VARS['SC_OPTION S']['GLOBAL']['cliKeys']['getmail_cli'] = array('EXT:mail2news/getmail_cli.php','_CLI_mail2news');

If you are upgrading from mail2news version 1.9.4 or earlier, you should remove this line from typo3conf/localconf.php

FAQ

none so far

Configuration

Configuration settings

All configuration of the extension is done in the extension manager. (See screenshot) You will find the following settings:

((Unknown Property))

a

b

c

Default value

Servername

a

Servername

b

Servername for POP3 or IMAP mailserver to fetch email from.

c

Username

a

Username

b

Username of the POP3 or IMAP account you want to use for this extension. In most cases it will be best to use an account that is unknown to other people. Because this account allows you to publish directly on your website. To avoid spam on the account, it is also best not to use often used words before the '@' like info, helpdesk or admin.

c

Password

a

Password

b

Password of the POP3 or IMAP account.

c

IMAP

a

IMAP

b

If set, IMAP protocol will be used to fetch mail, otherwise POP3

c

1 (=IMAP)

SSL

a

SSL

b

If set, the connection will be made using SSL encryption

c

0

self_signed_certificate

a

self_signed_certificate

b

Set this if you use SSL with a self-signed certificate

c

0

portno

a

portno

b

Port number to connect to mail server. In most cases you won't have to set this because the extension will use the default port numbers for IMAP and POP3 with or without SSL.

c

0

Delete after processing

a

Delete after processing

b

If set, mail that is processed by the extension will be deleted from the account – ATTENTION: in this first release of the extension this option should always be set, otherwise the extension will import the same messages over and over again

c

1

Also delete rejected messages

a

Also delete rejected messages

b

If set, then messages that were not processed (e.g. because sender did not match allowed senders) will also be deleted by the script. This is useful when you use a dedicated email account only for this script (which is recommended)

c

0

((Unknown Property))

a

b

c

Allowed senders

a

Allowed senders

b

Only email from accounts that match one of the values specified here will be processed. You can specify multiple addresses comma separated like this:

name@email.com,person.blog@domain.nl,other.blog@domain.nl

or use parts of the addresses to match more than 1 address:

name@email,.blog@domain.nl

This will for instance match all the addresses above, plus e.g.

anyotherperson.blog@domain.nl as well as name@email.org

c

@

Concatenate text parts

a

Concatenate text parts

b

In the default setting only the first text part is imported, which skips separate signature parts. Check this box to import and concatenate (glue together) all text parts of multi-part messages.

c

0

Maximum image size in kb

a

Maximum image size in kb

b

Images above this size are not imported into tt_news

c

300

Maximum attachment size in kb

a

Maximum attachment size in kb

b

Attachments above this size are not imported into tt_news

c

500

Allowed file extensions for images

a

Allowed file extensions for images

b

Comma separated list of image extensions, like png,jpg,gif. Files with extensions used here will be filtered to news images and will NOT end up at the news file attachments.

c

png,jpg,gif

Allowed extensions other attachments

a

Allowed extensions other attachments

b

Comma separated list of file extensions, like pdf,odt,doc, etc.Extensions that are already listed in 'Allowed file extensions for images' will be ignored here.

c

avi,csv,doc,eps,mdb,mp3,mp4,mpg,mov,odt,ods,otf,pdf,pps,ppt,rtf,sxw,sx c,swf,txt,ttf,wks,wpd,wps,xls,xml

Record type

a

Record type

b

Choose what type of records mail2news should create by default – tt_news or t3blog – you can override this (and all other settings) in the mail2news importer records.

c

Default categories

a

Default categories

b

Comma separated list of titles or ID's of default categories (ttnews or t3blog) for items created by this extension, used if no (valid) category has been set in the email message

c

Storage ID

a

Storage ID

b

The ID of the BE folder where news or blog messages are stored

c

Backend user ID

a

Backend user ID

b

The ID of the back-end user that will own the created records (cruser_id)

c

Hide new messages

a

Hide new messages

b

If set, news items created by this extension will be hidden. This is useful when there is a risk that unauthorized people will send email to the account specified in the settings above.

c

1

Clear cache of these pages

a

Clear cache of these pages

b

Comma separated list of page IDs where the created news records will be shown, of which the cache needs to be cleared for the new items to be shown. E.g. 101,214,62Note: you cannot use the clearCacheCmd options 'all' or 'pages'!

c

TS Config Options (New in version 2.0 !): fieldMarkers

fieldMarkers are a very important new feature: they replace the settings 'Category line identifier' and 'Subheader line identifier' from the 1.x.x versions. With the fieldMarkers you can use TS Config to define markers like #cat# (but could be anything), with which you can define the categories (tt_news or t3blog) that should be added to new imported message.

Examples:

mod.tx_mail2news.fieldMarkers.categories = #cats#

will allow you to write an email message like this:

Subject: This is the subject-line

#cats# Frontpage, important news, 12This is the content of this message, of course it doesn't deal with anything specific and nothing particular.

This message will get imported with:

title = This is the subject-linebodytext = This is the content of this …

The message will get the following categories attached: 'Frontpage' and 'important news' (if they exist, otherwise these categories will be ignored) and the category with uid 12. You can freely mingle category uid's and category names – if you use a number, mail2news assumes it's a uid.

The key fieldMarkers.categories is specifically defined for mail2news in order to process multiple categories and this works both for tt_news and t3blog.

Store data in any field

It gets even better: you can create fieldMarkers for almost any field that exists in the tables in which you'll be importing (currently tt_news for news items, or tx_t3blog_post and tt_content for blog items).

For example:

mod.tx_mail2news {fieldMarkers {// process categories after the marker #cat#categories = #cat#// the line beginning with #short# will be imported in the tt_news field 'short' (subheader)short = #short#// This line:// #start# 5-10-2011 14:15// will be translated to a unix timestamp to make TYPO3 understand the date// (following your servers locale settings) and the result is stored in field tx_newscalendar_calendardatetx_newscalendar_calendardate = #start#tx_newscalendar_calendardate.parseDate = 1tx_newscalendar_calendardate_end = #end#tx_newscalendar_calendardate_end.parseDate = 1}}

Transform date notation to unix timestamp

As shown above, you can even use fields added by other extensions, like newscalendar in this example. The property parseDate = 1 is a special function in mail2news to recognize dates written as strings and transform them to unix timestamp format (otherwise you would have to write timestamps in your email, quite uncomfortable...)

Set fixed values (or defaults if marker is not mentioned in message)

TS Config Option: fieldPresetValues

With fieldPresetValues you can define fixed values which should be stored in a certain field. With t3blog for example, your message content will be stored in a tt_content record. If you include a picture, it will always be in the same position from the text. With the following TS Config settings you can define the value that will be used by mail2news for the tt_content field 'imageorient':

mod.tx_mail2news {fieldPresetValues {// example: set the imageorient field (tt_content, for use with t3blog) to value 16 by default.imageorient = 16}}

Now if you also want to be able to choose this value for each specific message, you just define a fieldMarker as well:

mod.tx_mail2news {fieldPresetValues {imageorient = #orientation#}}

Now you can use the line #orientation# 2 above your email message to override the default value 16 which we've just set, and fill the field 'imageorient' with the value '2' for this record.

For backward compatibility with older mail2news versions:

To make mail2news behave exactly like in version 1.9.9 and below, set this in TS Configuration in root page:

mod.tx_mail2news {fieldMarkers {categories = ##short = &&}}

What will be processed?

You can use as many fieldPresetValues and fieldMarkers as you wish. fieldMarkers override fieldPresetValues , if the marker is found in the message of course.

  • The first lines of the body of the email message are scanned for markers
  • as many lines are scanned as there are fieldMarkers defined (if you have 4 fieldMarkers defined, the first 4 lines of the email message will be scanned for markers)
  • a marker should start at the first position of a line in an email
  • if a marker is recognized in those lines, that line, and all lines above will be stripped from the record content
  • a marker could be anything (the format '#marker#' is just a convention) but can not contain white space characters
  • if a marker is recognized, all content that follows on that line will be trimmed (from starting and ending spaces) and processed

Reference

Tutorial

Recommendations

  • To avoid spam, use a dedicated email account with an unusual name, which is not published anywhere and which you only use for using this extension

Known problems

Issues with mail2news can be reported at http://forge.typo3.org/projects/extension-mail2news/issues Please check if it is not already reported!

To-Do list

Ideas for later:

  • Add authorization code to filter spam or unauthorized messages
  • Add basic text formatting options
  • Add HTML-mail import
  • Add hooks or other method to add custom importer types, e.g. for EXT:news

ChangeLog

((generated))

See file Changelog

Credits

  • Thanks to Jonas Dübi for constructive feedback

    Sponsoring:

  • Thanks to Etienne Girardet (www.pacificografik.de, Berlin) for sponsoring the initial version and added features

  • Thanks to Sebastian Wendler (Pro Tours AG, Hannover) for feature sponsoring

  • Thanks to the Dutch Trade Union CNV for supporting the new features for importing to t3blog, adding the fieldMarkers feature with the parseDate option.

14