EXT: Mailing List Archive¶

Categories¶

Before anything else you might like to create a few categories you can assign to mails and FAQ/HOWTO items. If so, simply create such records in the Storage folder configured for the branch of the website.

These categories can be assigned to mails and FAQ items during management. It will look like this in the archive, being used in both the category selector box and in the listing:

Add content element type "Insert plugin"¶

Simply insert a content element on the page where the archive should be:

Set the type to this:

Configure the plugin¶

Then comes a very important part - configuring the plugin to the mailing list:

Select which email-header field in the mails from the mailing list server to select on. Here the sender mail is used which is constantly set to "typo3-english-admin@lists.netfielders.de" (#2) when the mails are for the "typo3-english" mailing list.

The value to match the "Select field" (#1) against.It works like this: When mails are moved from the "_inmail" table to the "_ml" table (from the "inbox" to the archive) only mails where the "Select field" value matches the "Select value" are selected - thus its a filter retrieving mails only for the list you want (assuming that the "_inmail" table may receive mails from lots of various mailing lists. On typo3.org that means all the other lists as well.)

The email address to which new messages or replys are sent. Used for the post/reply forms in the archive.

The prefix string used in the subject line. This is used to clean up the subject line. Make sure its correct case and all.

The list "supervisors". These frontend users are super-managers. Basically that means they are allowed to capture/manage threads that other managers actually "own". Supervisors can also make threads "sticky" - managers cannot.

The list of "managers" are frontend users which can manage the threads, thus cleaning up the content, removing reply text, hiding posts, moving posts, categorizing threads and FAQ items.

The starting point of the mailing list archive. Probably you want the FAQ and Mail items to be on the same page as the plugin - so just leave this option. Otherwise you can point out another page where those are stored. But only one page - not many, not recursively.

CMD¶

pidList¶

tx_newloginbox_pi3-showUidPid¶

storeCompressedOriginalContent¶

enableFinalDesperateTryToLocateThreadBySubject¶

messageDividers.[0..x]¶

faq_email¶

readmail¶

_CSS_DEFAULT_STYLE¶

_LOCAL_LANG.[langkey].[ll-key]¶

_DEFAULT_PI_VARS.[piVar-key]¶

[tsref: plugin.tx_maillisttofaq_pi1]

results_at_a_time¶

maxPages¶

transfer_probability¶

transfer_amount¶

textarea_style¶

catSelSize¶

catSelTop¶

catSelNoHeader¶

showHiddenOTmsgHeaders¶

replyIndented_BreakNumChar¶

expireDaysForUnanswered¶

lgdIndicator¶

lgdIndicator.limit_1¶

daysBeforeDirectNotificationsWhenReply¶

[tsref: plugin.tx_maillisttofaq_pi1.listView]

admin_email¶

admin_name¶

cc_email¶

regards¶

[tsref: plugin.tx_maillisttofaq_pi1.faq_email]

probability¶

url¶

password¶

number¶

enableFeed¶

[tsref: plugin.tx_maillisttofaq_pi1.readmail]

The category selector¶

In the category selector box you would find the extension categories listed like this:

... and as a manager you would be able to select any extension as category:

"user_local"¶

Step one: Create "user_local" extension.

I rushed to the Extension Manager and used the Kickstarter Wizard to create a new extension. The extension key was selected as "user_local". The "user_" prefix is reserved for site specific extension and is therefore a good choice for an extension which is never going to be shared with anyone - since it is simply a practical container for all the stuff related to this website.

The Kickstarter basically looked like this:

This creates nothing but a very blank framework for the extension.

Step two: Create the class file

So in the new extension folder I created a PHP file for the extension class (codelisting in prev. chapter):

Step three: Create a "ext_localconf.php" file

Then I created the "ext_localconf.php" file so that I could configure the system to use my class extension file. This was the code listing:

<?php

$TYPO3_CONF_VARS['FE']['XCLASS']['ext/maillisttofaq/pi1/class.tx_maillisttofaq_pi1.php'] =
        t3lib_extMgm::extPath('user_local').'class.ux_tx_maillisttofaq_pi1.php';


?>

Step four: Set priority

Now, at this point the extension is ready to install. And remember that the idea is that this extension could be used for even more site specific additions.

Since this is the case - that the extension is site specific and probably going to extend other extensions etc. - then we might set the priority flag to "bottom" so that the extension will always be loaded as one of the last extensions. This is done by editing the file "ext_emconf.php" in the extension root directory:

$EM_CONF[$_EXTKEY] = Array (
        'title' => 'LOCAL STUFF',
        'description' => 'Extension which contains local overriding stuff etc.',
        'category' => 'misc',
        'author' => 'Kasper Skårhøj',
        'author_email' => 'kasper@typo3.com',
        'shy' => '',
        'dependencies' => '',
        'conflicts' => '',
  'priority' => 'bottom',
        'module' => '',
        'state' => 'stable',
        'internal' => '',
...

(For more informaton about these flags there is a description of the keys on this page .)

Step five: Install it

Then, install it!

Or if you changed it while it was installed, clear the cache files:

Technical overview¶

Generally the archive works like this:

Getting mails into the database

• Someone sends a mail to the mailing list. This mail arrives on the list server which forwards it all all the subscribers. This is how mailing lists work.
• One of the subscribers to the list is your archive email , lets say that is "mailarchive@my_list_archive.com".
• The mail server which receives messages to "mailarchive@my_list_archive.com" must be configured to pass all mails sent to this address on to a PHP shell script (included with this extension) which will insert the mail into the MySQL database, more precisely the table "tx_maillisttofaq_inmail"

This was the hard part. The rest is pure PHP-only.

Organizing mails in threads

• You insert the archive plugin on a page on your website. Then depending on the probability configured in TypoScript the archive will look into the _inmail table (where the raw mails are) and see if there are new mails for the archive.
• If there are new mails the plugin will read the mails, parse them for content and headers, create relations to the existing threads and the archive is thereby updated.
• During this process the plugin will look into the "fe_users" table and try to find a fe_user (from the current storage folder) with the email address matching the sender - and if found, create a relation to the mail.

Database¶

The plugin is build on two principles which should make it work efficiently.

All content is stored in a table separate from the meta information of the messages in the archive. In other words the archive listing is done purely by looking up in a "header-table" (tx_maillisttofaq_ml) and when the threads are displayed, the message content is looked up from the content table (tx_maillisttofaq_mlcontent) which is related 1-1 with the header table (referred to as the "_ml" table in the source)

All root-messages (thread starters = tx_maillisttofaq_ml records with the field "parent" set to zero) will contain all content from the reply messages and itself in the field "all_content" of their tx_maillisttofaq_mlcontent-records. This is a cache of the thread content updated each time new messages arrive in the thread and thus making it easy to program the searching of content within a thread since only the root message needs to be searched.

The "inmail.phpsh" script¶

This script is not a normal PHP script which is executed by a webserver. It's a shellscript and it requires you to have a PHP- version compiled as a binary available in /usr/bin/php

This is how the first two lines of the script look:

#! /usr/bin/php -q
<?php

This instructs the shell to let the script "/usr/bin/php" execute the further contents of the file (which is plain PHP code).

The script outputs nothing, it just takes input from "stdin" (which is the same as content piped into the script from command line):

// MAIL CONTENT
$filename = "php://stdin";

Further it's very important to notice that the script expects the file "_dir_config_for_php_shell_script.php" to be located in the site root of the website. This script contains a simple line:

<?

$HTTP_ENV_VARS["_"]="/[absolute-path-to-site-document-root]/typo3conf/ext/maillisttofaq/inmail.phpsh";

?>

This script simply defines the absolute path to the inmail.phpsh script. This is needed for the script to perform correctly and this environment variable, $HTTP_ENV_VARS["_"], is not always available, therefore it must be set manually.

The steps you have to take regarding this script is:

Make sure there is a PHP binary, "/usr/bin/php"

Create the file "_dir_config_for_php_shell_script.php" in the TYPO3 website root.That means "/[absolute-path-to-site-document- root]/". In this file set the variable $HTTP_ENV_VARS["_"] to "/[absolute-path-to-site-document- root]/typo3conf/ext/maillisttofaq/inmail.phpsh" (the extension is expected to be locally installed as you can see!)

Make sure permissions allows the 'inmail.phpsh' script to be executed from the shell.

Setting up the mail server¶

The next thing is to set up the mail server to pipe content into this new script.

I can give an example of how this is done for Postfix and Qmail:

Postfix

• In the file, "/etc/postfix/virtual" you define that an email address is forwarded to an alias. A line in this file could look like this:

  mailarchive@my_list_archive.com    mailarchive
  

• In the file, "/etc/postfix/aliases", you configure a mail alias like this:

  mailarchive:     |/[absolute-path-to-site-document-root]/typo3conf/ext/maillisttofaq/inmail.phpsh
  

Qmail

Qmail uses ".qmail-xxx-yyy" files to control the action of mail addresses and with them you can direct content into the script instead of into a mail box. I'm not fully sure (since I tried this long time ago) but if you have a .qmail file for the email address something like this inside should do the trick:

|  /[absolute-path-to-site-document-root]/typo3conf/ext/maillisttofaq/inmail.phpsh

BTW: Qmail actually delivered the env var $HTTP_ENV_VARS["_"] mentioned above so the fix with the file "_dir_config_for_php_shell_script.php" was needed only for Postfix in my tests.

If anyone has comments, confirmations of these methods or additions for other mail servers (windows?) then please let me know.

Debugging that mails are piped into the inmail.phpsh script¶

You can test the settings by sending a mail to the email address of course. If you don't see a new record appear in the " tx_maillisttofaq_inmail " table before long, then you might want to debug this whole thing.

One thing you could do is to insert a line in the inmail.phpsh script which sends you an email when executed:

mail (

"your_email@email.email",

"INMAIL",

"TEST"

);

If you execute the script from command line you should receive an email from the script right away. Then, try to send a mail to the email address that gets piped into this. Do you get a mail then? If so, the script gets executed and you might want to find out why the mail content is not inserted into the database. Questions like "Is the mail content received by the script?" or "Do I get any errors when inserting the record" are interesting. You can debug it likewise by sending emails from the script to yourself containing error messages. Or write a log file of course.

The _inmail table¶

When mails are received by the inmail.phpsh script they are inserted in the table "tx_maillisttofaq_inmail "

This is done by the main part of the inmail.phpsh script which looks like this:

$dir = dirname($HTTP_ENV_VARS["_"]);
define("PATH_typo3", dirname(dirname(dirname($dir)))."/typo3/");
define("PATH_site", dirname(PATH_typo3)."/");
define("PATH_t3lib", PATH_typo3."t3lib/");
define("PATH_typo3conf", PATH_site."typo3conf/");   // Typo-configuraton path
define("TYPO3_MODE","BE");


if (substr($dir,strlen(PATH_site))!="typo3conf/ext/maillisttofaq")    {
        die("Wrong path... This '".substr($dir,strlen(PATH_site))."' should be the last part of '".$dir."'");
}
require(PATH_t3lib."class.t3lib_div.php");
require(PATH_t3lib."class.t3lib_extmgm.php");
require(PATH_t3lib."config_default.php");
if (!defined ("TYPO3_db"))    die ("The configuration file was not included.");

require_once (PATH_t3lib."class.t3lib_readmail.php");

// Connect to the database
$result = @mysql_pconnect(TYPO3_db_host, TYPO3_db_username, TYPO3_db_password);
if (!$result)   {
        die("Couldn't connect to database at ".TYPO3_db_host);
}



// MAIL CONTENT
$filename = "php://stdin";

$content = t3lib_div::getUrl($filename);
if (trim($content))     {
        $readMail = t3lib_div::makeInstance("t3lib_readmail");
                // Split mail into head and content
        $mailParts = $readMail->fullParse($content);

        $query="INSERT INTO tx_maillisttofaq_inmail (mailcontent,from_email,to_email,reply_to_email,sender_email,message_id,subject) VALUES (
                '".addslashes($content)."',
                '".addslashes($mailParts["_FROM"]["email"])."',
                '".addslashes($mailParts["_TO"]["email"])."',
                '".addslashes($mailParts["_REPLY_TO"]["email"])."',
                '".addslashes($mailParts["_SENDER"]["email"])."',
                '".addslashes($mailParts["message-id"])."',
                '".addslashes($mailParts["subject"])."'
                )";
        $res = mysql(TYPO3_db,$query);

The table is defined like this:

CREATE TABLE tx_maillisttofaq_inmail (
        uid int(11) unsigned DEFAULT '0' NOT NULL auto_increment,
        mailcontent mediumblob NOT NULL,
        from_email varchar(80) DEFAULT '' NOT NULL,
        to_email varchar(80) DEFAULT '' NOT NULL,
        reply_to_email varchar(80) DEFAULT '' NOT NULL,
        sender_email varchar(80) DEFAULT '' NOT NULL,
        message_id varchar(80) DEFAULT '' NOT NULL,
        subject tinytext NOT NULL,
        deleted tinyint(4) unsigned DEFAULT '0' NOT NULL,
        PRIMARY KEY (uid)
);

When the mail is read from stdin it is parsed by the class "t3lib_readmail" so that header information like from,to,reply-to and sender addresses are known. Further the message id and subject are extracted. All this information is inserted into the table in the fields with corresponding names (see above).

The "deleted" flag¶

The "deleted" flag is by default zero, but when the frontend plugin reads a mail from the "tx_maillisttofaq_inmail" table (basically the "inbox" of this extension) and inserts it into the "tx_maillisttofaq_ml" table, then the flag is changed to "1".

So all mails in the " tx_maillisttofaq_inmail " table with the deleted flag set to zero has be processed and can be deleted if you like!

Production webserver¶

This server must pull content from the combined mail-/webserver. In TypoScript Setup field you insert this:

plugin.tx_maillisttofaq_pi1.readmail = 1
plugin.tx_maillisttofaq_pi1.readmail.url = http://[mail-web-server-site-url]/1422.0.html?

By default the production server will try and fetch mail content by a probability of 5% (1 out of 20 hits). This is configurable in TypoScript. Likewise you can configure a password and a max number of mails to fetch.

Combined web-/mailserver¶

The webserver with the dummy TYPO3 site having the Mailing List Archive plugin inserted has the URL "http://[mail-web-server-site- url]/1422.0.html?" in this example.

By setting this TypoScript in the Setup field..:

plugin.tx_maillisttofaq_pi1.readmail.enableFeed = 1

... the plugin will feed a data stream with mail records to the production server pulling it from there (instead of displaying the webpage with the archive). You can configure a password as well.

The parameter that triggers this is "&tx_maillisttofaq_pi1[readMails]=1"

Debugging the pulling action¶

This can be done on the production server by using the Admin Panel as an admin-user. Normally you would see a message like this:

Reading mails (pulling) from the combined web-/mailserver was not triggered since the probability for that to happen is 5%

But if it happens (can be configured with "plugin.tx_maillisttofaq_pi1.readmail.probability = 5") then it looks like this:

You can see the URL which was requested and the result below. In this case 5 messages was received and inserted in the local "_inmail" table on the production server.

When such a transfer happens the local reading of mails from the "_inmail" table is also triggered ("Transferring 50 mails..."). Notice that this query also happens once in a while (prob. 10%) on its own. Further the question selects mails from "_inmail" table based on the sender email address, here "typo3-german- admin@lists.netfielders.de". Thus you can see that the extension can receive mails from many mailing lists and split them out to various plugins on the website(s) if they are just configured to select mails from different lists.