Email System Configuration
This chapter covers system configuration necessary so that CiviCRM can send and receive email. This is a complex task which requires system administrator level skills. Correct configuration is crucial to keep your server off spam lists and black lists. See Set Up of the Email section for tasks necessary to set up the sending of messages once the server settings have been configured.
Some parts of the configuration are core CiviCRM functionality (basic sending and receiving of emails) whereas others (mass mailings) require that the CiviCRM component be enabled.
You will need to be able to change the configuration of your DNS, create email accounts, configure a cron job, read the headers of email messages, and possibly change the configuration of your SMTP server.This chapter assumes you are running CiviCRM on a Linux server and that you are comfortable working with the shell and running some simple commands. Most of these steps will be similar on other operating systems, but you will need to adapt them to your system and tools.
The configuration described works fine for mailings to up to about 10,000 people. If you plan on sending email to hundreds of thousands of contacts, you should benchmark several options and consider a dedicated SMTP server. This more complex configuration is outside the scope of this book.
In this chapter we'll use an external Gmail mailbox address to test configuration. So the first step is to create a Gmail account if you don't have one already; alternatively, you can use another address for testing the procedures in this chapter, but you will need to be able to view the source of the mails you receive.
Once your system is properly configured, you are going to run periodically (for instance every 10 minutes) two different programs:
- EmailProcessor.php to check if you have received new bounces, and flag the invalid contacts
- civimail.cronjob.php to send all the emails that might be queued for sending.
Configuring outbound email
Outbound email setting are configured at: Administer > Configure > Global settings > Outbound Email. The choices here are:
- mail(). This is the default option and if it works for you, you should use it.
- SMTP. If you have a dedicated external mail server, specify its details here. Bounce messages generated with SMTP are slightly more complete than the ones from mail(), but there is no practical benefit to using SMTP if you can use mail().
- Sendmail. This option is kept for compatibility with older CiviCRM versions.
- Disable Outbound Email. Works as expected.
After making a choice, send a test email to your account on Gmail and verify that you receive it.
If you receive the following error message, you'll need to configure a default FROM email address (covered in the chapter on CiviMail configuration).
Sorry. A non-recoverable error has occurred. The site administrator needs to enter a valid 'FROM Email Address' in Administer -> Configure -> Domain Information. The email address used may need to be a valid mail account with your email service provider.
Once you have received the email, you will need to view the source. This is done in Gmail by clicking on "Show original" in the email you receive.
The email should contain headers that resemble the following.
Received: from yourmailserver.example.org (xxx.example.org [220.127.116.11]) by mx.google.com with ESMTP id e31si4519230wej.3.2010.04.26.00.38.16; Mon, 26 Apr 2010 00:38:17 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of email@example.com designates 18.104.22.168 as permitted sender) client-ip=22.214.171.124
- The "Received: from" header should correspond to your mail server and be properly configured. It might contain information about your hosting provider instead of your domain name. This is not a problem as long as the mail server is properly configured. If you have a dedicated IP address for your server, you should try to configure a reverse DNS that represents your organisation instead of the default name.
- The "Received-SPF" header should list "pass" or "neutral". Sender Policy Framework is described later in more detail.
Sending mass mailing is resource intensive. We don't recommend sending email messages from budget hosting providers. The time you will spend troubleshooting will often cost more than upgrading to a more professional host. Check with your hosting provider to find out whether they limit the number of email messages you can send and whether they run PHP in safe mode.
Some of your recipients' mail servers use DNS based blacklisting services (DNSBL) which keep a blacklist of IP addresses likley to send spam. Mail from these servers will be flagged as spam and not reach its intended destination. If your server is blacklisted (for instance, because enough of your recipients flagged your email as spam, or because another website on your server has been flagged as spam), you will need to contact the organisations that have blacklisted you and convince them to unlist you.
They are several websites that help you testing whether you are in a DNSBL. A web search for "blacklisting email" will turn some up. Test regularly to find whether you are on a blacklist.
Configuring Sender Policy Framework (SPF)
By default, the Internet allows any mail server to send any email claiming to be from anyone. This makes it easy for spammers to forge addresses and send spam using your email address (or any other). SPF allows you to create a special DNS record listing the IP addresses of the mail servers that can legitimately send email from @yourdomain.org.
If your domain name already has an SPF record, make sure that it includes the IP address of your CiviCRM mail server (which might be a different from the host used for the web server or from your mail servers), and if it doesn't, add this IP address.
If you don't have an SPF record, consider adding one. You will need to add at least your mail server and CiviCRM server (if they are different) to the SPF record.
You can read more about SPF at http://www.openspf.org.
Configuring inbound email
This section explains the configuraiton of the handling of incoming emails for 1. handlings bounces or 2. autofiling emails.
CiviCRM can automatically receive the bounced email notifications and flag your contacts accordingly. Based on the type of errors (mail server is unreachable, mailbox full, mailbox doesn't exist, dns error...), CiviCRM will either directly set the contact as on hold or wait until several mailings are bounced with the same error for the same contact.
You need to set up two things: a mailbox to receive bounced email messages, and a cronjob that will read periodically this mailbox, and update your contacts and your mailing reports in civicrm.
The bounce email address is an "invisible" email address visible only in the email message's envelope (hidden fields that precede the headers and message added by the user). Choose any name you like that is meaningful to you. In this example we have chosen return, so the email address we need to set up on a mail server for example.org is firstname.lastname@example.org.
Verify that your account is properly set up by sending a test email from Gmail to the return account.
CiviCRM handles bounces as follow: for each email sent, a new unique "invisible" sender address is created using the variable envelope return path (VERP). When CiviCRM receives a bounce, it looks at the invisible sender address to decide which email bounced. Contacts will be marked as on hold when their email bounces. Further messages to those addresses won't be sent.
CiviCRM can automatically retrieve email from a specified inbox and file it as an email activity against contacts corresponding to sender and recipients of the email. New contacts are created for emails not already in the database.
There are two ways to do this that can be used one or both at the same time.
Option 1) Set up a special email address for your organization, e.g. email@example.com. Users can then add this address in the Bcc field for your outbound emails they will get autofiled in CiviCRM as described above. No one who receives the email will see this special address if the Bcc field is used.
Option 2) Set up a folder in your IMAP Inbox where you can drag emails that you want filed in CiviCRM. This works with both inbound and outbound emails. (Option 2 requires that your email be set up using IMAP.)
See the wiki page for further emails http://wiki.civicrm.org/confluence/display/CRMDOC40/Autofiling+email+activities+via+EmailProcessor.
Types of email accounts for incoming email
There are several ways of configuring your bounce mailbox:
Sub-addressing: Your mail service might allow you to append a +tag or -tag qualifier to your e-mail address (e.g., firstname.lastname@example.org). Several mail servers, including Gmail, Yahoo! and Postfix provide this sub-addressing by default.
Try to send yourself an email, adding a +tag or -tag. If you received received the mail you sent with a tag, it means that you can directly use the mailbox you created (email@example.com in our example) as the VERP.
Catch-all address: If sub-addressing doesn't work on your mail server, you need to define the mail account you created (firstname.lastname@example.org) as the "catch-all" account. Every mail sent to an address that isn't a real mail account will end up there, including all the bounced email messages.
External address: If neither of the preceding methods works, consider creating a new account on a service such as Gmail and use it to receive the bounced emails. You will have to set filters in this account so it doesn't discard as spam all the bounced email it will receive.
Adding an incoming email account for handling bounces or autofiling to CiviMail
Once you have created your email account to receive bounces or emails for autofiling, you need to set up CiviMail so it knows how to read it: Administer > CiviMail > Mail accounts as the default email address.
- Specify the mail server, username, and password you used when creating the account.
- The local part is the account you created with '+' appended , e.g., "return+".
- The email domain is the domain of your email address (example.org).
- You can leave the return path empty.
- If your mail server supports it, specify IMAP protocol and check SLL, otherwise use POP.
- You can speficy an IMAP folder in the source field using the syntax INBOX.CiviMail. Note: Some exchange servers may not be configured in a compatible way. In that case, you can configure a script like fetchmail and use Maildir.
- In the Used for? field you can choose whether you want to use the email account for Bounce Processing or Email-to-Activity Processing (or Autofiling). You can have multiple accounts specified for autofiling but only one for bounce processing. This will be marked as default.
Once the Bounce Processing mailbox is configured, you will need to configure CiviMail to empty it, read all these bounced messages and identify the related bounced contacts. This is performed by the bin/EmailProcessor.php program. We recommend testing the bounce process by running this program directly before setting up CiviCRM to process the bounced email messages automatically. For instance, try entering the following URL into a browser to test the program, substituting the details for your invisible email account:
Read the chapter on cron for the details of these parameters.
If CiviCRM can't properly connect to the mailbox, it displays an error message:
ezcMailTransportException: An error occured while sending or receiving mail.
followed by some extra information about the precise problem, such as:
- Failed to connect to the server
- The POP3 server did not accept the password: -ERR [AUTH] Username and password not accepted
- The IMAP server did not accept the password: -ERR [AUTH] Username and password not accepted
Once you have verified that CiviCRM can properly handle the bounce, you can set it up to automatically process the replies and bounces on a regular basis.
The different options to set up this periodical task are described on the Scheduling the job section below.
Configuring cron jobs for scheduled mass mailings
Mass mailings are generated via the web interface and queued to be sent by a background cron job, a process that periodically checks whether there are any mailings waiting to be processed. The program that cron runs is bin/civimail.cronjob.php. This section explains how to schedule it to run on a regular basis
If you need to send some email from CiviCRM right away, without waiting for the cron job, you can trigger the sending process by visiting the http://example.org/civicrm/mailing/queue&reset=1 URL. Use this capability sparingly. It could utilize a lot of server resources and cause CiviCRM to slow down noticeably. The administrative settings for sending email are usually configured to minimize the load on the server, and the cron job is a more efficient way to send mass email.
Scheduling the job
To handle both outgoing email and bounced email, you should run a process several times an hour. On Linux and other Unix or Unix-like systems, this is done by configuring a cron job.The cron job needs to run using an account recognized by your Drupal or Joomla! server. Create an account dedicated to this task (e.g., mailprocess), give it a long, secure password (e.g.,seol-lzprm42amv-psyc) and grant it access on CiviCRM and CiviMail. Do not change the account password without changing the password in the configuration files of this cron job.
To set up your cron job, you need to understand how cron works specifically on your CiviCRM server. But as example of setting up cron, you can log in as your dedicated cron user and type the following in the shell:
and then enter the following line:
*/5 * * * * php absolute_path_to_civimail.cronjob.php
Press CTRL-D to save the job and exit crontab. This example runs the PHP file every five minutes.
The civimail.cronjob.php program has two modes: one for running the file directly from the shell and one for loading the program from the web server. Use the direct shell method whenever you can. But if for technical reasons (not enough access rights, non-working php-cli, etc.) you can't run the programs from the shell, use the web server method.
From the shell
First, find out whether php-cli is installed. From the shell, type php -v. If you see (cli) in the result, as in:
PHP 5.2.3-1ubuntu6.5 (cli) (built: Feb 11 2009 19:55:53) Copyright (c) 1997-2007 The PHP Group Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies with eAccelerator v0.9.5.3, Copyright (c) 2004-2006 eAccelerator, by eAccelerator
This means you have php-cli installed and you should use it, because it has several advantages:
- You can run a PHP script at a lower priority than your web server, so that even if it takes a lot of CPU, it won't interfere with the regular users of your site.
- You can set different memory limits for the php-cli process and the PHP process used by your web server.
- You avoid the overhead of the web server and the HTTP layer.
- You won't have any timeout problems.
# This must be set to the directory where civicrm is installed. CIVI_ROOT=/var/www/civicrm USER=www-data MAILTO="email@example.com" # Location of the PHP Command Line Interface binary. nice -19 forces to run at a lower priority than the web server PHP=nice -n19 /usr/bin/php #line to be modified according to the informations below PARAMS= -sdefault -umailprocess -pseol-lzprm42amv-psyc #cronjob send # m h dom mon dow command */5 * * * * cd $CIVI_ROOT;$PHP bin/civimail.cronjob.php $PARAMS */15 * * * * cd $CIVI_ROOT;$PHP bin/EmailProcessor.php $PARAMS
The user that run the scripts (www-data in this example) needs to be able to write into the temporary folder. Your configuration might specify a different user.
You don't have to run both scripts at the same frequency. The preceding crontab file verifies every 5 minutes whether mail messages need to be sent, but only every 15 minutes whether bounced email needs to be processed.
- The site you used, which is -sdefault on Drupal. If you run multiple CiviCRM sites on a single server, you need to specify your site's domain, such as -sexample.org.
- The user login account (-umailprocess).
- The password you defined (-pseol-lzprm42amv-psyc).
From the web server
You can access the two processes from pages on your web server using commands such as:
wget -O - -q -t 1 --post-data='name=mailprocess&pass=seol-lzprm42amv- psyc&key=yoursiteinstallkey' http://www.example.org/sites/all/modules/civicrm /bin/civimail.cronjob.php wget -O - -q -t 1 --post-data='name=mailprocess&pass=seol-lzprm42amv- psyc&key=yoursiteinstallkey' http://www.example.org/sites/all/modules/civicrm /bin/EmailProcessor.php
This works like visiting the web pages in your brower, but can be run automatically as shell commands.
For security reasons, you need to add an extra key parameter, defined in your civicrm.settings.php file. Read the chapter on the REST interface for more information about this parameter.