CiviCRM

System Configuration

This chapter covers CiviMail system configuration.  This is a complex task which requires system administrator level skills.  Correct configuration is crucial to keep your server off spam lists and black lists.

You will need to be able to change the configuration of your DNS, create email accounts, configure a cron job, read the headers of email messages, and possibly change the configuration of your SMTP server.

In this chapter, we assume you are running CiviCRM on a Linux server and that you are comfortable working with the shell and running some simple commands.  Most of these steps will be similar on other operating systems, but you will need to adapt them to your system and tools. 

The configuration described works fine for mailings to up to about 10,000 people. If you plan on sending email to hundreds of thousands of contacts, you should benchmark several options and consider a dedicated SMTP server. This more complex configuration is outside the scope of this book.

In this chapter we'll use an external Gmail mailbox address to test configuration. So the first step is to create a Gmail account if you don't have one already; alternatively, you can use another address for testing the procedures in this chapter, but you will need to be able to view the source of the mails you receive.

Once your system is properly configured, you are going to run periodically (for instance every 10 minutes) two different programs:

  • EmailProcessor.php to check if you have received new bounces, and flag the invalid contacts

Configuring outbound email

Outbound email setting are configured at: Administer > Configure > Global settings > Outbound Email. The choices here are:

  • mail().  This is the default option and if it works for you, you should use it.
  • SMTP.  If you have a dedicated external mail server, specify its details here. Bounce messages generated with SMTP are slightly more complete than the ones from mail(), but there is no practical benefit to using SMTP if you can use mail().
  • Sendmail.  This option is kept for compatibility with older CiviCRM versions.
  • Disable Outbound Email. Works as expected.

After making a choice, send a test email to your account on Gmail and verify that you receive it.

If you receive the following error message, you'll need to configure a default FROM email address (covered in the chapter on CiviMail configuration).

Sorry. A non-recoverable error has occurred.
The site administrator needs to enter a valid 'FROM Email Address' in Administer ->
 Configure -> Domain Information. The email address used may need to be a valid
 mail account with your email service provider.

Once you have received the email, you will need to view the source.  This is done in Gmail by clicking on "Show original" in the email you receive.

The email should contain headers that resemble the following.

Received: from yourmailserver.example.org (xxx.example.org [12.45.120.30])
        by mx.google.com with ESMTP id e31si4519230wej.3.2010.04.26.00.38.16;
        Mon, 26 Apr 2010 00:38:17 -0700 (PDT)
Received-SPF: pass (google.com: best guess record for domain of
    youremail@example.org designates 12.45.120.30
    as permitted sender) client-ip=12.45.120.30

In particular:

  • The "Received: from" header should correspond to your mail server and be properly configured.  It might contain information about your hosting provider instead of your domain name. This is not a problem as long as the mail server is properly configured.  If you have a dedicated IP address for your server, you should try to configure a reverse DNS that represents your organisation instead of the default name.
  • The "Received-SPF" header should list "pass" or "neutral".  Sender Policy Framework is described later in more detail.

  • Sending mass mailing is resource intensive.  We don't recommend sending email messages from budget hosting providers.  The time you will spend troubleshooting will often cost more than upgrading to a more professional host.  Check with your hosting provider to find out whether they limit the number of email messages you can send and whether they run PHP in safe mode.

    Some of your recipients' mail servers use DNS based blacklisting services (DNSBL) which keep a blacklist of IP addresses likley to send spam.  Mail from these servers will be flagged as spam and not reach its intended destination.  If your server is blacklisted (for instance, because enough of your recipients flagged your email as spam, or because another website on your server has been flagged as spam), you will need to contact the organisations that have blacklisted you and convince them to unlist you.

    They are several websites that help you testing whether you are in a DNSBL. A web search for "blacklisting email" will turn some up. Test regularly to find whether you are on a blacklist.

    Configuring Sender Policy Framework (SPF)

    By default, the Internet allows any mail server to send any email claiming to be from anyone. This makes it easy for spammers to forge addresses and send spam using your email address (or any other). SPF allows you to create a special DNS record listing the IP addresses of the mail servers that can legitimately send email from @yourdomain.org.

    If your domain name already has an SPF record, make sure that it includes the IP address of your CiviCRM mail server (which might be a different from the host used for the web server or from your mail servers), and if it doesn't, add this IP address.

    If you don't have an SPF record, consider adding one. You will need to add at least your mail server and CiviCRM server (if they are different) to the SPF record.

    You can read more about SPF at http://www.openspf.org.

    Configuring the return channel to manage bounces

    You will have contacts that have invalid emails, and CiviCRM can automatically receive the bounced email notifications and flag your contacts accordingly. Based on the type of errors (mail server is unreachable, mailbox full, mailbox doesn't exist, dns error...), CiviCRM will either directly set the contact as on hold or wait until several mailings are bounced with the same error for the same contact.

    You need to set up two things: a mailbox to receive bounced email messages, and a cronjob that will read periodically this mailbox, and update your contacts and your mailing reports in civicrm.

    The bounce email address is an "invisible" email address visible only in the email message's envelope (hidden fields that precede the headers and message added by the user). Choose any name you like that is meaningful to you. In this example we have chosen return, so the email address we need to set up on a mail server for example.org is return@example.org.

    Verify that your account is properly set up by sending a test email from Gmail to the return account.

    CiviCRM handles bounces as follow: for each email sent, a new unique "invisible" sender address is created using the variable envelope return path (VERP). When CiviCRM receives a bounce, it looks at the invisible sender address to decide which email bounced. Contacts will be marked as on hold when their email bounces. Further messages to those addresses won't be sent.

    Setting up an email account to receive bounces

    There are several ways of configuring your bounce mailbox:

    • Sub-addressing: Your mail service might allow you to append a +tag or  -tag qualifier to your e-mail address (e.g., return+test@example.org). Several mail servers, including Gmail, Yahoo! and Postfix provide this sub-addressing by default.

      Try to send yourself an email, adding a +tag or -tag. If you received received the mail you sent with a tag, it means that you can directly use the mailbox you created (return@example.org  in our example)  as the VERP.

    • Catch-all address: If sub-addressing doesn't work on your mail server, you need to define the mail account you created (return@example.org) as the "catch-all" account. Every mail sent to an address that isn't a real mail account will end up there, including all the bounced email messages.

    • External address: If neither of the preceding methods works, consider creating a new account on a service such as Gmail and use it to receive the bounced emails. You will have to set filters in this account so it doesn't discard as spam all the bounced email it will receive.

    Adding the account on CiviMail

    Once you have created your email account to receive the bounces, you need to set up CiviMail so it knows how to read it: Administer > CiviMail > Mail accounts as the default email address.

    • Specify the mail server, username, and password you used when creating the account.
    • If your mail server supports it, specify IMAP and SLL, otherwise POP.
    • You can leave the return path empty.
    • The email domain is the one for your email address (example.org).
    • The local part is the account you created with '+' appended , e.g., "return+".
    • Check the default checkbox. (If not, in Civicrm 3.2.5, CIvimail will ignore this information, construct a Return-Path lacking a domain, generating a 501 error: "recipient is not recognized" from SMTP.

    Once this mailbox is configured, you will need to configure CiviMail to empty it, read all these bounced messages and identify the related bounced contacts. This is performed by the bin/EmailProcessor.php program. We recommend testing the bounce process by running this program directly before setting up CiviCRM to process the bounced email messages automatically. For instance, try entering the following URL into a browser to test the program, substituting the details for your invisible email account:

    http://example.org/sites/all/modules/civicrm/bin/EmailProcessor.php?name=username&pass=
          password&key=your_site_install_key

    Read the chapter on cron for the details of these parameters.

    If CiviCRM can't properly connect to the mailbox, it displays an  error message:

    ezcMailTransportException: An error occured while sending or receiving mail.

    followed by some extra information about the precise problem, such as:

    • Failed to connect to the server
    • The POP3 server did not accept the password: -ERR [AUTH] Username and password not accepted
    • The IMAP server did not accept the password: -ERR [AUTH] Username and password not accepted

    Once you have verified that CiviCRM can properly handle the bounce, you can set it up to automatically process the replies and bounces on a regular basis.

    The different options to set up this periodical task are described on the Scheduling the job section below.

    Sending mass mailings

    Mass mailings are generated via the web interface and queued to be sent by a background cron job, a process that periodically checks whether there are any mailings waiting to be processed. The program that cron runs is bin/civimail.cronjob.php. This section explains how to schedule it to run on a regular basis

    If you need to send some email from CiviCRM right away, without waiting for the cron job, you can trigger the sending process by visiting the http://example.org/civicrm/mailing/queue&reset=1 URL. Use this capability sparingly. It could utilize a lot of server resources and cause CiviCRM to slow down noticeably. The administrative settings for sending email are usually configured to minimize the load on the server, and the cron job is a more efficient way to send mass email.

    Scheduling the job

    To handle both outgoing email and bounced email, you should run a process several times an hour. On Linux and other Unix or Unix-like systems, this is done by configuring a cron job.

    The cron job needs to run using an account recognized by your Drupal or Joomla! server. Create an account dedicated to this task (e.g., mailprocess), give it a long, secure password (e.g.,seol-lzprm42amv-psyc) and grant it access on CiviCRM and CiviMail. Do not change the account password without changing the password in the configuration files of this cron job.

    To set up your cron job, you need to understand how cron works specifically on your CiviCRM server. But as example of setting up cron, you can log in as your dedicated cron user and type the following in the shell:

     crontab -e

    and then enter the following line:

    */5 * * * * php absolute_path_to_civimail.cronjob.php

    Press CTRL-D to save the job and exit crontab. This example runs the PHP file every five minutes.

    The civimail.cronjob.php program has two modes: one for running the file directly from the shell and one for loading the program from the web server. Use the direct shell method whenever you can. But if for technical reasons (not enough access rights, non-working php-cli, etc.) you can't run the programs from the shell, use the web server method.

    From the shell

    First, find out whether php-cli is installed. From the shell, type php -v. If you see (cli) in the result, as in:

    PHP 5.2.3-1ubuntu6.5 (cli) (built: Feb 11 2009 19:55:53)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
    with eAccelerator v0.9.5.3, Copyright (c) 2004-2006 eAccelerator, by eAccelerator

    This means you have php-cli installed and you should use it, because it has several advantages:

    • You can run a PHP script at a lower priority than your web server, so that even if it takes a lot of CPU, it won't interfere with the regular users of your site.
    • You can set different memory limits for the php-cli process and the PHP process used by your web server.
    • You avoid the overhead of the web server and the HTTP layer.
    • You won't have any timeout problems.
    The following is complete cron configuration to handle CiviCRM's mail requirements:
    # This must be set to the directory where civicrm is installed.
CIVI_ROOT=/var/www/civicrm
USER=www-data
MAILTO="you@example.org"
# Location of the PHP Command Line Interface binary. nice -19 forces to run at a
lower priority than the web server
PHP=nice -n19 /usr/bin/php

#line to be modified according to the informations below
PARAMS= -sdefault -umailprocess -pseol-lzprm42amv-psyc
#cronjob send
# m h  dom mon dow   command
*/5 * * * * cd $CIVI_ROOT;$PHP  bin/civimail.cronjob.php $PARAMS
*/15 * * * * cd $CIVI_ROOT;$PHP bin/EmailProcessor.php $PARAMS

    The user that run the scripts (www-data in this example) needs to be able to write into the temporary folder. Your configuration might specify a different user.

    You don't have to run both scripts at the same frequency. The preceding crontab file verifies every 5 minutes whether mail messages need to be sent, but only every 15 minutes whether bounced email needs to be processed.

    PARAMS contains:

    1. The site you used, which is -sdefault on Drupal. If you run multiple CiviCRM sites on a single server, you need to specify your site's domain, such as -sexample.org.
    2. The user login account (-umailprocess).
    3. The password you defined (-pseol-lzprm42amv-psyc).

    From the web server

    You can access the two processes from pages on your web server using commands such as:

    wget -O - -q -t 1  --post-data='name=mailprocess&pass=seol-lzprm42amv-
  psyc&key=yoursiteinstallkey' http://www.example.org/sites/all/modules/civicrm
  /bin/civimail.cronjob.php

wget -O - -q -t 1  --post-data='name=mailprocess&pass=seol-lzprm42amv-
  psyc&key=yoursiteinstallkey' http://www.example.org/sites/all/modules/civicrm
  /bin/EmailProcessor.php

    This works like visiting the web pages in your brower, but can be run automatically as shell commands.

    For security reasons, you need to add an extra key parameter, defined in your civicrm.settings.php file. Read the chapter on the REST interface for more information about this parameter.



your comment:
name :
comment :

If you can't read the word, click here
word :