The API

CiviCRM has a stable comprehensive API (Application Programming Interface) that can be used to access much of the functionality of CiviCRM. The API is the recommended way for any external programme to interact with CiviCRM.

Why use the API?

If you're not familiar with APIs, you might ask why you would use the API when you could access the core functions (e.g. the BAO files) directly?

The answer is that the API is guaranteed to remain stable between releases, and warn developers well in advance of any changes, where as BAOs offer no such guarantee.

Version 3 is one of the seven wonders of CiviCRM.  It is much more consistent than version 2 and has much improved test coverage: it should be used for all new projects.

Learning about (and experimenting with) the API V3

The best place to learn about the API is your own test install of CiviCRM.  Be wary about using the API on your live site because the the API does not ask for confirmation, and therefore you can easily make significant changes to your data.

Each install comes with two tools that are useful for this.  The API parameter list, which shows all available entities which can be manipulated by the API and is available at http://[CIVICRM_URL]/civicrm/api/doc, and the API explorer, which is available at http://[CIVICRM_URL]/civicrm/api/explorer.

How to use the API

There are at least 5 different ways of using an API function:

1. as a php function, to run your own code on the same server as CiviCRM 
   
2. via the AJAX interface, to be called from JavaScript code
3. via the REST* interface, can be called from another server via http calls
4. as a Smarty function to add data to templates 
   
5. from drush on the command line for Drupal installations. 
   

Calling the API in PHP 

If you write custom code that is going to run within the same environment as CiviCRM, calling the API as a PHP function is the recommended way of modifying or fetching data. You will see examples of API calls throughout the developer documentation. For example, you could create several tags from a script rather than doing it manually from the CiviCRM admin page. The code below shows you how to do it using the API. 

The recommended way to call the API in PHP is

require_once 'api/api.php';
$result = civicrm_api ($entity,$action,$params); 

where $entity is the object you want to manipulate (a contact, a tag, a group, a relationship), and$action is get (to retrieve a list or a single entity), create (to create or update if you provide the id), delete, update and getFields (that provide the list of fields of this entity, including the custom fields).

So with real data:

require_once 'api/api.php';
$contact = civicrm_api('Contact','Get',array('first_name' => 'Michael', 'last_name' => 'McAndrew', 'version' =>3)); 

Calling the API from smarty

CiviCRM defines a smarty function {crmAPI} which can be used to call the API from the template layer.

{crmAPI var="ContactS" entity="Contact" action="get" version="3" first_name="Michael" first_name="McAndrew" }
{foreach from=$ContactS.values item=Contact}
  <li>{$Contact.example}</li>
{/foreach} 

Calling the API via javascript

It also defines a javascript function crmAPI() which can be used to call the API from javascript. 

 $().crmAPI ('Contact','get',{'version' :'3', 'first_name' :'Michael', 'first_name' :'McAndrew'}}
  ,{ success:function (data){    
       $.each(data, function(key, value) {// do something  });
     }
 });

Calling the API via Ajax or REST

You can also call the API via Ajax or REST.  Have a look at the api explorer for examples of how you call the API with these methods.

Calling the API from an external server via the REST API

The syntax and principle are identical to the other methods, but with one major difference: the external server needs to authenticate itself before being able to use any API functions.

To call the API with a browser, use:

http://www.example.org/path/to/civi/codebase/civicrm/extern/rest.php?q=civicrm/<function>

The first call is to authenticate the caller on the CiviCRM server:

https://eg.org/path/to/civi/codebase/civicrm/extern/rest.php?q=civicrm
   /login&name=user&pass=password&key=yoursitekey&json=1

The key is in the CIVICRM_SITE_KEY variable defined in your civicrm.settings.php file.

Note: On the first call, you might get an error message like "This user does not have a valid API key in the database, and therefore cannot authenticate through this interface". This means that you need to generate a key for the user, and add it to the api_key field in the civicrm_contact table in the database.

{"is_error":0,"api_key":"as-in-your-setting.civicrm.php",
   "PHPSESSID":"4984783cb5ca0d51a622ff35fb32b590",
   "key": "2e554f49c9fc5c47548da4b24da64681b77dca08"}

It returns a session id. You can then use any API adding the extra param PHPSESSID = the value returned by the log-in call.

For example:

http://www.example.org/civicrm/ajax/rest?fnName=civicrm/contact/search&json=1&key=
yoursitekey&PHPSESSID=4984783cb5ca0d51a622ff35fb32b590

Note: An action that modifies the database (such as creating a contact or group) should have the parameters passed as POST, not GET. The REST interface accepts both a GET and a POST, but you should follow the web standard and use POST for things that alter the database. You'll win the respect of your peers and the admiration of your friends and family.

Autocomplete and search contacts (API v2)

Note that the instructions below are for API v2.  

If you create a profile for individual contacts that contains the current employer, you might want to add an autocomplete on that field, such as you have on the normal contact edit form. When a user starts to type the name of the current employer, the system will attempt to autocomplete the name from your database of organisation contacts. 

For security reasons, the Ajax interface is restricted to users who have access to CiviCRM - otherwise, it would be fairly easy for anyone to download all the contacts you have in your database. So that's the first thing we check for here: 

{if $session->get('userID') > 0}

<script type="text/javascript" src="/civicrm/{$config->resourceBase}js/rest.js"></script>{literal}
<script>
jQuery(document).ready(function($){
  $('#current_employer').crmAutocomplete({params:{contact_type:'Organization'}});
});
</script>
{/literal}

{/if}

You might want to add additional filters. For instance in a profile "new volunteer from a member", you want to populate the list only with the organisations that belong to the group "members" (group id 42).

$('#current_employer').crmAutocomplete({params:{contact_type:'Organization',group:42}});

Extending the API

It should be noted that it's extremely easy to extend the api if you want to add some funky features: add a Magic.php file under api/v3, create a function civicrm_magic_getpony and voila, you have civicrm_api ("magic","getpony"), the smarty, the ajax, the REST, and it's also directly usable from the api explorer at /civicrm/ajax/doc 

BASIC SET UP

When writing CiviCRM customizations and extensions make sure you don't do it in a production environment! You don't want to interrupt your organisations or users' work or corrupt the data in the database. Instead you should always create a separate local development environment where you can do whatever you want to CiviCRM without disturbing anyone else.

If your computer runs Linux or Mac OS X, running CiviCRM on your local machine is pretty easy. If your computer runs Windows, you have a little more work to do, but it's not impossible - you could consider running Linux inside a virtual machine. There's a free program called VirtualBox that makes this pretty easy. We recommend running the latest version of Ubuntu Linux inside the virtual machine. You can download that here: http://www.ubuntu.com/

Installing CiviCRM

http://wiki.civicrm.org/confluence/display/CRMDOC40/Installation+and+Upgrades outlines pre-installation requirements and gives step by step instructions for installing CiviCRM.

Installing from SVN

CiviCRM developers use SVN to collaborate.  If you are doing a lot of development in collaboration with the core team or other developers, you might want to install from SVN.

Instructions on installing from SVN are available here:http://wiki.civicrm.org/confluence/display/CRMDOC40/Installing+CiviCRM+from+subversion+%28SVN%29+repository

Using source control management

It is good practice to put any code customisations that you write under version control.  Keeping your code in a repository means that you have access to all previous versions of the code and can revert back if anything goes wrong (which with any complex project, will inevitably happen one day).  Although CiviCRM uses SVN, we recommend you consider using Git for any code that you write.  CiviCRM will migrate to Git at some point in the future.

Putting your source code in a public repository is even better as it makes it public from day one and easy to share.

Logging emails to a file

If you have a local test environment set up and want to test processes that involve sending emails, you can set CiviCRM up to log emails to a file rather than sending them.  To do this, modify your CiviCRM settings (http://civicrm.settings.php) and add following line:

define('CIVICRM_MAIL_LOG', '/path/to/mail.log');

Developing Custom Searches

A custom search is a method of providing new functionality within the standard CiviCRM navigation structure. This chapter looks at how to develop a Custom Search.

Custom searches produce a screen showing a set of contacts, from where you can execute actions such as sending email, printing mailing labels, and all the other actions that are available for contact search results within CiviCRM. The results page displays the same as any other search results page, such as results delivered from an Advanced Search, however, a predefined set of functions controls which information is delivered to the result page.

Custom Searches follow the Hollywood principle, "Don't call me, I'll call you." In this case CiviCRM calls your functions at the appropriate time.

When to use Custom Searches

A custom search is the right choice when ...

• you need to access the Actions list after running your search, to send email, add the contacts to a group or other actions (since the list of actions and list of tokens can be extended with developer-created hooks, the combination of a custom search + custom action + custom token is a powerful tool for implementing a special requirement. If you are interested in creating new actions or tokens, read the section on CiviCRM hooks in the chapter Extending CiviCRM.)
  
• you want to create a Smart Group based on the parameters of the custom search
• you want to use the search results to drive a mass mailing in CiviMail
• you want results that can be sorted by clicking any column heading in the results page.

A custom search is the wrong choice when ...

• you can use the Advanced Search or the Search Builder to create the same results
• the information you need is not primarily accessible from the CiviCRM database tables (for example, the information needs to be retrieved from a third-party system or a different database)
• the information you need does not include the CiviCRM contact ID (for example, information related to summarised event income). 
  

CiviReport is a better choice if ...

• you need to schedule and send the entire result as an email to one or more people
• you need a summary/detail drill-down style of results page where the detail is not just the contact summary screen
• you want to use the results as a dashlet on your CiviCRM dashboard
• you need multiple report break areas within the results page, such as subtotals after every 5 records as well as grand totals
• you need to include information not related to contacts.
  

Getting started creating a new custom search

In this section we will create a new custom search called "BirthdaySearch" that will find all contacts whose birthdays fall in June. 

Custom searches are written using PHP and SQL. Very little knowledge of PHP is needed, as you start with a template file and only make minor changes. In many cases the only changes are the SQL Select statement and which columns to display in the results.

Plan and test

Before writing the code, it is important to plan and test the SQL query and verify the results. It is valuable at this stage to review the database tables and test the SQL select statements within the database using an SQL tool such as PHPMyAdmin.

It may be helpful for you to review the information at:

• Useful CiviCRM SQL Queries: http://wiki.civicrm.org/confluence/display/CRMDOC/Useful+SQL+queries
• CiviCRM Data Architecture: http://wiki.civicrm.org/confluence/display/CRMDOC/CiviCRM+data+architecture 
• CiviCRM wiki: http://wiki.civicrm.org/confluence/display/CRM/CiviCRM+Wiki

Important Note

It is important to include the contact_id field from the table civicrm_contact in your select statement. Even if you do intend to display the contact_id field, include it. The reason is that the view and edit links need the value for the cid attribute of the url. Without a value, you can not edit or view information about the contact. Follow the select statement below for guidance.

Setting the filepath

Your custom search files can be stored almost anywhere, but you must tell CiviCRM where these files are. 

1. Within CiviCRM, go to: Administer > Configure > Global Settings > Directories
2. Fill in the Custom PHP Path Directory. This needs to be an absolute path to your PHP directory, such as /home2/jsmith/public_html/civicrm_custom_code/
   
3. Create a copy of an existing custom search file, such as the EventAggregate.php file. You will find this file at: 
   <joomla root>/administrator/components/com_civicrm/civicrm/CRM/Contact/Form/Search/Custom  
   or
   <drupal root>/sites/all/modules/civicrm/CRM/Contact/Form/Search/Custom
4.  Place the EventAggregate.php file in the directory: /home2/jsmith/public_html/civicrm_custom_code/CRM/Contact/Form/Search/Custom
5. Rename the copied EventAggregate.php file to BirthdaySearch.php

Understanding and updating the search code

Start by opening the file BirthdaySearch.php in a text editor. The first change needed is to change the class declaration to:

class CRM_Contact_Form_SearCustom_UpcomingBirthdays
   implements CRM_Contact_Form_SearInterface {
  

Functions you will probably need to modify:

• function_construct: this controls the columns that are part of the results page.

• function all: this function is responsible for returning the SQL select statement to execute that gets the entire set of information. The select statement MUST include a field named "contact_id". Normally it also contains fields named "name" and "sort_name". For example:

if ( $onlyIDs ) {
     $select  = "DISTINCT civicrm_contact.id as contact_id,
      civicrm_contact.display_name as name";
    } else {
    $select = "DISTINCT civicrm_contact.id as contact_id, CONCAT(
     monthname(civicrm_contact.birth_date) , ' ', day(civicrm_contact.birth_date))
     as sort_name , civicrm_contact.display_name as name, 'birthday' as oc_type" ;

    }
   
    $from  = $this->from( );
    $where = $this->where( $includeContactIDs ) ;
    $sql = "SELECT $select FROM  $from WHERE $where ";

⁞

• function from: this function is responsible for returning the from clause of the SQL select statement. It is normally called from the "all" function as well as by CiviCRM.
  

• function where: this function is responsible for returning the where clause of the SQL select statement.
  

• function buildForm: this controls the results title as well as the parameters that are available to the person running the search. For this birthday search, you may want the user to be able to choose the month.   

Functions you may need to modify:

• function summary: this function is needed if some or all columns have summary information, such as total number of birthdays in June.

• function alterRow: this function allows you to alter the contents of a piece of information before the results are displayed.

• templateFile: this function returns the name of the Smarty template that CiviCRM will use to present the results. For most purposes, the template CRM/Contact/Form/Search/Custom/Sample.tpl will suffice. 

Prepare to run the custom search

Before you can run the custom search, CiviCRM needs to be informed that it exists. This is accomplished by the following steps:

1. Go to: Administer > Customize > Manage Custom Searches.
2. Scroll to the bottom of the page and click the button New Custom Search.
3. Provide the class path as: CRM_Contact_Form_SearCustom_BirthdaySearch
4. Provide the title as: Birthday Search

The new Birthday Search should now appear in (and can be run from) the list of custom searches in the navigation menu. It will also appear on the page reached by going to Search > Custom Searches.

The new custom search will not appear in the black navigation menu unless the navigation menu is edited.  This can be done by going to Administer > Customize > Navigation Menu.

Testing the custom search

 Always test the following behaviors of the new search:

• Test for a variety of form values, especially for invalid data. Errors in validation can lead to serious security breaches. Just because there is a drop-down list of valid months, do not assume that only valid months are passed to your custom search. Also test for values with apostrophes and other special characters.
  
• Test the previous and next page links for a variety of different size results. 
•  Test the first/last page links for a variety of different size results. 
•  Make a Smart Group from it and send a CiviMailing to that group.
•  Test other actions in the Action menu such as Send an Email, or create PDF letters with mail merge tokens.

<?xml version="1.0" encoding="UTF-8" ?>
<extension key="org.civicrm.activity" type="search">
  <callback>ActivitySearch</callback>
  <name>Activity Search</name>
  <description>
   This custom search allows to search through activities
   and returns activities as results (not contacts as
   regular search).
  </description>
  <url>http://civicrm.org</url>
  <license>AGPL</license>
  <maintainer>CiviCRM Core Team &lt;noreply@civicrm.org&gt;</maintainer>
  <releaseDate>2010-09-01</releaseDate>
  <version>1.0</version>
  <compatibility>
   <ver>3.3</ver>
   <ver>3.4</ver>
  </compatibility>
  <develStage>beta</develStage>
  <comments>For support, please contact project team on the forums. (http://forum.civicrm.org)</comments>
</extension>

org.civicrm.activity/
|-- ActivitySearch.php
|-- README.txt 
|-- templates
|   `-- ActivitySearch.tpl
`-- info.xml

DEBUGGING

When your code isn't doing what you want it to do, it's time to debug.  There are lots of options for debugging and there is lots you can do without setting up a sophisticated debugging environment.  This chapter contains some simple debugging tips and tricks to get you started and also instructions on setting up XDebug, which is the recommended debugging tool for CiviCRM when you have bugs which you are finding it really hard to squish.

Printing PHP variables

CRM_Core_Error::debug($name, $variable = null, $log = true, $html = true); can be called to print any variable. It is a wrapper around print_r($variable); which does some clever stuff, but print_r($variable); is often all you need.

Following your print_r(); with and exit; to stop the script execution at that point is often useful or necessary.

Debug mode

CiviCRM has a debug mode which you can enable to give you quick access to a couple of useful diagnostic tools, including access to all the smarty variables that make up a page, and access to the backtrace (read more about backtrace below). It also provides shortcut methods to empty the file based cache and session variables.

These tools are activated by adding parameters to the URL that makes up the page, e.g. &backtrace=1. Go to Admin > Global config > Debugging to enable debug mode and read more about the available options.

Printing smarty variables

When debug mode is enabled, you can display all the smarty variables that have been added to the page by adding  &smartyDebug=1 to any URL.

Backtrace

A backtrace is a list of all the functions that were run in the execution of the page, and the php files that contain these functions.  It can be really useful in understanding the path that was taken through code, what gets executed where, etc.

If display backtrace is enabled in Debugging options then the backtrace will be displayed when an error occurs.

You can also force the backtrace to be printed at any point in the code by adding a call to "CRM_Core_Error::backtrace();"

Clearing the cache

Using Drupal, you can clear all caches with the drush command civicrm-cache-clear.

Alternatively, you can call the following two methods:

• CRM_Core_Config::clearDBCache();
• CRM_Core_Config::cleanup();

which clear the database and file cache respectively.

Check the queries fired by Dataobject

define( 'CIVICRM_DAO_DEBUG', 1 );

XDebug

XDebug is our main recommendation for developers that want to go into hardcore debugging. Readers familiar with what a debugger is and how it works should feel free to skip ahead to the "Setting Up XDebug" section.

What is a debugger?

A debugger is a software program that watches your code while it executes and allows you to inspect, interrupt, and step through the code. That means you can stop the execution right before a critical juncture (for example, where something is crashing or producing bad data) and look at the values of all the variables and objects to make sure they are what you expect them to be. You can then step through the execution one line at a time and see exactly where and why things break down. It's no exaggeration to say that a debugger is a developer's best friend. It will save you countless hours of beating your head against your desk while you insert print statements everywhere to track down an elusive bug.

Debugging in PHP is a bit tricky because your code is actually running inside the PHP interpreter, which is itself (usually) running inside a web server. This web server may or may not be on the same machine where you're writing your code. If you're running your CiviCRM development instance on a separate server, you need a debugger that can communicate with you over the network. Luckily such a clever creature already exists: XDebug.

Setting Up XDebug

XDebug isn't the only PHP debugger, but it's the one we recommend for CiviCRM debugging.

The instructions for downloading and installing XDebug are here: http://xdebug.org/docs/install

Those instructions are a bit complex, however. There is a far simpler way to install it if you happen to be running one of the operating systems listed here.

Debian / Ubuntu Linux

sudo apt-get install php5-xdebug

Red Hat / CentOS Linux

sudo yum install php-pecl* php-devel php-pear

sudo pecl install Xdebug

Mac OS X

sudo port install php5-xdebug

Next Step for All Operating System

Tell XDebug to start automatically (don't do this on a production server!) by adding the following two lines to your php.ini file (your php.ini file is a php configuration file which is found somewhere on your server.  Calling the phpinfo() function is probably the  easiest way to tell you where this file is in your case.

xdebug.remote_enable = On
xdebug.remote_autostart = 1

Once XDebug is installed and enabled in your PHP configuration, you'll need to restart your web server.

Installing an XDebug Front-End

After you have XDebug running on your PHP web server, you need to install a front-end to actually see what it is telling you about your code. There are a few different options available depending on what operating system you use:

All Operating Systems

NetBeans is a heavyweight Java IDE (Integrated Development Environment). It offers lots of features, but isn't exactly small or fast. However, it is very good at interactive debugging with XDebug. And since it's written in  Java, it should run on any operating system you want to run it on. You can find it at http://www.netbeans.org/

After installing NetBeans, open your local CiviCRM installation in NetBeans and click the Debug button on the toolbar. It will fire up your web browser and start the debugger on CiviCRM. You may went to set a breakpoint in CRM/Core/Invoke.php to make the debugger pause there. For more information, see the NetBeans debugging documentation.

Mac OS X

A much lighter-weight option for Mac users is a program called MacGDBp. You can download it here: http://www.bluestatic.org/software/macgdbp/

After installing MacGDBp, launch it and make sure it says "Connecting" at the bottom in the status bar. If it doesn't, click the green "On" button in the upper-right corner to enable it. The next time you access CiviCRM, the web browser will appear to hang. If you click on MacGDBp, you'll see that it has stopped on the first line of code in CiviCRM. From there you can step through the code to the part you're interested in. But it's probably a better idea to set a breakpoint in the part of the code you're interested in stopping at. See the MacGDBp documentation for more information.

EXTENSIONS

CiviCRM has an extensions framework that allows you share your custom searches, reports and payment processors with the community. Before you being work on writing an extension make sure someone else hasn't already contributed something similar by checking the list of extensions on http://directory.civicrm.org/.

How to create an Extension 

Before you start, make sure your extensions are enabled! You need to configure a directory, where your extensions will be stored. To do that, you need to go to: "Administer -> Configure -> Global Settings -> Directories", fill in "CiviCRM Extensions Directory" field and click "Save" at the bottom of the page. It's best to choose a directory outside of $civicrm_root, to avoid potential problems during upgrade. Then follow the seven steps below.

1. Choose a unique key for your extension

Every extension has a unique name called the extension key. It's built using Java-like reverse domain naming to make it easier with identifying unique extensions and giving developers more flexibility on providing different extensions. So for example, if your website is civiconsulting.com and you're developing custom search showing only event payments, you can choose: com.civiconsulting.search.eventregistration as the key.

2. Create an extension directory

Go to your extensions directory and create the directory named after your key. This will be the place where you'll put all the extension related files. When you're done, it will also be put into the zip archive for further distribution.

3. Create the info.xml file

Now you need to create a file which will describe your extension so we know what a given extension does, who wrote it, who supports it and quite a few other things. It needs to be called info.xml. It's in XML format and it's relatively simple. Here's an example:

<?xml version="1.0" encoding="UTF-8" ?>
<extension key="org.civicrm.activity" type="search">
  <callback>ActivitySearch</callback>
  <name>Activity Search</name>
  <description>
   This custom search allows to search through activities
   and returns activities as results (not contacts as
   regular search).
  </description>
  <url>http://civicrm.org</url>
  <license>AGPL</license>
  <maintainer>CiviCRM Core Team &lt;noreply@civicrm.org&gt;</maintainer>
  <releaseDate>2010-09-01</releaseDate>
  <version>1.0</version>
  <compatibility>
   <ver>3.3</ver>
   <ver>3.4</ver>
  </compatibility>
  <develStage>beta</develStage>
  <comments>For support, please contact project team on the forums. (http://forum.civicrm.org)</comments>
</extension>

Here's a quick description of the XML elements:

• extension - enclosing element: everything needs to sit inside of it. Attributes are:
  • key: unique name of the extension (should match the name of directory this extension resides in). We're going with Java like reverse domain naming to make it easier with identifying unique extensions and giving developers more flexibility on providing different extensions.
  • type: one of "search", "payment", "report" for now, meaning that this extension is - respectively - a custom search, payment processor or custom report.
• callback - the name of the class that should be called in order to run this extension. Following the namespacing convention in CiviCRM, it will also be the name of the php file. Above example means that the file $extension_dir/search/ActivitySearch.php should exist and contain Extension_SearActivitySearch class. Extension_Search part of the namespace is obligatory, the rest is under extension author's control.
• name - well, that one's easy. It's a name of the extension.
• description - easy as well.
• url - address to extensions website with documentation, more information, etc.
• license - the name of the license under given extension is offered.
• mainainer - self explanatory, hopefully. Email address inside required.
• releaseDate - date when given extension has been released on.
• version - version of the extension.
• develStage - if you want to push out you extension in unstable state, here's the opportunity to be sincere about it. Allowed values are: "alpha", "beta", "stable". Beta is the default if you don't define develStage - 'cause you must have been unsure, if you forgot to provide such crucial piece of information. :-)
• compatibility, ver - lists compatible versions. It cares only about first two elements of version name and ignores the rest to avoid the need for frequent updating. So if your extension is compatible with CiviCRM 3.3, also means it supports 3.3.beta4. However, if you state it supports 3.3.beta4 (<compatibility><ver>3.3.beta4</ver></compatibility>), it won't support any other version - no go on 3.3.1.
• comments - last one, the opportunity to say anything you want. :-)

Since we love things to be localised, we've added that too and you can provide the extension description in other languages - the optional localisedInfo section of the info file does that:

<extension ...>
    ...regular stuff
    <localisedInfo>
        <pl_PL>
            <description>Opis po polsku.</description>
            <maintainer>Zespół CiviCRM &lt;noreply@civicrm.org&gt;</maintainer>
            <comments>Wsparcie dla tego rozszerzenia dostępne na forum CiviCRM (http://forum.civicrm.org).</comments>
        </pl_PL>
        <pt_BR>
            ...
        </pt_BR>
    </localisedInfo>
</extension>

Please note though, this only supports name, description, maintainer and comments sections - all other extension information defined here will be ignored.  

4. Develop your extension!

Depending on the type of extension that you're building, you will need to follow different instructions. Please refer to separate chapters for:

• Creating A Custom Search Extension
• Creating A Report Template Extension
• Creating A Payment Processor Extension

5. Test your extension

Make sure your extension works as planned! Once you have the info.xml file, you will be able to turn the extension on and off in the Manage CiviCRM Extensions screen - use that to see what errors you're getting and react accordingly.

6. Package your extension

Once you're done with developing and testing, you need to put all the contents of your extensions directory into a zip file. 

7. Submit your extension for public distribution

We are working on the extension submission system, but in the meantime, just get in touch with us via info@civicrm.org  with your extension or on the extensions board of the developer forum. We'll then do some testing and if everything works fine, we'll put your extension into the public distribution. Once there, everyone will be able to install it once they get to "Manage CiviCRM Extensions" screen.

BEFORE YOU START

Before you start on any code to extend CiviCRM, it is really important to discuss your ideas with the community. Here are a few of the reasons why this is a good idea:

• It may have been done already
• You'll get suggestions and advice on suitable ways to approach the problem
• Other people will know what you are doing, and be able to contact you if they want to collaborate

A typical pre-development workflow will start with a discussion on the forum about what you want to do. Here you'll receive feedback from other members of the community and core team about your ideas. You might be lucky and find out that there is a already a way to do what you want using the user interface (and that no coding is necessary). Or it might be that someone has done something similar already and all that is required is a few tweaks to their code. 

If and when you have confirmed that some coding is required, it is good practice, even for relatively small projects, to write

• a requirements document which describes in detail what you want the finished code to do
• a specification that outlines how you are going to meet these requirements with CiviCRM

The requirements are typically written to be understandable to end users, and the specification can be thought of as a translation of those requirements into the language of CiviCRM. Both requirements and specification should go on the wiki.

Once you've written the requirements and specification document, you should go about soliciting feedback.  Get feedback on the requirements from your end users and feedback on the requirements and the specification from anyone that is willing to comment. To encourage more discussion, you can write a post on CiviCRM's blog, tweet it out with the #civicrm hashtag, tell similar CiviCRM users and organisations and so on.  The more feedback you can get the better.

If you get stuck writing your requirements and specification, or would like to get more background, have a look at some requirements and specifications written by others - there are plenty on the wiki.

Make it happen

If you need or would like extra resources for your code improvement, you should consider a 'make it happen' (MIH for short).  

Make it happen is a crowd-sourcing initiative for CiviCRM, which incidentally, is built using CiviCRM.  Around 15 MIH's were selected for the 4.0 release, and more Make it Happens are likely to be selected for future releases.  MIH work is carried out by the core team or trusted consultants.  You can see a list of current MIH online at http://civicrm.org/mih.  If you think your project would make a good MIH, discuss it with the core team.

Hooks

Hooks are a common way to extend systems. The way they work in CiviCRM is that, at key points in processing - such as saving a change or displaying a page - CiviCRM checks to see whether you've "hooked in" some custom code, and runs any valid code it finds.

For example, let's say you want to send an email message to someone in your organization every time a contact in a particular group is edited. Hooks allow you to do this by defining a function with a specific name and adding it to your organisation's CiviCRM installation. The name of the function indicates the point at which CiviCRM should call it. CiviCRM looks for appropriate function names and calls the functions whenever it performs the indicated operations.

Hooks are a powerful way to extend CiviCRM's functionality, incorporate additional business logic, and even integrate CiviCRM with external systems. Many CiviCRM developers find themselves using them in nearly every customization project.

A good test for whether or not to use a hook is to ask yourself whether what you're trying to do can be expressed with a sentence like this: "I want X to happen every time someone does Y."

How to use hooks

How you use hooks depends on whether you're using CiviCRM with Drupal or Joomla!.

Using hooks with Drupal

Check the CiviCRM wiki page for the most up-to-date information on setting up hooks with Drupal: http://tiny.booki.cc/?hooks-in-drupal http://wiki.civicrm.org/confluence/display/CRMDOC/CiviCRM+hook+specification#CiviCRMhookspecification-Proceduresforimplementinghooks%28forDrupal%29  

In order to start using hooks with a Drupal-based CiviCRM installation, you or your administrator needs to do the following:

1. Create a file with the extension .info (for instance, myhooks.info) containing the following lines. Replace the example text in the first 2 lines with something appropriate for your organization:
   

   name = My Organization's Hooks
   description = Module containing the CiviCRM hooks for my organization
   dependencies[] = civicrm
   package = CiviCRM
   core = 6.x
   version = 1.0
   

2. Create a new file with the extension .module (for instance, myhooks.module) to hold your PHP functions.
3. Upload both the .info and .module files to the server running CiviCRM, creating a new directory for them under /sites/all/modules (for instance, /sites/all/modules/myhooks/) inside your Drupal installation. The directory name you create should be short and contain only lowercase letters, digits, and underlines without spaces.
4. Enable your new hooks module through the Drupal administration page.

Using hooks with Joomla!

Check the CiviCRM wiki for the most up-to-date information on setting up hooks with Joomla!: http://tiny.booki.cc/?hooks-in-joomla (http://wiki.civicrm.org/confluence/display/CRMDOC/CiviCRM+hook+specification#CiviCRMhookspecification-Proceduresforimplementinghooks%28forJoomla%29)

In order to use hooks with Joomla!, you or your administrator needs to do the following:

1. Create a file named civicrmHooks.php to contain your functions.
2. Create a new directory anywhere on your server outside of your Joomla! / CiviCRM installation (so that upgrades will leave your hooks alone). /var/www/civicrm_hooks might be a good choice.
3. Upload the civicrmHooks.php file to the directory you just created.
4. Go to: CiviCRM Administer > Global Settings > Directories. Change the value of Custom PHP Path Directory to the absolute path to the new directory (e.g., "/var/www/civicrm_hooks" if you used that suggestion in the earlier step).

Refine what you want to act upon

When you create a hook, it will be called for all the types of entities. For instance, a civicrm_post is called after the creation or modification of any object of any type (contact, tag, group, activity, etc.). But usually, you want to launch an action only for a specific type of entity.

So a hook generally starts with a test on the type of entity or type of action. For instance, if you want to act only when a new individual contact is created or modified(does this match the code?), start your civicrm_post hook with:

if ($objectName != "Individual" || $op != "edit") {
  return;
}

On the other hand, if you want to run multiple hooks within the same function, you don't want to return from any single hook. Instead, you can nest the entire code for each hook within your test:

if ($objectName == "Individual" && $op == "edit") {
  // Your hook
}

Pitfalls of hooks

Because you have little control over what CiviCRM passes to your hook function, it is very helpful to look inside those objects (especially $objectRef) to make sure you're getting what you expect. A good debugger is indispensable here. See the Developer Tips & Tricks chapter at the end of this section for more information on setting up a debugger for your development environment. 

Examples of using hooks

Some example hooks follow. Consult the hooks reference documentation on the CiviCRM wiki to see the full extent of what you can do: http://wiki.civicrm.org/confluence/display/CRMDOC/CiviCRM+hook+specification

In all of these examples, you'll put the code we provide into your myhooks.module file if using Drupal, or the civicrmHooks.php file if using Joomla!. Be sure to upload the file after each change to the appropriate location on your server to see the new code take effect.

Additionally, if you are using Drupal and add a new hook to an existing module, you will need to clear the cache for the hook to start operating. One way of doing this is by visiting the page Admin > Build > Modules.

Sending an email message when a contact in a particular group is edited

In order to have CiviCRM tell you when a contact is edited, define the civicrm_pre hook. This lets you see the incoming edits as well as the values of the existing record, because you may want to include that information in the email.

If you are using Drupal, create a function named myhooks_civicrm_pre. If using Joomla!, create a function named joomla_civicrm_pre. We'll assume you're using Drupal for the rest of the example, so please adjust the code accordingly if you're using Joomla! instead. 

<?php

function myhooks_civicrm_pre( $op, $objectName, $objectId, &$objectRef ) {
    # configuration stuff
    $theGroupId = 1; # group id we want the contacts to be in
    $emailRecipient = 'johndoe@example.org'; # person to e-mail

    # Make sure we just saved an Individual contact and that it was edited
    if ($objectName == "Individual" && $op == "edit") {

        # Now see if it's in the particular group we're interested in
        require_once 'api/v2/GroupContact.php';
        $params = array('contact_id' => $objectId);
        $groups = civicrm_group_contact_get( $params );
        $found = false;
        foreach ($groups as $group) {
            if ($group['group_id'] == $theGroupId) {
                $found = true;
            }
        }
        # Exit now if contact wasn't in the group we wanted
        if (! $found) {
            return;
        }

        # We found the contact in the group we wanted, send the e-mail
        $emailSubject = "Contact was edited";
        $emailBody = "Someone edited contactId $objectId\n";
        # Here's where you may want to iterate over the fields
        # and compare them so you can report on what has changed.
        mail( $emailRecipient, $emailSubject, $emailBody );
    }
}

Validating a new contribution against custom business rules

If you have experience with other hook-based systems, you might think that the civicrm_pre hook is the one to use for validations. But this is not the case in CiviCRM because, even though the civicrm_pre hook is called before the record is saved to the database, you cannot abort the action from this hook.

This is where validation hooks come in. When you return true from a validation hook, CiviCRM saves the new or updated record. When you return an error object instead, CiviCRM aborts the operation and reports your error to the user.

An example follows of using a validation hook to validate new contributions against a business rule that says campaign contributions must have a source associated with them. In this example, we'll assume you are using Joomla!, so if you are using Drupal instead, be sure to change the function name accordingly.

<?php

function joomla_civicrm_validate( $formName, &$fields, &$files, &$form ) {
    # configuration stuff
    $campaignContributionTypeId = 3; # adjust for your site if different

    $errors = array();

    # $formName will be set to the class name of the form that was posted
    if ($formName == 'CRM_Contribute_Form_Contribution') {
        require_once 'CRM/Utils/Array.php';
        $contributionTypeId = CRM_Utils_Array::value( 'contribution_type_id',
            $fields );
        if ($contributionTypeId == $campaignContributionTypeId) {
            # see if the source field is blank or not
            $source = CRM_Utils_Array::value( 'source', $fields );
            if (strlen( $source ) > 0) {
                # tell CiviCRM to proceed with saving the contribution
                return true;
            } else {
                # source is blank, bzzzzzzzzzzzt!
                # assign the error to a key corresponding to the field name
                $errors['source'] =
        "Source must contain the campaign identifier for campaign contributions";
                return $errors;
            }
        } else {
            # not a campaign contribution, let it through
            return true;
        }
    }
}

Automatically filling custom field values based on custom business logic

This example uses a hook to write some data back to CiviCRM. You can make a custom field read-only and then set its value from a hook. This is very handy for storing and displaying data that are derived from other attributes of a record based on custom business logic.

For example, let's say you are storing employee records and you want to auto-generate their network login account when new employees are added. By doing it in your code, you can enforce a policy for login account names. For this example, let's say the policy is first initial + last name. So if your name is Jack Frost, your network login name would be jfrost.

Add a new read-only custom field to CiviCRM called "Network Login" and then find its ID. You can find it either by:

• Checking the civicrm_custom_field table in your CiviCRM database.
• Editing a contact and check the name of the Network Login field.

Now that we have our Network Login field, let's see how to populate it automatically with a hook. We'll switch back to the Drupal naming convention for this example.

Note that we use the civicrm_post hook here because we need the new contact record's ID in order to save a value to one of its custom fields. New records don't have an ID until they have been saved in the database, so if we ran this code in the civicrm_pre hook, it would fail.

<?php

function myhooks_civicrm_post( $op, $objectName, $objectId, &$objectRef ) {
    # configuration stuff
    $customId = 74;

    if ($objectName == 'Individual' && $op == 'create') {
        # generate the login
        $firstName = $objectRef->first_name;
        $lastName = $objectRef->last_name;
        $firstInitial = substr( $firstName, 0, 1 );
        $networkLogin = strtolower( $firstInitial . $lastName );

        # assign to the custom field
        $customParams = array("entityID" => $objectId,
            "custom_$customId" => $networkLogin);
        require_once 'CRM/Core/BAO/CustomValueTable.php';
        CRM_Core_BAO_CustomValueTable::setValues( $customParams );
    }
}

Custom mail merge token

The CiviMail component lets you customise a bulk email message using mail merge tokens. For instance, you can begin your message with, "Hi, {recipient.first_name}!" and when John Doe receives it, he'll see, "Hi, John!" whereas when Suzy Queue receives it, she'll see, "Hi, Suzy!" You can find out more details about working with custom tokens on the CiviCRM wiki: http://wiki.civicrm.org/confluence/display/CRMDOC/Mail-merge+Tokens+for+Contact+Data.

Besides the built-in tokens, you can use a hook to create new custom tokens. Let's make a new one that will show the largest contribution each recipient has given in the past. We'll use Drupal syntax again for this one.

<?php

# implement the tokens hook so we can add our new token to the list of tokens
# displayed to CiviMail users
function myhooks_civicrm_tokens( &$tokens ) {
    $tokens['contribution'] =
        array('contribution.largest' => 'Largest Contribution');
    /*  just array('contribution.largest'); in 3.1 or earlier */
}

# now we'll set the value of our custom token;
# it's better in general to use the API rather than SQL queries to retrieve data,
# but in this case the MAX() function makes it very efficient to get the largest
# contribution, so let's make an exception
function myhooks_civicrm_tokenValues( &$details, &$contactIDs ) {
    # prepare the contact ID(s) for use in a database query
    if ( is_array( $contactIDs ) ) {
        $contactIDString = implode( ',', array_values( $contactIDs ) );
        $single = false;
    } else {
        $contactIDString = "( $contactIDs )";
        $single = true;
    }

    # build the database query
    $query = "
      SELECT contact_id,
         max( total_amount ) as total_amount
      FROM   civicrm_contribution
      WHERE  contact_id IN ( $contactIDString )
      AND    is_test = 0
      GROUP BY contact_id
    ";

    # run the query
    $dao = CRM_Core_DAO::executeQuery( $query );
    while ( $dao->fetch( ) ) {
        if ( $single ) {
            $value =& $details;
        } else {
            if ( ! array_key_exists( $dao->contact_id, $details ) ) {
                $details[$dao->contact_id] = array( );
            }
            $value =& $details[$dao->contact_id];
        }

        # set the token's value
        $value['contribution.largest'] = $dao->total_amount;
    }
}

Fixing bugs

So you've found a bug and you want to fix it. What next? The first thing you should do is congratulate yourself on stepping up to fix the bug. The second thing you should do is to make sure that you are familiar with the content of this chapter so that you can fix the bug in the best possible way.

The three things that make up a great bug fix are:

1. a detailed bug report
2. a patch file
3. tests

Writing good bug reports

The best bug reports give lots of background and context. Don't forget that the way you are using CiviCRM is most likely very specific to your organisation. The more background you can give on the bug, the better.

The best bug reports clearly state

• What you did
• What you expected to happen
• What actually happened

Creating patches

A patch is a text based file format that shows the difference between one or more files.  Once you have submitted your patch, CiviCRM core developers can apply it to the development version of CiviCRM.

You should create your patch against the appropriate version of CiviCRM.  If you are in doubt as to which version this is, there'll likely be someone on IRC that can give you a immediate answer.  Otherwise ask on the forum.

The standard way to create a file is by using the diff command. 

$diff /path/to/old/file.php /path/to/new/file.php > changes.patch

The above will create a patch called changes.patch that you can attach to your issue.

There are cleverer techniques that you can use to generate patches if you are using version control systems.

Access to SVN

SVN is a free/open source version control system. It manages the changes made to the source code by multiple developers. SVN allows you to recover older versions, examine the history of how the code has changed, and receive and publish changes.

If you are submitting a lot of changes (patches), feel free to ask on IRC for an SVN account so you can directly apply your code contributions.

Adding tests to bug reports

US Federal law prohibits the submission of a patch to the CiviCRM code base without a suite of tests that have been shown to demonstrate the correct functioning of the code. It is also a contravention of European Union directive E842317 and enshrined in the UN universal declaration on human rights Article 31.  Well not quite! But you might think so if you asked the CiviCRM community, and for good reasons.  

Clever CiviCRM developers know that it is in their interest to submit one or more tests with their bug reports.  There are at least three important reasons to submit a comprehensive set of tests with your patch.

• it minimises the chances that you will be bitten by the same bug in the future as CiviCRM developers are committed to having the test suite complete successfully before each point release.
• it providers developers with an easy way to find out whether they have actually fixed your bug
• they are often the best way to explain complex user interface bugs.

Importing data

The CiviCRM import system can be used to import contacts, activities, contributions, event participants, and memberships. You can extend the import system to allow it to parse different data sources, such as XML, JSON, Excel, or OpenDocument. The import system is especially useful when you have legacy data in another system (such as a spreadsheet), or have data you need to migrate from one system to another.

The import system has a few important limitations. It is currently not very good at importing:

• Large data sets
  
• Data sets which contain information for more than component (such as contact data that also contains contributions)
• Data sets which should be imported into more than one group or tag (currently, you can choose only one group or tag for the entire import rather than specifying the value in a field of the imported data set).

These limitation apply to any extensions you build on the import. To overcome these limitations, one strategy would be to write a custom script using the CiviCRM API that parses the incoming data and makes the appropriate API calls to import it.

Data Sources

The import system supports two data sources out of the box. CSV and SQL. CSV is exported by most spreadsheets, many web sites, and a lot of legacy systems. SQL is the language of relational databases.

Here's your first insider tip on the import system: Every import is an SQL import. The CSV import data source, for example, starts out by dumping the CSV into a temporary database table and then running an SQL import on those data. If you write a custom data source, it's important to remember this because it will save you a lot of mental energy. You don't need to worry about formatting or validating the import data at all. Just get it into a database table and let CiviCRM take it from there.

When would you want to write an import data source? It might be useful if, say, a client were migrating from another CRM to CiviCRM and wanted to import their data themselves. Another example would be to support a different import file format.

To make things easy on yourself, check for an existing PHP library that can read the format of the data you want to import. If you fine one, use that library to read the data into a database table.

Data Source API

The data source API is essentially a set of functions you should implement in a new PHP class. Many software developers refer to this as an "interface" or "protocol." The abstract class that defines this interface is located in CRM/Import/DataSource.php. You should also look at the CRM/Import/DataSource/CSV.php file, because this implements the interface for the CSV import feature.

Here are the functions you will implement and what they should do:

getInfo(); # should return an array with the title of your data source plugin

preProcess( &$form ); # if you need to set anything up before the form is
displayed, this iss the place to do it

buildQuickForm( &$form ); # here you should build a form snippet that asks the user
for the appropriate information to access the incoming data

postProcess( &$params, &$db ); # this is where you dump the incoming data into the
database table

After defining your import data source, you need a new Smarty template to create your form snippet. This is pretty specific to the type of import data source you're defining. Let's look at an example of a real custom data source.

Example Data Source

This example of a custom import data source reads a JSON file and imports its contents as new contacts. This file should be named JSON.php and be added to the CRM/Import/DataSource/ directory.

<?php

/* My awesome JSON importer of dubious utility and efficiency (but still awesome!)
 *
 * JSON should look like this:
 * {
 *   "contacts" : [
 *      {
 *        "first_name" : "Foo",
 *        "last_name" : "Bar",
 *        "other_field" : "baz"
 *      },
 *      {
 *        "first_name" : "Other",
 *        "last_name" : "Contact",
 *        "something_else" : "yep"
 *      }
 *   ]
 * }
 */

require_once 'CRM/Import/DataSource.php';

class CRM_Import_DataSource_JSON extends CRM_Import_DataSource
{
    function getInfo()
    {
        return array('title' => ts('JavaScript Object Notation (JSON)'));
    }

    function preProcess(&$form)
    {
        # nothing to do here, but we still define the function
    }

    function buildQuickForm(&$form)
    {
        ### In this function we're calling a lot of QuickForm functions.
        ### If you're unfamiliar with that library, it might be good
        ### to look up the documentation for it.

        # define a hidden field that tells the system which
        # data source class we're using
        $form->add( 'hidden', 'hidden_dataSource', 'CRM_Import_DataSource_JSON' );

        # grab the config object so we respect some system settings
        $config = CRM_Core_Config::singleton();

        # get the max upload file size from the config
        $uploadFileSize = $config->maxImportFileSize;
        $uploadSize = round( ( $uploadFileSize / (1024*1024) ), 2 );

        # assign the max upload file size to a template variable
        # see the template documentation for more info on this
        $form->assign( 'uploadSize', $uploadSize );

        # add the file selection field to the form
        $form->add( 'file', 'uploadFile', ts('Import Data File'),
            'size=30 maxlength=60', true );

        # now set the max file size so the form enforces it
        $form->setMaxFileSize($uploadFileSize);
        $form->addRule( 'uploadFile',
            ts('File size should be less than %1 MBytes (%2 bytes)',
            array(1 => $uploadSize, 2 => $uploadFileSize)),
            'maxfilesize', $uploadFileSize );

        # not a very smart rule, but it'll do for now
        $form->addRule( 'uploadFile', ts('Input file must be in JSON format'),
            'utf8File' );

        # make sure we end up with a file after the form posts
        $form->addRule( 'uploadFile', ts('A valid file must be uploaded.'),
            'uploadedfile' );
    }

    function postProcess(&$params, &$db)
    {
        # grab the name of the file that was uploaded
        $file = $params['uploadFile']['name'];

        # call a helper function we'll define below to parse the JSON
        $result = self::_JsonToTable( $db, $file,
            CRM_Utils_Array::value( 'import_table_name', $params ) );

        # grab the import table name (CiviCRM determines this for us)
        $table = $result['import_table_name'];

        # create a new ImportJob object for our table
        require_once 'CRM/Import/ImportJob.php';
        $importJob = new CRM_Import_ImportJob( $table );

        # the ImportJob modifies the table name a bit, so let's update it
        $this->set( 'importTableName', $importJob->getTableName() );
    }

    /**
     * We define this function just to keep things cleaner, the import data source
     * interface doesn't look for it; it's a private function. We use an
     * underscore at the beginning of the name to indicate this.
     */
    private static function _JsonToTable(&$db, $file, $table )
    {
        # read the JSON into a string variable
        $jsonString = file_get_contents($file);
        if (!$jsonString) {
            # oops, reading the file didn't work, generate an error
            CRM_Core_Error::fatal("Could not read $file");
        }

        # grab the config object again
        $config = CRM_Core_Config::singleton();

        # this is a bit presumptuous of us, but oh well
        $db->query("DROP TABLE IF EXISTS $table");

        # create the table where we'll store the incoming data
        $create = "CREATE TABLE $table LIKE civicrm_contact";
        $db->query($create);
        # drop the id column because the import system will add one
        $dropId = "ALTER TABLE $table DROP COLUMN id";
        $db->query($dropId);

        # decode the JSON and INSERT the records one by one;
        # it might be more efficient to build one big multi-insert,
        # but we'll leave that as an exercise for the reader

        ### BE CAREFUL THAT YOU DON'T RUN THIS ON A MASSIVE JSON FILE!!
        ### It creates one big object from the entire JSON file, so you'll quickly
        ### eat up every bit of memory PHP can use if you try to import a large
        ### file.

        # this requires that the JSON PECL extension is installed and
        # enabled in PHP
        $importObj = json_decode( $jsonString, true );

        # loop through each record in the JSON object, put it into an SQL query,
        # and insert it into the database
        foreach ($importObj['contacts'] as $newContact) {
            $fields = array_map( '_civicrm_mysql_real_escape_string',
                array_keys( $newContact ) );
            $sqlFields = "(" . implode( ',', $fields ) . ")";
            $values = array_map( '_civicrm_mysql_real_escape_and_quote_string',
                $newContact );
            $sqlValues = "VALUES (" . implode( ',', $values ) . ")";

            # construct the query and run it
            $sql = "INSERT IGNORE INTO $table $sqlFields $sqlValues";
            $db->query($sql);
        }

        # get the import tmp table name and return it
        $result = array( );
        $result['import_table_name'] = $table;
        return $result;
    }
}

# Another couple private helper functions we define
function _civicrm_mysql_real_escape_and_quote_string( $string ) {
    return _civicrm_mysql_real_escape_string( $string, true );
}

function _civicrm_mysql_real_escape_string( $string, $quote = false ) {
    static $dao = null;
    if ( ! $dao ) {
        $dao = new CRM_Core_DAO( );
    }
    $returnString = $quote ? "\"{$dao->escape( $string )}\"" :
        $dao->escape( $string );
    return $returnString;
}

And here's the Smarty template. It should be saved in templates/CRM/Import/Form/JSON.tpl.

<fieldset><legend>{ts}Upload JSON File{/ts}</legend>
  <table class="form-layout">
    <tr>
        <td class="label">{$form.uploadFile.label}</td>
        <td>{$form.uploadFile.html}<br />
            <div class="description">{ts}File format must be JavaScript
              Object Notation (JSON). File must be UTF8 encoded if it contains
              special characters (e.g. accented letters, etc.).{/ts}</div>
            {ts 1=$uploadSize}Maximum Upload File Size: %1 MB{/ts}
        </td>
    </tr>
  </table>
</fieldset>

Summary

The CiviCRM Import system has a few limitations, but it is by far the easiest way for end users to get data from other systems into CiviCRM. When a client has ongoing data import needs and wants non-technical users to initiate and manage the imports, writing a custom data source may be a good solution.

If the import system cannot handle the type or amount of data you need to import, your options are to write a separate import program that calls the CiviCRM API, write your own improvements to the import system, or sponsor other developers to write improvements. If you write or sponsor improvements to the import system, make sure you contribute them back to the core system! The CiviCRM development team would be happy to discuss potential improvements with you. You can find them in the #civicrm IRC channel on irc.freenode.net.