The CiviCRM import system can be used to import contacts, activities, contributions, event participants, and memberships. You can extend the import system to allow it to parse different data sources, such as XML, JSON, Excel, or OpenDocument. The import system is especially useful when you have legacy data in another system (such as a spreadsheet), or have data you need to migrate from one system to another.
The import system has a few important limitations. It is currently not very good at importing:
- Large data sets
- Data sets which contain information for more than component (such as contact data that also contains contributions)
- Data sets which should be imported into more than one group or tag (currently, you can choose only one group or tag for the entire import rather than specifying the value in a field of the imported data set).
These limitation apply to any extensions you build on the import. To overcome these limitations, one strategy would be to write a custom script using the CiviCRM API that parses the incoming data and makes the appropriate API calls to import it.
The import system supports two data sources out of the box. CSV and SQL. CSV is exported by most spreadsheets, many web sites, and a lot of legacy systems. SQL is the language of relational databases.
Here's your first insider tip on the import system: Every import is an SQL import. The CSV import data source, for example, starts out by dumping the CSV into a temporary database table and then running an SQL import on those data. If you write a custom data source, it's important to remember this because it will save you a lot of mental energy. You don't need to worry about formatting or validating the import data at all. Just get it into a database table and let CiviCRM take it from there.
When would you want to write an import data source? It might be useful if, say, a client were migrating from another CRM to CiviCRM and wanted to import their data themselves. Another example would be to support a different import file format.
To make things easy on yourself, check for an existing PHP library that can read the format of the data you want to import. If you fine one, use that library to read the data into a database table.
Data Source API
The data source API is essentially a set of functions you should implement in a new PHP class. Many software developers refer to this as an "interface" or "protocol." The abstract class that defines this interface is located in CRM/Import/DataSource.php. You should also look at the CRM/Import/DataSource/CSV.php file, because this implements the interface for the CSV import feature.
Here are the functions you will implement and what they should do:
getInfo(); # should return an array with the title of your data source plugin preProcess( &$form ); # if you need to set anything up before the form is displayed, this iss the place to do it buildQuickForm( &$form ); # here you should build a form snippet that asks the user for the appropriate information to access the incoming data postProcess( &$params, &$db ); # this is where you dump the incoming data into the database table
After defining your import data source, you need a new Smarty template to create your form snippet. This is pretty specific to the type of import data source you're defining. Let's look at an example of a real custom data source.
Example Data Source
This example of a custom import data source reads a JSON file and imports its contents as new contacts. This file should be named JSON.php and be added to the CRM/Import/DataSource/ directory.
And here's the Smarty template. It should be saved in templates/CRM/Import/Form/JSON.tpl.
The CiviCRM Import system has a few limitations, but it is by far the easiest way for end users to get data from other systems into CiviCRM. When a client has ongoing data import needs and wants non-technical users to initiate and manage the imports, writing a custom data source may be a good solution.
If the import system cannot handle the type or amount of data you need to import, your options are to write a separate import program that calls the CiviCRM API, write your own improvements to the import system, or sponsor other developers to write improvements. If you write or sponsor improvements to the import system, make sure you contribute them back to the core system! The CiviCRM development team would be happy to discuss potential improvements with you. You can find them in the #civicrm IRC channel on irc.freenode.net.