Thinking About Your Data

It takes time and patience to figure out where to put data in a new system, even if you're starting fresh. It's even harder if you have pre-existing data. The purpose of this chapter is to familiarize you with how data is managed in CiviCRM and to get you thinking about where to "plant" your data in CiviCRM.

Garbage in...garbage out...

CiviCRM depends upon the integrity of the data that is stored in it: where it is and what "shape" it is in. You might be familiar with the expression: "garbage in... garbage out". The following are examples of data that is, for one reason or another, "garbage".

• Inconsistencies in data entry. Recording a state for one person as California and for another as CA makes it difficult when you try to search for everyone in that state. Use one or the other. Being inconsistent makes it difficult to find specific data, and results in "ugly" reports and mailing labels.
  
• Out-of-date information is also "garbage": If you've sent mailings that have been returned to you either because the address was completely wrong or because the person had moved to a different address, and you haven't updated the address data for those contact records, then the address information for those contacts is also garbage.
  
• Data that isn't well organized results in sending duplicate mailings. For example if all correspondence from John and Jane Smith indicate that they are a couple living in the same house, yet you continue to keep separate records with no indication that thay are part of the same household, then you will continue to send duplicate mailings.
  

Clean data is data that appears in the same way consistently throughout the system. Even making a poor choice of the 'standard' for how to store data for a particular field is better then storing it in a haphazard way.

Organized data makes it easy to find the contacts that you need for a mailing, an event or any other purpose.

Whether you're moving old data into CiviCRM or starting from scratch, take time before you define data fields to think through the way your data reflects your organization's work and take time to standardize the way that certain data is recorded in your system. Also take time to learn about the purpose of the fields in the new system so that you will enter or import data into the right place and in a consistent manner.

Choosing and structuring your data in CiviCRM

If you have pre-existing data that you want to put into CiviCRM, you might find it useful to think of a process like moving your house or apartment. Part of the packing process should include asking "Do we really need this?" People use apartment moving as a chance to say, "This stuff is too old; let's trash it and get some new stuff once we have moved in."

To apply this metaphor to your data, look for fields that have no purpose, such as old organizational divisions that you've abandoned or office locations in a facility you no longer occupy.

Many fields will have continuing value. One organization, for instance, in a financial pinch decided to make use of an old list of founding donors who had not given money for many years. It turned out that these "lapsed" donors still had strong emotional ties to the organization that they had founded and they came to its financial rescue. So in that case saving old data was crucial.

While you're evaluating data place it into a structure that allows you to manipulate it later. In our moving metaphor, this is like packing things into boxes based on which room you'll want them in after you move. There are several CiviCRM concepts you can apply at this stage, even if you don't know CiviCRM very well:

• CiviCRM's basic, top-level item of information is the record and there are three record types: individual, household and organization
• The record type determines which fields will appear when someone views a record of that type. For example: you won't see Organization name in a household-type record
• "Contact" is the term that is used to refer to all record types: individuals, households and organizations but when you add a new record, or do an import, you must specify the record type.
• A record might have data such as donation amounts and dates and events attended. When it comes to importing the data, you will need to first import the contact and then import the donation or participation information into CiviContribute and CiviEvent.

Of course, if you're moving into a house you don't know very well, or one that you've seen only in plans or photos, you may find when you get there that stuff you had intended for the lounge would go better on the landing or in the hall. So part of the process is ensuring that you leave options open to move things around until you have properly settled in.

It is strongly recommended that you analyze your data needs as described here before you begin defining data groups and fields through the CiviCRM interface. Start by making a list of each existing data field that you want to transfer and any new fields that you wish to start tracking in CivCRM. Compare each field with CiviCRM's core contact fields and try to find one that is the appropriate match for your data field. If you do not find an appropriate match in core fields, then create a custom field or a tag: core fields have a specific purpose in the system and only data that matches that purpose should be present. You can then classify the remaining fields as custom data fields, groups, or tags.

What does CiviCRM offer?

At its core, CiviCRM has a number of predefined data fields. These fields should always be used before creating custom fields because core fields are tied to specific inbuilt functionality. Core fields include basic information about the contact such as:

• Name, nickname, greeting, title
• Website, email addresses, phone numbers, IM account name
• Addresses
• Communication preferences
• Demographics

Before continuing with the analysis of your existing data or required data needs, you should familiarize yourself with the core contact fields available. You may do this by viewing a contact in CiviCRM and clicking "edit," which will display all available fields for that contact type. At a later date you may choose to disable certain fields that will not be used by your organization. This is configurable through the Administer CiviCRM » Global Settings pages.

Wherever possible, you should seek to make use of the pre-defined contact fields before examining other options. However, it is very likely that your needs will extend beyond the options available in the core contact fields, since you may want to be capturing information about individual or organisation's characteristics, interests, history, etc. In such cases you will make use of CiviCRM's custom data tools - or make use of the Groups or Tags options.

For example, if your existing data includes a list of people who are on a Committee - you could replicate this information in CiviCRM through either a custom data field, or by adding those contacts directly in to a Group.

Extending core data

It is very likely that the information you want to collect will extend beyond the predefined fields that come with CiviCRM.

Custom data tools provide you the ability to create as many additional fields of varying types as you need, and to group and display them based on your preferences. These fields can then be used in Profiles to expose or collect data through website forms.

If you have already undertaken a data structure analysis you will have identified the fields you need in addition to those that come predefined in CiviCRM. You should have also reviewed the function and benefits of Groups and Tags, as the characteristics of these tools may be better suited for certain data management and organizational needs than custom data fields.

Custom data sets

Custom data fields are organized into sets referred to in CiviCRM as custom data groups (to avoid confusion with Groups within this manual we will refer to them here as 'sets'). These custom data sets can be associated with ALL contact types, with a specific contact type (e.g. Individuals), with a specific component (CiviMember, CiviEvent), or with other elements such as Relationships and Groups.

NOTE: It is important to carefully consider what record type your custom data group will be applied to. You will not be able to change the record type after creating your data group.

When creating field sets you should ask: How will these fields be used? What types of contacts or records will these fields be appropriate for? Will they have broad applicability, or are they relevant to a specific contact type, event, contribution type, etc. Taking the time to think through these questions will help keep your application screens as relevant and clear of superfluous fields as possible. For example, if your field set contains contact characteristics such as a field for the “color of eyes,” you would associate with the “Individual” contact type, rather than the generic “Contacts” option, as this field would be irrelevant to Organization and Household contact types.

Depending on how many custom data fields you are creating, you should also give consideration to grouping the fields topically. For example, you may have 20 custom fields that will all be associated with Individual contacts, 12 of which are fields relating to an online membership directory. Rather than group all 20 fields in a single custom data set, you may want to split them into two groups—one for the directory-related fields, and a second for more general “Individual Details.”

Configuring custom data sets

You create custom data sets and fields through Administer CiviCRM » Custom Data » New Group of Custom Fields. Using this form you identify the title of the data set, specify what type of records the data set will be used for, select the display characteristics, and enter help text. 

 

Group Name
For 'inline' display custom groups, this name will appear as the field set legend. If this group uses the 'tab' display style, the name will be used for the navigation tab.

Used For

There are many options in terms of how specifically you can define what the Data Fields will be used for. You can use this option to ensure that the fields are only available in the relevant places. The full set of options include:

• Activity: May be assigned to all activities or to a specific activity type, such as Meeting or Phone Call.
• Contacts: Applied to ALL contact types.
• Contributions: May be assigned to all contributions or to a specific contribution type, such as Donations or Event Fees.
• Events: Applied to the actual event itself, not the participant registration record, and can be assigned to all events or a specific event type (e.g. Conference or Fundraiser)
• Grants
• Groups: Displayed in the Group Settings (note that these fields are not searchable)
• Households: Applied to household contact types only.
• Individuals: Applied to individual contact types only.
• Membership: May be assigned to all membership records or to a specific membership type.
• Organizations: Applied to organization contact types only.
• Participants: Applied to the participant registration record. There are three options for participant fields—general fields applied to all registration records, role-type fields assigned to a specific participant role, and event participant fields assigned to a specific event.
• Pledges
• Relationship: May be assigned to all relationship records or to a specific relationship type, such as Spouse of or Employee/Employer of.

As you can see, custom data sets and fields provide tremendous flexibility and control over your data needs, allowing you to create and assign fields to virtually every organizational element within CiviCRM.

Multiple Records
Data sets usually only have a single set of data associated with a contact record – for example, a person has blue eyes or a person has brown eyes. However, there are times when you may need to track multiple sets of records for a single contact. CiviCRM provides this functionality for custom data sets assigned to contacts (all contact types or a specific type). For example, you may want to track a person’s post-secondary educational history, collecting information about the schools they attended, date of graduation, degree level, and area of study. A single person may have multiple records tracking their undergraduate and post-graduate work.

To make use of this option you select the "Does this Custom Data Group allow multiple records?" option when creating the custom data set. Note that this option:

• can only be applied to a data set, not to a particular field
• the data set must be displayed in a tab (inline display is not supported)
• can only be used for Contacts, not for Activities, Contributions, etc.
• cannot be used in Profiles, such as those used for Events
• cannot currently be exported.

Display Style
Data field sets are either displayed 'inline' on the contact summary page (the Summary tab), or as a new Tab sitting along the top of the contact record, along with Summary, Contribution, Group, Note etc. Select 'Tab' for less frequently accessed and/or larger sets of fields). NOTE: This setting applies to Custom Data sets used for Contact records only. Custom data sets used for components, relationships, or other resources, are always displayed inline. Also note that custom data sets configured to handle multiple records must be displayed in a tab.

Other settings
You can specify that you want the field set to be Collapsed on initial display. If you check this box only the title for this field set will be displayed when the page is initially loaded as the fields are hidden. This is helpful for field sets that are less frequently used as it reduces how much screen real estate is used when the page opens.

You can also set whether the Field set is active and can provide explanatory text displayed above or below the field set.

Custom Data Fields

After defining your custom data set, you must create the actual data fields.

As you work through the settings for each field, think carefully about how you want to display and use the information. Each field type provides unique characteristics regarding how the field displays, how data is stored, and how it can be searched. For example using Checkboxes rather than Multi-Select enables you to use OR as well as AND searches in Advanced Search.

 

Field Label
This is a unique field name used in screen displays and when you export data. When using Fields in a Profile you can overwrite the Field Label – so choose names that make sense for Administrators, and then give more user friendly names when exposing them in Profiles.

Data and Input Field Type

Select the type of data you want to collect and store for this contact. Then select from the available HTML input field types (choices are based on the type of data being collected). You can also specify the database field length here, where applicable.

• Alphanumeric (text or numbers)
  • Text (simple text form field)
  • Select (drop-down box; one selection)
  • Radio (list of options; one selection, NB there is a pre-built Yes/No data type)
  • Checkbox (list of options with checkboxes; multiple selections)
  • Multi-select (list of options in a single box; multiple selections using Control-Click)
• Integer
  • Text/Select/Radio
• Number
  • Text/Select/Radio
• Money
  • Text/Select/Radio
• Note (longer text; multiple lines)
  • TextArea (plain text, multi-line box)
  • RichTextEditor (multi-line box with WYSIWYG editor for html styling)
• Date (select dropdowns) (can include hours and mins)
• Yes or No (radio buttons)
• State/Province (selection options as defined in global settings)
  • Select dropdown (single selection)
  • Multi-select box
• Country (selection options as defined in global settings)
  • Select dropdown
  • Multi-select box
• File (browse to select and upload file)
• Link (active hyperlink)

Multiple Choice Options
For field types that involve selecting from a set of multiple options – Select, Radio, Checkbox and MultiSelect – you are given the option of using an existing set of options which you've already created for another custom field or creating a new set.

If you choose to use the same set of options for several fields you will be notified when making any changes that this will be affecting an option set that is used by several fields.

When you create a new set you have the option of initially entering up to ten multiple choice options in a table. If you need more than ten options, you can create an unlimited number of additional choices after saving this new field by using the Edit Multiple Choice Options link.

You may modify the label, order and active status of any multiple option at a later date through Custom Data » View and Edit Custom Fields » Edit Multiple Choice Options.

If desired, you can also mark one of the choices as the default option. The option 'label' is displayed on the form, while the option 'value' is stored in the contact record. The label and value may be the same or different.

Inactive options are hidden when the field is presented.

Order
As the name would suggest, the order field controls the order in which the fields appear. You may assign the order in the field edit form, or use the up/down icons on the main field listing table to adjust the field presentation. By default, new fields appear at the bottom of the field list within a set.

Default Value
Where applicable, you may designate a default value for a field. Note the format required for date fields (YYYY-MM-DD).

Field Help
In most cases, your field name is self-explanatory in describing the purpose of the field. But in those cases where there is some ambiguity, or where you wish to help regulate how a certain field is used, you may enter help text. The help text appears below the form field. Note that the text identified here will appear in all uses of the field in administration pages and will be inserted as the default help text when fields are assigned to a profile. When used in a profile, the help text may be removed or changed without impact on the original custom field definition.

Required
As you begin defining each of your custom fields, you will undoubtedly come across some fields that you wish to designate as a required field. By selecting this option, you ensure that whoever is completing the form must provide a value for this field before submitting the form. Failure to do so will result in an error message directing the person to complete the required field(s).

Is this Field Searchable
CiviCRM’s advanced search page is a powerful tool for running complex queries on your records. The tool includes a custom field panel which displays custom fields configured to be searchable, as defined in this option. As with other areas of CiviCRM, it is worth thinking through how you will use the particular field you are defining. While you may be tempted to simply mark every field as searchable, doing so may unnecessarily clutter the advanced search custom field panel, when in fact certain fields will likely never be used in that way. You may toggle this option on or off at any time, so don’t be overly concerned about arriving at a final decision during this initial field-definition stage.

Active
The “active” option determines if the field is disabled or enabled within the CiviCRM interface. Disabling a field does not remove the data from the system—it simply hides the field from the contact interface.

While this option may seem rather mundane and obvious, it can perform very valuable functions when managing your data, especially if you are migrating from an existing database system.

For example, you may have certain fields in your existing database that you would like to transfer to the new system for historical data keeping purposes but plan to then deprecate or migrate to a new data structure in CiviCRM. Suppose you are importing membership records from an MS Access database. Each record in Access has a unique id (key) field, which has no direct benefit in CiviCRM. Rather than ignoring it altogether, you could create a custom field to hold the value, import the records, and then disable (un-activate) the field, thereby hiding it from view and minimizing the interface clutter. Though not visible to users, the field value is stored in the system and can be referenced at a later date if you need to—for instance if you ever needed to investigate archived data for a possible discrepancy or compare the field value with a printed record.

View Only
The last option on the field creation form allows you to designate a field as visible but uneditable. There are two general uses for this field: to store data imported from another system that you want available for reference to the user, but do not want them to be able to modify; or as a data container that is controlled through a custom PHP hook rather than through the user interface. CiviCRM has a number of hooks defined that allow developers to modify data, as well as customize forms and screens, without modifying the CiviCRM codebase. Visit the documentation online for more specific information about available hooks and how to implement them.

After completing the field configuration options, click save to record the field and return to the field listing for your current field set, or click “save and new” to save the field and begin defining a new field.

With the exception of the data and input field type selection, all of the configuration options may be modified after your initial creation of the field. You may also find it useful to preview your custom fields, as well as the set of custom fields as you are defining them. This is particularly useful for checking the layout of radio button and checkbox fields with a large number of choices.

Choosing between custom fields, groups and tags

Data fields, groups, and tags are three main ways to associate information with contacts. While it can be tempting to just create a custom data field for every attribute of your data, take time to learn about the alternatives. They offer powerful functionality that you may miss out on if you rely only on custom data. Furthermore, using data fields for information stored more appropriately as groups or tags can slow your system. Finally, proper use of groups and tags makes it much easier for administrative staff to maintain the records.

Some tips that may help you choose :

• Data that can take a wide range of values, such as a person's address or biography, should be stored in an alphanumic custom data field
• Custom Data fields can be grouped and displayed on their own Tab on the contact's record
• As the name implies, Groups are used to group contacts. For instance, you'll probably assign board members to one group, staff to another, volunteers to a third, and so on. If you use Drupal you can assign permissions based on group membership. You can also define a group that CiviCRM automatically adds contacts to and deletes contacts from, based on some characteristic. This feature is called a Smart Group.
• If you plan to use CiviMail for mass mailings and you want certain contacts to get a particular mailing, those contacts must be assigned to a Group. For instance, you may want a press release to go only to certain contacts; those contacts should be assigned to a particular group. This group could be a Smart Group,
• Both Tags and Groups can be structured hierarchically. For instance, a group/tag labeled Regions can have a subgroup/subtag for each geographic region your organization covers. (See "Case study in hierarchical tags" later in this section.)
• Tags support more powerful search options than data fields or groups. For instance, visitors can search through multiple tags with both AND and OR operands. Data fields support only lists of words (which is effectively the same as an AND operand), except for fields represented as checkboxes, which support OR operands.
• Tags have a more sophisticated user interface than data fields or groups The interface allows the visitor to add and remove tags without reloading the page in edit mode.
• Custom Data fields can be assigned to a specific record type (e.g., only households), whereas tags will be assigned to all types once the tags are defined.

WARNING: Once you choose a record type in which your custom data set appears, you can't change the type. If you find later that you need to change the record type, you either have to delete and recreate the data set or get someone with database experience to look at the representation of the record types in your underlying database and copy the data manually to the new record type.

Relationships

Relationships are another way of capturing data for information that connects one contact to another. Go to Administer CiviCRM » Relationship Types to familiarise yourself the standard relationships included in CiviCRM. You can define additional types of relationships as needed.

Prior to importing data, you will need to pay attention to how you handle relationships such as employer-to-employee and household-to-individual. During the import you can create these if you understand what needs to be done.

Relationships are also the basis for providing inherited memberships, which is covered in more depth in the section on Memberships.

Creating Your Data Structure in CiviCRM

Once you have completed the analysis of your data needs, you may want to begin creating your custom data groups and fields, groups, and tags. Take time to read the sections and documentation that further details how these functions work, but in summary:

• For custom data groups and fields, go to Administer CiviCRM and select "Custom Data" in the "Customized" section
• For groups, click on CiviCRM Home and the select "Manage Groups"
• For tags, go to Administer CiviCRM and select "tags" in the "Option Lists" section

When creating a custom data group field you will need to decide how you want it to be displayed on the screen: this is what the user/guest will see. A variety of "web form elements", such as text box, date, radio buttons, checkboxes, or drop-down selection list, are availabe. Just pick the one that suits your needs.

Next Steps

After you have:

• analyzed your existing data
• cleaned it up by standardizing and organizing itr
• removed duplicate records
• decided what fields in CiviCRM you will be using
• created any custom fields, groups, and tags that you will need in CiviCRM
• decided the number and layout for your import files
  

You will then be ready to import your existing data.