26 QUESTIONS TO HELP RESEARCH BUYERS OF ONLINE SAMPLES

This answer might help you to form an opinion about the relevant experience of the sample provider. How long has the sample provider been providing this service and do they have for example a market research, direct marketing or more technological background? Are the samples solely providedfor third party research, or does the company also conduct proprietary work using their panels?

The description of the type of source a provider uses for delivering an online sample might provide insight into the quality of the sample. An actively managed panel is one which contains only active panel members - see question 11. Note that not all online samples are based on online access panels.

The answer to this question may simplify the comparison of online sample providers in the market.

Combining panellists for different types of usage (like direct marketing) might cause survey effects.

The inclusion of hard-to-reach groups on the internet (like ethnic minority groups, young people, seniors etc.) might improve the quality of the sample provided.

The type of rewards and proposition could influence the type of people who agree to answer a questionnaire or join a specific panel and can therefore influence sample quality.

The panel attrition rate may be an indicator of panellists’ satisfaction and (therefore) panel management, but a high turnover could also be a result of placing surveys which are too long with poor question design. The method of calculation is important because it can have a significant impact on the rate quoted.

The opt-in process might indicate the respondents’ relationship with the panel provider. The market generally makes a distinction between single and double opt-in. Double opt-in describes the process by which a check is made to confirm that the person joining the panel wishes to be a member and understands what to expect.

Confirmation of identity might increase quality by decreasing multiple entries, fraudulent panellists, etc.

Extended and up-to-date profile data increases the effectiveness of low incidence sampling and reduces pre-screening of panellists.11

The size of the panel might give an indication of the capacity of a panel. In general terms, a panel’s capacity is a function of the availability of specific target groups and the actual completion rate. There is no agreed definition of an active panel member, so it is important to establish how this is defined. It is likely that the new ISO for access panels which is being discussed will propose that an active panel member is defined as a member that has participated in at least one survey, or updated his/her profile data, or registered to join the panel, within the last 12 months. The type and number of sources might be an indicator of source effects and source effects might influence the data quality. For example, if the sample is sourced from a loyalty programme (travel, shopping, etc.) respondents may be unrepresentatively high users of certain services or products.

The sampling processes for the sample sources used are a main factor in sample provision. A systematic approach based on market research fundamentals may increase sample quality.

Survey results can sometimes be influenced by the wording used in subject lines or in the body of an invitation.

The reward or incentive system might impact on the reasons why people participate in a specific panel and these effects can cause bias to the sample.

Frequency of survey participation might increase conditioning effects whereas a controlled survey load environment can lead to higher data quality.

Not complying with local and international privacy laws might mean the sample provider is operating illegally.

The sample provider usually stores sensitive and confidential information on panellists and clients in databases. These need to be properly secured and backed-up, as does any confidential information provided by the client.

A quality management system is a system by which processes in a company are described and employees are accountable. The system should be based on continuous improvement. Certification of these processes can be independently done by auditing organizations, based for instance, on ISO norms.

The ICC/ESOMAR International Code requires special permissions for interviewing children.

Many providers work with third parties. This means that the quality of the sample is also dependent on the quality of sample providers that the buyer did not select. Transparency is a key issue in this situation. Overlap between different panel providers can be significant in some cases and de-duplication removes this source of error, and frustration for respondents.

It is not that uncommon for a panellist to be a member of more than one panel nowadays. The effects of multi-panel membership by country, survey topic, etc., are not yet fully known. Proactive and clear policies on how any potential negative effects are minimized by recruitment, sampling, and weighting practices is important.

Panel response might be a function of factors like invitation frequency, panel management (cleaning) policies, incentive systems and so on. Although not a quality measure by itself these rates can provide an indication of the way a panel is managed. A high start rate might indicate a strong relationship between the panel member and the panel. A high drop-out rate might be a result of poor questionnaire design, questionnaire length, survey topic or incentive scheme as well as an effect of panel management. The new ISO for access panels will likely propose that participation rate is defined as the number of panel members who have provided a usable response divided by the total number of initial personal invitations requesting members to participate.

This type of data per respondent increases the possibility of analysis for data quality, as described in ESOMAR’s Guideline on Access Panels.

When the sample provider is also hosting the online survey, preliminary data quality analysis and validation is usually preferable.

Respondent satisfaction may be an indicator of willingness to take future surveys. Respondent reactions to your survey from self-reported feedback or from an analysis of suspend points might be very valuable to help understand survey results.

One might expect a full sample provider debrief report, including gross sample, start rate, participation rate, drop-out rate, the invitation text, a description of the field work process, and so on.