26 questions to help research buyers of online samples
These questions, in combination with additional information, will help researchers consider issues which influence whether an online sampling approach is fit for purpose in relation to a particular set of objectives; for example whether an online sample will be sufficiently representative and unbiased. They will help the researcher ensure that they receive what they expect from an online sample provider
Company profile
1. What experience does your company have with providing online samples for market research?
This answer might help you to form an opinion about the relevant experience of the sample provider. How long has the sample provider been providing this service and do they have for example a market research, direct marketing or more technological background? Are the samples solely provided for third party research, or does the company also conduct proprietary work using their panels?
Sample Source
2. Please describe and explain the types of source(s) for the online sample that you provide (are these databases, actively managed panels, direct marketing lists, web intercept sampling, river sampling or other)?
The description of the type of source a provider uses for delivering an online sample might provide insight into the quality of the sample. An actively managed panel is one which contains only active panel members - see question 11. Note that not all online samples are based on online access panels.
3. What do you consider to be the primary advantage of your sample over other sample sources in the marketplace?
The answer to this question may simplify the comparison of online sample providers in the market.
4. If the sample source is a panel or database, is the panel or database used solely for market research? If not, please explain.
Combining panellists for different types of usage (like direct marketing) might cause survey effects.
5. How do you source groups that may be hard-to-reach on the internet?
The inclusion of hard-to-reach groups on the internet (like ethnic minority groups, young people, seniors etc.) might improve the quality of the sample provided.
6. What are people told when they are recruited?
The type of reward and proposition could influence the type of people who agree to answer a questionnaire or join a specific panel and can therefore influence sample quality.
Panel Recruitment
7. If the sample comes from a panel, what is your annual panel turnover/attrition/retention rate and how is it calculated?
The panel attrition rate may be an indicator of panellists’ satisfaction and (therefore) panel management, but a high turnover could also be a result of placing surveys which are too long with poor question design. The method of calculation is important because it can have a significant impact on the rate quoted.
8. Please describe the opt-in process.
The opt-in process might indicate the respondents’ relationship with the panel provider. The market generally makes a distinction between single and double opt-in. Double opt-in describes the process by which a check is made to confirm that the person joining the panel wishes to be a member and understands what to expect.
9. Do you have a confirmation of identity procedure? Do you have procedures to detect fraudulent respondents at the time of registration with the panel? If so, please describe.
Confirmation of identity might increase quality by decreasing multiple entries, fraudulent panellists, etc.
10. What profile data is kept on panel members? For how many members is this data collected and how often is this data updated?
Extended and up-to-date profile data increases the effectiveness of low incidence sampling and reduces pre-screening of panellists.
11. What is the size and/or the capacity of the panel, based on active panel members on a given date? Can you provide an overview of active panellists by type of source?
The size of the panel might give an indication of the capacity of a panel. In general terms, a panel’s capacity is a function of the availability of specific target groups and the actual completion rate. There is no agreed definition of an active panel member, so it is important to establish how this is defined. It is likely that the new ISO for access panels which is being discussed will propose that an active panel member is defined as a member that has participated in at least one survey, or updated his/her profile data, or registered to join the panel, within the last 12 months. The type and number of sources might be an indicator of source effects and source effects might influence the data quality. For example, if the sample is sourced from a loyalty programme (travel, shopping, etc.) respondents may be unrepresentatively high users of certain services or products.
Panel and sample management
12. Please describe your sampling process including your exclusion procedures if applicable. Can samples be deployed as batches/replicates, by time zones, geography, etc? If so, how is this controlled?
The sampling processes for the sample sources used are a main factor in sample provision. A systematic approach based on market research fundamentals may increase sample quality.
13. Explain how people are invited to take part in a survey. What does a typical invitation look like?
Survey results can sometimes be influenced by the wording used in subject lines or in the body of an invitation.
14. Please describe the nature of your incentive system(s). How does this vary by length of interview, respondent characteristics, or other factors you may consider?
The reward or incentive system might impact on the reasons why people participate in a specific panel and these effects can cause bias to the sample.
15. How often are individual members contacted for online surveys within a given time period? Do you keep data on panellist participation history and are limits placed on the frequency that members are contacted and asked to participate in a survey?
Frequency of survey participation might increase conditioning effects whereas a controlled survey load environment can lead to higher data quality.
Policies and Compliance
16. Is there a privacy policy in place? If so, what does it state? Is the panel compliant with all regional, national and local laws with respect to privacy, data protection and children e.g. EU Safe Harbour, and COPPA in the US? What other research industry standards do you comply with e.g. ICC/ESOMAR International Code on Market and Social Research, CASRO guidelines etc.?
Not complying with local and international privacy laws might mean the sample provider is operating illegally.
17. What data protection/security measures do you have in place?
The sample provider usually stores sensitive and confidential information on panellists and clients in databases. These need to be properly secured and backed-up, as does any confidential information provided by the client.
18. Do you apply a quality management system? Please describe it.
A quality management system is a system by which processes in a company are described and employees are accountable. The system should be based on continuous improvement. Certification of these processes can be independently done by auditing organizations, based for instance, on ISO norms.
19. Do you conduct online surveys with children and young people? If so, please describe the process for obtaining permission.
The ICC/ESOMAR International Code requires special permissions for interviewing children.
20. Do you supplement your samples with samples from other providers? How do you select these partners? Is it your policy to notify a client in advance when using a third party provider? Do you de-duplicate the sample when using multiple sample providers?
Many providers work with third parties. This means that the quality of the sample is also dependent on the quality of sample providers that the buyer did not select. Transparency is a key issue in this situation. Overlap between different panel providers can be significant in some cases and de-duplication removes this source of error, and frustration for respondents.
Partnerships and Multiple Panel Membership
21. Do you have a policy regarding multi-panel membership? What efforts do you undertake to ensure that survey results are unbiased given that some individuals belong to multiple panels?
It is not that uncommon for a panellist to be a member of more than one panel nowadays. The effects of multi-panel membership by country, survey topic, etc., are not yet fully known. Proactive and clear policies on how any potential negative effects are minimized by recruitment, sampling, and weighting practices is important.
data quality and validation
22. What are likely survey start rates, drop-out and participation rates in connection with a provided sample? How are these computed?
Panel response might be a function of factors like invitation frequency, panel management (cleaning) policies, incentive systems and so on. Although not a quality measure by itself these rates can provide an indication of the way a panel is managed. A high start rate might indicate a strong relationship between the panel member and the panel. A high drop-out rate might be a result of poor questionnaire design, questionnaire length, survey topic or incentive scheme as well as an effect of panel management. The new ISO for access panels will likely propose that participation rate is defined as the number of panel members who have provided a usable response divided by the total number of initial personal invitations requesting members to participate.
23. Do you maintain individual level data such as recent participation history, date of entry, source, etc., on your panelists? Are you able to supply your client with a per job analysis of such individual level data?
This type of data per respondent increases the possibility of analysis for data quality, as described in ESOMAR’s Guideline on Access Panels.
24. Do you use data quality analysis and validation techniques to identify inattentive and fraudulent respondents? If yes, what techniques are used and at what point in the process are they applied?
When the sample provider is also hosting the online survey, preliminary data quality analysis and validation is usually preferable.
25. Do you measure respondent satisfaction?
Respondent satisfaction may be an indicator of willingness to take future surveys. Respondent reactions to your survey from self-reported feedback or from an analysis of suspend points might be very valuable to help understand survey results.
26. What information do you provide to debrief your client after the project has finished?
One might expect a full sample provider debrief report, including gross sample, start rate, participation rate, drop-out rate, the invitation text, a description of the field work process, and so on
Facebook Twitter