Decision-Makers' Guide: The 12 killer Questions
The research industry is providing more holistic consumer evidence to aid decision-makers
Let’s develop a ‘charter’ that gives clients the ammunition to evaluate the real quality of consumer evidence.
A document prepared by Dr DVL Smith, Director, DVL Smith Group
on behalf of the ESOMAR Developing Talent Project
This article follows up on the piece in December’s Research World about whether we should now consider preparing some form of ‘Charter’ that would help clients know what ‘telling’ questions to ask to establish the robustness of the consumer evidence they are using for decision-making. The introduction of a Charter would put our industry in the driving seat when it comes to ‘adjudicating’ on the (new) rules that should be applied when determining what constitutes sound evidence based decision-making.
We must avoid an ‘anything goes’ approach to evidence
This is an important development as we are edging towards a point where what we have in place to evaluate the quality of what we do as market researchers is going to look very thin and outdated. This is because these days it is difficult for market researchers to stay close to the gold standard of ‘orthodox’ market research. We often have to go beyond the literal consumer evidence and make various creative interventions to provide customer insights. So if we do not develop something along the lines of a ‘Research Charter’ there is a danger of an ‘anything goes’ approach to evidence based decision-making emerging.
To get industry-thinking underway about possible content, of such a Charter, in this article we provide some illustrations of the types of questions that could be asked by clients to investigate each of the subject areas that we have outlined in the article in Research World. These questions are at the very early stage of development and will need considerable refinement before we arrive at the ‘killer’, telling questions that will give the Charter its bite and edge.
The 12 Killer Questions
1. To what quality level was the research conducted?
Up front in the Charter – and a comparatively straightforward task - is the issue of framing the questions the decision-maker should ask to in order to establish the overall quality assurance standard to which the research was conducted. This line of enquiry would include simple questions to establish whether the research was conducted by agencies affiliated to different industry organisations – where membership is known to be a proxy for attaining a particular quality standard.
2. What impact will the consumer evidence have on the final decision?
The Charter could then include some questions to help the decision-maker establish the impact a particular piece of consumer evidence will make on the final decision. In some cases the consumer evidence will be a critical driver of the decision. But in other situations the consumer evidence could play a much smaller part in the decision-making process. Therefore we need to provide the decision-maker with questions they can ask their (co)management and market intelligence specialists so that at the outset the decision-maker is clear about the ‘sensitivity’ of his/her decision to a particular item of consumer evidence.
|
3. Was the research design fit-to-purpose?
We need to frame some questions to allow the decision-taker to establish where the research evidence they are considering using falls on the market research ‘radar’. Is this a study that represents the highest level of quality and thinking the industry can offer. Or was this piece of evidence always intended to be no more than a light hearted ‘snippet’ of evidence to throw into the media melting pot. The questioning to get to grips with this issue needs to focus on the important issue of whether the study struck the right balance between understanding the issue under investigation at an appropriate level of depth, rather than having gone no further than providing a general overview. In short, the Charter needs some questions to help the decision-maker decide how well the research study struck the balance between ‘shallowness’ and ‘depth’. Do we have a research study that has been structured in a way that can make a serious contribution to the putative decision? Or is this a research study where from the outset many critical problem definitions were never fully resolved, and/or the research structure chosen to deal with the problem was never fully thought through, resulting in the decision-maker ending up with superficial, unsophisticated, off-target, structurally flawed, or ‘thin’ evidence. (And this issue needs to take into account the budget that has been allocated to this problem.)
|
4. What was the ‘agenda’ behind the study?
It is important for the data user to establish the motivations behind those who commissioned (and also executed) the survey. We need to equip our end users with some penetrating questions they can ask the commissioners and/or suppliers of their data to establish whether any of these ‘predilections’ may have interfered with the independence of the research approach taken. The decision-maker needs to know if the overall ‘stance’ adopted for the study could have created any major systematic biases. We need questions to help the decision-maker understand the way the architects of the survey ‘see the world’, and assess whether this has skewed the outcome.
|
In what context was the research conducted?
Even on the assumption that the motives behind the commissioning of a piece of evidence were entirely ‘legitimate’, and this has not introduced any kinds of bias, there remains the related issue of the context in which the research was conducted. Experienced market researchers know that when it comes to interpreting data, drilling down into ever increasing levels of detail only goes so far in improving our understanding of what is really going on. Invariably, the road to truth lies in understanding the wider context in which the research study was conducted. So, the decision-maker needs to know whether his evidence has been influenced by the ‘prevailing context’ at the time at which the evidence was being collected?
|
5. To what extent did the research accurately reflect the target group being researched?
This above question will, of course, be familiar to experienced researchers. First, there is the issue with quantitative research, of whether the research design was structured in a way to ensure that the relevant respondents were not excluded from the study. Then there is the need to establish the extent of ‘sample bias’: are there people in this target who were correctly included in the definition of the survey population, but who then did not respond? These are issues that lead us into questions about the achieved level of ‘response’ (or, in the case of quota samples, ‘strike rate’). And this leads us onto further questions about what such a level means with regard to the interpretation of the results. This is all, of course, very familiar and well documented territory. So, here the challenge for the Charter is not so much knowing what questions to ask, but in finding ways of constructing penetrating, yet accessible, questions that will alert decision-makers to any data that has been faithfully reported, but because it carries unacceptably high degrees of sample bias, is nonetheless dangerous to use for key decisions.
|
6. To what degree has the interview medium affected the results?
The market research industry has been incredibly successful in adapting data collection methods to meet clients’ information needs. But there is perhaps a danger of this pragmatism being accompanied by a lack of resolve to spell out for the end data user the precise implications of using different types of interview medium to solve a particular problem. For example, we all know that online surveys are in many ways the saviour of our industry. They provide a highly effective, low-cost way of investigating various issues. But, by the same token, there is also a rich body of psychology that tells us about how people may behave differently - on certain surveys on certain topics - when embedded in an electronic (internet type) communication, as opposed to how they might behave when operating on, say, a face-to-face basis. So, it is critical that we do not brush these possible effects of the interview medium under the carpet. It is important for us as an industry to explain in a transparent way what may happen in different interview scenarios. So, we must equip the decision-maker with the set of questions they should ask to help them establish to what degree (if at all) the interview method has influenced the findings. And, following on from this, there is the issue of exactly how the data analyst has ‘compensated’ for any such shortfall in the subsequent interpretation.
|
Was the ‘dialogue’ that took place between the respondent and the interviewer/researcher reasonably natural, or was it contrived and/or stilted?
Building on the above point, there is the issue of the ‘character’ of dialogue that took place between the respondent and the interviewer/researcher. The process of asking questions and listening to answers as the basis of understanding what people do and think (and establishing how they may behave in the future) is, at best, a coarse instrument. But we do the users of our data no favours by glossing over some of the ‘challenges’ our industry faces in making sense of the respondent's ‘literal’ responses to our qualitative and quantitative questioning. Thus, the Charter will need to include questions to help the data user establish just how appropriate and successful the use of various types of ‘direct’, and ‘indirect’, questioning techniques have been in going beyond respondent's tendency to reel off generalised, what we may term ‘platitudes’, rather than their ‘true’ attitudes. This calls for us to equip end users with some questions that will help them determine the overall quality and meaningfulness of the dialogue that took place between the respondent and the interviewer. It is critical for the decision-maker to reassure themselves that they understand the ‘frame of reference’ within which individuals were giving answers to questions. Much poor quality market research is the result of surveys that have not done justice to an individual respondent’s knowledge: the research has just not allowed participants to fully explain their viewpoint. The result is often mindless tables of yea, or nay, giving data that could lead the data user widely astray when it comes to understanding a particular market, or group of consumers. So, it is important for the decision-maker to be clear about the salience of the issues that were being presented to individuals.
|
7. Did the interpreters of the research evidence really understand the critical ‘research effects’ at work?
Experienced market researchers will, of course, pass their qualitative and quantitative evidence through what we might call various ‘knowledge filters’ in order to establish whether there were any key ‘research effects’ at work that materially affected the interpretation of the evidence. These ‘knowledge filters’ have been built up by the market research industry over decades of practical research experience. Seasoned researchers are aware of a whole array of issues, such as the quality of a respondent’s memory in recalling details of different types of past behaviour. This is familiar territory to the market research community. But the challenge, in developing a Charter, is how to equip decision-makers with some penetrating questions they could ask of their data suppliers to unearth any critical ‘research effects’, and allow them to check that an informed appreciation of the ‘effects’ has been factored into the interpretation of the data.
|
8. Was the appropriate analysis approach deployed?
The Charter needs to include questions that help the decision-maker establish whether the ‘appropriate’ approach to the analysis was pursued. This is clearly a massive and challenging area for the Charter. But we should not shrink from the task of bringing down quite big and complex technical ideas into simple to understand concepts that can inform users’ evaluation of their evidence. The end user needs to be reassured that the analyst has constantly ‘toggled’ between the initial emerging picture and each successive piece of evidence that became available. The data user needs to know that the analyst has not simply selectively looked for evidence that supports a prior theory, but instead has gone through a rigorous iterative process.
|
To what level of technical excellence were the key techniques executed?
Building on this, there is then the issue of the level of technical excellence that was applied to the analysis, for instance, establishing that the most appropriate stats test has been applied, and so on. This line of enquiry is tricky because this is very much the domain of the market research specialist. But the Charter should still not shy away from designing some questions that will focus decision-makers on the key principles, while also alerting them as to when to draw in experts to evaluate critical technicalities.
|
9. What level of ‘business conceptualisation’ has been brought to the presentation of the consumer evidence?
It is important for today’s marketing intelligence industry to bring their consumer evidence ‘alive’ for the decision-maker by presenting their evidence in the context of appropriate business frameworks and models. Such ‘conceptualisations’ give consumer evidence more meaning for a senior decision-making audience. Here the Charter could serve to encourage decision-makers to ‘demand’ from their data supplier that their consumer evidence has been set in the right business context.
|
What degree of ‘creative stretch’, and leading edge thinking, has been applied to the literal consumer evidence?
Extending the above point there is the issue of the degree of ‘creative stretch’, and leading edge thinking, that was applied to the original evidence. Increasingly, it is expected that market researchers will get underneath the ‘literal’ consumer evidence and make some form of ‘creative leap’ in a bid to pinpoint fresh insights. Today there is an expectation that market researchers will go beyond the respondent’s world by looking at parallel markets, and by applying various creative techniques, in order to get a wider understanding of what the immediate consumer evidence is really telling us. Therefore the Charter needs to include questions that will reassure the decision-maker that the interpretation of their evidence has benefited from wider, creative, ‘off-the-wall’ contextual thinking. This needs to be done alongside providing reassurances that sufficient caution and prudence has also been exercised in ‘stretching’ the data in this way.
|
10. How well has all the available evidence been integrated into an impactful compelling story, and have the outcomes of the study been presented in a way that will influence the decision-making audience?
The market research industry has made great strides with regard to the quality of its presentations. Gone are the days when market researchers could be accused of giving uninspiring, dry, boring, tedious, data-laden presentations. Today market researchers can be proud of the energy, enthusiasm and flair that they bring to most of their presentation. The industry has moved on from the days when it just analysed isolated, solitary datasets, and only focused on what was ‘pure’ valid and reliable evidence. Today we are much better at building arguments from multiple sources of both orthodox, and more ‘imperfect’ evidence. We are becoming skilled at weaving together different evidence to find creative solutions to business problems. But the bar keeps getting raised. Thus it is now expected that market researchers will go beyond the skills needed to give a good presentation. It is now also expected that they will focus on their part in the ultimate goal of explaining how the research evidence can successfully impact on the bottom-line profitability of the client company. So the Charter will need questions to help the data user reassure themselves that everything possible has been done by the supplier to constructively ‘impact’ on the decision-making process.
|
11. To what extent has the data been presented in the context of what we know about this genre of evidence-based decisions?
Today it is generally accepted that suppliers of market research data need (when appropriate) to extend their role to that of a ‘decision facilitator’. Decision-takers now expect to be guided through the potential decision-making ‘minefields’ they face. Specifically the end users of our data are entitled to expect that the supplier of their data has alerted them to the typical errors that may be made with this type of evidence-based decision-making. So, suppliers need to go beyond simply making recommendations based on their research findings and take personal responsibility for making sure their evidence is actually being utilised in an informed way.
|
How well has the end decision been ‘framed’ according to the personality and ‘style’ of the end decision-maker?
Developing the decision-facilitation point still further, a CEO once said, "Once I have got a feel for the choices, making the decision is straightforward". So this brings us on to the way the research to be used for a decision has been ‘framed’ by the supplier for the end decision-maker. Has the agency - given what we know about the personality, style and methodological predilections of the decision-maker – structured the evidence and arguments in the best possible way to facilitate an intelligent and informed decision? Specifically, this ‘framing’ involves making sure that the decision-maker is clear about: the ‘safety’ of each piece of evidence that impacts on their decision; aware of the precise implications of interpreting a piece of evidence in a particular way; and is told about the overall successful track record of this genre of consumer evidence being used in this decision scenario.
|
12. Has the tactical feedback from the latest research study on this topic been incorporated into a wider ‘meta-analysis’ in order to identify any over-arching strategic trends?
It is important for the decision-maker to be aware of the power of bringing together – in the form of a ‘meta’ analysis - lots of different tactical insights to see whether, when taken in the round, this wider contextual appreciation of the issue improves the richness of our understanding and/or unearths a bigger overarching trend, concept or insight. So the Charter will need questions that will encourage the decision-maker to demand that this wider contextualisation of what we know already about this issue is carried out.
|
KEEP ME INFORMED |