Picture the scene: a CMO faced with a rapidly changing market needs to get to the heart of why too many target prospects are buying from their competitors. The goal is to identify how the company should finesse the proposition so that more prospects entering the top of the funnel make their way to the point of purchase and beyond.
Rather than second guess the market, the CMO commissions a large-scale quantitative study to help understand how core personas in a variety of target segments take decisions at key stages of the buyer process.
It might be the only time in the annual planning cycle that the CMO gets the chance to take the pulse of market sentiment, so they are keen to maximise the value the company can get from the research.
Having agreed the research objectives, audience and methodology, the next step is to agree the questionnaire. And it’s here that for some CMOs the wheels can start to fall off, all be it often unwittingly.
MISTAKE #1 – TELLING THE RESEARCHER WHAT QUESTIONS TO ASK
It’s surprising how many times clients approach researchers with a list of fully fleshed questions they want to put to their market.
Quite often they do it with a sincere wish to be helpful, or to save time but it can often be counterproductive.
Think of it like this; you wouldn’t hire an expert ad agency to run an ad on TV that you’d filmed yourself, would you?
How to avoid mistake #1
Provide the researcher with the business context in which you are commissioning the research.
Don’t pre-judge what questions you think you should be asking. Tell the researcher what it is you want to find out, from whom, why, and what you will do with the information when the results come in.
Be clear about what you need to find out but leave the writing of the actual questions to the researcher.
A good researcher well-versed in your business context should be able to craft a succinct questionnaire that intelligently draws out the themes you wish to explore and give you the answers you need.
They will structure it logically, phrase questions in a way that maintains the respondents’ engagement and avoid asking questions that lead to respondent bias.
In return, you should get genuine insights that help to drive your business forward, rather than a set of numbers that simply confirm existing prejudices within your business.
MISTAKE #2 – ASKING TOO MANY QUESTIONS
As we’ve said, the research being commissioned will likely be the only opportunity the CMO gets to gain a window on their customers’ world. This is true in B2B, where research budgets can be much more restrictive than in the B2C world.
This brings with it a huge temptation to ask as many questions as possible to wring every last drop of value from the project.
Paradoxically, this too can be counterproductive (though for some reason not all agencies seem to be keen for their clients to understand why.)
The reason is to do with the mechanics of quantitative research.
Sad to say, but a fact nonetheless, most respondents taking part in a quantitative survey are far less interested in the outcome than the marketers that commission the research, even if the respondents are incentivised.
This means that researchers and their clients can’t expect participants to commit to giving their undivided attention and full consideration to every question.
You only have to listen to a respondent being taken though a lengthy, directionless questionnaire to sense the increasing frustration at how long the process is taking. The disinterest in their voice rises as their concentration diminishes, and the thought they give to answering each question consequently declines with each passing minute. The value that could be gained from asking the questions simply slides away.
It can get even worse with questionnaires that are completed online. Within a remarkably short time, respondents can get bored and start to click on random answers just to get to the end.
Expert researchers can generally spot the point at which this starts happening as the response data starts to get flatter and flatter. But what they can’t do is filter out the respondents that were genuinely paying attention versus the ones that got bored.
As a result, the client is left with results to multiple questions that are inconsequential or, worse still, unreliable, contradictory or misleading.
How to avoid mistake #2
In B2B research, a good researcher, given a solid brief, should be able to construct a meaningful quantitative survey of 15 questions that gets to the nub of the client’s core issues and takes no more than 15 minutes to complete.
(To be clear, a question that gives six statements and asks a respondent to review each in turn and indicate to what extent they agree or disagree with each would count as a single question.)
Choosing your questions wisely and keeping your questions crisp has the added advantage of speeding the rate of data collection. That’s because the longer the questionnaire, the greater the drop off in completion rates. Therefore, the more respondents the researcher approaches to complete the questionnaire, the longer it takes to get a quorate number of responses. (It’s for this reason that shorter questionnaires tend to cost less to put into field, by the way).
This is particularly important in B2B research if the personas you wish to survey are relatively niche. High respondent drop-off is less of an issue if you are researching UK households who buy mayonnaise, for example. If some respondents drop out, there’s probably another 20 million households you could go out to.
But if you need to survey Procurement Directors in the UK telco infrastructure supply chain, there may only be a respondent universe of a few hundred people qualified to answer your questions. It’s therefore imperative they remain engaged throughout the process.
MISTAKE #3 – EXPECTING THE RESPONDENT TO DO ALL THE THINKING
When devising multi-answer questions, it’s always tempting to add a catch all “Other (please write in)” response option.
The thinking is straightforward enough: “We may have missed something important – the respondents will tell us if this is the case”.
However, most clients are disappointed in how few respondents bother to complete the “Other” box.
As we’ve already explained, respondents’ commitment to completing the questionnaire can fall frustratingly short of the client’s commitment to uncovering the answers they need. So questions that ask respondents to think and type for themselves tend to be given less thought than questions where the answer range has already been thought through for the respondent to consider.
Not only that, having a question with an open answer field option creates a challenge in accurately interpreting the answers. That’s because this kind of question ends up blending prompted and unprompted answer sets in the same question. Consider it this way; had any relevant “please write in” answer been included in the original answer set, would more respondents have chosen that option as an answer?
How to avoid mistake #3
The best way to mitigate this outcome is to do some qualitative interviews with your core audience prior to constructing the quantitative questionnaire. A skilled interviewer can then use the insight to tease out what the most likely answer set will be for a quant questionnaire.
The second-best way to approach the problem is to pilot the questionnaire with a relatively small sample of respondents prior to rolling out the full survey. This is good practice in any case as it can help show up unanticipated flaws in the questionnaire logic or help reveal where an unintentionally ambiguously worded question is throwing up some bizarre answers.
By reviewing where pilot respondents have completed any “Please write in” answers, the client and researcher can decide whether these extra answers are sufficiently insightful to be included in the prompted answer set when the questionnaire is rolled out.
Though it’s worth noting that even doing this can distort the validity of the final data. That is unless the pilot respondents’ answers are disregarded and then replaced with an equivalent cohort of additional respondents, ie all final respondents see the same answer set.
In summary, if you are thinking of commissioning market research:
- The best research briefs inform the researcher what the client needs to find out, not what questions need to be asked.
- The best questionnaires are crisp and engaging. Remember that over a certain boredom threshold, the more questions asked, the less reliable the answers will be to the later questions, and the more contradictory the data.
- It’s always wise to do some qual research prior to running the quant, even if it’s only amongst four or five subject matter experts. It’ll tease out angles that neither the client nor the researcher have anticipated and ultimately lead to better insights.
If you need help shaping your research strategy, we can help just contact us.
- The 3 key mistakes marketers make when commissioning quantitative research – and how to avoid them - February 27, 2023
- How to keep your head, and grow your business in a recession - January 19, 2023