Market Research: Is There Just Too Much of It?
In his latest post, B&T regular Robert Strohfeldt argues that despite all the market research, is it really telling adland what it needs to hear?
Change. The one variable that is constant. And usually, change means better. No one changes to something that was worse than what they had before. For those who hanker for the “good old days”, think about what it would have been like to go to the dentist 100 years ago?
DAYS OF INNOCENCE
Way back in 1981, when I first joined a market research firm fresh out of university, I was engaged initially as a mathematician to ensure proper random sampling.
Research firms had their own field force back then and most quantitative surveys were done door to door, face to face. We worked from Collection Districts from the ABS. A major issue was cost – an interviewer was given a strict set of dwellings and if nobody, or the appropriate person, was not home, they were to try three times before replacement a dwelling was allowed.
Decent sample size, quantitative studies were too expensive for all but the large companies. Anecdotal findings, coupled with experience, were often all small businesses had.
There was also the obscene amount of paper used – imagine a six-page questionnaire for 1000 respondents.
The number of successful interviews achieved in a day was “bugger all” and a sample of 500 or 1000, took ages just to complete the field work. And of course, the cost was often seen as prohibitive for the perceived results. (That the results many times reinforced what the marketing team believed they already knew, was deemed a waste of money).
In the early 80’s there were not any PCs – coded questionnaires (coding is a valuable art that has been lost) were taken to a company who produced a computer print- out. Often, after working through the first run, additional reports were produced – “Can we see how the breakdown how of respondents in question four answered question six?”. On top of this any statistical analysis was done separately, again after all report runs had been completed.
Often, the alternative was “Let’s just run a few groups”. Great, an-depth discussions with 30 odd people was not really an accurate indication of anything, bar the lack of understanding of how research was best used.
Many clients would baulk at spending $30,000-$50,000 to optimise a $5 million campaign.
By 1983 I had graduated to running groups and have fond memories of running focus groups in peoples’ kitchens. The respondents were genuinely thrilled that large consumer companies sought out their opinions and engaged in discussions accordingly.
It was a novelty and as such they gave honest answers and when the topic was exhausted, that was it, the group was over. (Easy to spot) Yes, they were paid, but many times I had respondents state they would have participated even if not paid.
Many golden nuggets of consumer thinking (and statements, or in today’s terminology, content) were gleaned from them.
COST AND TECHOLOGY. AGENTS OF CHANGE
Telephone surveys took over from door to door sampling – no- where near the cost of sending interviewers out into the field.
For market researchers, their most valuable natural resource is respondents. Telephone interviews soon became problematic thanks to telemarketers.
“Hi, I am doing a survey on house cladding”. Several question later they knew your type of dwelling and if you were a candidate for cladding. And in no- time there was a knock at the door, or sometimes a phone call, from a person trying to sell you cladding.
And as telemarketing grew, many people had experienced having their dinner disturbed by someone trying to sell them something, camouflaged as market research.
“Piss off” became a frequent response. Many households grew tired of receiving telephone calls, be they from a genuine research organisation or a telemarketer.
By this time, PCs were in all research companies and the questionnaires were on screen. As the interviewer “filled in” the questionnaire on screen, the first run of data was being done simultaneously. Not anything to marvel at today, but when this first became available, it was revolutionary and research companies tried to push all quantitative surveys into telephone surveys.
But all the while, researchers’ most valuable natural resource, willing and uncorrupted respondents, was gradually being eaten away.
WHO HAS NOT BEEN SURVEYED?
Fast forward to 2019 and I doubt there is a person over the age of 18 who has not been surveyed in some form. As more and more businesses move online, they see it as an opportunity to tag a few research questions at the end.
The quality of the research results is totally dependent upon the quality (accuracy) of the sample – is it representative of the group you want to survey? “Normally distributed, proportionate to size”, was a phrase used in every quantitative study back when door to door was the norm. The sample would be “reweighted” to the exact proportions of the various demographic groups in the population being studied.
Research is the same as computers i.e. rubbish in = rubbish out.
I have no doubt that reputable research firms adhere to strict sampling disciplines, but this does not solve the problem of “too much research”.
I had lunch recently with two friends who have nearly 80 years of market research experience between. They felt that too much research kills our natural resource – the unbiased and honest respondent. Our chat formed the genesis of this article.
Many large research companies today run panels – tens of thousands of people who, for a reward (often cash), are willing to be respondents. By having all the relevant demographic information of each panel member and by filtering via certain behavioural questions, they can supply a ready- made sample.
But just how random is this group of people? They can distort focus group discussions – a couple may attend up to two groups each a month for $80 (or more) per group. And they are also paid to participate in quantitative studies. This can give them up to even $320 a month extra cash in hand, nothing to be sneezed at.
They are paid to give an opinion and so they do, but whether this is what they really believe, or what they think they should say, no one can truly tell.
“Do you buy jam” – “yes”
“What brand did you last buy?”. The brand manager for Cottee’s jams is deeply involved in the brand, but the average shopper only gives it a cursory consideration when shopping. “Last brand? Um, Cottee’s”. It may not be, but as they are paid for an opinion, they better have one. It could have just as easily been another brand that was on special. But Cottee’s is top of mind and gets the vote.
SEVEN SOUNDS LIKE A GOOD NUMBER
And then there is the classic “How likely would you be to recommend…….?” The Nett Promoter Score. I read not long ago the new MD of a large private health insurance company was going to link all senior executives’ bonuses to Nett Promoter Scores. Poor bastards!
Mark Ritson wrote a column recently about CEOs’ messing up marketing and advertising- what a colleague refers to as “Seagull Management” – they fly in, shit over everything and fly off.
This was probably a good example. The MD had probably heard of The Nett Promoter score and thought “Bloody brilliant. I will introduce that”.
Where once a survey was something rare and genuinely regarded as being worthwhile, people are asked to rate something nearly every day. And hence we have “Seven sounds like a good number”. Nett Promoter Scores may have been worthwhile years ago before the natural resource of “virgin” or at least “rarely used respondents” had been used up.
Now, people rattle off numbers between five and seven, unless something extraordinarily good or bad had happened to them.
A/B testing is another hoary chestnut. Not the place to detail an in-depth dissertation on the problems with this technique. It has become so commonly used that I would prefer to take the opinion of an experienced and intuitive creative person to finalise a headline or ad concept, over the answer given by some spreadsheet A/B testing technique.
Survey Monkey has allowed smaller businesses to conduct their own research. Sometimes this does throw up useful information, but often it is conducted by people with no experience in sampling and interviewing. The analogy is of the person who acts as their own legal counsel has a fool for a client.
We know that thanks to Google, Social Media and almost no end of information sources, consumes are far better informed than ever.
I have seen focus groups in which the respondents were acting more as creative directors – “It would have been more impactful if you had used morphing”.
Testing communications against their objectives is becoming more and more testing.
Data seems to have usurped research as a source of insights. In many companies, the market research and data insights are separate departments. Where once we referred to data as Primary Research – all available information (data) was analysed prior to any market research being developed – what do we know, or think we know, what gaps are there is our knowledge etc. Market research and data were all part of marketing intelligence. There was no demarcation between the two, they worked together seamlessly.
IS THERE AN ANSWER?
There have been many fantastic new research techniques and programmes developed over the past 40 years. That is a given. But it does not address the single biggest problem facing any form of market research. As with data, quality should always take precedence over quantity.
No matter how good the technique, if the sample is a dud, then the results will turn out the same. Nielsen introduced, a number of years ago, a qualitative research technique they called “Delta Qual”. Derived from forensic psychology, its aim was to determine what people actually did when grocery shopping, not what they thought they did.
Not a simple or cheap technique, it required two moderators. Though this approach is more likely than many techniques to find the truth, it is still heavily dependent upon the quality of the respondent.
“Killing the goose that laid the golden egg”, was how one researcher described the overuse of research.
This is a topic in which it is far easier to identify the problem than it is to provide an answer. Cliched as it is, the answer is “each case is different”.
Prior to any research study, be aware of the fact that many of the respondents are jagged and give an answer off the top of their head. If you have received feed back from say 5,000 people and 4,000 of them are just going through the motions, then the results will be rubbish.
How to identify and access the respondents who are genuinely interested in providing answers? Of course, this then raises the problem of statistical validity. Make sure you know the demographic and relevant behavioural breakdown of your target group and then reweight the results accordingly.
With qualitative research, speak to the people who are either very happy, or very pissed off.
Have you ever been to a restaurant where the waiter or waitress continuously interrupts to ask, “How is everything”? Maybe we need to learn from this. Stop asking questions after every customer contact and learn to identify those who really do have something to say.
Latest News
EMG/Gravity Media Names New Managing Director
The makers of Australia's Got Talent has been a magnet for senior hires in recent times.
Opinion: Shark Tank’s Sabri Suby Slams Politicians Jumping On TikTok Ban Bandwagon
Are politicians actually scaremongering about TikTok or just fed up of you doom-scrolling in the work bogs?
TV Ratings: “Ivana Was Robbed” – Final Three Contestants Revealed On Australian Idol
B&T's unsure if these Idol contestants look extra young or whether we are as old as we feel.
Veridooh Snatches Prashand Menon From ScentreGroup
Out-of-home OOH) and digital media leader Prashand Menon (lead image) has joined OOH ad tech company Veridooh as group business director within NSW and QLD. Menon joins Veridooh from his most recent role as ScentreGroup’s group business director and brings with him more than 10 years of experience in Australian media. Previously, he held positions […]
Creative Content Studio OK COOL Answers APAC’s Call, Opening In Melbourne
Life set to get extra confusing for new agency staffers mistaking OK COOL for a pass-ag housemate text message.
Harry’s Cafe de Wheels & Hipages Launch New Pie To Fuel Tradies While On The Tools
The pies should apparently be better for tradies' health, apparently but won't be better for on-site ribbing.
Women Leading Tech’s Woman Of The Year Megan Dalla-Camina: What’s The True Cost Of Cutting DEI Spending?
Cutting DE&I spending might seem easy but you might change your mind when faced with Dalla-Camina's ire.
Thrive Introduces A Batch Of New Senior Leadership Roles
You could say that Thrive is set to thrive with these new hires. But that would be too obvious.
It’s Coming Home! Sponsors Make Subbuteo World Cup Finals Free
No idea what Subbuteo is? You clearly had a more exciting childhood than our editorial team.
Media Bigwigs Bet On Predictive Intelligence Platform Prophet In $5M Seed Funding
Prophet, profit, geddit? Must be laugh-a-minute in this startup's office.
CommsDeclare Brands Shell Energy An “Olympic Gaslighter” & Urges Agencies Not To Pitch For Creative Account
And we thought the gaslight Olympics took place in the Coogee Bay Hotel smoking area!
AFL Wins Battle With Warner Bros. As New AFL Team Is Revealed
There hasn't been this much wrangling over a cartoon animal since, well, ever!
BBC Podcasts Go Global With Amazon Music
Ironically, BBC podcasts unlikely to be available in the Amazon given the lack of signal.
Johnson & Johnson’s Former Consumer Healthcare Arm Reviews Creative
J&J inspired by Scottish term for knowledge 'ken' in its rebrand; nothing says 'wise' like a shot of Listerine.
Kat & Co Transforms Into THE WORLD OF Experiential Agency
After more than twelve years of producing experiences for some of the world’s most innovative brands, including Hermes, Van Cleef & Arpels, Louis Vuitton, and Tiffany & Co, Kat & Co has quietly metamorphosed into THE WORLD OF. THE WORLD OF is an experiential agency at the juncture of culture, technology and the human condition. […]
Adobe Brings Firefly Generative AI Into Substance 3D Workflows
Still unclear of what generative AI actually is? Steer clear of this story.
Clemenger BBDO Introduces Saucy Campaign With Declan Cleary & Lee Kum Kee
When we heard this was a saucy campaign, we thought it be the barbecue and ketchup showdown we'd been waiting for.
Gawk Acquires 40 Billboard Locations From Transad
Gawk makes it move out of VIC into NSW with acquisition. Has promised to stop going on about Melbourne nightlife.
All About Attribution: Commission Factory Releases New eBook
Ever wanted to dive deep into channel attribution? Consider this the perfect springboard.
SMG Studio Launches Risk: Global Domination X Dune Part Two
Through a licensing agreement with Legendary Entertainment, SMG Studio inked the rights to adapt Dune: Part Two into a game based on Risk: Global Domination, Hasbro’s interactive board game. Inspired by Denis Villeneuve’s cinematic masterpiece, Dune: Part Two, this new digital adaptation, RISK: Global Domination X Dune: Part Two, is poised to captivate players from […]
RMIT Study Reveals That 73% Of Employees Don’t Believe Gen AI Is Relevant
Noticing staff switching off when you tell them about gen AI? Understand why here.
Australian Eggs Launches Egg-citing New Campaign Via Connecting Plots
How do you like your eggs in the morning? We usually skip them and go straight for the darts and whisky.
Future Lions 2024 Invites Young People To Be Heard With Spotify
Future Lions, of course, is the AKQA-backed student initiative at Cannes Lions, not a feeder for the rugby team.
QMS: Taylor Swift, Lunar New Year & Mardi Gras Were Boon For OOH Market
Taylor Swift, Lunar New Year and the Sydney Gay and Lesbian Mardi Gras Festival have had a dramatic impact on the number of people moving in and around Sydney over the past month, drawing big crowds into the heart of the city and its surrounding suburbs. New data from digital outdoor company QMS reveals a […]
Legend John McEnroe Joins Pepperstone’s Team Of Global Tennis Ambassadors
Despite the deal, McEnroe remains confused about difference between above- and below-the-line marketing.
LinkedIn Launches New Thought Leader Ads For Companies To Promote Member Posts
Though LinkedIn couldn't get any more insufferable? You thought wrong.
Latrell Mitchell Smacked On Wrist After Dropping Multiple F-Bombs In Post-Match Triple M Interview
Andrew Abdo has confirmed he is not angry... just disappointed.
TV Ratings: “When You Do Love Yourself, You’re Amazing” – Is This MAFS Most Heart-breaking Split?
Hold the press. MAFS groom and bride actually shown to be nice people.
Opinion: The Only Drag Marketers Should Be Worried About
Thought Transactional Drag was a lifeless club night in Newtown? Read this and you'll think again.
AWARD School Welcomes Class Of 2024
It's best to think of AWARD School like Survivor but with pitch decks instead of swimwear.
Interest In Motorsports Reaches New Heights As Formula 1 Grand Prix Revenue Expected To Surpass $100 Million
Need to get a dull sports client some quick results? Rustle up a Netflix series and watch the dollars roll in.
Chair Rob Murray To Depart As SCA Considers ARN’s Latest Takeover Bid
Struggling to make heads or tails of the SCA & ARN saga? Add this to your woes.
Adland Legend John Singleton Separates From Seventh Wife
Singo calls it quits with seventh wife as his race with Rupert Murdoch to become the most-married man continues.
ACCC Eyes Search Competition, Citing ‘Decreasing’ Quality & Rise In Generative AI
The only decline in search quality we notice is when B&T doesn't appear at the top of Google.
“Imagine A Career Above The Clouds”: Royal Australian Air Force Launches New Campaign Via VML
Don't worry about the forthcoming war with China - at least you'll get a bird's-eye view of the apocalypse.
MKTG Launches PR & Talent Division Realigning Business Structure
Dentsu‘s MKTG Sports and Entertainment has unveiled a significant evolution, including the launch of a specialised PR and Talent Marketing division, investment in expanding its Creative Services group, and a centralised approach to brand and rights holder servicing with increased production capabilities and a strengthened in-stadium sports presentation team. Lead Image: MKTG CD Ross Allen; […]