Market Research: Is There Just Too Much of It?
In his latest post, B&T regular Robert Strohfeldt argues that despite all the market research, is it really telling adland what it needs to hear?
Change. The one variable that is constant. And usually, change means better. No one changes to something that was worse than what they had before. For those who hanker for the “good old days”, think about what it would have been like to go to the dentist 100 years ago?
DAYS OF INNOCENCE
Way back in 1981, when I first joined a market research firm fresh out of university, I was engaged initially as a mathematician to ensure proper random sampling.
Research firms had their own field force back then and most quantitative surveys were done door to door, face to face. We worked from Collection Districts from the ABS. A major issue was cost – an interviewer was given a strict set of dwellings and if nobody, or the appropriate person, was not home, they were to try three times before replacement a dwelling was allowed.
Decent sample size, quantitative studies were too expensive for all but the large companies. Anecdotal findings, coupled with experience, were often all small businesses had.
There was also the obscene amount of paper used – imagine a six-page questionnaire for 1000 respondents.
The number of successful interviews achieved in a day was “bugger all” and a sample of 500 or 1000, took ages just to complete the field work. And of course, the cost was often seen as prohibitive for the perceived results. (That the results many times reinforced what the marketing team believed they already knew, was deemed a waste of money).
In the early 80’s there were not any PCs – coded questionnaires (coding is a valuable art that has been lost) were taken to a company who produced a computer print- out. Often, after working through the first run, additional reports were produced – “Can we see how the breakdown how of respondents in question four answered question six?”. On top of this any statistical analysis was done separately, again after all report runs had been completed.
Often, the alternative was “Let’s just run a few groups”. Great, an-depth discussions with 30 odd people was not really an accurate indication of anything, bar the lack of understanding of how research was best used.
Many clients would baulk at spending $30,000-$50,000 to optimise a $5 million campaign.
By 1983 I had graduated to running groups and have fond memories of running focus groups in peoples’ kitchens. The respondents were genuinely thrilled that large consumer companies sought out their opinions and engaged in discussions accordingly.
It was a novelty and as such they gave honest answers and when the topic was exhausted, that was it, the group was over. (Easy to spot) Yes, they were paid, but many times I had respondents state they would have participated even if not paid.
Many golden nuggets of consumer thinking (and statements, or in today’s terminology, content) were gleaned from them.
COST AND TECHOLOGY. AGENTS OF CHANGE
Telephone surveys took over from door to door sampling – no- where near the cost of sending interviewers out into the field.
For market researchers, their most valuable natural resource is respondents. Telephone interviews soon became problematic thanks to telemarketers.
“Hi, I am doing a survey on house cladding”. Several question later they knew your type of dwelling and if you were a candidate for cladding. And in no- time there was a knock at the door, or sometimes a phone call, from a person trying to sell you cladding.
And as telemarketing grew, many people had experienced having their dinner disturbed by someone trying to sell them something, camouflaged as market research.
“Piss off” became a frequent response. Many households grew tired of receiving telephone calls, be they from a genuine research organisation or a telemarketer.
By this time, PCs were in all research companies and the questionnaires were on screen. As the interviewer “filled in” the questionnaire on screen, the first run of data was being done simultaneously. Not anything to marvel at today, but when this first became available, it was revolutionary and research companies tried to push all quantitative surveys into telephone surveys.
But all the while, researchers’ most valuable natural resource, willing and uncorrupted respondents, was gradually being eaten away.
WHO HAS NOT BEEN SURVEYED?
Fast forward to 2019 and I doubt there is a person over the age of 18 who has not been surveyed in some form. As more and more businesses move online, they see it as an opportunity to tag a few research questions at the end.
The quality of the research results is totally dependent upon the quality (accuracy) of the sample – is it representative of the group you want to survey? “Normally distributed, proportionate to size”, was a phrase used in every quantitative study back when door to door was the norm. The sample would be “reweighted” to the exact proportions of the various demographic groups in the population being studied.
Research is the same as computers i.e. rubbish in = rubbish out.
I have no doubt that reputable research firms adhere to strict sampling disciplines, but this does not solve the problem of “too much research”.
I had lunch recently with two friends who have nearly 80 years of market research experience between. They felt that too much research kills our natural resource – the unbiased and honest respondent. Our chat formed the genesis of this article.
Many large research companies today run panels – tens of thousands of people who, for a reward (often cash), are willing to be respondents. By having all the relevant demographic information of each panel member and by filtering via certain behavioural questions, they can supply a ready- made sample.
But just how random is this group of people? They can distort focus group discussions – a couple may attend up to two groups each a month for $80 (or more) per group. And they are also paid to participate in quantitative studies. This can give them up to even $320 a month extra cash in hand, nothing to be sneezed at.
They are paid to give an opinion and so they do, but whether this is what they really believe, or what they think they should say, no one can truly tell.
“Do you buy jam” – “yes”
“What brand did you last buy?”. The brand manager for Cottee’s jams is deeply involved in the brand, but the average shopper only gives it a cursory consideration when shopping. “Last brand? Um, Cottee’s”. It may not be, but as they are paid for an opinion, they better have one. It could have just as easily been another brand that was on special. But Cottee’s is top of mind and gets the vote.
SEVEN SOUNDS LIKE A GOOD NUMBER
And then there is the classic “How likely would you be to recommend…….?” The Nett Promoter Score. I read not long ago the new MD of a large private health insurance company was going to link all senior executives’ bonuses to Nett Promoter Scores. Poor bastards!
Mark Ritson wrote a column recently about CEOs’ messing up marketing and advertising- what a colleague refers to as “Seagull Management” – they fly in, shit over everything and fly off.
This was probably a good example. The MD had probably heard of The Nett Promoter score and thought “Bloody brilliant. I will introduce that”.
Where once a survey was something rare and genuinely regarded as being worthwhile, people are asked to rate something nearly every day. And hence we have “Seven sounds like a good number”. Nett Promoter Scores may have been worthwhile years ago before the natural resource of “virgin” or at least “rarely used respondents” had been used up.
Now, people rattle off numbers between five and seven, unless something extraordinarily good or bad had happened to them.
A/B testing is another hoary chestnut. Not the place to detail an in-depth dissertation on the problems with this technique. It has become so commonly used that I would prefer to take the opinion of an experienced and intuitive creative person to finalise a headline or ad concept, over the answer given by some spreadsheet A/B testing technique.
Survey Monkey has allowed smaller businesses to conduct their own research. Sometimes this does throw up useful information, but often it is conducted by people with no experience in sampling and interviewing. The analogy is of the person who acts as their own legal counsel has a fool for a client.
We know that thanks to Google, Social Media and almost no end of information sources, consumes are far better informed than ever.
I have seen focus groups in which the respondents were acting more as creative directors – “It would have been more impactful if you had used morphing”.
Testing communications against their objectives is becoming more and more testing.
Data seems to have usurped research as a source of insights. In many companies, the market research and data insights are separate departments. Where once we referred to data as Primary Research – all available information (data) was analysed prior to any market research being developed – what do we know, or think we know, what gaps are there is our knowledge etc. Market research and data were all part of marketing intelligence. There was no demarcation between the two, they worked together seamlessly.
IS THERE AN ANSWER?
There have been many fantastic new research techniques and programmes developed over the past 40 years. That is a given. But it does not address the single biggest problem facing any form of market research. As with data, quality should always take precedence over quantity.
No matter how good the technique, if the sample is a dud, then the results will turn out the same. Nielsen introduced, a number of years ago, a qualitative research technique they called “Delta Qual”. Derived from forensic psychology, its aim was to determine what people actually did when grocery shopping, not what they thought they did.
Not a simple or cheap technique, it required two moderators. Though this approach is more likely than many techniques to find the truth, it is still heavily dependent upon the quality of the respondent.
“Killing the goose that laid the golden egg”, was how one researcher described the overuse of research.
This is a topic in which it is far easier to identify the problem than it is to provide an answer. Cliched as it is, the answer is “each case is different”.
Prior to any research study, be aware of the fact that many of the respondents are jagged and give an answer off the top of their head. If you have received feed back from say 5,000 people and 4,000 of them are just going through the motions, then the results will be rubbish.
How to identify and access the respondents who are genuinely interested in providing answers? Of course, this then raises the problem of statistical validity. Make sure you know the demographic and relevant behavioural breakdown of your target group and then reweight the results accordingly.
With qualitative research, speak to the people who are either very happy, or very pissed off.
Have you ever been to a restaurant where the waiter or waitress continuously interrupts to ask, “How is everything”? Maybe we need to learn from this. Stop asking questions after every customer contact and learn to identify those who really do have something to say.
Latest News
Sydney Comedy Festival: Taking The City & Social Media By Storm
Sydney Comedy Festival 2024 is live and ready to rumble, showing the best of international and homegrown talent at a host of venues around town. As usual, it’s hot on the heels of its big sister, the giant that is the Melbourne International Comedy Festival, picking up some acts as they continue on their own […]
Global Marketers Descend For AANA’s RESET For Growth
The Australian Association of National Advertisers (AANA) has announced the final epic lineup of local and global marketing powerhouses for RESET for Growth 2024. Lead image: Josh Faulks, chief executive officer, AANA Back in 2000, a woman with no business experience opened her first juice bar in Adelaide. The idea was brilliantly simple: make healthy […]
Is Meta’s New AI Chatbot Too Left-Wing?
Meta's chatbot accused of being left-wing after being caught wearing a Che Guevara T-shirt & listening to Billy Bragg.
TV Ratings (23/04/2024): Why Did No One Tell Angela That Farmer Wants A Wife Is Set On A Farm?
As wonderful as this headline is, let's face it, we all know an 'Angela', don't we?
PubMatic Unveils New AI Partnership To Turn Social Posts Into Ads For Any Digital Channel
Here's some nifty tech for turning social posts into ads. Assuming said posts aren't one-star character assassinations.
Intuit Mailchimp Makes A Splash With Its First Australian Brand Campaign
Ever laugh along at a gag you didn't get so as not to appear dumb? Get ready for more feigning with this new work.
GumGum’s Rob Hall: Advertisers Can No Longer “Rely On Binary Descriptions” Of Consumers
If anyone's got their finger on adtech's pulse, it's Rob Hall. He also avoids using the good paper in the office printer
Mastercard Nabs Florencia Aimo From Marriott International
Marriott International's Florencia Aimo jumps from the hotel business to the exploitative credit card one.
Bastion Agency Appoints Cheuk Chiang As New ANZ CEO
Cheuk Chiang takes the reins over at Bastion Agency. But not the rains down in Africa.
Spotlight On Sponsors: Major Sponsorship Wins After A Disappointing Week In Sport
B&T continuing our deep dive into local sport sponsorships & that's despite not a single offer of a free ticket as yet.
Macca’s Marketing Director, Samantha McLeod On Big Mac Chant: “What Was Once Old Is Now Cool Again”
Macca's using the power of nostalgia in latest Big Mac campaign. Well, only for those who've ever eaten one sober.
World Premiere Of Midnight Oil: The Hardest Line To Open Sydney Film Festival 2024
Oil's biopic to open Sydney Film Festival. Here's hoping Molly Meldrum will take his pants down at the premiere.
Entries Are Now Open For The 2024 Brandies, IntelligenceBank’s Annual Brand Marketing Awards
The Brandies are, of course, a prestigious marketing gong and not the mystery tipple favoured by nannas everywhere.
The Fred Hollows Foundation Appoints Ardent For PR
Yes, we all like to have a joke at PR's expense. But sometimes it does important work, like this.
AI, eCommerce & Marketing Specialists Are In Increased Demand By Businesses, New Data From Fiverr Shows
Has your philosophy & anthropology degree left you with nothing but a huge HECS debt? Here's what you should've studied.
Perth’s First 3D Anamorphic Billboard Arrives Courtesy Of oOh!media
Do you love a buzzword? Now you can add anamorphic to the list as it relates to billboards, not a colleague's ears.
MasterChef Australia & Crown Resorts Launch Unique Dining Experience With ALUMNI
A pop-up restaurant staffed by MasterChef contestants! That's fine dining prices for first-year apprentice chef cuisine!
Amanda Laing Announces Resignation From Foxtel Group
Foxtel's chief commercial & content officer heads for the exits. Read nice things the bosses said about her right here.
The Lost Letters From Our Diggers: News Corp Unveils ANZAC Day Special
It's nice when brands respectfully acknowledge ANZAC Day.
Howatson+Company Acquires Akkomplice
Large indie acquires a slightly smaller indie. Much like a shark eating a tuna, just with less thrashing and blood.
Google Delays Third-Party Cookie Deprecation Again
In good news for the sale of picture library biscuit photos, Google continues to tease over the end of cookies.
Education A Low Priority For Aussies More Concerned With Cost Of Living Forethought Study Reveals
Study finds Aussies cutting back on education due to cost of living. Booze & Uber Eats sales remain largely unaffected.
“I’m Still The Same Person That I Was”: Rikki Stern Says “Fucc It” To Cancer Stereotypes
B&T always happy to promote the anti-cancer cause. Even brands that massively overdo it with the hot pink.
The Unapproved Climate Certification Allegedly Causing Mass Greenwashing
Are you left flummoxed in the canned tuna & free range eggs aisle? Just wait till this green certification gets up.
TV Ratings (22/04/2024): Fans Mock “Over The Top” Reaction To New MasterChef Judges
MasterChef returns for its 2024 season. B&T stands by putting peppercorns in Gravox & no one will be any the wiser.
Dentsu Restructure: Muddle, Harvey & Johnston Take Leadership Baton As Bass & Yurisich Exit
A large broom has swept through Dentsu's local ops this morning, taking with it some big names & the air con's cobwebs.
Industry Shares Trends Shaping The Industry This International Creators Day
B&T's asking adland creators to reveal their top trends. And it's not good news for your Jenny Kee cardigan collection.
Mable Extends HOYTS Sensory Screenings Partnership
Mable has extended its HOYTS sensory screening partnership. Vigorously defends its two-star Oppenheimer review.
Orphan Launches ‘They Need Our Help. We Need Yours’ For Children’s Cancer Institute
Anything to do with childhood cancers has B&T's 110% support. That said, we do ignore the red meat & alcohol warnings.
Smile Team Orthodontics & Keep Left Collaborate On Smile-Inducing Campaign
As parents would attest, given the cost of orthodontics you'd expect this campaign to be a lavish production indeed.
Opinion: How Video Calls Neglect Learning Diversity
Need an excuse to duck out of a video call this arvo? Show this to your boss.
DoubleVerify Achieves First-Of-Its-Kind Responsible AI Certification From TrustArc
DoubleVerify receives responsible AI certification. However, not its robotic vacuum that's been seen menacing the cat.
Smile For A Good Cause: The Social Media Campaign Giving Back To The Community
Are you known as the office Austin Powers? More for you teeth than shagability? Get snappy new fangs with this news.
Elon Musk Mocks Albo After ESafety Wins Court Injunction Against X
Albo's 2024 from hell continues - Rabbitohs in crisis, down in the polls and now feuding with world's richest man.
Real Estate Developer In Hot Water Over “Sexually Exploitative” OOH Campaign
Real estate agents again tops in the 'least trusted profession' polls, nudging used car salesmen & ad creatives.
Epsilon’s Shane Hanby: Post-Cookie Era Relies On “Teamwork” Between Brands, Marketers & Tech
This pro predicts more "teamwork" in a post-cookie era. Which spells bad news for the uncooperative or plain stubborn.