Market Research: Is There Just Too Much of It?
In his latest post, B&T regular Robert Strohfeldt argues that despite all the market research, is it really telling adland what it needs to hear?
Change. The one variable that is constant. And usually, change means better. No one changes to something that was worse than what they had before. For those who hanker for the “good old days”, think about what it would have been like to go to the dentist 100 years ago?
DAYS OF INNOCENCE
Way back in 1981, when I first joined a market research firm fresh out of university, I was engaged initially as a mathematician to ensure proper random sampling.
Research firms had their own field force back then and most quantitative surveys were done door to door, face to face. We worked from Collection Districts from the ABS. A major issue was cost – an interviewer was given a strict set of dwellings and if nobody, or the appropriate person, was not home, they were to try three times before replacement a dwelling was allowed.
Decent sample size, quantitative studies were too expensive for all but the large companies. Anecdotal findings, coupled with experience, were often all small businesses had.
There was also the obscene amount of paper used – imagine a six-page questionnaire for 1000 respondents.
The number of successful interviews achieved in a day was “bugger all” and a sample of 500 or 1000, took ages just to complete the field work. And of course, the cost was often seen as prohibitive for the perceived results. (That the results many times reinforced what the marketing team believed they already knew, was deemed a waste of money).
In the early 80’s there were not any PCs – coded questionnaires (coding is a valuable art that has been lost) were taken to a company who produced a computer print- out. Often, after working through the first run, additional reports were produced – “Can we see how the breakdown how of respondents in question four answered question six?”. On top of this any statistical analysis was done separately, again after all report runs had been completed.
Often, the alternative was “Let’s just run a few groups”. Great, an-depth discussions with 30 odd people was not really an accurate indication of anything, bar the lack of understanding of how research was best used.
Many clients would baulk at spending $30,000-$50,000 to optimise a $5 million campaign.
By 1983 I had graduated to running groups and have fond memories of running focus groups in peoples’ kitchens. The respondents were genuinely thrilled that large consumer companies sought out their opinions and engaged in discussions accordingly.
It was a novelty and as such they gave honest answers and when the topic was exhausted, that was it, the group was over. (Easy to spot) Yes, they were paid, but many times I had respondents state they would have participated even if not paid.
Many golden nuggets of consumer thinking (and statements, or in today’s terminology, content) were gleaned from them.
COST AND TECHOLOGY. AGENTS OF CHANGE
Telephone surveys took over from door to door sampling – no- where near the cost of sending interviewers out into the field.
For market researchers, their most valuable natural resource is respondents. Telephone interviews soon became problematic thanks to telemarketers.
“Hi, I am doing a survey on house cladding”. Several question later they knew your type of dwelling and if you were a candidate for cladding. And in no- time there was a knock at the door, or sometimes a phone call, from a person trying to sell you cladding.
And as telemarketing grew, many people had experienced having their dinner disturbed by someone trying to sell them something, camouflaged as market research.
“Piss off” became a frequent response. Many households grew tired of receiving telephone calls, be they from a genuine research organisation or a telemarketer.
By this time, PCs were in all research companies and the questionnaires were on screen. As the interviewer “filled in” the questionnaire on screen, the first run of data was being done simultaneously. Not anything to marvel at today, but when this first became available, it was revolutionary and research companies tried to push all quantitative surveys into telephone surveys.
But all the while, researchers’ most valuable natural resource, willing and uncorrupted respondents, was gradually being eaten away.
WHO HAS NOT BEEN SURVEYED?
Fast forward to 2019 and I doubt there is a person over the age of 18 who has not been surveyed in some form. As more and more businesses move online, they see it as an opportunity to tag a few research questions at the end.
The quality of the research results is totally dependent upon the quality (accuracy) of the sample – is it representative of the group you want to survey? “Normally distributed, proportionate to size”, was a phrase used in every quantitative study back when door to door was the norm. The sample would be “reweighted” to the exact proportions of the various demographic groups in the population being studied.
Research is the same as computers i.e. rubbish in = rubbish out.
I have no doubt that reputable research firms adhere to strict sampling disciplines, but this does not solve the problem of “too much research”.
I had lunch recently with two friends who have nearly 80 years of market research experience between. They felt that too much research kills our natural resource – the unbiased and honest respondent. Our chat formed the genesis of this article.
Many large research companies today run panels – tens of thousands of people who, for a reward (often cash), are willing to be respondents. By having all the relevant demographic information of each panel member and by filtering via certain behavioural questions, they can supply a ready- made sample.
But just how random is this group of people? They can distort focus group discussions – a couple may attend up to two groups each a month for $80 (or more) per group. And they are also paid to participate in quantitative studies. This can give them up to even $320 a month extra cash in hand, nothing to be sneezed at.
They are paid to give an opinion and so they do, but whether this is what they really believe, or what they think they should say, no one can truly tell.
“Do you buy jam” – “yes”
“What brand did you last buy?”. The brand manager for Cottee’s jams is deeply involved in the brand, but the average shopper only gives it a cursory consideration when shopping. “Last brand? Um, Cottee’s”. It may not be, but as they are paid for an opinion, they better have one. It could have just as easily been another brand that was on special. But Cottee’s is top of mind and gets the vote.
SEVEN SOUNDS LIKE A GOOD NUMBER
And then there is the classic “How likely would you be to recommend…….?” The Nett Promoter Score. I read not long ago the new MD of a large private health insurance company was going to link all senior executives’ bonuses to Nett Promoter Scores. Poor bastards!
Mark Ritson wrote a column recently about CEOs’ messing up marketing and advertising- what a colleague refers to as “Seagull Management” – they fly in, shit over everything and fly off.
This was probably a good example. The MD had probably heard of The Nett Promoter score and thought “Bloody brilliant. I will introduce that”.
Where once a survey was something rare and genuinely regarded as being worthwhile, people are asked to rate something nearly every day. And hence we have “Seven sounds like a good number”. Nett Promoter Scores may have been worthwhile years ago before the natural resource of “virgin” or at least “rarely used respondents” had been used up.
Now, people rattle off numbers between five and seven, unless something extraordinarily good or bad had happened to them.
A/B testing is another hoary chestnut. Not the place to detail an in-depth dissertation on the problems with this technique. It has become so commonly used that I would prefer to take the opinion of an experienced and intuitive creative person to finalise a headline or ad concept, over the answer given by some spreadsheet A/B testing technique.
Survey Monkey has allowed smaller businesses to conduct their own research. Sometimes this does throw up useful information, but often it is conducted by people with no experience in sampling and interviewing. The analogy is of the person who acts as their own legal counsel has a fool for a client.
We know that thanks to Google, Social Media and almost no end of information sources, consumes are far better informed than ever.
I have seen focus groups in which the respondents were acting more as creative directors – “It would have been more impactful if you had used morphing”.
Testing communications against their objectives is becoming more and more testing.
Data seems to have usurped research as a source of insights. In many companies, the market research and data insights are separate departments. Where once we referred to data as Primary Research – all available information (data) was analysed prior to any market research being developed – what do we know, or think we know, what gaps are there is our knowledge etc. Market research and data were all part of marketing intelligence. There was no demarcation between the two, they worked together seamlessly.
IS THERE AN ANSWER?
There have been many fantastic new research techniques and programmes developed over the past 40 years. That is a given. But it does not address the single biggest problem facing any form of market research. As with data, quality should always take precedence over quantity.
No matter how good the technique, if the sample is a dud, then the results will turn out the same. Nielsen introduced, a number of years ago, a qualitative research technique they called “Delta Qual”. Derived from forensic psychology, its aim was to determine what people actually did when grocery shopping, not what they thought they did.
Not a simple or cheap technique, it required two moderators. Though this approach is more likely than many techniques to find the truth, it is still heavily dependent upon the quality of the respondent.
“Killing the goose that laid the golden egg”, was how one researcher described the overuse of research.
This is a topic in which it is far easier to identify the problem than it is to provide an answer. Cliched as it is, the answer is “each case is different”.
Prior to any research study, be aware of the fact that many of the respondents are jagged and give an answer off the top of their head. If you have received feed back from say 5,000 people and 4,000 of them are just going through the motions, then the results will be rubbish.
How to identify and access the respondents who are genuinely interested in providing answers? Of course, this then raises the problem of statistical validity. Make sure you know the demographic and relevant behavioural breakdown of your target group and then reweight the results accordingly.
With qualitative research, speak to the people who are either very happy, or very pissed off.
Have you ever been to a restaurant where the waiter or waitress continuously interrupts to ask, “How is everything”? Maybe we need to learn from this. Stop asking questions after every customer contact and learn to identify those who really do have something to say.
Latest News
TV Ratings (18/04/2024): I’m A Celebrity Wins Prime Time And Key Demos
Aussie viewers can be a harsh lot at times. Only days after Ellie Cole bled her heart out, she has been sent packing.
Effie&co Launches New ConnectAsia Division To Help Aussie Brands Market To Asian Consumers Overseas & At Home
Not provided is advice on using chopsticks and not spilling ramen down your shirt.
Cashrewards Sets Out Stall For New CMO
Thinking of applying for the Cashrewards CMO gig? Here are some insider tips that, yes, are tantamount to cheating.
‘I Ask For The TV Industry To Stand Up And Defend Itself’ – Seven Boss James Warburton Steps Down
The Seven supremo heads for the exits after five years. Here's hoping the Spotlight team organised the farewell bash.
Poh! Jamie! Adriano! Paramount ANZ reveals its tasty plans for this year’s MasterChef
It's your fan's guide to this year's MasterChef! Although no tips on how to pronounce crudités or use a un fait-tout.
Dentsu’s iProspect Partners With MOOD Tea Ahead Of May Campaign Launch
We love a Mood Tea here at B&T. Although we do store old screws and nails in the International Roast caterer's tin.
Opinion: When Culture Starts Eating Itself: Navigating The Age Of Self-eating Nostalgia
Born boss David Coupland asks is adland going through a nostalgia period? But please, no repeats of Best Of Red Faces.
Who’s Going To Cannes?! The TikTok Young Lions Winners!
It's Aussie adland's next gen! They're off to Cannes with high hopes of bringing back a Lion & a foot-long Toblerone.
Adobe Launches Express Mobile App With Firefly AI
Want to be the coolest kid at Friday staff drinks but forgot your retro Nikes? This new Adobe wizardry may do the trick.
ThinkNewsBrands & IMAA Extend News Publishing Education In Brisbane
Industry duo takes its publishing roadshow to Brisbane. Was disappointed no male attendees were wearing walk socks.
B&T Chats With Wavemaker’s Provocative Pioneers On Their Cross-Pacific Sojourn
B&T TV heads to Wavemaker's Sydney digs to interview two staffers from its New York & LA digs. If that makes sense?
HoMie & Champion Launch “Give One. Get One” Campaign Supporting Youth Homelessness Via Town Square
Much like the fête's prized chutney wears a blue winners sash, so too should this top initiative from HoMie & Champion.
Thinkerbell Takes Us Back To Summer In Latest Work For XXXX
This beer ad wants to take you back to summer! Just minus any chance of a shark attack on your morning bus commute.
Cannes Lions Unveils 2024 Programme Featuring Queen Latifah, Jay Shetty & P&G’s Mark Pritchard
Are you one of the lucky ducks heading to Cannes in June? Check out the headliner acts you'll be queueing hours to see.
Scroll Media Recruits Costa Panagos From Twitch
Costa Panagos set to bring South American flair to the Scroll offices. Assuming that he is, indeed, South American.
Year13, Microsoft & KPMG Australia Launch AI Course For Gen Zs
Born around the 2000s? Need to amp up your AI creds? This guide's for you (although it's not really that age specific).
General Motors Snares Heath Walker From Scania
Do you rage about oversized American cars on our roads? You need to bail up Heath Walker at parties & industry events.
VML Launches New “Envoyage” Brand For Flight Centre
VML unveils new brand for travel operator Flight Centre. Alas, no sign of those paid actors pretending to be pilots.
Subaru Places Media Account Up For Review
Subaru puts media up for review, as adland journos get set for mandatory "agency drives off with..." headline.
TV Ratings (17/04/2024): Contestants Faced With Harsh Realities As Alone Australia Heats Up (Or Cools Down)
Alone still doing the business for SBS. Overly long train journeys not doing the business, but they persist anyway.
Ben Fordham Loses Number One Spot As Ray Hadley Celebrates 156th Ratings Win
The radio numbers are in! Discover who's off for a boozy lunch today & who's waiting for the dreaded HR death knock.
Gourmet Ice Cream Brand Connoisseur Launches New “Thrill Your Senses” Iteration, Via SICKDOGWOLFMAN
Rattling the old "truth in advertising" adage comes this ice-cream spot full of noticeably thin people.
Paramount’s Global Sales Boss: ‘Australia’s Converged Model Is A Blueprint For How I’d Like All Of Our Markets To Be’
Paramount's global sales boss gives local sales ops the thumbs up. Didn't weigh-in on the Lisa Wilkinson debacle.
TikTok Starts Testing Its Instagram Rival In Australia
In exciting news for piano playing cats & brattish pranks in shopping centres, TikTok unveils its Insta rival plans.
Man Wrongly Named By Seven As Bondi Killer Hires Lawyers
Struggling to save for a house deposit? Why not get wrongly identified by Sunrise!
Smartsheet Appoints Indie Agency Sandbox Media To Its Media Account
Can't stand your colleagues? Like to dob them in when they miss a deadline? These work management platforms are ideal.
Boss Not Letting You Come To Cannes In Cairns? Use This Business Case To Convince Them!
Stingy boss won't spring for a ticket to Cairns? Add this to your persuasive argument repertoire. Or grovel.
Alt/shift/ Brisbane Builds Portfolio With Ausbuild Creative, PR, Content & Social Account Win
The Brisbane comms/PR agency lands constructor Ausbuild. Also hoping for a discount on its new glass conservatory.
Young Guns Versus The Old Guard: Who Adds More Value to Our Industry?
Cannes In Cairns poking this hornet's nest in a lively debate. Just so long as the oldies can get up the stairs.
70% Of Aussies Don’t Have Green Power Plans ENGIE Says In Major Brand Campaign Via HERO
Are you the notorious "light leaver on-er" in your flatshare? Quell any infighting with this green energy news.
PrettyGood Launches Offering Brand & Media Solutions For Australasian SMEs
B&T applauds the charitable nature of this new agency. Although we'd hate to see it impact any Chrissie present sends.
A Blunt End: Dolphins Medicinal Cannabis Sponsorship At Risk
Yes, it's another NRL drug story. Yet, thankfully it doesn't involve coke in Kuta during the off-season.
Slew Of New Creative Hires At Leo Burnett Australia
Ahhh, all black! The outfit of choice for agency creatives, David Jones staff and everyone in Melbourne.
Under Armour Unveils Local “Live in UA” Campaign
American apparel brand set for yet another tilt at the Aussie market, as Nike declares "we'll see about that".
Pepsi Launches New Look, Refreshing Classic Fashion Staples Via Special PR
Are you always the bridesmaid, never the bride, as the old saying goes? How do you think Pepsi feels?
Pure Blonde Returns To A Place Purer Than Yours In New Campaign Via The Monkeys
B&T's always been a huge fan of the 'drink yourself thinner' diet plan. So big thanks to Pure blonde, vodka & tequila.