Aussie Market Mix Modelling (MMM) firm Mutinex has launched a potentially ground-breaking product for the MMM market, giving hope for moving the industry away from black box modelling frameworks towards a degree to accountability, transparency and improved governance standards.
Its new Open MMM Validation Framework is the industry’s first open-source, vendor-neutral toolkit that tests and compares any MMM solution.
The Open MMM Validation Framework is a universal testing suite that works with any model, regardless of methodology or code access, and measures business-relevant outcomes such as predictive accuracy, ROI stability, and cross-validation performance.
The Open MMM Validation Framework also allows procurement teams to demand independent validation in RFPs, giving marketers the tools to make decisions with greater confidence and clarity.
Speaking with B&T, Mutinex CEO and co-founder Henry Innis hailed the new framework as the “right thing to do” and an initiative that will elevate the standard of the entire industry.
“With a rapid proliferation of MMM tools and frameworks, without any objective way to evaluate them and understand whether or not they’re fit for purpose, we saw it as a really, really big gap,” he said.
“We saw it as important to bring some kind of framework to the fore as we weren’t seeing anybody else do it… If we built a black box tool, everyone’s going to ignore it. We took the decision to open-source it. We made a judgement that giving away the IP, whilst it would cost us something long-term, would build a better and more sustainable industry. A rising tide lifts all boats.
“As the category leader, you have to lead and it was our obligation to do so.”
The new framework follows “years” of work at Mutinex elevating the rigour of its own models. The team felt that these learnings should be available to all for the good of the industry. Innis said that a small but crack team spent months dedicated to getting the framework ready, being taken off other work in order to get it ready to ship.
Joseph Kang, Mutinex’s model scale lead, led the project, with help from senior data scientists Sam McCormick and Phil Clark and data scientist Benjamin Dobrin. Innis himself and Peter Photinos, Mutinex’s head of data science and product, also worked directly on the framework.
“We know transparency is key for customers because they need to trust the model,” Photinos said.
“If they trust the model, they’ll evangelise the use of it within their organisations. Unless you’re transparent about why the model is good—but also when it’s bad—they’re not going to be able to build that trust within their organisation. It’s about pushing the envelope a bit and forcing people to be more transparent with the results because the whole industry is heating up and there’s a lot more players entering the field.
“We don’t want people to get burned by MMM as a whole. If they don’t go with Mutinex, they go with someone else and they have a crappy model, they spend a heap of effort, make bad decisions and then they’re not seeing any growth, they cook the industry and that’s not good for anyone.”
Blake Rand, group head of insights at Domino’s said it was a “clear demonstration” of Mutinex’s “commitment to transparency”.
MMM rivals Analytic Partners, Google’s Meridian and Prophet have all backed Mutinex’s Open MMM Validation Framework, though with some caveats from Analytic Partners and Prophet. Agency groups including WPP Media, Publicis-owned Atomic 212°, Omnicom and independent econometrics firm Accurace have also announced their support for the tool. Innis said he was “relieved” to see the support from across the market after it made the choice to be “collaborative and radically transparent”.
“There’s not an established benchmark for MMM,” said Innis.
“Are we saying what we’ve done is perfect? No. We are saying that it’s very important to have transparent sets of benchmarks that we’re all able to use to evaluate these models clearly and objectively, rather than data science models that may be managing billions of dollars in spend through pitch decks,” he continued.
“To me, it is incumbent on the industry to rally together to build a better standard and to build governance. The industries that develop trust are the ones that have good, transparent governance protocols. They’re the industries that end up being relied upon as infrastructure, not gimmicks. Our intention is to build the MMM industry up as an industry that is infrastructure to major brands and a co-pilot to major brands, rather than where we have been which is a cottage industry that has bad governance standards that are wildly different vendor-to-vendor.”
One may question why Mutinex, which Innis himself said is the “industry leader” gets to, in essence, check everyone else’s homework. But that would be to mis-read the importance of Mutinex open-sourcing the code and its call with open arms for collaboration.
“When you have an open-source product, someone could take this standard, commercialise it themselves and not reference our name at all. We have not even put ‘Mutinex’ in the name of the package. We do not expect Mutinex to own the code ongoing. We expect it to be an open-source standard with multiple contributors across all competitors,” Innis.
“We have no intention of trying to own this as a piece of tooling or kit that is owned by us under license. We are giving away the code base to ensure that we can evaluate and lift the standards.
“If people want to think that way, we’d love to see your code because, frankly, I see a lot of people are wanting to critique products and innovations. Australians love criticism where they can. But what I’m not seeing is a lot of people showing receipts. Maybe the industry would be better if we started living the values of being transparent with clients and building a robust industry,” said Innis.
“We want people to contribute,” added Photinos, “we want people to add new tests, improve tests, iterate and push us right. This is in an imitative to raise the bar for the entire industry. We would love someone else to push us, raise the bar and don’t light a fire under us as well.”
The Mutinex team said it will be running test coverage across its client base, meaning a “large portion” of the industry will be adopting the standard from the get-go. It hopes more competitors, beyond those who have already voiced support in Mi3, will also climb aboard to raise standards.
The battle between MMM businesses has been heating up over the last 12 months. In January, Google launched its open-source Meridian MMM. It’s deeply integrated with Google’s channels, as you’d expect, but also offers marketers the chance to compare over other channels.
Last month, Mutinex launched Data MAITE, an AI-powered auto-labelling feature for its DataOS product, aiming to reduce the time marketers and its agencies spend preparing data for modelling by up to 95 per cent.
It has also invested heavily in building out an chatbot-like feature, known as Hendren, within its platform. Mutinex marketing science director Nicky Barton told B&T previously that the volume of requests had grown rapidly, particularly in the three months leading into Christmas.
“In the past quarter we’ve seen a 210 per cent increase on our query volume, including a 109 per cent increase in November alone…it’s gone crazy,” he said.
Fellow Aussie MMM player Prophet launched around four years ago and as of October listed half a dozen clients in sectors including retail, insurance, on demand delivery, gambling and wagering and cyber security.
It raised $5 million in funding earlier this year and is being backed by heavy hitters, including Antony Catalano, executive chair of View Media Group and former Domain CEO; Seek co-founder Matt Rockman; and former Dentsu CEO Cheuk Chiang.
