The football grand final predictions made by Twitter users are an example of why we should take consensus with a grain of salt when evaluating consumer research, says strategist Thomas McGillick.

The central assumption of the phenomenon known as the wisdom of the crowd is that the average of a group of independent predictions is likely to be more accurate than any randomly selected individual prediction. The bigger the crowd, the better the prediction. It’s part observable effect, part misleading heuristic, and it’s the reason we so rarely challenge consensus among respondents in consumer research.

There are plenty of reasons why a group of consumers might struggle to accurately determine the potential effectiveness of an advertisement, and some of them have been on show over the last two weeks, as the collective wisdom of Twitter repeatedly swung and missed when predicting the outcomes of both football grand finals.

In the Australian rules final we had an excellent opportunity for the wisdom of the crowd to work as it is supposed to. The large upset means that the collective wisdom in the crowd of predictions should be seen in (if not an accurate prediction) a more accurate, or moderate prediction than that made by a bookmaker. The opposite happened.

Collating every tweet appearing the hours leading up to the opening bounce hashtagged #AFLGF resulted in an average prediction erroneously in favour of the Swans, that predicted a larger winning margin than that predicted by the major online bookmakers. Twitter had the Swans by 15.5 and the bookies on average had the Swans by 10.5.

So the crowd was wrong, so were the bookies, but in an instance where you would expect the crowd to be more accurate, they were more wrong.

Not surprisingly, Twitter also got the first try scorer and man of the match predictions wrong.

Turning to the NRL grand final and we have another surprising result. Not an upset this time, but a larger winning margin than the bookies and tipsters predicted. In this case the twitter hive mind correctly picked the favourite (Souths) as the winner, but consensus was less clear. Twitter predicted a Rabbits victory by 5.1, while the bookies margin was an equally impossible 8.5. The actual margin was a much larger 24 points, so again, the crowd was less accurate than the bookies.

The mistakes these two groups made are actually typical of groups reaching consensus, and they happen in consumer research all the time. The AFL crowd reached a consensus that was too strong, and so their overconfidence in the Swans lead them to overestimate the score line.

On the other hand, the NRL crowd didn’t reach a strong consensus, and so they were more conservative in their prediction.

Ultimately what this means is, just because a group of people agree on something, doesn’t mean that what they agree on is correct.