Statistics and data guru Nate Silver has said most data analysts are “oversensitive” to random fluctuations in their data, allowing it to skew their conclusions.
The statistician, author of The Signal and the Noise, also told an audience at the South by SouthWest festival the art of prediction has not gone “all that well” for society in the past, and admitted he is “something of a sceptic” of them.
Silver came to public prominence from the film Moneyball, about his project to create a perfect baseball team based on stats rather than their perceived performances.
He also said marketers will take a “long time” to properly assimilate data from tools like Google Social Search and Facebook into their thinking.
He added: “Having open data is a good thing for society. I don’t think they will make big breakthroughs straight away, it will take a while for society to get things in place to produce useful insights.”
He also said people give “more meaning than they deserve” to the “random fluctuations” which happen in most data sets, arguing that then skews the conclusions.
He also urged companies not to revert to slashing budgets for things like research and development and staff training in tough economic times, arguing they are the areas of most use in the long-run.
Silver also debunked the notion we are in a “big data singularity”, adding: “Really I think that prediction has not gone all that well in society, and it’s still making slow progress in most fields.”
Public sector areas like education, city planning and even the prison systems are what he predicts will be the next area statisticians turn their eye to, adding correctional systems could look to data to try and predict which prisoners are less likely to re-offend if released
