The famous statistician George Box once said “Statisticians, like artists, have the bad habit of falling in love with their models”. Mathematicians working with big data can often come to believe that their model is flawless – because based on the huge amounts of data they have, it is! However, no data set, however large, can cover every eventuality, and no model is ever perfect – the important question is whether they are good enough to be useful.
Writing about the financial crisis, economist and Financial Times columnist John Kay observed “The people who built [the models] – the mathematics PhDs – didn’t know very much about the world. The people who knew about the world didn’t understand the mathematics. Both groups had inappropriate confidence in the value of these models. They aren’t useless – but models can only illuminate the world, never be a substitute for judgement”
The decision makers – the people who know about the real world – often fall into two camps. The first group are totally convinced of the infallibility of their expertise – they can think of lots of examples where they knew the right answer, and at least one or two edge cases where the computer models would get it wrong. These guys are victims of the “availability heuristic”, in the words of Nobel Prize winner Daniel Kahneman. For them, the memorable stories are more important than the evidence. They often feel that using a decision support system undermines their professional expertise, and puts them in danger of being replaced by machines. The second group put all of their trust in the models – “the quants have already worked all this out”. They outsource their critical thinking to the model, because it’s easier that way. They can think of lots of examples where the model got it right… and there’s just so much to do!
Neither of these options is healthy. Too much trust in big data analytics is part of what led to the financial crisis (including the infamous models of mortgage defaults which couldn’t accept negative house price growth). Too little trust is what lets tired clinicians at the end of a shift dismiss the patient with the early symptoms of a heart attack, or sales people leave millions of dollars on the table through missed cross-selling opportunities.
Luckily however, all of the issues above can be tackled by one thing – better communication. Big data isn’t a one man show. The analytics whizz-kids need to talk to the decision makers to find out when their models can’t or won’t work, not just rely on the available data. They also have a sales job to do – they understand the limitations of their model better than anyone, so it is up to them to convince the decision makers that the system is there to help them – not replace them. The decision makers need to talk to the mathematicians too, to understand when it is safe to rely on the model and when it is dangerous. They need to take the time to explain what happens in the real world, and treat the mathematicians as equals, not techies who should be hidden in the back room, or academics with their heads in the clouds.
The aim should be to set up a cycle of positive re-enforcement. Modellers will create better models if they discuss their assumptions and approximations with the decision makers who know their domain inside out. The decision makers will make better decisions, and learn more about what works in their domain with the support and help of big data analytics. The more they talk, the better the decisions and the more money the business makes.
The bottom line: big data analytics shouldn’t be outsourced to an ivory tower. It’s dangerous – and expensive. Businesses need an analytics partner they can talk to.
Tags: Actionable Intelligence, Big Data, data mining, predictive analytics
I couldnt agree more with this Blog. The whole term of Big Data has blindly followed and marketing led fashion all over it, like knowledge management, and Networked Enabled Connectivity before it, and Cyber currently sat in the shadows on the other side of the data centre.
What data science and knowledgeable decision makers do get from these approaches – which dont have to be big, is help asking the right questions.