Being involved in business consultancy and education has a number of interesting by-products. One of them is that I am constantly being asked for my views of the latest strategic analysis tool, financial metric or two-by-two matrix, usually by people who have a vested interest in promoting its virtues.
A great response to these queries is to ask the question ‘how do you know it works?’ It is a slightly dangerous strategy; you always run the risk that people will take your scepticism personally, and on one occasion I have narrowly avoided being punched. However, once over this hurdle, the answers you get are always very interesting and usually very similar.
Most people will begin to recite the numerous times they have used this idea, and it has ‘worked’ for them. As if racking up instances of success is the same as proof. I am a chemist by training, and this isn’t the way that the scientific community go about proving a proposition. Whenever a new scientific theory or model is proposed, the peer community instantly set about trying to disprove or break the model. In other words ‘falsity’, or more specifically the absence of falsity, is seen to be far more powerful evidence than a thousand confirmation events.
Sadly, this doesn’t seem to be the way that the economic, finance and commercial world operates, and this often leads to catastrophic errors. In 2007 many of the world’s largest banks had racked up massive positions in a whole range of complex financial instruments. Their key probability analysis tool, called Value at Risk (Var) had decreed that the risks in these portfolios were ‘acceptable’. In 2009, once the dusk had begun to settle after the biggest financial crash since the Great Depression, various commentators began to understandably ask how bankers could have put so much faith in a model which had effectively crashed the world. The rather piqued response from these otherwise very intelligent finance professionals has been to quote a long list of instances prior to 2007 when VaR worked perfectly! And in stable, normal market conditions it does work. The problem is that no scenarios had ever been run through the VaR model to see how it coped with extreme events. In other words, it had never been tested for failure.
So add the question ‘how do you know it works?’ to your commercial toolkit, and then listen to the response you get. Look for the models and ideas that have been challenged, pulled apart and tested for failure, and are still here. Don’t be seduced by pretty matrices and clever theories. As Albert Einstein once said, ‘Elegance is for Tailors’.
