The democratisation of statistics

I’ve spoken here often of the democratisation of the tools of banking. Anyone can be a banker, thanks to Peer to Peer lending. Anyone can do payments cross boarder, thanks to PayPal and similar. And anyone can do customer service, thanks to account aggregation and financial advice sites.

But the democratisation of tools is not limited to banking. And one example which I think many people have missed is the democratisation of the tools of statistics.

“There are lies, damned lies and then there are statistics”  said Benjamin Disraeli in a particularly famous diatribe about the misuse of mathematical statements to bolster weak arguments. Disraeli was twice the Prime Minister of the United Kingdom, and it were his words which were then popularised by Mark Twain in his 1907 work Chapters from my Autobiography.

Disraeli’s words are increasingly relevant today. Who, for example, has not naively used Excel to create a trend line to support a business case and then wondered why the real world result didn’t match the projection? 

One thing I’m always surprised about, though, is that people don’t spend a little more time understanding the statistical tools which are available to them easily these days. When one is an innovator, for example, you are dealing with phenomena that are amenable to statistical treatment. The arrival of new ideas, when captured in a structured manager, is eminently describable in a statistical sense, for example. 

So having a grip on statistics is something you almost certainly want to do. Because of democratisation, anyone can do statistics, whether they are educated in it or not.

At the level of an individual business case, I regularly see methods which are unnecessarily simplistic, given the tools available today. When one needs to forecast the likely take up of a new product, for example, it is de rigour to provide base, optimistic and pessimistic cases. One makes three point guesses and then crosses fingers.

But Monte-Carlo methods, for example, can take us much further. Monte-Carlo is a technique that lets you simulate a business case thousands of time (if you like) and then aggregate the results. With Monte-Carlo, you can make statements such as “it is less than 10% likely we will make a loss of greater than X”, or “it is 90% likely that we will make a profit of at least Y”. 

Such statements are powerful when you want to convince stakeholders to buy. At least they are, as soon as stakeholder realise that you’ve used a rigorous process to get to your predictions.

When I was doing my doctoral work on forecasting innovation curves, I used a Monte Carlo method to get my results. I had to code a cluster of machines to make it work. But that was then.

Today, dong stuff like Monte Carlo is simplicity itself. There are plug-ins for Excel that take your current models you’d have used anyway and Monte-Carlo them. All you have to do is specify a few extra parameters and out pop the results. Don’t believe me? Do a search for Crystal Ball, now owned by Oracle.

The point of all this is that there is now the opportunity for more sophistication in the methods we use than ever before.

Here’s my advice. The next time you want to do a business case or any other kinds of analysis of unpredictable phenomena, try something different. Don’t guess three times and cross you fingers.

Try a simulation, do a Monte-Carlo, or run a Delphi survey, or any one of hundreds of other techniques that used to be out of reach.

If nothing else, people will marvel at your brilliance and sophistication. But most importantly, you’ll be showing everyone that it is possible to raise the bar on the way we run our businesses, and you’ll be doing it without much effort.

2 Responses to“The democratisation of statistics”

  1. October 12, 2008 at 2:56 am #

    James I largely agree… stats are not the exclusive province of mathematicians any more. Tools like KXEN (no affiliation) make it possible for anyone to use sophisticated analytical methods without fear of failure.
    Nonetheless there are two concerns that I believe are important and relevant to talk to here. First of these is that just because anyone can use the tools doesn’t mean everyone should. Just like everyone should not become a web programmer just because it is easy. Obvious but important.
    Second point is that banks need to work on transforming raw data into useful information on which to base models. This has been my mission for the last decade, developing customer account level profitability models for banks and more recently perfecting a method for identifying the behavioural structure of flow of funds within and among accounts. The latter method improves model predictiveness by 30% and more, so is at least as important an idea as the dmocratization of technique.

  2. October 12, 2008 at 9:38 am #

    Hi Dave,
    Yes, I completely agree with your points. But I think my main point is that the bar is going up, however slowly.
    Surely, given the ability to do it, and the capacity to learn how, people ought to make the time to keep up.
    Surely, the fact that easy tools to create web sites (such as this blogging platform I use) has contributed to the development of good content.
    Why would it not be true for analysis?

Leave a Reply

Your email address will not be published. Required fields are marked *

(Required)

Proudly powered by WordPress   Premium Style Theme by www.gopiplus.com