Why is the failure rate so high in innovation?

Why is it there's such a high failure rate in innovation efforts? 

Why is it I run around talking about our (the departments) 10% success rate on new ideas and think that's a good performance?

And why is it that other organisations are thrilled when they get even one new thing out the door?

I think the answer is we've all programmed ourselves to think this innovation thing is hard, and since it is hard, you should only be successful some of the time.

I also think all this programming we're giving ourselves has become something of a self-fulfilling prophesy.

I mean, the reality of large organisations is that most of the preconditions for success are already present. There is never a lack of great ideas, for example. Ideas are bubbling around everywhere, and all you have to do is collect them. Neither do large organisations lack people who want to get things done: getting magic to happen is really a question of linking them up and giving them permission to do things.

Most importantly though – large organisations are paying more than lip service to their desire to be more innovative. In the last week, for example, I learned that two more banks have started new innovation programmes complete with massive budgets.

So why, why, why does this stuff keep failing?

My experience at Lloyds with Innovation Market, and more latterly at the DWP where we've built largely the same thing (we call it IdeaStreet) suggests part of the answer, to me at least.

Innovation is a social activity.  People want to talk and share. They want to co-create. If you want people – of their own free will – to give you innovation, you must create a system that lets them link up with each other independently of any central effort and concentrate on the things they care about.

Why? I suspect potential innovators don't want to be little islands of greatness, trying to push uphill against the combined weight of everyone else trying to make things stay the same. That's very hard work; probably too hard most of the time, when you also have to do a day job.

When you have this situation, you only get innovation in the presence of individual heroics by very special, very rare people. Most organisations have too few of them to ever get reliability out of their innovation efforts. And, at the very best, you're only going to get one or two really meaningful things before your heroes burn out.

This, then, is the cause of all the failure: all those little islands having to push against the big corporate machine. You hear so many stories where even those at the top of organisations can't get anything done because all the machinery at lower levels frustrates their innovation efforts.

I say, if even those at the top of an organsiation can't mandate innovation, then you have little choice but to start innovation from the bottom up.

That's why building social innovation systems are important.  People down the pecking order are very powerful agents for change when they band together to do something.  And its the reason why central innovation efforts that only invest in big game changing things usually fail.

16 Responses to“Why is the failure rate so high in innovation?”

  1. August 18, 2010 at 4:55 pm #

    Hi James
    Great blog, nice to know DWP are keenly seeking staff ideas on how to change. I do think innovation is also created by knowledge, in short, knowledge limited to the individuals. That tends to be solely an internal issue.
    DWP is doing some good stuff lately. But, structure, as you say, does stifle innovation. This is particularly true of larger organisations where innovation can be welcomed but the engagement channels do not exist.
    Innovation could also be led by external observations. However, insight of current process and also access to change champions is often stifled by engagement layers. This is typical of the SI and BPO delivery models where the existing incumbents control and protect the status quo for their commercial benefit as opposed to customer and internal user needs.
    2 suggestions. First, a web url where ideas could be posted by smaller SMEs who need to get their message across to those who could make it happen.
    Second, promote some come and meet us days to exchange knowledge on what you do and how you do it, with follow ups to allow new shoots to blossom.
    The big boys can use interchange programmes and secondment to gain privileged insight.
    The innovation community, in agile smaller companies with interesting ideas often find they cannot reach first base.
    I commend you on your open approach to driving that through and wish you luck
    Kind Regards

  2. Stephen
    August 18, 2010 at 6:24 pm #

    @Chris An interesting point about SI’s and outsourcing, People are starting to notice the negative impact procurement policies can play, in terms of innovation not just in an individual organisation but the UK economy as a whole. An article today in Computer Weekly re-enforces this point.
    I am just taking a macro view of James point about ‘people at the lower levels banding together’ your points on procurement models, and the wider issues raised in the Computer Weekly article e,g, perhaps SME’s having to band together in order to influence and change how the bigger machine works.
    Perhaps my tangental view has gone too far?

  3. August 19, 2010 at 4:37 am #

    Thankyou – and thanks for those ideas. I shall think about and talk about internally.

  4. August 19, 2010 at 8:28 am #

    You still don't get it, do you James?
    You're not, nor ever have been – involved in any innovation efforts. You're simply overstating what are simple product enhancements and pretending they're "incremental innovations".
    James, accept it, they're tweeks. the best you can force past a regime that simply doesn't believe its outmoded and a support section too lazy to welcome change even if it was dropped on them from a great height.
    OK, that sounds harsh, I accept. but go take a listen at Simon Wardley's excellent presentation and just try to suspend your tendency for proclaiming your opinions as absolutes and look at this from another standpoint. I have. This guy is absolutely inspired.
    Real innovation rarely comes from inside an organisation. Its forced in by market forces (when it has to happen for survival – as you alluded to recently). Too many vested interests to protect the norm with their efforts aimed at maintaining the status quo.
    The problem is, the Public Sector with no competition has no market to force it to do anything, so unless something terrible happens (like running out of taxpayer money to support you), you stay as-is.
    One great example of true innovation is the highly successful US Government Cloud, the GSA.
    But guess what, this was an external initiative brought in by an amazing US-CIO and its revolutionised the American Public Sector.

  5. August 19, 2010 at 9:37 am #

    I suppose, Neil, it would depend on how you define "innovation". One mans tweaks are another's greatness I suppose. But whether you call them incremental or tweaks or anything else, if you do enough of them they add up pretty lucratively.
    Of course that was not the point of my post.
    The real point was that if you do things to empower the
    Sent from my iPhone

  6. August 19, 2010 at 10:10 am #

    James, I know the presentation is long at 30 minutes and seems focussed on Cloud, but only from the standpoint of innovation.
    Jump to the 17 minutes point and take it from there. Hopefully, you'll see a great explanation of why innovation fails, not because of the ideas, but because of the process needed to float them – agility – is diametrically opposed to business-as-usual which is about suppressing variance.
    I've never seen a presentation (nor ever listened to one multiple times) that describes this so clearly. I would seriously welcome your comments on it.
    It does deserve a full end to end listen when you have time.

  7. August 21, 2010 at 10:21 am #

    Your para number 2 is precisely my view too. It makes me unpopular when I say it in the Department, though.

  8. August 22, 2010 at 6:41 pm #

    I disagree. Your view does not go far enough. I know of at least 3 innovative suppliers within DWP whose services were turned off, despite having been acquired as part of a conformacne to an original set of requirements. Sadly, the supplier handed control of 'continued implementation' to the SI, leaving DWP spending over £2 million per annum on non-innovative body-shopping costs rather than the 'innovation i delivery' tey began with. Ergo, it is not just the procurment phase which requires innovation, it is the continued supplier engagement and delivery phase. SIs would be less likely to continue their behaviour if the seniors were accessible to gauge continued benchamarking of what they achieve as outcomes and the cost and process of delivering those outcomes. Traditionally, once the SI takes control, the customer is powerless to effect change unless such change is visible as a requirement. The SI acts as a gatekeeper to prevent such change occurring, thus squaring the circles on protected revenues and ever decreasing servie qualities without furtherinvestment from the customer.
    SMEs can band together as much as they like, but unless the true contract owner, the customer, finds a proactive way of engaging in direct touch with the innvoation community, which includes end customers and front line staff just as much as it involves the supply community, then opportunity to protect, evolve and improve services will be lost. Large companies make long term investments in their service infrastructure, their global patnerships and their shareholders. They need a return on their invertment, and do all they can to ensure they gain a long term return to the detriment of the stakeholder community. This is not to the benefit of anyone but the corporation. Innovation requries speed, vision and flexibility, even found these witin an SI?
    The procurement models already exist, within the European rules and also within Central Government at policy level. ICT teams rely upon procurement to guide them towards their desired outcome, but yet again,without expert insight, how can a CTO know their procurement team had a better option, but failed to execute. They themselves want to be needed, busy, and retain control. So whilst remunerated the same way, where is their incentive to show short cuts? Hit the efficeincy target and no more, because any performance beyond the curve is not rewarded. These are some of the cornerstones of public service which prevent innovation from ever occuring.
    SMEs contribute more in coporation tax than the top suppliers, they do not move their profits offshore and they want to grow and emply more staff, which could mean DWP themselves had to pay out less as a pot. They haemorrage enough cash already on ridiculously expensive re-eomplyment strategies with stakeholders who train people back to work. Why they never square the circle and emply SMEs who employ people straight off benefits is a question perhaps for the department. Given the rules define the requirement as the most advantageous econimic decision in the round, it's difficult to see why DWP, as a payee of benefits only to the unemployed, could not adopt this premise of being advantageous as reducing it's 'negative outflow' by having less customers as a result of it's purchasing decisions, it not only fits procurement law, it actually achieves the department's current objectives.

  9. August 22, 2010 at 6:47 pm #

    I too have worked within government. I applaud your attempts to activate innovation. Neil, I agree with your outcome based approach to demonstrating 'the art of the possible'. however, Kanes is right in what he says. There are innumeerous internal blockers to amek these things happen. This guy is on the web, wanting to make a difference. It's our job ehtically and operationally to assist in showing how he might achieve that, rather than attacking the system he works within. He did not invent this environment, but it seems to me he is trying to do what he can to make it happen, and seeking insight as to how to get there. As Stephen says, collaboration is the key to this. James could have kept his views to hiomself, stayed offline, and not sought out critique. I commend him for doing so and wish more CTOSs and CIOs had such a minsdet to making things happen!

    • August 20, 2011 at 1:10 pm #

      Wait, I cannot fathom it being so stargihftorward.

  10. August 22, 2010 at 6:49 pm #

    I am gald that is of use. Try and find a way of talking about them externally too. That is where you will find the real power to make things happen. Too many people internally have enough to lose to pay lip service and put you in a loop of prevention

  11. Stephen
    September 1, 2010 at 12:17 pm #

    I still think the strategic leadership of an organisation has a responsibility and accountability for defining a vision of the future. There also needs to be a set of values that helps shape the desired culture. Taking risks in a risk averse or threatening environment can be very unrewarding.
    Some organisations are re-active e.g. they will only innovate or change when it is absolutely necessary. If an organisation can only embrace change once it is thrust upon it, then some big questions need to be asked about leadership and perhaps the reward system.
    Change may be initiated top-down or bottom-up, outside-in, but eventually it needs to go inside-out and get everyone involved to help make it work?

  12. Andrew Rouse
    September 3, 2010 at 12:20 pm #

    When you look at failure rate do we assess why it failed? Is it the process stifled it and us BAU types sapped the thing to death or was it that only 10% were actually innovative and any use and the other 90% needed to die or were just rehashed stuff passed off as innovation?
    Have to agree will alot of the posts that innovation comes from the inside. Unfortunately from my experince we rely on the "big" consultancies to come in to tell us what to do which is normally what we already know but weren't listened to. Plus whatever they come up with is always a way for them to further extract vast sums from us.
    Perhaps the next innovation we need is to make it easier to get the SME level companies in for this work and use them, they can't be as expensive as the usual suspects and we might even get something innovative rather than income generating?
    Just to pick up on the risk thing which often raises itself here. As a Government Department we have to be so wary of risk because we are genuinely looking to protect the public purse and our failures come with big numbers and bigger headlines. Hence we need to see what innovation is but probably pick up when it is stable? Not easy for the IT gadget brigade to comprehend as it isn't shiny & new but probably safer for DWP? Or perhaps I'm just old fashioned in my thinking, happy to be trolled by your contributors on this.

  13. September 6, 2010 at 11:50 am #

    Most of the things that don't' make it die because 1: they don't contribute to the business objectives 2: they are squashed because of politics 3: no one is interested. I don't think of anything off the top of my head that was a re-hash, but even if it was, I'd suggest that rehashing is a perfectly reasonable innovation strategy. Why not copy what's happened in the past if there is some advantage to be derived from doing so?


  1. Quora - December 21, 2011

    How should managers approach failure in the innovation process?…

    Fast fast, fail often. The thing is, most stuff you try in an innovation effort will fail. The fist thing to do is accept and plan for it. Planning for failure means you have to have a balanced portfolio of things you’re working on. The ones that succ…

  2. Quora - December 21, 2011

    At what point in the effort to innovate do most initiatives fail?…

    Yes it has. In fact I wrote a book on this subject in banking. I studied 200 or more bank innovation programmes, and came to one conclusion: they all fail because their innovators failed to get a connection to the money quickly enough. In other words, …

Leave a Reply

Your email address will not be published. Required fields are marked *


Proudly powered by WordPress   Premium Style Theme by www.gopiplus.com