The occult insignificance of meaningless numbers

May 19, 2009

I’ve been in Belgium and Luxembourg for the last two days delivering two half day seminars for the EU. All journeys by train which allowed me to catch up a lot of work, although I will confess that on the journey from Brussels to Luxembourg I dozed and watched the unfolding scenery remembering past trips to the Grand Duchy. Many moons ago when I was setting up a logistics software business one of our important early clients was NAMSA, the supply agency of NATO. We handled stock forecasting and inventory management for weapons systems; an interesting bit of Operational Research there, we had to create a whole new category of Lumpy Lumpy to handle some of their slow moving parts. There is a whole story about the trip there to close that contract which involves myself and team ending up perfectly parked on the hard shoulder of the M25 approaching the Guildford exit at around 0200 in the morning; sounds ordinary until I tell you that we were upside down and the facing the oncoming traffic, but (thank god) not on fire and with everyone alive. That’s a story for another day however (the roof rails saved us).

What I want to talk about today (and tomorrow). Is my concern, nay passion for governments of whatever ilk to adopt complexity theory and new research/reporting approaches such as our work on impact based measurement, citizen consultation and policy formation.

There is a very simple reason for this which is in part ideological. I think we face a real set of choices in the world at the moment, between carrying on within the current paradigm and radical change. The consequence of not changing will be an increasing inability to provide the basic public services on which any civilised society depends for its existence. For me that is free at the point of entry, access to health services and education for all, together with a safety net of basic housing and income (especially where children are involved). I am less doctrinaire about what political system delivers that, and many of the polarities of my student radical days are no longer with us or even appropriate. I am also realistic enough to know that we will never fully achieve that goal. I am also old enough to know that doctrinaire and/or utopian political movements will not achieve anything approaching that result. But its equally true that if we don’t do something then we will fall into the pit that the acolytes of Ayn Rand find so desirable, namely the tyranny of self interest. I know I need to write a blog on that evil woman (so described by Chomsky from the left and Buckley from the right) but I need to cleanse my soul before and after and that will require some time.

The trouble at the moment is that we moving away from that goal rather than towards it. For every step forward we take at least two strides backwards. We face the challenge of meeting increasing legitimate demands for social services with decreasing real time resources. That brings with it questions of rationing, control and measurement which, however well intentioned, conspire to make the problem worse rather than better. For me this all comes back to one fundamental error, namely we are treating all the processes of government as if they were tasks for engineers rather than a complex problem of co-evolution at multiple levels (individuals, the community, the environment etc.). This was compounded for me over the last two days with the discovery that the invidious tentacles of sick stigma are starting their poisonous intrusion into the body politic of the European Union. While there are aspects of government which would benefit from process based control, there is nothing that you can legitimately do with sick stigma that you can’t do as well with a much cheaper BPR tool and some common sense. There is no need to create a high priesthood of black belts to enforce the occult insignificance of meaningless numbers.

Ok rant over, but I’m not withdrawing anything! The above mentioned fundamental error needs some elaboration in the context of the difference between an ordered and complex system. In an ordered system, the level of constraint makes agent interaction predictable, The system can be modelled and desirable outcomes defined. So where you are dealing with processing student loan forms, car tax renewal or counting the swabs out of a patient at the end of an operation you can use engineering type approaches. Nothing wrong there, in fact such approaches are to be desired in those contexts. On the other hand where you are dealing with a complex adaptive system such an approach is a living breathing disaster. A complex system is one in which the co-evolutionary process between agents and system (people and government being one manifestation of that) is such that any future state is inherently uncertain, cannot be modelled and defining outcome based targets produced perverted behaviour.

So what is wrong? I’m sure this is incomplete but here are my seven errors:

  1. You get what you measure, so if you set a target humans will achieve the target at all costs, ignoring context or the unstated goals that the outcome based target was attempting to achieve. If you measure teachers on their learning plans then you will get learning plans, however you may not get learning. Teachers who inspire pupils to learn are not measured on their ability to motivate but rather on a set of indirect indicators whose links to learning are dubious at best. Triage in the accident and emergency department of a hospital is about making medical decisions, not processing everyone within a defined time period.
  2. Outcome based measurement can make people far too comfortable. it’s all to easy to achieve an explicit target, especially if you can turn off an empathy (or at least suppress it). To be able to site back and point to a clear set of targets achieved can all to easily be used an excuse for not really making a difference. I have heard one UK minister argue a year or so ago that we had just seen the best ever year for the health service, only to be destroyed by the anecdotal evidence of medical and patient experience. The scary thing is that she couldn’t understand that there was a problem. Just like the social work director who had achieved all her targets but was still the scapegoat for an horrific case of child abuse.
  3. A mechanical approach is by its nature dehumanising in its effect on people and inhuman in its impact on society. People are not robotic units who should be forced (in other than limited circumstances) to follow rigid processes and comply with idealised rules. When you deal with people you need to flexibility to adapt to the local context. The mental impact is high as well. I remember just after my parents had died within ten days of each in a cancer ward, sitting in a workshop with a group of health administrators who were in tears; they knew that achieving their targets was in some cases killing people or causing unnecessary suffering but they had little choice if they were to keep their jobs. A week or so ago a couple of nurses had broken every rule in the book to tell me what was really happening with my mother as a result of which her sister was able to be there when she died. They did the right thing, and we had built enough of a relationship over the previous two months for them to trust me.
  4. You waste an awful amount of resource just managing the measurement system. That is both real money and time. Why are we spending money on screens & computers to monitor waiting time when key medical equipment and drugs are rationed? How many people are engaged in inspecting schools rather than teaching? How many headmasters are now administrators without any teaching responsibility? How many questionnaires do hard pressed front line staff have to fill out every week to satisfy the insatiable appetite of the measurement junkies? How many people have died because we treated hospital cleaning as a contracted service rather than as a part of the overall nursing/medical provision? How many entrepreneurs have not received start up funding because they weren’t good at filling out the forms and playing the game?
  5. We try and solve issues with idealistic fail-safe designs rather than allowing systems to evolve. This is every present in the use of technology, and deeply ironic as modern IT systems allow for far greater messiness and adaptability. The patient record system in the UK is a good example of this, building a massive system on how things should be rather than creating a sound infrastructure then allowing different capabilities to evolve. They also don’t do joined up thinking. Why wasn’t the national identity card used to allow people to carry their patient records around with them? Flexible and adaptable, thinking at the right level of granularity and a damn sight more secure. God, I’d even take an implanted chip if it made things work better. In a modern computer age, you distribute capability you don’t centralise it.
  6. Re-organisation is a disease and an excuse. It’s the knee jerk reaction to any failure that ends up breaking your jaw with the recoil. It’s easy with hindsight to see how things went wrong, and to design something around your retrospectively coherent interpretation of the facts. The trouble is you don’t know everything anyway, you only know what was reported and that was influenced considerably by the adverse outcome in the first place. Smashing things together, spending weeks creating pretty organisation charts may seem satisfying, and allows those who execute the plan to postpone delivery while the reorganisation takes place but it rarely works. It takes human beings a few years to settle into any new structure anyway and informal networks, so critical to trust, take similar periods to evolve. This not a factory floor, its people’s lives, social interactions, their work, their values and in many case their meaning that you are playing with.
  7. Communication is all up and down the chain, ironically this mediates information to senior decision makers so they are immunised from the real data they need, and also from the consequences of their actions. The expectation is that people in the field will report upwards, and work within centrally determined rules. They will of course be consulted first, and then generally ignored. Plans and actions are then determined, communicated and measurement systems put in place. Those who execute them learn quickly to feed back what is expected, building up an level of inauthenticity in the system which is dangerous at all levels of the organisation. There are few if any people in this, at any level who are evil, bad things are done for the best of all possible motives, good things often happen accidentally despite the system not because of it.

OK so that’s a set of examples of what is wrong, what is the alternative? Well I can’t tackle all of that, but complexity gives us a good shift in the right direction on how to make decisions and how to organise, while narrative based research allows us to move from targeting outcomes to impact based measurement. I’ll start work on outlining those tomorrow if I get the time.

One response to “The occult insignificance of meaningless numbers”

  1. […] energy and increases vulnerability to the risk of ideological imposition of a pet hypothesis (policy determined evidence), despair, perverse incentives in targets and a host of other all too common errors.  The […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Posts

About the Cynefin Company

The Cynefin Company (formerly known as Cognitive Edge) was founded in 2005 by Dave Snowden. We believe in praxis and focus on building methods, tools and capability that apply the wisdom from Complex Adaptive Systems theory and other scientific disciplines in social systems. We are the world leader in developing management approaches (in society, government and industry) that empower organisations to absorb uncertainty, detect weak signals to enable sense-making in complex systems, act on the rich data, create resilience and, ultimately, thrive in a complex world.
ABOUT USSUBSCRIBE TO NEWSLETTER

Cognitive Edge Ltd. & Cognitive Edge Pte. trading as The Cynefin Company and The Cynefin Centre.

© COPYRIGHT 2024

< Prev

thanks for reading

I just wanted to sneak in a quick thankyou and an invitation to those who ...

More posts

Next >

Backwards and forward

Although I have been working with Cognitive Edge techniques for the last 5 years, I ...

More posts

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram