What evidence?

October 24, 2007

Its funny how a quickly written blog often produces interesting reactions, while one that takes several hours to write is ignored (well maybe not ignored, but not cross referenced or commented on). I was very tired last night and took my call from the University of Texas in bed following a raid on the minibar fully prepared to fall asleep as soon as I put the phone down; fortunately for the students it was not a video-conference. However after answering questions I was awake again and had an idea for a blog which would have nagged at me and prevented sleep, so five minutes on the keyboard produced yesterday’s attempt at wisdom in respect of a consultant starting out on their career.

Waking at 0430 (0530 hotel shuttle, 0630 plane) I logged on to discover several comments already in place, and more had arrived as I checked in between flights in Copenhagen along with several emails. Not only that but my RSS threw up a posting from Tom Davenport on the subject of Evidence-Based Advice and I realised that serendipity was in play. I also realised that I was picking up indirectly on some of my earlier thoughts on professionalism in management. Settling in on the flight to Helsinki with my sixth coffee of the morning I decided to pick up on the Davenport article as a starting point to a more considered reflection on the subject: which will probably get no attention whatsoever!

Now its impossible to work with Larry Prusak without knowing Tom, and I have also spoken at several conferences with him outside of the IBM Institute for Knowledge Management (IKM) and on one memorable occasion in St Charles when Andersen not only paid me to speak at an internal KM event, but insisted I tell the Water Engineer’s Story in full knowledge that it was in part an attack on their consultancy arm. This seemed discourteous given the fee they were paying and the luxury of the stretch limo that had picked me up from O’Hare airport the previous night, but the if the client asks ….

A brief discourse on Davenport (with a minor polemic on Cohen)

Tom has a considerable record in the field of management. Many of us think he was the real founder of Business Process Re-engineering rather than Hammer & Champy. Working Knowledge is one of the classics and the Prusak/Davenport combination always worked well: Tom’s discipline with Larry’s humanity. Prusak/Cohen was a far less successful combination, in fact I would go so far as to say that Cohen tended to trivialise Larry’s thinking and in IKM days (from my perspective) he played the politics in a less than constructive manner in the style of a tabloid journalist rather than a thinker. Enough of that, back to the subject.

Tom has produced a succession of books, such as the Attention Economy and more recently Competing on Analytics. His HBR article To Make the Best Decisions, Demand the Best Data also picks up on the general thrust of his blog, the need for evidence, interpreted as data, and data analysis. The opening paragraph summarises his view well:

You may have heard of “evidence-based medicine,” the notion that medical decisions should be based on clinical research and evidence. What scares me about the term is the implicit suggestion that medicine is currently based on something other than evidence. The same squeamishness occurs when I hear the term “evidence-based consulting”.

He goes on to argue that to date most consultancy has been based on experience, and that in consequence the holy grail of consulting has been to grow some gray hair so that clients believe your pronouncements. His blog argues that in future we will need data, and he regrets the failure of the large consultancies to gather it over the years. Of course this feeds into the central arguments and prescriptions of both Competing on Analytics and the HBR article.

Evidence-based and the role of data

No one in their right mind (a category which does not include the more extreme manifestations of social constructivism and post-modernism) would deny the value and need for evidence. However I am less sure of value of contrasting experience with evidence, or of linking evidence in all cases to data. In effect Davenport is operating (as he does in all his books) on a single ontology, namely that of order. In doing this he is operating in the main stream of management science so my criticism here is not personal but general. Multi-ontology sense-making, a phrase I created (well I think I did and can find no other references that predate my use) argues that different ontologies (defined as the nature of systems based on the relationship between cause and effect) require different approaches to evidence, analysis and action. Tom Stewart, editor of the HBR in his November letter summarises this well when he says:

The situations leaders encounter most fall into none of these classes but rather into a fourth: circumstances that are complex, where the truth is not immediately evident even to an expert but emerges over time, where cause- and-effect relationships are not well established, where positive results come from offering incentives rather than issuing commands, and where, consequently, the tools of influence and decision making are subtle and ill-defi ned

More on that next week when both his letter and the article by myself and Mary Boone is available. For the moment my point is simple – evidence, as data, in the sense that Tom uses it is a feature and a requirement of ordered systems, but it is inappropriate in any complex system; not only inappropriate but down right dangerous. We can see the impact in the public services in the UK (in particular health and education) where the inappropriate use of a limited concept of evidence is stifling innovation and enterprise, a situation made worse by the limiting concept of outcome based targets.

Evidence in complex systems

So lets move on, and look at what sort of evidence would be appropriate in a complex ontology. Remember that in a complex system we have high sensitivity to starting conditions (so even if you have the data, minor changes could produce radically different outcomes), order is emergent and non repeating (so evidence of the past cannot provide foresight), and local connectivity counts more than the system as a whole. Some people argue that all concept of measurement, evidence and targeting should be abandoned when you are dealing with a complex system, but I see this is an unnecessary pessimism and/or a form of Art-Ludditism. We don’t have full answers yes. but we do have indicators of answers. I can summarise these as follows:

  1. Any complex system is subject to first and second order constraints (I sometimes approximate these as boundaries and attractors). Now we can measure boundary stability and we can to a degree measure the number and strength of attractors. To do so provides evidence of the limitations of co-evolution within a system.
  2. We are getting closer to narrative based measure of impact, which link to more accurate and contextually aware descriptions of the current situation, and where small safe-fail interventions are possible and where they such interventions can be monitored to allow rapid response amplification of dampening of emergent patterns.
  3. Understanding the limitis of evidence, and the concept of sufficiency of information to take an experimental and evolutionary approach to the development of future strategy mitigates the question of a strict data driven concept of evidence, but also places a requirement for management discipline. It is not about gut feel or intuitive/emotional decision processes. It is about using the right sort of science. Realising that we are not in a newtonian universe but one in which uncertainty, phase shifts, coalescence and emergence are key concepts that can be used by managers, and can be used with confidence give their science base.

So I think there is hope, more than that I think there are exciting possibilities. However to realise those possibilities we need to shift away from single ontology, ordered thinking and open up to new ways of thinking. It is an area where we could do with some of Tom’s discipline and application, but I suspect (and hope to be proved wrong) that he is too wedded to order.

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Posts

About the Cynefin Company

The Cynefin Company (formerly known as Cognitive Edge) was founded in 2005 by Dave Snowden. We believe in praxis and focus on building methods, tools and capability that apply the wisdom from Complex Adaptive Systems theory and other scientific disciplines in social systems. We are the world leader in developing management approaches (in society, government and industry) that empower organisations to absorb uncertainty, detect weak signals to enable sense-making in complex systems, act on the rich data, create resilience and, ultimately, thrive in a complex world.

Cognitive Edge Ltd. & Cognitive Edge Pte. trading as The Cynefin Company and The Cynefin Centre.


Social Links: The Cynefin Company
Social Links: The Cynefin Centre
< Prev

December master class on Leadership

- No Comments

As we get close to the publication date of the HBR article on Leadership and ...

More posts

Next >

Melissie Rumizen

- No Comments

Today would have been Melissie's birthday, author of Complete Idiot's Guide to Knowledge Management, friend ...

More posts

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram