Power laws & abductive research

I promised a more substantive post about our Friday session at the AoM on new forms of research, and I want to do it by summarising aspects of Max Boisot’s final presentation, which did an excellent point of putting everything in context. He neatly summed up the three types of inference in this picture.

• Deduction (shown as the small dark dots) delivers certainty and is based on logic.
• Induction (blue areas) which delivers probability and is based on repeatable events
• Abduction (the wider turquoise area) delivers plausibility and is based on coherence with prior experience.

He rightly headed the slide Complements, not alternatives. Abductive techniques (and SenseMaker® is an example of a tool to support such an approach) are in effect means of generating coherent hypotheses under conditions of uncertainty. The coherence word is key here, just because we don’t know everything it doesn’t mean that all ideas have equal value. Creationism for example remains incoherent to the facts, while evolution is coherent, even if it doesn’t account for everything and still contains errors. In complexity based strategy coherence is a key test for each safe-fail experiment.

Before he reached that point Max had built a highly coherent narrative around the idea of dots and linkages. I’ve shown the whole sequence on the right. Dots here are events, and the way we join things up is by narrative. We make sense of the world by explaining connections between things, the patterns that form from the dots and the linkages. It’s simplified here of course with only four dots, as the dots increase the number of possible linkages increases. With four dots there are six linkages and sixty-four possible patterns, if we go up to 10 dots then there are 45 links and 35,184,370,000,000 patterns. But we will come back to the volume issue later.

In the second slide, we see the difference between deduction and induction. With deduction there is only possible answer, it can be computed. With induction we can predict a range of events based on what we already see. The trouble is that the world is not always this simple. In practice (slide 3) we have inconclusive patterns and it’s all too easy to ignore events and linkages that don’t fit the patterns of our expectation.

Abduction on the other hand (slide 4) is all about choosing between different narratives, or allowing competing narratives to run in practice to test for coherence, another way of saying safe-fail experimentation.

Now, this leads us on to the integration of this with the whole power law concept. Most risk is assessed on the basis of a normal or Gaussian world in which means make sense and events that fall beyond a certain number of standard deviations can be ignored. However in nature if we plot a log scale of frequency against size then we get a power law as shown in the picture below. Also known as a Pareto distribution here was have to account for the fact that events that are outliers in a Gaussian world tend to have a higher probability given the fat tail of a Pareto distribution, we can’t afford to ignore them, but we are dealing with small sample sizes which reduce the value of induction.

Max’s integrative slide shows this progression from the world of Gaussian distributions that dominate business school thinking, to the more uncertain Paretian world in which we are dealing with higher levels of uncertainty.

Max’s conclusions (with some paraphrasing on my part) were:

• We have to match the inferential strategy we adopt to the level of uncertainty we face.
• Uncertainty is the surface manifestation of complexity at work
• Abduction is the most tentative of the three inferential strategies
• It allows anticipation but not prediction
• In an increasingly complex world we increasingly have to settle for anticipation

2020 Postscript and apology

We have had two major web upgrades since I wrote this post – each has lost illustrations and attribution.  I’ve restored two pictures here, but the opening one is the one I really want and I can’t find that.  Attempts are being made to recover material so hopefully, this will get some priority