Evidence?

Somehow an image from the Salem Witch trials with a powerful image of patriarchy seems appropriate to continuing by discussion of what constitutes evidence in a complex system. Miller’s The Crucible was one of the first plays I ever saw as a teenage in the old Chester Gateway and it made a lasting impression on me. The evidence of of a corrupt and corrupting testimony overlaid onto existing prejudices and needs constituted proof enough to allow ‘strange work for a Christian girl to hang old women’ to become a terrible reality; but with the best of intentions.

Now I have no problem whatsoever with the concept of evidence based policy, in either the public or the private spheres. My issues are (i) that people too often confuse evidence with proof, and (ii) that what constitutes evidence is very different in complex systems to complicated ones. Given that all social systems have high degrees of complexity we therefore have a problem with any notion of evidence that assumes repeatability of cause and effect even within similar contexts let alone different ones. Human systems have levels of complexity that shift us beyond modelled patterns of behaviour. We learn from and interact with systems in multiple unmanageable ways. Check in on my post about cobra, butterfly and hawthorn effects if you want some very basic illustrations.

One we both understand and internalise the concept that most human systems are disposition not causal then it logically follows (as I suggested yesterday) that much of what is evidence will be an emergent property of interactions with and within the system over time; not some innate feature of the system that is constrained sufficiently to allow repetition. A randomised control trial is cool for testing drugs in some cases. The physiological reaction of our bodies to those drugs and to any invading agent is a level where there is little conscious control of social interaction. But when we try and move it to the social domain we trespass on both scientific and ethical issues. For system level impact we can dismiss RCT a priori while recognising that it may have utility for aspects of that system. In the development sector this is an increasingly problem given the demand for defined outcomes and the RCT obsession (which has many aspects of the witch trial) and the lack of effective critical opposition. It has become a norm through which funding is given as it gives the comfortable appearance of predictability and creates a lot of research and consultancy revenue. Sorry to be cynical there, but it needs to be said.

So what constitutes evidence within a complex system? Well one of the Centre for Applied Complexity research programmes which we identified yesterday in a great meeting at Bangor University will look specifically at this, and also aim to create some (sic) evidence. But for the moment there are some clear steers on what we should look for, or the process we should adopt to create sustainable practice with the right evidence base:

1. We need to start with a dispositional map of the current situation: what aspects are stable;  where is there potential for phase shift or incremental change?   What can we change where we can monitor the impact of change over time?  We always start with how things are in complexity; yes a sense of direction for future travel but not a precise goal.
2. What does natural science say is possible and impossible?  Within those boundaries what are the lowest energy costs of replication?  Can we change that by changing the fitness landscapes?
3. What coherent actions can we take that might shift the system in the right direction (what is right is for another day).  Can we create safe-to-fail experiments to achieve that shift?
4. Have we got an assemblage/strange attractor aspect of this system which indicates that a social construct now exists independently of its contractors?  How stable is it?
5. What aspects of success (again a question for another day) are repeatable at what level of granularity?  What type of information exchange is necessary or possible between the grainsI?  How would we monitor the properties that emerge from those interactions?

And that is just for starters. At the moment this is a little academic so the next stage is to shift it to practice. But that will be practice informed by theory, not theory determined by conventional current practice or a dominant trope of belief.