Needles in Haystacks

The banner shot here was taken by me last Sunday in the late afternoon of what was a delightful hike; one of those days where showers are followed by sun and even the low hills/mountains of the Black Mountains assume a mysterious air.  I spotted the sheep outlined against the horizon and prayed it would stay there while I got the camera out of it’s waterproof carry bag and thankfully it did. I had to take it with the 20mm lens I had in as there was no time to switch to the telephoto but that worked well.  Earlier on that day we had searched the eastern slope of Garreg Las to try and find two Napoleonic era millstones that were meant to be there.  Well, I say we in practice the rest of settled in while Paul went hunting, something we have learned to do over the last few months.  He normally finds things, like the Ogham script boundary stones above Merthyr all of which are interesting but this occasion he failed.  Back in the 19th Century Millstones were quarried and then shaped on-site before being shipped to the Mill.  In this case, the end of the war had probably reduced demand and these two had been left abandoned in partial states of completion.  One of the reasons he couldn’t find it is that Garreg Las is a delightful summit with multiple quarries, boulder fields and has a generally rocky aspect.  Paul was, as we said, trying to find a needle in a haystack.  Of course, sometimes you are lucky.  A couple of weeks back on my five and a half-day walking break in South Cornwall I happened to notice a stray rubber Ferrel lodged in a step while descending to a cwm.  These things are easy to lose and it is always useful to have a spare so I picked it up and put it in my pocket.  The next day I found that the rubberised tips on my trekking poles had worn through.  I use these rather than the carbide tips for coastal walking as a lot is on lanes so it avoids the click-click of metal but they wear out faster.   Now, this was serious as I had no spares, and given the state of my knees, one pole at least is essential.  That small observation saved the next few days as I could fall back to one pole, and use the Ferrel on it to keep me going without damaging the more fragile end of the pole and thus avoided an expensive replacement.

So why this somewhat rambling discussion?  Well, I want to return to what has always been a driving theme of our work from the early days of DARPA work when I was still in IBM, through the post-IBM period, the Singapore Government IRAHS project, and now Crisis Management; namely the need for weak signal detection.  The ability to see something small and inconsequential in the here and now, that in the future, with the benefit of hindsight you will realise is more significant than you thought at the time.  If you look at the broad history of this in the literature and elsewhere then searching for needles in haystacks is entirely approach, so to are being lucky, of spotting something at the right time and in the right place, or contrast suddenly making you aware of something that would otherwise be lost in the detail.  It is also important to understand that weak signal detection is broad in need and application.  It’s not just about counter-terrorism, it is about sensing issues with your workforce, colleagues, or customers before they become overt.  With the benefit of hindsight, everything becomes clear but hindsight will not result in foresight of itself so we need to think differently about the problem.

See – Attend – Act

I devised the above some years ago when working with Gary Klein in Singapore for the Ministry of Defence – we were both associated with their innovation unit at the time.  In far too many approaches to decision making the assumption is that if you put the right information in front of the right people at the right time and they have the right competence and the right training then they will make the right decision. in practice this is far from the case.  You may not even see data in plain site, even if you see it then you may not pay attention to it, and even if you pay attention you may not choose to, or may not be able to act on it.  So in any decision support system, you need to separate the three and think about all three as distinct if deeply interconnected aspects requiring a different design.  No this is something on which I posted back in 2015 and at that time I loaded a proposal I had written for a client which was then, and still is a paper in the process of creation and which you can read in full here.  It describes five key factors that are a part and parcel of being human and that we have to work with, not try and pretend can be removed. As originally written they are:

1. We know best …
   The danger of expert bias, the more we know about something, the more competent we are the less likely we are to see something that falls outside the bounds of that expertise. Sometimes known as inattention blindness this is not something that can be trained out of people, it is a part of what we are as a species
2. Swamped by data

   Sheer volume of data may prevent us joining up the dots, after an adverse or favorable outcome it is easy to know what is relevant and see some causal connection but at the time it is simply a matter of luck. Hindsight is a wonderful thing but it doesn’t of itself lead to foresight
3. Mediation and interpretation
   Excessive mediation, interpretation, and screening of data before it reaches the decision-maker means that raw data is stripped out or summarised based on the assumptions and knowledge base of the interpreter. Middle management issues may also subconsciously eliminate material that does not match, or which implicitly and explicitly threatens their interests. The decision-maker may realise the significance of data because they see the bigger picture, their sub-ordinates may not
4. The courtier syndrome
   In most organisations of any size, power is based on the ability to influence the decision-maker which frequently leads to bad news not being shared or access to disruptive influences being prevented. This is no fault of the decision-maker per se, but it appears to emerge over time in government and industry alike.
5. It worked for me before …

   We all have a tendency to interpret data to match what has worked for us before and to act accordingly, without investigating in any depth. We evolved as a species to make decisions very quickly using remembered past experiences and then to justify those decisions post hoc. In overall evolutionary terms, it makes a lot of sense but the downside is that novelty may be easily missed, and the more successful we are the easier it is to miss.

I would now extend that to pick up on something Gary Klein advocates and I fully support, namely that so-called cognitive biases are should be called cognitive heuristics.  Basically evolution tends to energy efficiency and the heuristics have evolved over time to do that.  The question of inattentional blindness (described in the linked material) makes sense – most of the time it is very unlikely that there will be pictures of a gorilla in an X-Ray so it is not unreasonable that people miss it.  We evolved to make decisions very quickly using the minimal amount of information and privileging our most recent experiences.  If something large and yellow is running towards you at high speed then the last thing you want to do is scan all available data, look up a catalogue of the flora and fauna of the African veldt, and having identified ‘lion’ lookup best practice case studies on how to avoid being eaten by lions.  As I joke at conferences, at that point the only document of any use will be the Book of Jonah from the Old Testament, the only example I know of a self-help manual on how to escape from the digestive tract of a large carnivore.  So we need to work with the way we are, not try and pretend we can change it on a sustainable basis.

The linked material provides some ideas of how that can be done with SenseMaker® but there are some basic principles around which different methods and tools have been developed and we are adding to that.  I plan to post on our use of Trios in this and related areas tomorrow and that will also link into my writing up the Triopticon, a formal way of bringing diverse perspectives into play.  There are also methods and tools that we have developed for what is called exaptive innovation in which existing capability is radically repurposed for novel use.  That technique uses abstraction to suggest novel connections between capability and brings those connections, with necessary ambiguity, to decision-makers for investigation. In effect, you can manage for serendipity but to do so you have to break away from the mundane.  You can also focus on anticipatory awareness, the ability to see novel connections.  Something by the way which is associated with both dyslexia and synesthesia.

In all of this, there is a basic principle at stake and it is illustrated by the banner picture.  Innovation, weak signal detection, seeing unusual potential is all about seeing things in contrast, so the thing ignored stands out.  It is also about allowing people to discover novelty for themselves something we call descriptive self-awareness.  As you have probably gathered, like my post last week on Leadership, this is the first in an occasional series of posts that at some stage will all come together in paper or hopefully the book

Hay by Idella Maeland on Unsplash