Research shows …

Before I return to my decision series I wanted to get a little bit of polemic off my chest. One of the most frequently misused statements on social media is the phrase “Research shows” and it rarely if ever comes with any reference to said research.  I challenge the authors from time to time and only once or twice have I had a proper reference.  Most of the time I either get pointed to some popular book or books that suffer from the same problem or to self-validating research, constructed with various levels of ethical intent for sales purposes.  Psychometric instruments are especially prone to this, they all seem to push out questionnaires at scale which already contain their assumptions in the questions themselves, they all end up validating their particular approach and they really can’t all be right!  Then we get a specious correlation or two.  I had that recently in respect of one of the said instruments where they claimed to show a correlation between leaders at their desired level of maturity (something I addressed in a blog post earlier this year) and a certain type of success.  Isolating one variable from many is questionable anyway, and even if the statistics are right the said property may itself be a property of the history that created the success, one emerged with the other it didn’t cause it.

I’m increasingly of the opinion that the use of statistics in a lot of management science is nothing but a Woozle effect at scale, a form of entrained thinking that is perverse in nature and from which people need to be weaned off as a matter of urgency. It’s the modern practice of patent medicines and snake oil.  Some of which may have to be taken seriously, in the immortal words of Terry Pratchett “I’ve never seen a rusty snake”.

This confusion of emergent properties with causal factors is a real issue, compounded by very limited sample sizes and a tendency to only study organisations held to be successful.  You know the way this goes, a set of organisations are selected that have the desired property and the researcher then conducts a survey or series of interviews or more frequently desk research on other people’s reports of their interviews.  From that process another set of properties is identified which are then identified as causal and recipes are created based on them – do these five things and you will achieve success.  Looking through too many airport bookshelves over the work I would hazard an estimate that over 90% of best-selling business books follow this formula.  One big idea, preferably with a catchy title, a lot of cases often shoehorned to fit the new theory through a partial selection of material, followed by a collection of platitudinous statements about a new practice that no one can really disagree with and a consultancy-led process of transformation that you need to buy.

One popular author I know creates a new grand idea every few years supported by the same or similar cases and then dines out on the speaker circuit proposing his next grand scheme for transformation.  They and their audience frequently fail to check that the last book, and those before it, all made similar claims.  I call this (with due acknowledgement to the Kinks) the Dedicated followers of Fashion approach and to be honest the main offenders are not the so-called thought leaders writing the books but the consultants who are dependent on a cycle of failed transformation projects to sustain their industrial business models.  Promising a future entry into heaven in return for ignoring the realities of the present is a common abuse of power, and many an author is complicit in the process.

One of the reasons executives like this approach is that it allows them to be seen as highly progressive in adopting the latest fashionable movement, but then (sic) allows them to move on before the consequences become clear and the cycle starts again.  Short-term thinking applies here and one of the things that impressed me about DataSciences when I joined was that it had a policy that General Managers had to stay in office for three years before they moved on and banning new major initiatives after the first year unless bonus and other payments were linked to its long term success.  People who argued that was unfair because they wouldn’t be there to make sure it worked were told, in no uncertain terms, that initiatives based on one person’s physical presence were not acceptable and it was their job to create a sustainable business unit.  We lost a lot of that when the Personal Director who instituted it decided to become a General Manager and then the management buyout, a forced merger and subsequent IBM takeover knocked it on the head for good but I still recommend that approach.

Another major issue with all these approaches is who you interview.  At the height of knowledge management, lots of people wrote cases and created practices based on interviewing those responsible for the knowledge function within organisations and believed what they said.  That is still going on today in multiple fields.  I recently had a case study based on an interview with the CEO of a company we have worked with where his claims were simply not backed up by the water cooler conversations and stories we recorded.  S/he had committed the classic error of deciding how things should be then assuming that would happen, and power means you will only hear what you want to hear so that gets negatively reinforced.  When I was at IBM we did a series of ethnographic enquiries into companies that were much lauded as exemplars of knowledge management and ground reality never matched the claims of the knowledge management function,

Retrospective coherence is also an issue there – the way people remember things even hours after they happened is largely determined by the political requirements of the here and now.  We have highly selective memories which is why we focus on lessons-learning, in real-time not retrospective capture of experiences. Any interviewer influences the person being interviewed and explicit questions mean you are prompting people as to the qualities you are interested in.  One of the things we do with SenseMaker® is to ask a question which contains no hypothesis and then put the qualities we are interested in as competing qualities on triads in order to see if, unprompted, those qualities are naturally present in people’s day-to-day experiences.  I recently saw the impact of that in discussion with one executive who was unhappy with a negative skew on one triad.  The desired quality had more or less no supporting material.  S/he then told me that when people had been explicitly asked about the said quality they had said it was present.  I think we got there in the end but you have to be very patient.  If you have strong communication about certain desired values people will learn, if prompted, to parrot them back to you.  The key measure is they are naturally present in people’s day-to-day experiences.  Actions, as they say, speak louder than words.

Part of all of this is an abuse of natural science, an attempt by management science to imitate the natural sciences, suffering from what Bennis and others have called Physics Envy which is a delightful play on words.  Human systems have two many factors and two many abstractions to be studied in the same way as ant behaviour or similar. Many of the classic experiments in psychology have not survived attempts at replication in consequence.  You can’t reduce the variables to achieve causal correlations in a complex adaptive system.  You gain adductive insights and you can use natural science as a constraint on current practice: the essence of naturalising sense-making.

So if someone says research shows a good starting assumption is to assume they are selling snake oil and they may or may not be duplicitous in doing so.  A interesting approach is to start with the research, if referenced, and then explore the consequences: often the claims simply don’t match.  There is one more last resort for the purveyors of serpentine lubricants, namely to claim that everything is unknowable and mysterious and can only be understood if you suspend believe and engage with them on a journey to enlightenment,  The number of people falling for that never ceases to amaze me.

The banner picture is cropped from an original by Sieuwert Otterloo on Unsplash  Skull image is by Square Frog from Pixabay