Reflections from a statistician

March 4, 2010

Well, I suppose a lapsed statistician is a more accurate description of my current status in the field of statistics – I haven’t proven a theorem in a quarter of a century, the last time I tested a hypothesis was two decades ago and as for data-analysis, well for that I now have SenseMaker Explorer!

When I started out as a statistician, there were no personal computers; we made very strong assumptions just to be able to calculate the results; non-parametric methods with fewer assumptions took all-night runs on the university mainframe for a simple hypothesis test. Then came the PC; suddenly exploratory statistics became possible – not that it was considered rigorous enough to be proper statistics back then. But I loved it – it was like being a detective, looking for structure in a mass of data, using dimension-reduction techniques to squash the information down into two or three dimensions that could be visualized and interpreted, looking for patterns in graphs and finally, when the data set gave up its secrets, finding out what the patterns I saw could actually correspond to in the real world! Unfortunately, in the first stage of analysis, the pattern often corresponded to a management decision nobody bothered to tell the poor statistician about, but that’s another story.

Sounds familiar? When I first saw the scatter plot matrix in Explorer, I felt like I’d come home – even better, I could now do the same kind of analysis on “soft” data too!

I was taken aback initially when I was told that statistical analysis is best applied in the chaotic domain, though – I had always thought of it as a very rational, analytical tool to help make sense of uncertainty. Well, that’s not contradictory then, is it? Agents acting independently… independent observations as a basic assumption for statistical methods … ok, so statistics could make sense of chaos ….

To my statistical mind, fitting linear (e.g. polynomial) regression models to data is an admission that you don’t know anything about the underlying mechanism, otherwise you would have used a more explanatory model. Linear models only represent correlations and no causality can be inferred from them; they only apply to the range covered by your experiments and cannot be extrapolated to different conditions, but they can indicate promising directions in which to search for better solutions – so again, isn’t this what one does in the chaotic domain?

So I have to acknowledge, statistical methods are applicable in the chaotic domain, especially the empirical type of methods (functional models and many other techniques fit better in the complicated domain). But I’ve never ever simply discarded an outlier, they are the most interesting species of data point!

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Posts

About the Cynefin Company

The Cynefin Company (formerly known as Cognitive Edge) was founded in 2005 by Dave Snowden. We believe in praxis and focus on building methods, tools and capability that apply the wisdom from Complex Adaptive Systems theory and other scientific disciplines in social systems. We are the world leader in developing management approaches (in society, government and industry) that empower organisations to absorb uncertainty, detect weak signals to enable sense-making in complex systems, act on the rich data, create resilience and, ultimately, thrive in a complex world.

Cognitive Edge Ltd. & Cognitive Edge Pte. trading as The Cynefin Company and The Cynefin Centre.


Social Links: The Cynefin Company
Social Links: The Cynefin Centre
< Prev

Serendipitous synchronicity

- No Comments

In our work at the Foundation, we have experienced one example after the other of ...

More posts

Next >

Information ecologies

- No Comments

An interesting day yesterday in New York. I was part of a small but ...

More posts

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram