What do you mean, evidence?

March 9, 2010

When I was a statistician, I had almost complete professional freedom, as the people I worked for or with did not consider themselves qualified to judge whether the approaches I took were the right ones. In the world of education, the opposite happens – as everyone has gone to school for more than a decade, everyone considers him- or herself an expert on how it should be done, and therein lies the rub. Many an intervention is tried, and even implemented on large scale, because it sounds like a good idea, with very little evidence as to its suitability for the particular context.

So the Foundation I work for set out to generate evidence to support solutions in all our programmes. That turned out to be easier said than done in something like a bursary programme. The sample sizes required to achieve acceptable power and discrimination using traditional statistical methods were simply prohibitive in terms of cost. All the ethical issues familiar in social research presented themselves; for example, how can we not provide the support we believe can make a difference to the bursars’ success just for the sake of having a control group in the experiment? And if we see something is not working, how can we not intervene just to get good data?

The approach to use safe-fail probes appears appropriate and solves some of the problems, but raises others. What we do know about students is that they often don’t realize they are in trouble until it’s too late, and that for many it’s a self-esteem issue to not ask for help, so expecting them to voluntarily join a support programme may not meet their real needs. Although the combination of narratives and quantitative data available from SenseMaker indeed convinces powerfully, the challenge still is to collect the narratives from a diverse enough population to see how the conclusions hold across different contexts before one can hazard any recommendations.

Then we get to the philosophical aspects of what constitutes convincing evidence for policy purposes. When I refer to evidence-based policy making, I often get a most cynical response. Case studies in the medical world show that a strong emphasis on data can lead to over-diagnosis and cause more problems than it solves.

So to get beyond implementing education policies just because they have worked elsewhere or seem like good ideas is not a simple problem; anyone who has experience with it or has an interest in grappling with it is most welcome to help us think about how to do it.

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Posts

About the Cynefin Company

The Cynefin Company (formerly known as Cognitive Edge) was founded in 2005 by Dave Snowden. We believe in praxis and focus on building methods, tools and capability that apply the wisdom from Complex Adaptive Systems theory and other scientific disciplines in social systems. We are the world leader in developing management approaches (in society, government and industry) that empower organisations to absorb uncertainty, detect weak signals to enable sense-making in complex systems, act on the rich data, create resilience and, ultimately, thrive in a complex world.

Cognitive Edge Ltd. & Cognitive Edge Pte. trading as The Cynefin Company and The Cynefin Centre.


Social Links: The Cynefin Company
Social Links: The Cynefin Centre
< Prev

Bed-time ritual

- No Comments

When my daughter was two or three years old, she asked for a story one ...

More posts

Next >

The vexed issue of language

- No Comments

In our work at the Foundation, we have developed a language of our own with ...

More posts

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram