What do you mean, evidence?

When I was a statistician, I had almost complete professional freedom, as the people I worked for or with did not consider themselves qualified to judge whether the approaches I took were the right ones. In the world of education, the opposite happens – as everyone has gone to school for more than a decade, everyone considers him- or herself an expert on how it should be done, and therein lies the rub. Many an intervention is tried, and even implemented on large scale, because it sounds like a good idea, with very little evidence as to its suitability for the particular context.

So the Foundation I work for set out to generate evidence to support solutions in all our programmes. That turned out to be easier said than done in something like a bursary programme. The sample sizes required to achieve acceptable power and discrimination using traditional statistical methods were simply prohibitive in terms of cost. All the ethical issues familiar in social research presented themselves; for example, how can we not provide the support we believe can make a difference to the bursars’ success just for the sake of having a control group in the experiment? And if we see something is not working, how can we not intervene just to get good data?

The approach to use safe-fail probes appears appropriate and solves some of the problems, but raises others. What we do know about students is that they often don’t realize they are in trouble until it’s too late, and that for many it’s a self-esteem issue to not ask for help, so expecting them to voluntarily join a support programme may not meet their real needs. Although the combination of narratives and quantitative data available from SenseMaker indeed convinces powerfully, the challenge still is to collect the narratives from a diverse enough population to see how the conclusions hold across different contexts before one can hazard any recommendations.

Then we get to the philosophical aspects of what constitutes convincing evidence for policy purposes. When I refer to evidence-based policy making, I often get a most cynical response. Case studies in the medical world show that a strong emphasis on data can lead to over-diagnosis and cause more problems than it solves.

So to get beyond implementing education policies just because they have worked elsewhere or seem like good ideas is not a simple problem; anyone who has experience with it or has an interest in grappling with it is most welcome to help us think about how to do it.