How to use evidence to make decisions

Policy-makers are often criticised for making decisions based on ideology rather than evidence. Here, Sam Freedman, who worked for years in the UK Department for Education, talks about ways it can be done.

Education is a social science. It will never give us the kind of proofs that are possible in physics or maths. On almost any given pedagogical controversy, you’ll be able to find at least one impressive-looking study to back up your prejudices. How then can a teacher, leader or policy-maker make decisions ‘on the basis of evidence’ when the evidence is so murky? There are those that argue the quest for ‘evidence-based education’ is entirely quixotic and we should focus instead on trusting the wisdom of experienced professionals.

This feels like a council of despair but it is a real problem, and it does worry me when well-intentioned practitioners base crucial decisions on a glance at a simple summary of the Education Endowment Foundation (EEF) or John Hattie meta-analyses.

The dangers of this approach were illustrated a few years ago when the EEF toolkit was originally published and the entry on teaching assistants indicated they had no impact. This was picked up by various newspapers – no surprise, given that more than £4 billion a year is spent on teaching assistants. The EEF was forced to put out a clarifying statement explaining that, while research suggests that, on average, teaching assistants do not have a positive effect on attainment, other studies showed that, if deployed in certain ways, teaching assistants can have a very significant impact.

And this is true of most of the other interventions in the toolkit – the averages hide huge variance that will depend on the exact structure of the intervention and, crucially, the context in which it is deployed. For instance, on ‘social and emotional learning’ the toolkit gives a positive rating overall; but an evaluation of the national social and emotional aspects of learning (SEAL) programme – which was poorly implemented in many schools – found no impact on attainment.

So how should we weigh up evidence when making decisions if it’s often contradictory and nearly always context-dependent? My starting point is to think of every question as a balance of probabilities rather than something with a right answer. Every piece of data then nudges the balance one way or the other; the better and more relevant the study, the bigger the nudge. Let’s say I want to know if I should introduce a uniform policy to my school. If a gold-standard randomised control trial (including schools like mine) published in my country shows that having a uniform makes a positive difference, that’s going to change the balance significantly. A small qualitative study from a developing country won’t push it far at all.

This way of thinking allows you to add your own experience and the qualitative feedback of colleagues into the mix. If the balance is fairly even, either because evidence of similar quality and context is contradictory or, more usually, because there just isn’t very much of it, then your own experiences can make the decisive nudge. The uniform example is a good one here. There isn’t much evidence to suggest it makes a difference or does any harm – so if in your school you feel it’s valuable, that’s enough to make the call. If there was strong evidence of harm, however, then that shouldn’t be outweighed by your own positive experience.

As a general heuristic, this is a useful model; but there’s still the problem of how to gather information. Teachers, and policy-makers have full-time jobs – how can they accurately calibrate the balance of probabilities without spending all their meagre spare time reading research? Given their lack of time, there’s no real choice but to start with meta-analyses like the EEF toolkit and Hattie.

But simply relying on summaries won’t give anything like the necessary nuance, so it’s vital to pick them apart and look at the collection of underlying studies. Often the first layer under the summary is another set of
issue-specific meta-analyses which have very helpful overviews of the existing evidence in their introductions. They should also help to identify which are the gold-standard evaluations in that area – which should have extra weight in your decision-making – as well as the context for the key studies. Typically, most of the best research comes from the US, so often there is a trade-off to be made between quality and context. Once you’ve done an initial review then it’s relatively easy to stay up to date by following a few key accounts on Twitter (my public ‘education’ list is a good starting place).

Perhaps the greatest challenge in doing this type of analysis is managing your own cognitive and political biases. If the evidence is genuinely unclear then using your own beliefs and experiences is the best available option. But if you rig the underlying analysis by favouring studies that support your existing opinion while finding reasons to dismiss those that don’t, then you’ll calibrate wrong in the first place. This tendency is apparent in all education debates – the recent one on grammar schools being an obvious example. Supporters, many of whom benefited from a grammar school education themselves, latch on to the evidence that selectively educated pupils do well, while ignoring the research showing that the system as a whole suffers.

It’s impossible to eliminate this instinct but we can at least become self-aware enough for the problem to weaken its hold over us. I would recommend everyone involved in education read Daniel Kahneman’s Thinking, Fast and Slow, which explains how we’re affected by cognitive biases, and Jonathan Haidt’s The Righteous Mind, which does the same for political/cultural biases. Philip Tetlock’s Superforecasting, which looks at how the best predictors of the future eliminate biases, is also worth a look.

To make the best use of evidence, decision-makers need to think of it as a way to calibrate the balance of probabilities that requires regular readjustment, rather than simply a way to identify whether something is right or wrong. They need to use meta-analyses and social media to be reasonably on top of the available data. And they need to do as much as possible to remove their irrational biases. Research will never give us the perfect answers; but if used right, it’s a hell of a lot more valuable than gut instinct and prejudice.