Effect Sizes and Why They Travel Badly

I remember sitting in a windowless CPD room in 2014, staring at a thirty-page white paper that promised to “revolutionise” our literacy outcomes, while the radiator behind me hissed and a Year 9 boy in the corridor started a rhythmic banging on a locker. The presenter spoke in that polished, academic dialect that sounds impressive in a lecture hall but evaporates the moment you step into a classroom with thirty kids and a broken photocopier. We were told this was the gold standard, but nobody actually explained how education research should be read by someone who has exactly fifteen minutes of planning time before the bell rings. It felt less like guidance and more like a burden of proof we were expected to carry without any actual tools.

I’m not here to give you a lecture on statistical significance or how to parse a longitudinal study. Instead, I want to show you how to strip away the jargon and find the bits that actually hold up when you’re exhausted on a rainy Tuesday afternoon. I promise to show you how to spot the practical gold from the theoretical fluff, so you can stop feeling guilty about the research you don’t have time to finish and start using the bits that actually work.

Evaluating Educational Study Validity Without the Academic Fluff

When you’re staring at a paper that claims a new technique will “revolutionise” student outcomes, your first instinct should be a healthy dose of suspicion. Most of these studies are conducted in sterile environments—perfectly behaved cohorts, small sample sizes, and researchers who haven’t stood in front of thirty Year 9s on a rainy Tuesday. When evaluating educational study validity, I don’t look at the flashy graphs first; I look at the context. If the study didn’t account for the sheer chaos of a real classroom, the results are essentially academic fiction.

You don’t need a PhD to start distinguishing correlation from causation in learning. Just because a class using “Strategy X” performed better doesn’t mean the strategy was the hero; it might just be that the teacher in that study was a seasoned veteran with a magic touch. Before you overhaul your entire scheme of work based on a single paper, ask yourself: does this actually hold up when the lesson is going sideways? If the methodology feels like it was designed for a laboratory rather than a school, treat it as a suggestion, not a command.

Critical Appraisal of Academic Papers for the Busy Teacher

When you finally find a spare twenty minutes to skim a paper, don’t feel obligated to parse every nuance of the methodology like you’re preparing a PhD thesis. Your goal isn’t to become a statistician; it’s to figure out if the findings will actually hold up when you’re standing in front of thirty Year 9s on a rainy Tuesday. A lot of the heavy lifting in a critical appraisal of academic papers comes down to looking at the context. If a study claims a massive breakthrough but was conducted in a controlled lab with high-achieving students and zero distractions, I’d take those results with a massive pinch of salt.

You also need to be wary of the “silver bullet” trap. Many papers are excellent at distinguishing correlation from causation in learning, but they often fail to mention that just because two things happened at once doesn’t mean one caused the other. If a study says “students who used X tool scored higher,” check if those students were already the most motivated ones in the cohort. We aren’t looking for perfection; we’re looking for the practical application of research findings that survives the chaos of a real school day.

Five Reality Checks for When You Actually Open That Paper

• Skip the Abstract and find the ‘Methods’ section immediately. I don’t care how much the introduction promises to “revolutionise pedagogy”; if they ran the study on thirty highly motivated university students in a controlled lab, it’s going to fall apart the moment you try it with thirty Year 9s on a rainy Tuesday afternoon.
• Look for the ‘Effect Size’ rather than the ‘P-value’. Academics love to tell you a strategy is “statistically significant,” which is a fancy way of saying it wasn’t just luck. But if that “significant” change only improved test scores by 0.5%, it isn’t worth the three hours of prep time it’ll take you to implement it.
• Check the sample size before you get your hopes up. If a study claims a new way of questioning is the “gold standard” but it was only tested on two classrooms in a leafy private school, keep your guard up. We need to know if it works in a school where the photocopier is broken and the heating hasn’t worked since November.
• Ask yourself: “Who paid for this?” If the research into a new digital learning platform was funded by the company that makes the software, take the findings with a heavy dose of salt. There’s a difference between genuine inquiry and a very expensive marketing brochure disguised as a journal article.
• Always look for the ‘Limitations’ section—and read it twice. The most honest researchers will tell you exactly where their theory hits a brick wall. If a paper doesn’t admit its own weaknesses, it’s not research; it’s a manifesto, and we’ve had enough of those in the staffroom already.

The Reality Check: What to Carry Out of the Paper and Into the Classroom

Stop looking for a magic bullet; most research tells you what can happen in a controlled environment, not what will happen when you’ve got twenty-eight teenagers and a broken projector. Look for the “transferability”—ask yourself if the study’s conditions even remotely resemble your Tuesday morning.

Prioritise the “so what?” over the statistical significance. A study might prove a tiny, measurable difference in test scores, but if implementing that strategy requires three hours of extra planning every week, it’s not a breakthrough; it’s just more unpaid labour.

Trust your gut when the data feels hollow. If a piece of research suggests a strategy that ignores the basic reality of classroom management or student rapport, it’s academic fluff. The best research doesn’t fight the reality of the room; it works with it.

The Reality Check

“Don’t let a beautifully formatted paper fool you into thinking it’s a roadmap for your Monday morning. If a piece of research describes a perfect laboratory setting with thirty compliant students and a researcher with nothing to do but observe, it isn’t a teaching strategy—it’s a fantasy. We shouldn’t be asking ‘is this true?’ so much as ‘is this true in a room where the radiator is clanking, the seating plan is a disaster, and I’ve only got forty minutes to make it count?'”

Ellery Mwangi-Selkirk

The Bottom Line

At the end of the day, reading research isn’t about becoming an amateur statistician or memorising p-values while you’re trying to eat a lukewarm sandwich in the staffroom. It’s about developing a healthy, rigorous scepticism. We’ve spent decades being told that every new study is a revolution, only to find out it’s just another shiny initiative that falls apart the moment it hits a Year 9 classroom on a rainy Tuesday. Remember to look past the academic jargon, question the sample sizes, and—most importantly—ask yourself if the findings actually account for the chaos of a real school environment. If a study assumes your students are perfectly behaved and your resources are infinite, it’s probably not worth your precious time.

Don’t let the weight of the literature make you feel like you’re doing it wrong. You don’t need to master every theoretical framework to be an exceptional teacher. Instead, use these tools to filter the noise so you can focus on what actually moves the needle for your pupils. The goal isn’t to follow every trend, but to build a pragmatic toolkit of ideas that have survived contact with reality. Trust your professional judgement, keep questioning the “experts,” and remember that the best research is the kind that actually works when you’re standing in front of thirty tired teenagers, trying to make a difference.

About Ellery Mwangi-Selkirk

Most educational advice is written for a classroom that does not exist — twenty motivated pupils, unlimited preparation time, no behaviour to manage. I write for the room you are actually in. What retrieval practice looks like when marking is already three weeks behind, why the seating plan matters more than the starter activity, and which of the last decade's initiatives left anything behind. Nineteen years taught me that the good ideas survive contact with a wet Wednesday and the rest do not.