“Oh look, Santa brought me an ILSA!”
Eemer Eivers
For many, December means Christmas, but for those interested in International Large-Scale Assessments (ILSAs), it is a Christmas of another sort. December is traditionally when ILSA results are released and we get to play with our new toys (the huge datasets!).
Like Christmas, ILSA releases have certain traditions. Expect a few psychometricians trying to explain complex statistical concepts such as concurrent calibration, or why measurement error does not mean someone made a mistake. Expect a stressed national research coordinator advising against a focus on country rankings in what are definitely not to be called “league tables”. And above all, expect an increasingly bored audience of journalists and policy-makers, waiting for the one thing they want to see … the league tables.
What’s wrong with league tables?
Achievement outcomes are an important part of ILSAs, but they are not the only outcome. And, like the lead actor in a teen drama, they are often misunderstood or unfairly picked on. The casual reader does not understand that differences of a few points are meaningless, that sampling methodology, weights, methods of assessment, and a variety of other factors unrelated to student achievement can affect country rankings.
Then there is the confusion of correlation and causation. You don’t need to be a psychometrician to guess that the correlation between national ice cream consumption and PISA performance is not causal (if only…). ILSAs are snapshot studies, not designed to establish causal relationships, but they offer many correlations that seem plausibly causal.
And that’s where education Ministers dreaming of becoming “the new Estonia” – who are themselves “the new Finland” – fall for the impressive charts so loved by people like OECD’s Andreas Schleicher. To a policy-maker’s ears, prediction and causation are the same thing (hint: they’re not). They look at the impressive correlations that ILSAs find between achievement and X, Y and Z and make wholesale and often ineffective or inappropriate changes to national education systems. Yet, realistically, there can be only one country at the top of a table. With so many countries taking part in ILSAs, the chances are it will not be yours.
So why bother with ILSAs?
Well first, when interpreted carefully, ILSAs provide very useful information about the skills that students can demonstrate. Second, and often overlooked, most ILSAs contain a treasure trove of contextual data. To understand education systems and how it affects young people’s lives, context matters.
The forthcoming TIMSS 2019 results will include country-level descriptions of education policy and national science and maths curricula, information about what happens in homes, schools, and classrooms, and some key population data. If considering reforms, that information should be granted as much importance as achievement data. Use ILSAs to see where student experiences and teacher practices in your country differ from those in others. Consider why that might be, and only then consider if change is needed.
National governments develop policies, but ILSAs can show how well those policies are implemented. For example, PIRLS 2016 showed that Maltese schools spend an unusually small amount of time on reading instruction (about half the international average). Time for major policy change? No, because a policy was already in place but was not being implemented. PIRLS shone a light on the disconnect between policy and practice, and measures are being taken bridge the policy-practice gap. In sum, like Santa Claus, ILSAs don’t always bring what you wanted, but used properly, they might just bring what you need.
Banner image by Arek Socha from Pixabay
About the author(s)
Eemer Eivers
Dr. Eemer Eivers is a research consultant and a Senior Research Fellow in the School of Arts Education and Movement in Dublin City University. Her interests include educational inequality, assessment, and international comparisons of educational systems. Having managed Ireland’s participation in multiple cycles of PIRLS, TIMSS and PISA, Eemer has a particular interest in the use (and abuse!) of ILSA data.
Let’s thank Santa or ILSA’s for bringing us what need 🙂