The Leaning Tower of PISA

So the 2018 Programme for International Student Assessment (PISA) results have been released by the OECD and the blame game has begun. Reasons for declining results include teacher resourcing, technology, social and economic struggles, and that student-directed learning is the devil's work.

The argument being that the structure of teacher-led instruction is closely associated with both student academic achievement and their sense of well-being. 

So what exactly are these tests and why are they so important? PISA tests the reading, maths and science skills of 15- and 16-year-olds in the OECD’s 78 member states, as well as a few other volunteer countries. The idea is that these results provide a means to directly comparing different educational systems. 

The aim of the assessment is that the wealth of new information provided will help identify why some school systems do so well and others not. The others would follow the lead of successful systems, causing results to rise across the board. Well, that was the plan...

So over the last 18 years, Singapore has done well. By 2018, the highest achiever overall is China—or to be more precise, the provinces and cities of Beijing, Shanghai, Jiangsu, and Zhejiang. Less well known but improving countries include Jordan, Poland, and Turkey. 

But despite this, few countries are improving over time, most have stayed still or declining. Part of the reason for the lack of overall progress is that schools have less influence over results than you would think. Culture and other social factors, such as adult literacy, matter more, meaning that even the most well-informed education minister can only make so much of a difference. This importance of culture can be seen in one of the new high achievers in PISA, Estonia, which has a long history of high levels of literacy, often promoted by the church. Immigration also appears to play an important role, with recent arrivals scoring below locals in most countries, reducing their overall scores. 

New Zealand Initiative research fellow Briar Lipson says teachers have lost their ability to lead, and the country's PISA rankings will continue to slide unless it redirects its approach to classroom learning.

"We could easily end up below the OECD average the longer we carry on with this vacuous curriculum ... the equity gap will only widen."

Rankings do matter, she says. The OECD's data is the only "reliable" standardized indicator New Zealand has.

But just how reliable are PISA tests as a measure of comparing education systems? PISA is a series of standardized tests that emphasize learning that can be easily measured. Questions have been asked on how samples of students are determined, and how some of the test questions are constructed.

These concerns are illustrated by Yong Zhao, a critic of the PISA assessment, whose research focuses on how globalization and technology affect education. Zhao states:

"[T]he foundation upon which PISA has built its success, has been seriously challenged. First, there is no evidence to justify, let alone prove, the claim that PISA indeed measures skills that are essential for life in modern economies. Second, the claim is an imposition of a monolithic and West-centric view of societies on the rest of the world. Third, the claim distorts the purpose of education".

His first criticism is that there is no evidence to justify, let alone prove, the claim that PISA indeed measures skills that are essential for life in modern economies. 

There is no research available that proves this [essential skills measured] assertion beyond the point that knowing something is always good and knowing more is better. There is not even research showing that PISA covers enough to be representative of the school subjects involved or the general knowledge-base. PISA items are based on the practical reasoning of its researchers and on pre-tests of what works in most or all settings — and not on systematic research on current or future knowledge structures and needs. (Hopmann, 2008, p. 438).

To illustrate the point made by Hopmann, in order to support the claim of a relationship PISA results and economic growth, PISA compared test scores in a given period (1964-2003) with economic growth during roughly the same period (1960-2000), but students who took the test were not contributing to the workforce. There is a lag time between students studying and then entering the workforce, so the evidence does not back up the claim.

Studies that compared test scores with economic growth in the subsequent periods using the same dataset and method found:

 “that the relationship between changes in test scores in one period and changes in economic growth for subsequent periods were unclear at best, doubtful at worst" (Komatsu & Rappleye, 2017, p. 183)

The OECD also has a very narrow version of what success looks like in an educational system. PISA treats economic growth as the sole purpose of education. Thus it only assesses subjects — reading, math, science, financial literacy, and problem-solving — that are generally viewed as important for boosting competitiveness in the global economy driven by science and technology. 

Education should incorporate a lot more than preparing consumers in an economy. Concepts such as citizenship, equity, curiosity and engagement, compassion, curiosity, physical and mental health are also critical. But as Sjøberg states: 

“[these aspects of the purpose of education] are often forgotten or ignored when discussions about the quality of the school are based on PISA scores and rankings” (Sjøberg, 2015, p. 113).

This narrow definition of the purpose of education may explain some interesting correlations seen in the PISA results. There is a persistent pattern of negative correlation between PISA scores and students’ interest and attitude. Many researchers have found that higher PISA scoring countries seem to have students with lower interest in and less positive attitudes toward the subject (Bybee & McCrae, 2011). As an example, the PISA science score has a significant negative correlation with the desire to become a scientist (Kjærnsli & Lie, 2011). Moreover, high PISA scoring education systems appear to involve more teacher-led didactic pedagogy (Shirley, 2017). Additionally, high PISA scores have been found to have a negative correlation with student wellbeing (Shirley, 2017).

The link between the last two points has been noted by Andreas Schleicher​, education and skills director for the OECD and education policy special adviser, who wrote in a November blog: 

"Maybe it's time to stop pitting teacher-directed instruction and student-oriented learning against each other, claiming the one is old-fashioned and stifling, and the other is forward-looking and enabling. Both approaches clearly have their place."

In summary, the promise of PISA was by measuring the skills and knowledge that matters in modern economies and in the future world, struggling education systems could improve through following the techniques of the more successful systems. However, after 18 years, this improvement has been negligible. Cultural and other social factors appear to have a larger impact than first thought. Additionally, upon closer examination, the fundamental claim of the link between particular skills and knowledge and economic growth is not supported by evidence. Finally, PISA espouses a narrow view of the purpose of education. The consequence is a trend of a celebration of more didactic pedagogy and the high PISA scores they appear to bring while ignoring the negative consequences on important human attributes and local cultures.

References:

Bybee, R., & McCrae, B. (2011). Scientific literacy and student attitudes: Perspectives from PISA 2006 science. International Journal of Science Education, 33(1), 7-26.

Hopmann, S. T. (2008). No child, no school, no state left behind: Schooling in the age of accountability. Journal of Curriculum Studies, 40(4), 417-456.

Kjærnsli, M., & Lie, S. (2011). Students’ preference for science careers: International comparisons based on PISA 2006. International Journal of Science Education, 33(1), 121-144.

Komatsu, H., & Rappleye, J. (2017). A new global policy regime founded on invalid statistics? Hanushek, Woessmann, PISA, and economic growth. Comparative Education, 53(2), 166-191.

Shirley, D. (2017). The New Imperatives of Educational Change: Achievement with Integrity. New York: Routledge.

Sjøberg, S. (2015). PISA and Global Educational Governance-A Critique of the Project, its Uses and Implications. Eurasia Journal of Mathematics, Science & Technology Education, 11(1), 111-127