The Organisation for Economic Co-operation and Development (OECD) runs the Programme for International Student Assessment (PISA), which is run every three years and assesses the abilities of a representative group of 15 year olds in reading, maths, and science.
Approximately, 80 countries including the UK took part in the last PISA tests between October 2018 and January 2019. For England, this involved a sample of 5,144 students from 170 schools. PISA allows us to compare how our children are performing compared to international competitors and it is the main resource to compare education outcomes across the four devolved nations. PISA is the only cross-national assessment measured on a regular basis.
The government published a summary of the results based on the latest 2018 tests, which stated:
“In PISA 2018, mean scores in England were significantly above the OECD averages in all 3 subjects. The mean scores in reading and science in England have not changed significantly over successive PISA cycles, but in mathematics, England’s overall mean score showed a statistically significant increase compared with PISA 2015.”
Such improvement is then taken by governments of all persuasions as a justification of their education policies and a green light to continue with more of the same. The PISA 2018 results and the maths result in particular were loudly applauded in the UK press.
Dissenting voices
This glowing picture of the PISA 2018 results is now being challenged by a new report from Professor John Jeffreys of UCL. Research conducted by the widely respected University College London (UCL) Institute of Education suggests that the results and subsequent rankings were distorted. The study, published in the current edition of the Review of Education, analyses results from the OECD PISA study and highlights serious concerns about the 2018 edition data for England, Wales, Northern Ireland and Scotland.
The report points to low levels of participation in the test and a disproportionate underrepresentation of lower achieving students. If the data was adjusted and a more accurate sample of the UK’s national school population was tested, England’s place on the PISA league table for maths results would fall 11 places. Scotland has also seemingly manipulated the tests to show its education system in the best possible light. The UCL paper states:
“Scotland changed the PISA test date in 2018 to later in the school year. This meant that half the sample were in a later school year (S5, equivalent to Year 12 in England) than in previous editions of the test. Yet this was not commented upon by either the Scottish government or by the OECD. A number of other anomalies emerge with the Scottish data – such as a very high number of “ineligible” pupils – which the Scottish government has not adequately explained or discussed.”
The report’s author John Jerrim is a professor of education and social statistics. He said:
“PISA is meant to be a representative study of 15-year-olds across the UK. But there are serious flaws with some children being excluded from the study, schools being unwilling to participate, and some pupils not showing up for the test. The reported average PISA mathematics score for England was 504 points – significantly above the average across industrialised (OECD) countries. But had a truly representative sample been drawn, I estimate England’s PISA mathematics score would have been about 494- roughly the same as the OECD average.”
He also calls on the OECD to improve its standards and data adjudication processes:
“Clear guidelines need to be put in place to ensure more transparent reporting in the future.”
If schools ignore the least able or those who simply vote with their feet and don’t turn up on the day, and if they have the right to refuse a pupils’ participation in the tests, how can we claim that the PISA tests are a true reflection of progress made by a cohort of pupils or say anything about the effectiveness of our schools nationally? Surely a cohort of pupils must include all abilities and not only those that schools are happy to enter for the tests?
Maths is a core subject in England’s school curriculum. The GCSE maths result matters to every student in England. Yorkshire Bylines recently highlighted the importance of this subject and how it can affect young people’s life chances at 16, such as progress on to T-levels or onto apprenticeship programmes run by the most successful companies.
PISA results matter
The effectiveness of our education system lies at the heart of our politics. Providing a good national education for all citizens is a key function of all the UK’s governments. Our economic future depends upon it. PISA results are used by political parties to justify their education policies and to berate the political opposition when the results are going well.
The accusation that the PISA tests do not paint an accurate picture in Scotland or England should raise alarm bells in both countries. If we want to measure our progress and compare it with other countries we should set the bar high and do it as accurately as we can. This report levels serious allegations that cannot be ignored in any part of the UK.
The news that the results may not be as good as they seem, that UK schools and PISA test entrants are skewed towards the most able, undermines public faith in our schools and in their political leadership. We must aim to set the standard for accuracy in the next round of tests. All our UK governments must compel chosen schools and pupils to participate if asked. Perhaps Professor Jerrim should be in charge of the sampling for PISA 2021 in the UK?
We all know now that we ignore advice from experts to our peril.
