The commenter “m” did some calculations to work out the relative performance of different countries in PISA vs. TIMSS, and in Math vs. Science.
m writes:
Dimension 1 is overall performance across all 4 (PISA Math, PISA Science, TIMMS Math, TIMMS science). Everything goes up with this dimension. Highest performers: Singapore, Chinese Taipei, Japan. Weakest performers: Turkey, UAE, Malta.
Dimension 2 separates stronger performers on TIMMS vs PISA: strongest performers on TIMMS relative to PISA are in order: Turkey, Korea, Russia, Hungary, United Arab Emirates, while strongest performers on PISA relative to TIMMS are: New Zealand, Canada, Australia, Norway, Italy. The five most balanced countries in the tradeoff are roughly: Slovenia, England, Hong Kong, Japan, USA.
Overperformance in TIMSS relative to PISA can arguably be used as a proxy for schooling quality, since it’s more dependent on academic/curricular skills than on raw intelligence. I am not surprised by the good figures for Korea, Russia, and to a lesser extent, the rest of East Asia and the post-Communist world. However, the UAE and Turkey are surprising.
Dimension 3 separates out Science nations vs Math nations: Most heavily Science vs Math: Slovenia, England, USA, New Zealand, Turkey and most heavily Math vs Science: Hong Kong, Malta, Korea, Italy, Norway.
As much as including TIMMS might be a worse proxy of “IQ” than just PISA, I have included in the above graphic a measure of using the PC1 overall performance score to convert to IQ, based on the assumption that England is 100 and Japan 104.3 as in your PISA conversion. There’s a bit of swing, not too much, compared to PISA alone.
m then extended his analysis to encompass Reading, which is unsurprisingly “less correlated with the other measures”:
as well as to the PIAAC Survey of Adult Skills:
Comment:
Singapore’s the biggest relative loser when the skills measure is rolled in as well, with the least advantage on PIAAC skills relative to TIMMS / PISA. Most other countries gain compared to the other PCA, as they are more advantaged relative to England and the East Asians on young people’s life skills than they are on young people’s education measures.
[Overperformance in TIMSS relative to PISA can arguably be used as a proxy for schooling quality, since it’s more dependent on academic/curricular skills than on raw intelligence]
Surely that effect, if it exists, would be confounded at least in the case of the TIMSS 4th grade tests, since they should be considerably younger than the PISA students and have had less time to benefit by schooling.
I was distracted by the thread about PISA hype. Rather than just hype, the PISA scores seem to be very significantly correlated with how the multinational corporate executives view the countries. The World Economic Forum had conducted the so called “Executive Opinion Survey” WEFEOS with rating from 0 to 100, high values being good. Regression with the PISA 2015 scores,
WEFEOS = +0.30018*Sci -77.3491; n=58; Rsq=0.4935; p=7.938e-10
WEFEOS = +0.267138*Math -60.6147; n=58; Rsq=0.4621; p=4.418e-09
WEFEOS = +0.283822*Read -69.0029; n=58; Rsq=0.4601; p=4.901e-09
As it stand there are already few countries with % at proficiency levels 5 and 6 greater than 20%. Any more abstract tests might fail to measure anything meaningful for many countries. There is already significant size country withdrawing from the PISA survey.
Roger Schank, 6 December: Getting people to think for themselves, becoming capable of employment, able to live happy lives, would seem to be better goals to me, but instead we have PISA.
https://educationoutrage.blogspot.com/2016/12/last-week-i-spoke-at-online-education.html
His rant: as with Trump, you should take him seriously but not always literally.
Regarding the value of testing, he has company. Various educationists have been pointing out that grading students has absolutely no basis in science. I’ve heard them saying that on radio. If you’re interested in references, ask Alfie Kohn: ….. While I understand the social mobility argument — a justification similar to what was offered for using standardized test scores as a criterion for university admission in the U.S. a century ago — the disadvantages [of exams] are now widely understood to swamp any benefits.
I asked him about how come the Chinese state was often at the same time both exam-centered and very, very capable (at least according to sinologists). He replied he doesn’t know.
Of course, one has to keep in mind that institutions and institutionalised social patterns – like tests – tend to have fairly short shelf lives.
“high levels of equity” between rich and poor pupils very important!
Please also see:
“”Even in rural areas and in disadvantaged environments, you see a remarkable performance.”
In particular, he said the test results showed the “resilience” of pupils to succeed despite tough backgrounds – and the “high levels of equity” between rich and poor pupils.”
http://www.bbc.co.uk/news/business-17585201
China: The world’s cleverest country?
Just came across an article about Ukraine’s TIMMS score:
http://voxukraine.org/2016/01/28/evaluating-ukraine-the-pisa-ranking-en/
Ukraine’s predicted PISA scores would be 478.6 in science (same as Iceland) and 460 in math (between Israel’s 466 and Greece’s 453).
The average of Ukraine’s predicted science and math PISA scores, 469, would be between the reading + science + math PISA averages of Slovakia (471) and Greece (466). In contrast, Russia’s readng + science + math average is 481, Serbia’s 447, Romania’s 440. Poland’s is 521.
Ukraine will be taking part in PISA in 2018.
I.e., doing marginally better than Turkey, the UAE, and perhaps even Romania. Lame.
So here’s the problem: One would assume that, based on the common legacy of its education system that, as with Russia and the other ex-Soviet regions, Ukraine would do relatively much better on TIMSS (PS. LOL, I always forget whether its two Ms or two Ss as well) than on PISA. So the regression analysis would probably be more accurate if done wrt to just the ex-Soviet countries.
In TIMSS 2011, Ukraine did an average (across Math/Science) of about 50 points (~7.5 IQ points) worse than Russia.
So I think expecting ~470 is quite optimistic, though still in line with my oft-made prediction that Ukraine is probably somewhere between Russia and the Balkans.
I suspect Ukraine will be more like a Romania/Bulgaria-like 450, with historical Novorossiya at a Stavropol/Rostov/Bryansk like 460, the far West at a Balkans-like level of not far above 400, and Kiev 1/2-2/3 S.D. above the country average, like Moscow, with perhaps 520.
But we’ll see. I am certainly looking forwards to December 2019. Let’s put it this way. Ukraine gets >460, you win; <460, I win? 🙂
Chernivtsi and Zakapatska oblast might be at a Balkans level but Lviv and Ivano-Frankivsk (Galicia) would probably be closer to Kiev. It’ll be interesting. Poor performance relative to IQ corroborates post-independence poor governance.