Preliminary Thoughts on TIMSS/PIRLS 2012

I had been meaning to post about this for a long time. Better late than never, I suppose.

The TIMSS and PIRLS are international assessments of academic ability in math, science and literacy that are conducted once every four years. They are similar to the PISA tests, although the latter are less purely academically focused and more a test of pure IQ.

Here are the results of TIMSS/PIRLS (h/t North Asian). And here are the results of PISA from 2009 for comparison.

As can be expected, they are highly correlated (r > 0.8 to be precise). This however makes the few differences all the more interesting. The gap between the East Asian countries and European countries, though substantial in PISA, is significantly greater in TIMSS/PIRLS. And most strikingly, both Russia and Israel go from being laggards in the OECD group to being at the forefront of the class.

  Math (PISA) Math (TIMSS)
Korea 539 613
Sweden 494 484
Russia 468 539
Israel 447 516

From performing more poorly than Turkey in the PISA reading test, Russia soars to take second global position in the PIRLS.

  Reading (PISA) Reading (PIRLS)
HK 533 571
Sweden 497 542
Russia 459 568
Israel 474 541

Meanwhile, some European countries, especially Sweden and Norway, plummet quite substantially.

What explains all this?

There are two possibilities. First, the TIMSS/PIRLS tests may have poorer samples than the PISA. For instance, we know from the latter that Moscow has a 10-point IQ lead over the rest of the country. If Muscovite pupils are over-sampled, then it’s quite feasible for the consequent result to be closer to say Hong Kong or Korea than to Greece or Turkey.

However, a second possibility is that the PISA-TIMSS/PIRLS gap is a proxy for differences in the quality of educational systems. It is more feasible to prepare for the TIMSS/PIRLS than it is for PISA, which is closer to an IQ test and is, as such, more difficult to improve through policy interventions. It is nowadays fashionable to lambast the ex-Soviet and East Asian school systems for “rote learning,” “stifling creativity,” and whatnot. However, the data shows that under these systems, pupils perform well above the levels they “should” as indicated by their underlying IQ levels. Meanwhile, in places where “creativity” and “self-expression” are given full bloom, where science lessons focus on the evils of plastic bags in between sermons on LGBT appreciation and the progressiveness of Islamic civilization, academic performance is somewhat less than what might expect based on the local students’ apparent IQ levels.

This all makes sense, I suppose. To be truly “creative” you first have to acquire a ton of skills and knowledge via the old method of applied hard work. Without that, “creativity” simply boils down to a sea of PoMo-waffling curmudgeons and MacBook-toting hipsters. And whoever needs that?

  • SH

    I am from Sweden, and I can report that our nation’s educational system (at least primary and secondary) is a complete mess with no signs of improvement. Over the past 20 years we have implemented a range of bizarre experimental reforms which has steadily eroded the quality of our educational system. Anyone can start an independent for-profit school, or a chain of such schools, and get financed by public funding (so called “free schools”). It have proved to be a very lucrative and safe business field for private equity firms which have channeled out large profits overseas to tax havens by slashing lessons hours and personnel, instead allowing the pupils to conduct their own “research”.
    Also, the teaching profession have a very low status and often a last resort for those with poor educational results, how cannot enter any other tertiary education. Teacher students today often have reading and writing difficulties! Administrative tasks has also expanded dramatically over the last few years, because of new policies issued by the government, with less time left for teaching and student support as a result. In Sweden, the municipalities are responsible for primary and secondary school and they have handled this task poorly.
    I certainly believe that Sweden’s poor performance in TIMSS is a reflection of its failed educational system.

    • SH

      UK (Cameron and tories) have recently shown a keen interest in our educational reforms and may seek to implement something similar in UK in the future.

  • Math is important as an indicator of intelligence but for most people it is just useless in their daily life. So maybe the fluffy stuff is more important for a countries economy.

  • Albo

    One caveat about reading tests and international rankings:

    This is why I’d rather focus on Science&Maths rankings.

    • charly

      Problem with Science and Math is that they are useless for most people after a certain level and their only use is as a kind of IQ test

      • Jen

        Much mathematics that is taught at school becomes specialised quite early in the lower and middle grades and is only really useful as a background for people wanting to pursue particular studies at university. What use is teaching quadratic equations and trigonometry to most 14 and 15-year-olds and calculus to most 16-year-olds unless they are all going to university to study engineering, physics and other areas that need this knowledge? For most youngsters, knowing probability and statistics, business mathematics and enough algebra and geometry to help them cope with most situations they are likely to encounter as adults and make sense of what they see and read in the news might be enough. Although probably a little over-teaching wouldn’t hurt because we don’t know if and when “enough” really is enough.

      • JT

        Most of what we’re taught are useless, not just Science and Math. Shakespeare? American History? How many of us actually get to use these knowledge at our place of work? Except for the very few, we only need rudimentary reading comprehension skills to get by in our jobs.

        What we do know is that science and math are now becoming more important. Surveys on the most promising careers/majors are mostly dominated by fields in science and math.

        Meanwhile, humanities have taken a turn for the worst with regards to job prospects.

        • charly

          They are not dominated by science & maths but by intelligence especially if you look at the number that work in those respected fields. You don’t need to know any science or maths to be a software engineer or accountant.

          The important things you learn in high school are more social interaction and learning to learn than any of the knowledge about science or history etc. Only exception i would say is foreign languages

  • ic

    AK: No I am not interested in off-topic pseudo-science. Please don’t spam.

  • Andrei

    about israel, israel decided to cheat big time to make headlines… the truth is that among all participants countries israel excluded by far the most students then any other country:

    page 9

    it excluded 16.4% of the students almost three times more then second place singapore with 5.7%

    • charly

      shouldn’t that be noticeable in the distribution of the results