What is comparative judgement?

Why assessing complex skills is so difficult is explained in this article. The good news is that there is an alternative assessment method that is more in tune with how the human brain works. How do you naturally make decisions, after all? Indeed: by making comparisons. And that is exactly what comparative judgement does.

If you are going to buy shoes, for example, chances are small that you will go to a shoe shop with a typed-out criteria list on which you are going to judge all the shoes. What you are more likely to do, and without thinking, is to compare pairs of shoes, or compare that one pair of shoes with an ideal image in your head.

When teachers and lecturers assess, they do the same. Give them one student’s product, and it is difficult to make a quality assessment. Offer them products in pairs, ask them to compare them with each other and yes, they will effortlessly answer the question of which product is the best of the two. Because of their expertise, they quickly see which opinion piece is better argued, which source research is more in-depth or which visual work is more creative.

Comparative judgement is making comparisons

And you do this almost without thinking. For example, it is impossible to estimate the altitude of an overflying aircraft. But if two planes fly over, you can quickly see which one is highest in the sky. So, using the same principle, you don’t assess students’ works as separate products, but in relation to each other. From two products, you choose the best one. By comparing all products in pairs this way, you will finally arrive at a ranking from ‘least good’ to ‘best’.

Comparative judgement is the counterpart of absolute judgement

With absolute judgement you look at one task, whereas with comparative judgement you situate the task in relation to other tasks. In addition, with comparative judgement you also choose for a holistic approach. Which means that you assess the task as a whole. This is in contrast to analytical methods that usually work with criterion lists: a piece of work is analysed in detail by looking at different aspects and sub-competences.

Comparative judgement reflects the consensus between assessors

And that without lengthy discussions, but simply by the statistical model (Bradley-Terry-Luce) underlying the method. It calculates a quality scale from lesser to better quality. It also becomes quickly clear which assessors deviate from the consensus. Who chooses regularly different and about which products are the opinions strongly divided? This provides useful information that can be further explored.