When is comparative judgement useful?

As a teacher, you have a range of assessment methods to choose from when assessing tests and tasks. Comparative judgement is one of them. But when is it a good idea to use this method? We will tell you in this article.

Complex competences

The choice of assessment method depends on the learning goals and the test or task you use to assess the extent to which your students achieve those goals. For example, knowledge is easily assessed using multiple-choice or closed-ended questions. Such tests are easy to assess reliably. Assessors will always arrive at very similar results, after all.

Comparative judgement is particularly interesting when the competence is too complex to be well captured in one or a few aspects. Think of reflection strategies, action plans, solution strategies, internship films, visual works, musical pieces, presentations, theses or portfolios. To try to grasp this complexity, a tendency towards the use of criteria lists and rubrics emerged. However, such analytical assessment methods often fall short in terms of reliability and validity. For instance, there is often overlap between different criteria or they are sometimes interpreted in different ways.

Therefore, for assessing complex competences, comparative judgement has received increasing attention in recent years. This method more closely matches the way the human brain works and is thus cognitively easier. As a result, assessors make more consistent decisions, which in turn leads to more reliable results.

A larger number of products

Comparative judgement with an assessment tool only makes sense from 16 products onwards. After all, a smaller number of products can still be perfectly ranked manually from least good to best. From 16 products onwards, it becomes confusing and thus more difficult to compile a ranking yourself. A tool that supports comparative ranking is then a useful aid.

In addition, it is important for the reliability of the assessment that products can be compared several times with several other products. To achieve sufficiently reliable results, a minimum number of 16 products is required. With fewer products, you need more comparisons than all possible pairs that can be formed. With a larger number of products, you already have enough data with a fraction of all possible combinations to reliably estimate the ranking.

Multiple assessors

In principle, you can also establish a rank order by yourself or with two assessors. However, it makes more sense to include multiple perspectives in the assessment. To ensure that all aspects of the competency are taken into account and thus that the validity of the assessment is on point, it is recommended to work with a minimum of four assessors.

Various formats

Comparative judgement is suitable for assessing different formats. And that because you are not bound by a rigid assessment framework. So not only texts, but also videos, music, drawings or images are suitable for comparative judgement. Suppose you assign your students to introduce themselves in a creative way. One sends in a video and the other a poem. With comparative judgement, these can be perfectly assessed together in an assessment.