Getting started with comparative judgement: this is what you’ll need

You’re excited to get started with comparative judgement. But where to start? We’re happy to help you on your way by listing the things you’ll need. Good luck! 

Students’ works

Logical, you might think. Nevertheless, we want to talk a little about the types of assignments here, since not all assignments are suitable for comparative judgement. Think of closed tasks such as closed questions, fill-in-the-blank questions or multiple-choice questions. After all, those types of tasks are easy to assess with an assessment key because there are usually only a limited number of answer options.

Comparative judgement is particularly interesting when the competence is too complex to be captured properly in one or a few aspects. Think of reflection reports, action plans, solution strategies, internship videos, visual works, presentations, thesises or portfolios.

How many assignments do you need for comparative judgement? Logically, comparative judgement can be done with as little as two. Up to 15 to 20 assignments can still be compared and ranked manually from least good to best. From 20 assignments and more, it becomes more difficult to keep track. A tool that supports comparative judgement can then be helpful.

Assessors

Of course, you could assess comparatively on your own. But if you want to rank the assignments with a certain reliability, it is better to have two assessors. And if you really want to guarantee that all aspects of the competence are taken into account and that validity is guaranteed, it is advisable to work with at least four assessors.

Where do you find these assessors? Comparative judgement does not require specific training. Anyone deemed capable of assessing the competence can be an assessor. They can be experienced teachers as well as novice teachers, but also (co-)students or external people from the field, such as internship mentors.

Of course, it all depends on the competence to be assessed. Suppose the students were assigned to design an advertisement. Anyone can judge whether the advertisement in question appeals or not. But to judge whether the advertisement took into account certain marketing principles, the assessor must have knowledge of marketing.

Time

This much is certain: assessing is a time-consuming job, whichever way you approach it. But if you think comparative judgement takes double the amount of time because you are grading in pairs, we would like to nuance this. For pairs with a clear difference in quality, it is quickly clear which product is the better one. However, if the products are of similar quality, it logically takes more time to make a decision. But the overall assessment time will never be longer than when assessing with a criteria list. Moreover, because criteria lists have to be developed and validated, and because people have to be trained to use them properly, comparative judgement actually saves time.

Comparing-tool

As mentioned earlier, you can perfectly rank a smaller number of assignments manually. But if you need to rank, say, 50 or 100 assignments, it is practically not feasible to randomly compose pairs yourself. A tool that supports comparative judgement is then a useful tool. Such a tool automates the process of pairing and enables fast and reliable pairwise comparison of products in an online environment.