5 tips for asking a good comparison question

When you assess comparatively, the question you ask while comparing is crucial. Examples of questions are “Which text is most informative?” or “Which illustration is best?”. The comparison question tells you what you are actually comparing. Not surprisingly, asking a good comparison question is very important. What should you pay attention to when formulating such a question? And what not to do? We help you on your way with a few tips.

Tip 1: keep it holistic

Comparative judgement means you assess holistically from your expertise. Instead of scoring separate parts of an essay or illustration, for instance, in comparative judgement you assess the interplay of all those parts. This starting point means that when formulating your comparison question, you should also think about the ‘width’ of your question.

A comparison question that is too specific nullifies the holistic character of comparative judgement. So if you want to assess argumentative texts, you don’t ask ‘In which text is the structure of paragraphs the best?’ Instead, you ask ‘Which argumentative text is better?’

Tip 2: ensure alignment with task and learning objectives

A good comparison question is about that which needs to be assessed. This sounds like an open door, but it still can’t hurt to pay attention to it. Suppose you ask students to solve a client’s problem. Do you then only assess how well the solution solved the problem, or is it mainly about the correct use of a particular strategy to solve the problem? The difference in focus creates a different comparison question. “Which product is the best solution to the problem?” is a different question from “Which product best applies the principles of design thinking?”.

At the same time, you might see some overlap in the example. That is not a bad thing and even necessary. A good application of the principles of design thinking will help students arrive at a good solution to the problem. That’s right. It’s all about where you put your focus while comparing. If you focus your question on the principles of design thinking, it means that applying those principles correctly will be the deciding factor in the choices you make when comparing. While at another point in your teaching practice, you might just want to know whether it is also a good solution to the problem. With a well-tailored comparison question, you ensure that your assessors have the right focus at the right time in the learning processes.

Tip 3: maintain a balance between holistic and clear

If you formulate a holistic comparison question, you run the risk of it becoming somewhat vague. Suppose you ask students to write a text. “Which text is better?” is then a very holistic question, but not very clear. You don’t specify what kind of text you are talking about. For example, it could be about an informative text, or it could be about a promotional or academic text. It depends on the situation whether a vague question causes problems, or not.

You can ask an unclear holistic question only if you know that your group of assessors means the same thing by a ‘better text’. If your group of assessors consists of a team of teachers who have worked together for years, you can generally assume that everyone has a clear idea of exactly what you mean. Still, you could specify a little further here. Perhaps then your question changes to, “Which scientific introduction is better?”. This question remains holistic but already reflects a lot more clearly what the comparison is about.

Tip 4: Align with your group of assessors

Okay, an experienced teacher team does get there with a very holistic question, but what if your team of assessors keeps changing or is partly inexperienced? Maybe you also use external assessors. In such a case, your comparison question might need more explanation. It works best then to jointly compare (sample) works in advance. Based on these comparisons, you can see the reliability of the resulting ranking in Comproved. You can then discuss with the assessors the reasoning behind the comparisons. Through this discussion, you come to a clearer shared understanding of the competence.

After you have formed a shared understanding, you can add an explanation of the competency being assessed in Comproved. This can be a description that you have agreed on together. Be careful: the pitfall is that you now start describing in detail again and draw up a kind of criteria list. Your description only serves as a reminder about the competence you are assessing, but does not serve as a checklist. By the way, did you know that with a (limited) list you can steer assessors just fine in what they do and do not include in the assessments? “Don’t pay attention to spelling!” you can perfectly add to the assignment if you don’t want it to be decisive in assessments. Research has shown that assessors are good at taking this into account during comparisons!

Tip 5: keep it simple!

In this article, we give tips on how to ask a good comparison question. But it is also important to keep it simple. If balancing between holistic and sufficiently clear, focused on the task and tailored to assessors results in you being stuck with a very complicated question, it might not work out either. So always ask yourself whether your grandmother would understand the question without much explanation, so that it doesn’t become too complex!

Ask your question!

With the tips above, hopefully you have been able to formulate a good comparison question. Do you like to discuss your question or are you looking for examples from your field? Then get in touch with us.