Programmatic assessment

Learning outcomes, competences, flexible education… all good new ideas, but then you as a teacher have the task of translating them into your daily practice. We want to link up as much as possible with the future working practice of the student, and yet not focus too much on evaluation but all the more on learning. So what does that look like concretely in education? For instance, how do we evaluate broad learning outcomes that require many underlying skills? No individual exam or test is going to ‘capture’ that entire learning outcome. And oh yes, it had to be mostly about learning. Programmatic assessment can be a solution for these problems. 

Because what if instead of focusing on optimising one exam or test, we combine different moments of measurement to form a holistic assessment of a student’s competence? That is what programmatic assessment is all about. In this article, we explain the concept in more detail.

Forming a reliable and valid picture of competence

Programmatic assessment is a way of working that is not easily explained in one sentence. Hence the following metaphor: Imagine taking a picture with your phone. The picture is made up of pixels. The pixels are all data points, points that give you information about the final image you see. If you have few pixels, the picture is blurred. You may not understand very well yet what you see. If you have many pixels, the image will be much clearer. Even if one pixel is in the wrong place, you will know what you see.

The pixels in the metaphor represent the evaluations or tests in an educational programme. In programmatic assessment, we no longer speak of evaluations, but of data points. In our traditional approach to evaluation, a data point sometimes consists of just one exam, with which we make a (sometimes far-reaching) claim about a student’s competence. One might wonder whether you then have a good picture of that competence. It is almost impossible to get the whole competence in that exam. In addition, there are all kinds of circumstances that can come into play: for example, if the student had a bad day, your judgement might be wrong. There were probably unwanted rater effects at work. There is no single exam with perfect reliability, validity, and impact on learning¹.

Programmatic assessment is assessing with data points

Programmatic assessment is a way of evaluating that makes you less dependent on the results of a single exam or test to make a decision. When you use programmatic assessment, you create a large number of data points on the basis of which you make a decision about the student’s competence.

Data points can be anything: a completed assignment, feedback on an elaboration in class, results of an internship, etc. All standalone data points have weaknesses in such areas as validity and reliability, but if you use enough data points, those weaknesses will average out and you will arrive at a well-founded decision.

The difference between feedback and a decision moment

Data points serve two functions: both a feedback function and a decision function. By working this way, evaluation is no longer a separate part of the learning process, but interwoven with education. The difference between evaluation and learning is blurred, so to speak; you want to ensure optimal constructive alignment².

Initially, data points are mainly used for formative assessment: where do students stand in their learning? Are they on the right track? What is still needed? This is where feedback plays an important role. An active role of students in the learning process is assumed, which also means that students increasingly collect and use the information about their learning themselves.

The actual decision on competence is made only when enough data points have been collected. So while feedback is generated based on one data point, a decision is never made based on one data point. In the end, the idea is that students already know what decision is going to come through the many feedback points along the way.

The importance of the decision determines how many data points are needed. If you determine whether someone has obtained a degree, the number of data points is obviously large. Such a decision is ideally made by a separately appointed committee that looks at the data points with an independent view.

If you want to know more about the practical workings of decision points and data points, read on here (in Dutch). In any case, if you want to start working with programmatic assessment, it is highly recommended to read up on it beforehand.

Thinking fundamentally different

Programmatic assessment requires a really different way of looking at evaluation. This may be difficult to imagine in concrete terms, so here’s an example: in an educational programme, the changeover is made to programmatic assessment. At the same time, a knowledge test which has major consequences is still being taken. If you don’t pass the exam, you don’t move on to the next year. Not even if your competences have been positively judged during the decision moment at the end of the year. Teachers are reluctant to use the exam as ‘just’ a data point. From a teacher’s point of view, this line of thinking makes perfect sense: if, for instance, students enter a profession where (lack of) knowledge can cause major consequences, you don’t want this to go wrong. From the traditional view of evaluation, you then naturally tend to stick to this exam.

Before you deploy the exam again anyway, you might ask yourself a number of questions, thinking from the programmatic assessment paradigm. For example: is this one exam valid and reliable enough to actually lead to a decision about the student’s knowledge and skills as the only data point? Besides, in working life, a student is unlikely to be asked to recite an entire list of factual knowledge. The knowledge is used in actions. So thinking from constructive alignment, you would also want to evaluate the application of that knowledge rather than its repetition. If you implement programmatic assessment, it is about forming a picture of the use of that knowledge by means of multiple data points. A knowledge test can possibly serve as a data point that says something about the student’s learning process, but for a complete picture that can lead to a decision, more information is actually needed in this situation.

At the same time, there are practical caveats to the above approach. Sometimes it is not possible (e.g. due to existing legislation) to simply make changes to existing evaluation methods. In such cases, it is sometimes decided to test certain competences programmatically, but to evaluate knowledge separately in the ‘traditional’ way. The transition to programmatic assessment is then partial.

Working with programmatic assessment

Programmatic assessment is a great way to focus more on student learning and less on summative assessment. At the same time, it becomes clear that our ‘old’ view of evaluation can still sometimes get in the way of applying programmatic assessment correctly³. This is not strange, but requires constant attention and active work. As with several educational innovations, this way of working also has its pitfalls. If you want to know more about possible pitfalls of this approach³, read on here (in Dutch).

In addition, it is important to realise that using programmatic assessment should not be an end in itself. If you are looking for a form of evaluation that is more about learning than about assessment, programmatic assessment can be a very valuable change. Just take a critical look at whether this way of working is actually suitable to achieve this goal within your own context.

Comparative judgement and programmatic assessment?

When using programmatic assessment, comparative judgement can be used, among other things, as a tool in the formative process. Comparative judgement means comparing works with each other. This is an intuitive process from which students can learn a lot⁴. For example, you could have students comparatively assess each other’s work. Perhaps students provide very different products (data points), but they can still be compared because the products are about the same learning outcome. Students can give each other (anonymous) feedback on the learning outcome based on all the possible examples they see. In doing so, they might also see that there are many different ways to work on a learning outcome. They can then create an action plan in Comproved based on the feedback to work further on their learning outcome. Comparative judgement can be easily done with our comparing tool. Want to know more about the formative use of comparative judgement? Download our guide here.

For the final summative decision moment, comparative judgement can also be interesting. Research shows that assessors in a decision committee can look at data points quite differently to reach their decision⁵. Comparative judgement can serve as a good tool to reach a reliable assessment with multiple assessors. Want to know more? Send us a message!

Literature

¹Van der Vleuten, C. P. M., Heeneman, S., & Schuwirth, L. W. T. (2021). Programmatic assessment. In J. Dent, R. M. Harden, & Hunt,D. (Eds.) A practical guide for medical teachers. (pp. 323- 327). Elsevier. 

²Torre, D. M., Schuwirth L. W. T., & Van der Vleuten, C. P. M. (2020). Theoretical considerations on programmatic assessment, Medical Teacher, 42(2), 213-220. https://doi.org/10.1080/0142159X.2019.1672863  

³Govearts, M., Van der Vleuten, C. P. M., & Schut, S. (2022). Implementation of programmatic assessment: challenges and lessons learned. Education Sciences, 12(10), 717. https://doi.org/10.3390/educsci12100717 

⁴Nicol, D. (2021). The power of internal feedback: exploiting natural comparison processes. Assessment & Evaluation in Higher Education, 46(5), 756-778. https://doi.org/10.1080/02602938.2020.1823314 

⁵Oudkerk Pool, A., Govaerts, M.J.B., Jaarsma, D.A.D.C. & Driessen, E.W. (2018). From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio. Adv in Health Sci Educ 23, 275–287. https://doi.org/10.1007/s10459-017-9793-y