Use of the test
Each test has the majority of items focused on the most recent case-units which the students have studied, but also includes a sample of items from earlier tests. So tests generated for second-year students will include items from first-year tests. This means that students have to accumulate knowledge, rather than replace old with new.
The most recent first-year test was taken 118 times by 77 students (46% of the cohort). However this figure represents the number of students that logged on to take the test. In fact students sometimes took the test in pairs and small groups so the actual number of students that used the test was higher. The mean score for all attempts was 59% (SD 16).
The most recent second-year test was taken 201 times by 71 students (40% of the cohort). Again more students actually used the test. The mean score for all attempts was 61% (SD 22).
With a further development of the database of items we will be able to demonstrate knowledge development in greater detail according to specific themes and subtopics.
Feedback from the students was very positive. They found it easy to access and take the test. The students' comments bore out the face validity of the format of the items - a clinical vignette followed by 5 options. They particularly liked the increased specificity of the feedback from the formative items and regarded it as an improvement on the feedback from the summative progress test.
Feedback from the staff that took the test was more critical of the technical item flaws in the questions. For example one commentator noted that the vignette was sometimes irrelevant, that the 'cover the options rule' was sometimes not employed, and that the topic assessed was sometimes not particularly relevant.
Future iterations of this project will employ a more detailed analysis of the test. It will explore in more detail the distribution of topics assessed, qualities of the items, and the effectiveness of the feedback. It will also employ a more sophisticated evaluation of the use of the test from the perspectives of the students, faculty and clinical practitioners.
