Introduction
In medicine, practical and clinical skills are typically assessed by the objective structured clinical examination (OSCE).1 Whilst well-designed OSCEs are reliable and valid, they are extremely expensive to run – requiring significant staff and patient time, travel, accommodation, equipment and venue costs.2
The sequential OSCE (sOSCE) aims to balance robustness in assessment with affordability.3–5 Wainer and Feinburg6 succinctly describe the sOSCE as a ‘shorter test with an adaptive stopping rule’. In the sOSCE all students undertake a screening test (Day 1), which commonly has a higher passing threshold than the traditional OSCE. Candidates identified as ‘failing’ or ‘borderline’ undergo a second examination (Day 2). This provides more evidence to determine a candidate’s true performance with pass/fail decisions being made from the entire examination sequence.4,7,8
Research to date has focused on psychometric properties of the sOSCE, with only two studies exploring students’ views of this examination format.8,9 However, students are key stakeholders in high-stakes medical examinations,10 and their perception of fairness in medical assessment is an important aspect of the examination’s utility.11 If students do not perceive an assessment to be ‘fair’, its face validity and acceptability are reduced.12,13 This perceived lack of fairness can result in reduced motivation, poorer exam performance and a distracted focus, whereby students study for the exam rather than the content.10,14 Moreover, individuals who perceive a process as unfair are more likely to challenge the organisation15 and criticise the process.16,17
Interestingly, papers reporting the perceptions of key stakeholders towards a sOSCE, have done so within larger studies where the focus was cost and psychometric qualities of this examination format.8,9 Smee et al.9 reported sOSCE positives in terms of resource savings and psychometric robustness; however, administrative challenges provided some descriptive data on negative staff and student perceptions. These negative perceptions relate to a lack of understanding of the sOSCE and the generation of false beliefs between staff and students regarding the relationship between performance on Days 1 and 2. Some 10 years later, Pell et al.8 studied the impact of the sOSCE on student performance, cost and reliability, but also reported some anecdotal data related to student morale. They identified that students undertaking Day 2 required a “great deal of support, both in reaffirming that they had not ‘failed’ and that they had a further opportunity to demonstrate their ability”.8
To date, no studies have focused specifically on student opinions and their perception of fairness of the sOSCE. Yet, as discussed earlier, the acceptability of an assessment is an important aspect of an examination’s utility,11 the lack of which may adversely influence learner motivation and satisfaction. Thus, to address this gap in the literature, the aim of the current study was to identify and explore student perceptions of, and attitudes towards, a newly introduced final-year sOSCE via a questionnaire survey.
Methods
Participants
A sOSCE was implemented for the first time in a medium-sized Scottish medical school in 2018. The medical programme is 5 years, integrated (i.e. no preclinical/clinical divide). Approximately three-quarters of the final-year medical student population (n = 154) entered medicine directly from high school. The other quarter were graduates on entry.
Students were familiar with the traditional OSCE as an assessment method, as it is used for formative and summative purposes from year 1 onwards. They had no prior experience of a sOSCE.
The year 5 cohort undertook the sOSCE as their final high-stakes exit examination, replacing a long-established 2-day traditional OSCE. The traditional OSCE consisted of eight stations on Day 1 and seven stations on Day 2. Students were provided with written and verbal information regarding the change to the sOSCE on multiple occasions throughout the preceding academic year.
Day 1 of the sOSCE consisted of 12 stations blueprinted to the curricula. Each station was 8 minutes long and all stations tested more than one skill required for a junior doctor (e.g. communication, examination and clinical reasoning skills), as per usual local OSCE station practices. A prespecified exemption standard was created using the Borderline Regression Method,18 which included passing ≥9 out of 12 stations and achieving the pass mark [+3 root mean square error (RMSE)].
Day 1 results were distributed 8 days after the examination. Day 2 took place 11 days after Day 1 results were provided, a total of 19 days after Day 1. Twenty seven students (17.5%) failed to meet the exemption criteria for Day 1 and were required to sit Day 2. Day 2 consisted of nine additional stations, which had all been used in previous exam diets with validated pass marks from these cohorts. To graduate, students sitting Day 2 had to meet the overall pass mark (+3 RMSE) and pass ≥14 out of 21 of the stations they sat. No re-sit was available for this examination.
Questionnaire development
Given the paucity of literature exploring student perceptions of a sOSCE, we used several sources of information to inform question development. First, to identify issues important to the students, a convenience sample of 47 final year students were asked the question ‘What do you think about the sequential OSCE?’ or ‘How do you feel about having a sequential OSCE?’. Second, we sought the opinions of five key members of the final-year research and assessment team to determine questions that would be useful in shaping future implementation of the sOSCE. Finally, issues identified from the literature regarding fairness and challenges associated with the sOSCE, such as stress and a sense of failure, were drawn upon.8,9
From these sources, 10 questions were created with the opportunity to answer using a five-point Likert scale of ‘Strongly Agree’, ‘Agree’, ‘Neutral’, ‘Disagree’ or ‘Strongly Disagree’. We used the Likert scale format as it is a well-established and reliable method, easy to construct and efficient for gathering large amounts of data.19,20 Two of the 10 questions were deemed purely locally relevant – asking students about dates/timing and impact on finances/housing – and so have not been included in this paper.
The questionnaire included an initial section explaining what a sOSCE is, how this compared to a traditional OSCE and why the format had changed. This ensured all students had the same knowledge of a sOSCE prior to completing the questionnaire. In addition to the 10 structured questions we also included an open comment option and encouraged respondents to use this to provide any additional comments about the sOSCE.
Data collection
We distributed the questionnaire after Day 1 of the sOSCE, but before the results were published. This meant all potential respondents were in the same position and did not yet know if they had to undertake Day 2. The questionnaire was paper-based, handed out and collected on the same day at a mandatory session for all final-year students. Paper format was used to maximise the number of responses.21
Responses were transcribed from paper and uploaded into Excel (Microsoft, USA). ‘Strongly agree’ and ‘agree’ responses were combined, as well as ‘strongly disagree’ and ‘disagree’, to create three discrete categories for ease of analysis and interpretation. All open comments were transcribed verbatim, and subject to inductive thematic analysis.22,23
Ethics
Our local ethics committee categorises questionnaire-based studies where the data is provided anonymously and on a voluntary basis as not requiring ethical approval. In addition, all students sign a disclaimer on entry to medical school stating their anonymous data can be used for educational research purposes.
Results
A total of 107 participants (69.5%) completed the survey. The results from the Likert-style questions are presented in Table 1. The majority of respondents agreed/strongly agreed with the following statements: ‘I feel stressed about the final year OSCE’ (98.1%), ‘If I have to take Day 2 of the OSCE I will feel like a failure’ (89.7%), ‘Day 2 of the sequential OSCE seems the same as a re-sit’ (78.5%), ‘Having fewer stations and exam days is positive’ (61.7%) and ‘The sequential OSCE is unfair to students who pass with a low mark’ (46.7%).
Table 1 Likert questionnaire responses
|
|
|
|
|
|
|
|
|
I feel stressed about the final year OSCE
|
105
|
98.1
|
1
|
0.9
|
1
|
0.9
|
|
If I have to take Day 2 of the OSCE I will feel like a failure
|
96
|
89.7
|
9
|
8.4
|
2
|
1.9
|
|
Day 2 of the sequential OSCE seems the same as a re-sit
|
84
|
78.5
|
6
|
5.6
|
17
|
15.9
|
|
Having fewer stations and exam days is positive
|
66
|
61.7
|
28
|
26.2
|
13
|
12.1
|
|
The sequential OSCE is unfair to students who pass with a low mark
|
50
|
46.7
|
28
|
26.2
|
29
|
27.1
|
|
I would prefer to have a 2-day OSCE where all students sit the same number of stations, and those who fail re-take the year
|
39
|
36.4
|
24
|
22.4
|
44
|
41.1
|
|
Day 2 of the sequential OSCE is a chance to prove yourself
|
38
|
35.5
|
26
|
24.3
|
43
|
40.2
|
|
The sequential OSCE is fairer than the previous system
|
25
|
23.4
|
39
|
36.4
|
43
|
40.2
|
The majority of respondents disagreed/strongly disagreed with the following statements: ‘The sequential OSCE is fairer than the previous system’ (40.2%), ‘Day 2 of the sequential OSCE is a chance to prove yourself’ (40.2%) and ‘I would prefer to have a 2-day OSCE where all students sit the same number of stations, and those who fail re-take the year’ (41.1%).
Three main themes were identified from thematic analysis of open comments: 1) heightened stress and pressure; 2) negative psychosocial impact; and, 3) concern regarding communication and change.
Heightened stress and pressure
Students were stressed by the ‘pressure on you to pass the first day’ (R9) and ruminated over the potential of having to do Day 2: ‘the thought of having to sit a second day makes me terrified’ (R66). Students reported the lack of a ‘clean slate’ (e.g. R61) for Day 2 (meaning their performance on Day 1 was still accounted for and not disregarded, thus they needed to make up for this on Day 2) resulted in increased stress as ‘the increased pressure of knowing that you performed poorly on day one will make day two incredibly stressful’ (R19).
Negative psychosocial impact
Respondents felt that having to take Day 2 would have ‘a very negative impact on confidence’ (R16), ‘morale’ (R17) and ‘mental health’ (R29), making you feel like ‘a failed product’ (R45) and ‘inadequate’ (R66). Students also felt it impacted on their peer relationships resulting in a ‘hostile competitive environment’ (R66) and a ‘worse atmosphere’ (R16): ‘The second day promotes a more competitive environment, pitting students against each other to try and pass on the first day’ (R69).
Concern regarding communication and change
Students reported the change as ‘frustrating’ (R36) referring to having been ‘screwed over’ (R18) and commenting that they ‘were surprised with the new experimental system’ (R36). They felt that they ‘had very little information prior to the week before the OSCE’ (R43) and that they ‘should’ve been informed about details of passing day 1/sitting day 2 a lot earlier than we were’ (R40). There was also evidence of confusion regarding the exemption criteria: ‘same pass mark? Or bottom 1/3 OR pass mark + 3 SEM or bottom 40?’ (R67).
On the other hand, some viewed the sOSCE positively: ‘Having a 2nd day has significantly reduced my anxiety prior to sitting the first day of the OSCE, despite my discomfort with the new process’ (R14) and ‘if you pass first time it’s great’ (R11).
Discussion
To the best of our knowledge, this is the first study purposively designed to gather stakeholder (student) views of a newly introduced, high-stakes sOSCE. We found that the majority of students who had sat the sOSCE but did not yet know the outcome of their assessment, did not think the sOSCE was fairer than a traditional 2-day OSCE. Respondents reported that if they had to take Day 2 of the sOSCE, they would feel like a failure and perceived Day 2 as a resit. The perception of Day 2 as a resit may have been exacerbated by the interval between Days 1 and 2, which was necessary to ensure rigour and ratification in the examination board processes. However, and somewhat contradictorily, only approximately one-third of respondents stated that they would prefer a 2-day OSCE where all students sit the same number of stations, and those who fail re-take the year. Contradictory responses may imply an ongoing lack of familiarity with this examination format, and all responses were against a backdrop of high levels of general exam anxiety.
Our findings are in keeping with Smee et al.,9 who also reported that approximately one-third of their students would not want another sOSCE. Thus, despite negative comments about increased pressure and stress, psychosocial impact, dislike of change and poor communication, the majority of the students are not wholly negative about the sOSCE and like having fewer exam days.
Again, in keeping with Pell et al.8 and Smee et al.,9 we identified a strong sense of failure associated with sitting Day 2. On entrance to medical school, students are unadjusted to failure.24 It is likely that students undertaking Day 2 may never have ‘failed’ before, and, therefore, may find this perceived ‘failure’ especially difficult, particularly given those who in Day 1 ‘passed with a low mark’ had to do Day 2 (in a traditional format OSCE these individuals would have had no obvious negative consequences as they would have passed). Drawing on the underperformance literature, it may be that if having to sit Day 2 is seen as ‘failing’, then having to do so may be blamed on the nature of the Day 1 exam or the cut-off for having to do Day 2, rather than because of any personal limitation or genuine weakness.25–27 This may explain many of the negative and contradictory attitudes identified in this study. Moreover, as this perceived ‘failure’ is in the middle of the sOSCE examination process, these attitudes may be detrimental to Day 2 performance, given that failing students disengage with self-regulated learning, select inappropriate learning strategies and resist engagement with appropriate support systems.27 Whether these attitudes are present or not in a sOSCE situation is currently unknown, but merit further investigation.
We identified an increased level of stress, pressure and anxiety associated with the sOSCE and its perceived lack of fairness. Numerous factors contribute to increased stress levels amongst medical students.28 As seen in our study and in the literature, exam pressure and test anxiety are significant contributors to medical student stress.29,30 Specific to this study, the unfamiliar nature of the assessment may have also contributed. In the right balance, stress can improve motivation and performance,31 but excessive stress in medical students can lead to increased levels of depression, anxiety, burnout, poorer academic performance, reduced empathy and impaired physical health.32–36 Given this, further investigation into specific aspects of the sOSCE that cause stress should be identified in order to reduce test anxiety and the associated negative sequalae. One obvious area to consider is communication and presentation of new assessment methods to stakeholders. This study and the previous work by Pell et al.8 and Smee et al.9 suggest clear communication regarding the change to the sOSCE, the examination format and exemption criteria, is essential to address misconceptions and (hopefully as a consequence) reduce assessment anxiety.
Our findings raise two general issues. The first of these is how best to manage change with medical students (in this case, changing a high-stakes assessment from one format to another). In retrospect, it may have been better to bring in this change earlier in the curriculum, for a lower-stakes or formative assessment. To have done so would have given the students insight into and experience of the format in a less anxiety-provoking context. The second issue is that of study design. This cross-sectional survey is useful for obtaining a ‘snapshot’ of an issue and creating a hypothesis for future studies. However, our study has several limitations. The first of these is responder bias, as only 69.5% of students completed the questionnaire. This response rate may be due to student fatigue, but is relatively good for this type of study.37,38 Second, as the freehand comments were not mandatory, only those with strong opinions may have answered this part of the questionnaire. Like all cross-sectional studies, our results may differ if repeated with different cohorts of students.39 Third, we developed our own questionnaire as it was not possible to find an existing tool fit for purpose, which would have facilitated comparison of our findings to other studies. Fourth, owing to the organisation of the assessment it was not possible to assess student opinions after Day 2, therefore, we do not know if/how their opinions changed after taking the second test. Finally, a qualitative study may have enabled a more in-depth exploration of student attitudes and views, allowing us to tease out issues related to the sOSCE from those related to assessment generally. A qualitative study may have also enabled the introduction of some theoretical framing into data collection, rather than the purely data/literature-driven questions used in the current study. For example, investigating the lack of perceived fairness using organisational justice theory may be useful. This would allow exploration of the lack of perceived fairness: that is, whether this is due to lack of procedural justice (fairness of the procedure), distributive justice (fairness of the outcome) or interactional justice (fairness of interactions and partnerships with staff).40,41
In conclusion, this is a single-site study of one cohort of students. It gives some insight into student attitudes towards the sOSCE, and the belief that having to do Day 2 represented ‘failure’ is of interest to those tasked with assessment processes in medical schools. A final-year OSCE is a high-stress assessment whatever the format. Changes in assessment procedure should be introduced with care to avoid additional pressure on candidates. If well managed, moving from a traditional OSCE to a sOSCE has the potential to save medical schools costs and to reduce the exam burden on students.
Acknowledgements
I would like to thank the year 5 team for their contribution to question development.
References
1 Sherazi MH, Dixon E. The Objective Structured Clinical Examination Review. Berlin: Springer; 2019.
2 Brown C, Ross S, Cleland J et al. Money makes the (medical assessment) world go round: the cost of components of a summative final year Objective Structured Clinical Examination (OSCE). Med Teach 2015; 37: 653–9.
3 Cookson J, Crossley J, Fagan G et al. A final clinical examination using a sequential design to improve cost-effectiveness. Med Educ 2011; 45: 741–7.
4 Homer M, Fuller R, Pell G. The benefits of sequential testing: Improved diagnostic accuracy and better outcomes for failing students. Med Teach 2018; 40: 275–84.
5 Currie GP, Sivasubramaniam S, Cleland J. Sequential testing in a high stakes OSCE: determining number of screening tests. Med Teach 2016; 38: 708–14.
6 Wainer H, Feinberg R. For want of a nail: why unnecessarily long tests may be impeding the progress of western civilisation. Significance 2015; 12: 16–21.
7 Muijtjens AM, van Vollenhoven FH, van Luijk SJ et al. Sequential testing in the assessment of clinical skills. Acad Med 2000; 75: 369–73.
8 Pell G, Fuller R, Homer M et al. Advancing the objective structured clinical examination: sequential testing in theory and practice. Med Educ 2013; 47: 569–77.
9 Smee SM, Dauphinee WD, Blackmore DE et al. A sequenced OSCE for licensure: administrative issues, results and myths. Adv Health Sci Educ 2003; 8: 223–36.
10 McCoubrie P. Improving the fairness of multiple-choice questions: a literature review. Med Teach 2004; 26: 709–12.
11 van Der Vleuten, Cees PM. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ 1996; 1: 41–67.
12 Memon MA, Joughin GR, Memon B. Oral assessment and postgraduate medical examinations: establishing conditions for validity, reliability and fairness. Adv Health Sci Educ 2010; 15: 277–89.
13 Duffield K, Spencer J. A survey of medical students’ views about the purposes and fairness of assessment. Med Educ 2002; 36: 879–86.
14 Derous E, Born MP. Impact of face validity and information about the assessment process on test motivation and performance. Le Travail Humain 2005; 68: 317–36.
15 Macan TH, Avedon MJ, Paese M et al. The effects of applicants’ reactions to cognitive ability tests and an assessment center. Person Psychol 1994; 47: 715–38.
16 Truxillo DM, Bauer TN, Campion MA et al. Selection fairness information and applicant reactions: a longitudinal field study. J Appl Psychol 2002; 87: 1020.
17 Goldman BM. Toward an understanding of employment discrimination claiming: an integration of organizational justice and social information processing theories. Person Psychol 2001; 54: 361–86.
18 Cameron I. University of Aberdeen Assessment Handbook [Internet]. 2018. file:///C:/Users/mes57/Downloads/handbook-iemds-healthcare-programmes-assessment.pdf (accessed 11/07/19).
19 Li Q. A novel Likert scale based on fuzzy sets theory. Expert Syst Appl 2013; 40: 1609–18.
20 Boone HN, Boone DA. Analyzing likert data. J Extension 2012; 50: 1–5.
21 Nulty DD. The adequacy of response rates to online and paper surveys: what can be done? Asses Eval High Educ 2008; 33: 301–14.
22 Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psych 2006; 3: 77–101.
23 Guest G, MacQueen KM, Namey EE. Applied Thematic Analysis. First ed. Thousand Oaks: Sage Publications; 2012.
24 McManus IC, Smithers E, Partridge P et al. A levels and intelligence as predictors of medical careers in UK doctors: 20 year prospective study. BMJ 2003; 327: 139.
25 Todres M, Tsimtsiou Z, Sidhu K et al. Medical students’ perceptions of the factors influencing their academic performance: an exploratory interview study with high-achieving and re-sitting medical students. Med Teach 2012; 34: e325–31.
26 Cleland J, Arnold R, Chesser A. Failing finals is often a surprise for the student but not the teacher: identifying difficulties and supporting students with academic difficulties. Med Teach 2005; 27: 504–8.
27 Patel R, Tarrant C, Bonas S et al. The struggling student: a thematic analysis from the self-regulated learning perspective. Med Educ 2015; 49: 417–26.
28 Benbassat J, Baumal R, Chan S et al. Sources of distress during medical training and clinical practice: suggestions for reducing their impact. Med Teach 2011; 33: 486–90.
29 Shaikh BT, Kahloon AF, Kazmi MF et al. Students, stress and coping strategies: a case of Pakistani medical school. Educ Health (Abingdon) 2004; 17: 346–53.
30 Hashmat S, Hashmat M, Amanullah F et al. Factors causing exam anxiety in medical students. J Pak Med Assoc 2008; 58: 167.
31 Pfeiffer D. Academic and Environmental Stress Among Undergraduate and Graduate College Students: a Literature Review. University of Wisconsin; 2001.
32 Guthrie E, Black D, Bagalkote H et al. Psychological stress and burnout in medical students: a five-year prospective longitudinal study. J R Soc Med 1998; 91: 237–43.
33 Hojat M, Mangione S, Nasca TJ et al. An empirical study of decline in empathy in medical school. Med Educ 2004; 38: 934–41.
34 Dyrbye LN, Thomas MR, Power DV et al. Burnout and serious thoughts of dropping out of medical school: a multi-institutional study. Acad Med 2010; 85: 94–102.
35 Midtgaard M, Ekeberg Ø, Vaglum P et al. Mental health treatment needs for medical students: a national longitudinal study. Euro Psych 2008; 23: 505–11.
36 Sohail N. Stress and academic performance among medical students. J Coll Physicians Surg Pak 2013; 23: 67–71.
37 Ali J, Goh A. Student perceptions of workplace-based assessment. Clin Teach 2017; 14: 319–24.
38 Nesbitt A, Baird F, Canning B et al. Student perception of workplace-based assessment. Clin Teach 2013; 10: 399–404.
39 Keyes KM, Utz RL, Robinson W et al. What is a cohort effect? Comparison of three statistical methods for modeling cohort effects in obesity prevalence in the United States, 1971–2006. Soc Sci Med 2010; 70: 1100–8.
40 Lizzio A, Wilson K, Hadaway V. University students’ perceptions of a fair learning environment: a social justice perspective. Asses Eval High Educ 2007; 32: 195–213.
41 Patterson F, Zibarras L, Carr V et al. Evaluating candidate reactions to selection practices using organisational justice theory. Med Educ 2011; 45: 289–97.
