Impact of Rater Training on Outcome of Formative Assessment-Mini-CEX in Gynecology Residents: A Quasi Experimental Study

Abstract

Introduction: The literature on rater training in medical education is diverse and limited. When implemented, rater training shows only minimal improvement in the psychometric properties of trainee evaluations. Further research is needed to evaluate the effectiveness of rater training in medical education.

Objective: To determine the impact of rater training on the reliability of the Mini-Clinical Evaluation Exercise (Mini-CEX).

Methods: In this quasi-experimental study three clinical scenarios were developed and recorded on video, each enacted by a standardized patient and a resident. A group of assessors was recruited to evaluate the resident's clinical skills using the Mini-CEX for each videotaped scenario. The training workshop was held for assessors. Each assessor was required to watch the videotaped clinical scenarios and rate residents before and after the rater training workshop. This ensured a minimum interval of three weeks between Mini-CEX rating sessions. Pre- and post-training assessment data were analyzed using SPSS version 23.

Results: Pre-training assessments exhibited poor reliability (Cronbach's Alpha = 0.090, 0.356, 0.006, respectively), which showed that the reliability of assessment was low, whereas post-training assessments showed significant improvement (Cronbach's Alpha = 0.809, 0.773, 0.739, respectively). Correlation analysis between pre- and post-training questionnaire scores was negative but significantly strong, i.e., r = -0.857 (p < 0.05). Post-training results indicated enhanced internal consistency, demonstrating the rater training efficacy in standardizing assessment practices.

Conclusion: This study emphasizes on the importance of rater training in achieving consistent and reliable evaluations in formative assessments.

Keywords: Medical Education, Mini-CEX, Rater training, Standardized Assessment, Workplace-based assessment