How to Implement Qualitative Skills Assessments (“Competency Exams”) as Part of the Assessment of Clinical Competence

For dental education, the assessment of clinical performance is critical. While clinical reasoning skills can be evaluated with Applied Didactic Assessments (e.g., OSCE) or, in some instances, by paper assessments, clinical skills can only be assessed quantitatively (number of produced procedures) or qualitatively (direct observation and assessment via rubrics).

This manual is intended to provide a step-by-step “how-to” manual for Qualitative Skills Assessments (QSA). These assessments are commonly called “Competency Exams” by many dental schools. The author considers this a misnomer because there are other ways to assess competence that by observed skills. However, in addition to clinical production, there is no more important of an assessment type for determining clinical competence than the QSA. There is also no assessment type that is more fraught with calibration errors and subjective grading errors than the QSA, so it is critical that they be performed well and with much forethought.

This manual will first discuss the areas of competence that can be assessed using this technique, and we will review the basics of clinical competence. Then we will discuss the components of every Qualitative Skills Assessment. Much detail will be discussed about the components of rubrics and how to write sound rubrics. This part is tedious, but is incredibly important to the process. Well-designed rubrics make the calibration and assessment easy and poorly-designed rubrics can simply be unfair to the student with unreproducible results. A brief overview of assessment-based inter-rater reliability calculations is provided to set the stage for discussions concerning multiple graders.

The rules and point values of the assessments are detailed in the following chapter with an emphasis on understanding passing and failing students in competency-based assessment versus grade-based assessment. The deployment issues and workflow involved with actually giving a QSA with multiple graders is provided along with a discussion about student self-assessment.

Finally, the data analysis of student performance, faculty performance (as graders), and assessment performance are all detailed with examples and best practices. At the end, the reader should be able to construct strong and reproducible Qualitative Skills Assessments.