Relationship Among United States Medical Licensing Step I, Orthopedic In-Training, Subjective Clinical Performance Evaluations, and American Board of Orthopedic Surgery Examination Scores: A 12-Year Review of an Orthopedic Surgery Residency Program

Objective

To improve the understanding of relationships among United States Medical Licensing Examination (USMLE Step I), Orthopedic In-Training Examination (OITE), Subjective Clinical Performance Evaluations, and American Board of Orthopedic Surgery Examination Part I (Abos-I) and Part II (Abos-II), which would help residency programs better achieve their educational mission.

Design

A 12-year descriptive study of retrospectively collected data.

Setting

One residency program with 47 resident participants.

Results

Residents that failed Abos-I and Abos-II had lower program mean OITE year-in-training (YIT) percentile rank scores. The program mean OITE YIT percentile rank score had a moderate relationship with Abos-I (% correct) score (r = 0.68, p < 0.0001) and an insignificant relationship with USMLE Step I (3-digit) score (r = 0.22, p = 0.13). Residents with upper quartile (≥220) USMLE Step I (3-digit) scores for our program had higher program mean OITE YIT percentile rank scores and Abos-I (% correct) scores than residents with lower quartile scores (≤202). Residents who scored in the upper quartile (≥55) for the program mean OITE YIT percentile rank score had higher Abos-I (% correct) scores than residents who did not. Residents who scored in the lower quartile for the third postgraduate year (PGY-3) program OITE YIT percentile rank score or for the program mean OITE YIT percentile rank score had a 5.2 and 5.8 time greater Abos-I failure risk, respectively. The program PGY-3 OITE YIT percentile rank score was the strongest Abos-I (% correct) score discriminator. Resident Abos-I (% correct), program mean OITE YIT, and program PGY-3 OITE YIT percentile rank scores were the strongest discriminators for Abos-II passage. Residents with a program mean OITE YIT percentile rank score ≥28, program PGY-3 OITE YIT percentile rank score ≥39, and USMLE Step I (3-digit) score ≥207 were more likely to pass Abos-I and II. Residents that had lower quartile USMLE Step I (3-digit) scores for our program had a 2.3 time greater Abos-I failure risk. Program residents with ≥2 below-average subjective clinical performance evaluations had lower Abos-I (% correct) scores but had similar Abos-I and II pass rates.

Conclusion

Our program uses the USMLE Step I (3-digit) score as a preacceptance estimate of likely supplemental guided mentoring needs. Program mean OITE YIT percentile rank and PGY-3 OITE YIT percentile rank scores help identify educational deficiencies and predict eventual Abos-I and II passage. Subjective clinical performance evaluations provide important supplemental information regarding professionalism, communication, and patient care skills.

Key Words: resident education, surgery, curriculum, admissions

Competencies: Medical Knowledge, Practice-Based Learning and Improvement, Professionalism