Continuous, Data-Rich Appraisal of Surgical Trainees' Operative Abilities: A Novel Approach for Measuring Performance and Providing Feedback
Background
We developed a convenient mechanism, Surgical Training and Assessment Tool (STAT), to accomplish detailed, continuous analysis of surgical trainees' operative abilities, and a simple method, Quality Based Surgical Training (QBST) for implementing it.
Methods
Using a web-accessed computer program, attending physicians and trainees independently assessed the trainee's operative performance after every operative (training) case. Global attributes of surgical knowledge, skill, and independence were assessed as well as the key technical maneuvers of each operation. A system of hierarchical, expandable menus specific to each of hundreds of different surgical procedures allowed the assessments to be made as detailed or as general as the users felt were necessary. In addition, freehand, unscripted commentary was recorded via an optional “remarks” box feature. Finally, an independently chosen, “overall” grade scaled F through A+ concluded each assessment.
Results
Over a 31 month period, 72 different users (52 trainees, 20 attending physicians) submitted 3849 performance assessments on 2424 cases, including 132 different case types and amassing 68,260 distinct data points. The mean number of data points per trainee was 1313; the median time spent per assessment was 60 seconds. Graphic displays allowed formative review of individual cases in real time, and summative review of long term trends. Appraisals of knowledge, skill, and independence were strongly correlated with and independently predictive of the overall competency grade (model r2 = 0.68; test of predictive significance p < 0.001 for each rating). Trainee and attending physician scores were highly correlated (> 0.7) with one another.
Conclusions
QBST/STAT achieves detailed, continuous analysis of surgical trainees' operative abilities, and facilitates timely, specific, and thorough feedback regarding their performance in theater. QBST/STAT promotes trainee self-reflection and generation of continuous, transparent, iterative training goals.
Key Words: Clinical competence, educational measurement, Journal of Surgical Education, quality
Competencies: Practice-Based Learning and Improvement, (Technical Skill) Patient Care, Medical Knowledge, Professionalism
To access this article, please choose from the options below
Sources of funding: Department of Surgery, the University of Chicago; Computation Institute, the University of Chicago; Educational Research Grant, the University of Chicago Pritzker School of Medicine.
Special thanks to the residents, fellows and staff of the University of Chicago, Department of Surgery, who have participated in STAT's development, and also to Stephen A. Roach for his contributions with respect to Managerial Science and quality management.
PII: S1931-7204(09)00159-7
doi:10.1016/j.jsurg.2009.10.001
© 2009 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.
