Explore the vast range of titles available.

Simulation-based training is increasingly being used for assessment and training of psychomotor skills involved in medicine. The application of artificial intelligence and machine learning technologies has provided new methodologies to utilize large amounts of data for educational purposes. A significant criticism of the use of artificial intelligence in education has been a lack of transparency in the algorithms' decision-making processes. This study aims to 1) introduce a new framework using explainable artificial intelligence for simulation-based training in surgery, and 2) validate the framework by creating the Virtual Operative Assistant, an automated educational feedback platform. Twenty-eight skilled participants (14 staff neurosurgeons, 4 fellows, 10 PGY 4-6 residents) and 22 novice participants (10 PGY 1-3 residents, 12 medical students) took part in this study. Participants performed a virtual reality subpial brain tumor resection task on the NeuroVR simulator using a simulated ultrasonic aspirator and bipolar. Metrics of performance were developed, and leave-one-out cross validation was employed to train and validate a support vector machine in Matlab. The classifier was combined with a unique educational system to build the Virtual Operative Assistant which provides users with automated feedback on their metric performance with regards to expert proficiency performance benchmarks. The Virtual Operative Assistant successfully classified skilled and novice participants using 4 metrics with an accuracy, specificity and sensitivity of 92, 82 and 100%, respectively. A 2-step feedback system was developed to provide participants with an immediate visual representation of their standing related to expert proficiency performance benchmarks. The educational system outlined establishes a basis for the potential role of integrating artificial intelligence and virtual reality simulation into surgical educational teaching. The potential of linking expertise classification, objective feedback based on proficiency benchmarks, and instructor input creates a novel educational tool by integrating these three components into a formative educational paradigm.

NRC publication: Yes

In procedural-based medicine, the technical ability can be a critical determinant of patient outcomes. Psychomotor performance occurs in real-time, hence a continuous assessment is necessary to provide action-oriented feedback and error avoidance guidance. We outline a deep learning application, the Intelligent Continuous Expertise Monitoring System (ICEMS), to assess surgical bimanual performance at 0.2-s intervals. A long-short term memory network was built using neurosurgeon and student performance in 156 virtually simulated tumor resection tasks. Algorithm predictive ability was tested separately on 144 procedures by scoring the performance of neurosurgical trainees who are at different training stages. The ICEMS successfully differentiated between neurosurgeons, senior trainees, junior trainees, and students. Trainee average performance score correlated with the year of training in neurosurgery. Furthermore, coaching and risk assessment for critical metrics were demonstrated. This work presents a comprehensive technical skill monitoring system with predictive validation throughout surgical residency training, with the ability to detect errors.

Modifiable technical errors are associated with adverse effects during surgery, poor outcomes, reoperation, and readmission. 1 Despite the importance of surgical technical expertise, there are currently no automated systems implemented in standard clinical practice to assess and monitor technical skills intraoperatively or post-procedure. 2 In most cases, there is no evaluation of surgeons’ performance unless done through visual evaluation by peer experts. CV applications enable previously unfeasible approaches to intraoperative surgical assessment, such as instrument presence detection, instrument tracking, surgical gesture, step, and phase detection, and intraoperative event prediction. 7 At our institution, the Children's National Hospital, Washington, D.C., we deploy AI to capture surgical performance data in real-time and store it on the cloud. A key advantage of this application is its convenient solution to automate both visual data collection and computer vision-based performance analytics.

Virtual reality surgical simulators are a safe and efficient technology for the assessment and training of surgical skills. Simulators allow trainees to improve specific surgical techniques in risk-free environments. Recently, machine learning has been coupled to simulators to classify performance. However, most studies fail to extract meaningful observations behind the classifications and the impact of specific surgical metrics on the performance. One benefit from integrating machine learning algorithms, such as Artificial Neural Networks, to simulators is the ability to extract novel insights into the composites of the surgical performance that differentiate levels of expertise. This study aims to demonstrate the benefits of artificial neural network algorithms in assessing and analyzing virtual surgical performances. This study applies the algorithm on a virtual reality simulated annulus incision task during an anterior cervical discectomy and fusion scenario. An artificial neural network algorithm was developed and integrated. Participants performed the simulated surgical procedure on the Sim-Ortho simulator. Data extracted from the annulus incision task were extracted to generate 157 surgical performance metrics that spanned three categories (motion, safety, and efficiency). Musculoskeletal Biomechanics Research Lab; Neurosurgical Simulation and Artificial Intelligence Learning Center, McGill University, Montreal, Canada. Twenty-three participants were recruited and divided into 3 groups: 11 post-residents, 5 senior and 7 junior residents. An artificial neural network model was trained on nine selected surgical metrics, spanning all three categories and achieved 80% testing accuracy. This study outlines the benefits of integrating artificial neural networks to virtual reality surgical simulators in understanding composites of expertise performance. •A neural network was used with a surgical simulator to understand aspects of expertise.•The ANN was trained on 9 selected surgical metrics and achieved 80% testing accuracy.•A novel implementation of the Connection Weights Algorithm on a multilayered ANN.•Results of the new implementation was compared to the permutation feature importance.

Despite advances in the assessment of technical skills in surgery, a clear understanding of the composites of technical expertise is lacking. Surgical simulation allows for the quantitation of psychomotor skills, generating data sets that can be analyzed using machine learning algorithms. To identify surgical and operative factors selected by a machine learning algorithm to accurately classify participants by level of expertise in a virtual reality surgical procedure. Fifty participants from a single university were recruited between March 1, 2015, and May 31, 2016, to participate in a case series study at McGill University Neurosurgical Simulation and Artificial Intelligence Learning Centre. Data were collected at a single time point and no follow-up data were collected. Individuals were classified a priori as expert (neurosurgery staff), seniors (neurosurgical fellows and senior residents), juniors (neurosurgical junior residents), and medical students, all of whom participated in 250 simulated tumor resections. All individuals participated in a virtual reality neurosurgical tumor resection scenario. Each scenario was repeated 5 times. Through an iterative process, performance metrics associated with instrument movement and force, resection of tissues, and bleeding generated from the raw simulator data output were selected by K-nearest neighbor, naive Bayes, discriminant analysis, and support vector machine algorithms to most accurately determine group membership. A total of 50 individuals (9 women and 41 men; mean [SD] age, 33.6 [9.5] years; 14 neurosurgeons, 4 fellows, 10 senior residents, 10 junior residents, and 12 medical students) participated. Neurosurgeons were in practice between 1 and 25 years, with 9 (64%) involving a predominantly cranial practice. The K-nearest neighbor algorithm had an accuracy of 90% (45 of 50), the naive Bayes algorithm had an accuracy of 84% (42 of 50), the discriminant analysis algorithm had an accuracy of 78% (39 of 50), and the support vector machine algorithm had an accuracy of 76% (38 of 50). The K-nearest neighbor algorithm used 6 performance metrics to classify participants, the naive Bayes algorithm used 9 performance metrics, the discriminant analysis algorithm used 8 performance metrics, and the support vector machine algorithm used 8 performance metrics. Two neurosurgeons, 1 fellow or senior resident, 1 junior resident, and 1 medical student were misclassified. In a virtual reality neurosurgical tumor resection study, a machine learning algorithm successfully classified participants into 4 levels of expertise with 90% accuracy. These findings suggest that algorithms may be capable of classifying surgical expertise with greater granularity and precision than has been previously demonstrated in surgery.

To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches. To determine how VOA and remote expert instruction compare in learners' skill acquisition, affective, and cognitive outcomes during surgical simulation training. This instructor-blinded randomized clinical trial included medical students (undergraduate years 0-2) from 4 institutions in Canada during a single simulation training at McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal, Canada. Cross-sectional data were collected from January to April 2021. Analysis was conducted based on intention-to-treat. Data were analyzed from April to June 2021. The interventions included 5 feedback sessions, 5 minutes each, during a single 75-minute training, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection. The 3 intervention arms included 2 treatment groups, AI audiovisual metric-based feedback (VOA group) and synchronous verbal scripted debriefing and instruction from a remote expert (instructor group), and a control group that received no feedback. The coprimary outcomes were change in procedural performance, quantified as Expertise Score by a validated assessment algorithm (Intelligent Continuous Expertise Monitoring System [ICEMS]; range, -1.00 to 1.00) for each practice resection, and learning and retention, measured from performance in realistic resections by ICEMS and blinded Objective Structured Assessment of Technical Skills (OSATS; range 1-7). Secondary outcomes included strength of emotions before, during, and after the intervention and cognitive load after intervention, measured in self-reports. A total of 70 medical students (41 [59%] women and 29 [41%] men; mean [SD] age, 21.8 [2.3] years) from 4 institutions were randomized, including 23 students in the VOA group, 24 students in the instructor group, and 23 students in the control group. All participants were included in the final analysis. ICEMS assessed 350 practice resections, and ICEMS and OSATS evaluated 70 realistic resections. VOA significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group (P < .001). Realistic Expertise Scores were significantly higher for the VOA group compared with instructor (mean difference, 0.53 [95% CI, 0.40 to 0.67] points; P < .001) and control (mean difference. 0.49 [95% CI, 0.34 to 0.61] points; P < .001) groups. Mean global OSATS ratings were not statistically significant among the VOA (4.63 [95% CI, 4.06 to 5.20] points), instructor (4.40 [95% CI, 3.88-4.91] points), and control (3.86 [95% CI, 3.44 to 4.27] points) groups. However, on the OSATS subscores, VOA significantly enhanced the mean OSATS overall subscore compared with the control group (mean difference, 1.04 [95% CI, 0.13 to 1.96] points; P = .02), whereas expert instruction significantly improved OSATS subscores for instrument handling vs control (mean difference, 1.18 [95% CI, 0.22 to 2.14]; P = .01). No significant differences in cognitive load, positive activating, and negative emotions were found. In this randomized clinical trial, VOA feedback demonstrated superior performance outcome and skill transfer, with equivalent OSATS ratings and cognitive and emotional responses compared with remote expert instruction, indicating advantages for its use in simulation training. ClinicalTrials.gov Identifier: NCT04700384.

Abstract BACKGROUND Virtual reality surgical simulators provide a safe environment for trainees to practice specific surgical scenarios and allow for self-guided learning. Artificial intelligence technology, including artificial neural networks, offers the potential to manipulate large datasets from simulators to gain insight into the importance of specific performance metrics during simulated operative tasks. OBJECTIVE To distinguish performance in a virtual reality-simulated anterior cervical discectomy scenario, uncover novel performance metrics, and gain insight into the relative importance of each metric using artificial neural networks. METHODS Twenty-one participants performed a simulated anterior cervical discectomy on the novel virtual reality Sim-Ortho simulator. Participants were divided into 3 groups, including 9 post-resident, 5 senior, and 7 junior participants. This study focused on the discectomy portion of the task. Data were recorded and manipulated to calculate metrics of performance for each participant. Neural networks were trained and tested and the relative importance of each metric was calculated. RESULTS A total of 369 metrics spanning 4 categories (safety, efficiency, motion, and cognition) were generated. An artificial neural network was trained on 16 selected metrics and tested, achieving a training accuracy of 100% and a testing accuracy of 83.3%. Network analysis identified safety metrics, including the number of contacts on spinal dura, as highly important. CONCLUSION Artificial neural networks classified 3 groups of participants based on expertise allowing insight into the relative importance of specific metrics of performance. This novel methodology aids in the understanding of which components of surgical performance predominantly contribute to expertise.

Virtual reality surgical simulators have facilitated surgical education by providing a safe training environment. Electroencephalography (EEG) has been employed to assess neuroelectric activity during surgical performance. Machine learning (ML) has been applied to analyze EEG data split into frequency bands. Although EEG is widely used in fields requiring expert performance, it has yet been used to classify surgical expertise. Thus, the goals of this study were to (a) develop an ML model to accurately differentiate skilled and less-skilled performance using EEG data recorded during a simulated surgery, (b) explore the relative importance of each EEG bandwidth to expertise, and (c) analyze differences in EEG band powers between skilled and less-skilled individuals. We hypothesized that EEG recordings during a virtual reality surgery task would accurately predict the expertise level of the participant. Twenty-one participants performed three simulated brain tumor resection procedures on the NeuroVR™ platform (CAE Healthcare, Montreal, Canada) while EEG data was recorded. Participants were divided into 2 groups. The skilled group was composed of five neurosurgeons and five senior neurosurgical residents (PGY4-6), and the less-skilled group was composed of six junior residents (PGY1-3) and five medical students. A total of 13 metrics from EEG frequency bands and ratios (e.g., alpha, theta/beta ratio) were generated. Seven ML model types were trained using EEG activity to differentiate between skilled and less-skilled groups. The artificial neural network achieved the highest testing accuracy of 100% (AUROC = 1.0). Model interpretation via Shapley analysis identified low alpha (8–10 Hz) as the most important metric for classifying expertise. Skilled surgeons displayed higher (p = 0.044) low-alpha than the less-skilled group. Furthermore, skilled surgeons displayed significantly lower TBR (p = 0.048) and significantly higher beta (13–30 Hz, p = 0.049), beta 1 (15–18 Hz, p = 0.014), and beta 2 (19–22 Hz, p = 0.015), thus establishing these metrics as important markers of expertise. Practice-Based Learning and Improvement. •The Artificial Neural Network (ANN) model is the best surgical expertise classifier.•Low alpha (8–10 Hz) is the most predictive EEG metric of surgical expertise.•It is possible to predict surgical expertise with 100% accuracy using only EEG data.•Skilled surgeons operate with higher overall Beta, Beta 1, Beta 2, and Low alpha.•Skilled surgeons operate with a lower theta/beta ratio.

Background: In procedural-based medicine, technical ability is a critical determinant of patient outcomes. Psychomotor performance occurs in real-time, hence a continuous assessment is necessary to provide action-oriented feedback and error avoidance guidance. Current continuous technical skill assessment systems are lacking expert performance-based assessment and accurate detection of trainee skills. This study outlines the development of a deep learning application for continuous monitoring of surgical technical skills and predictive validation on surgical trainee performance who are at different training stages. Methods: Fifty participants from 4 expertise levels (14 experts/neurosurgeons, 14 senior residents, 10 junior residents, 12 novices/medical students) performed a simulated tumour resection 5 times and a complex simulated brain tumour operation once on the NeuroVR simulation platform. A deep neural network was built using neurosurgeon and medical student data, learning the composites of expertise comparing expert and novice skill levels. Results: The trained algorithm continually tracked the surgical performance utilizing 16 performance metrics generated and provided an expertise score every 0.2 seconds. The average performance score was statistically compared among 4 expertise levels and differentiated trainee performance from expert and novice level performance. It also provided a differentiation between trainee levels, with the performance score correlated with the year of training in surgery. Conclusion: This work, to our knowledge, is the first technical skills continuous assessment application built using expert surgeon data, with predictive validity on surgical trainee performance. AI-powered surgical simulators offer a generalizable and objective continuous assessment of surgical bimanual skills in the aid of competency-based approach in surgery.