Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery

Training and assessment paradigms for laparoscopic surgical skills are evolving from traditional mentor–trainee tutorship towards structured, more objective and safer programs. Accreditation of surgeons requires reaching a consensus on metrics and tasks used to assess surgeons’ psychomotor skills. Ongoing development of tracking systems and software solutions has allowed for the expansion of novel training and assessment means in laparoscopy. The current challenge is to adapt and include these systems within training programs, and to exploit their possibilities for evaluation purposes. This paper describes the state of the art in research on measuring and assessing psychomotor laparoscopic skills. It gives an overview on tracking systems as well as on metrics and advanced statistical and machine learning techniques employed for evaluation purposes. The later ones have a potential to be used as an aid in deciding on the surgical competence level, which is an important aspect when accreditation of the surgeons in particular, and patient safety in general, are considered. The prospective of these methods and tools make them complementary means for surgical assessment of motor skills, especially in the early stages of training. Successful examples such as the Fundamentals of Laparoscopic Surgery should help drive a paradigm change to structured curricula based on objective parameters. These may improve the accreditation of new surgeons, as well as optimize their already overloaded training schedules.

How can video analysis help laparoscopic surgeons?

Automatic analysis of minimally invasive surgical (MIS) video has the potential to drive new solutions that alleviate existing needs for safer surgeries: reproducible training programs, objective and transparent assessment systems and navigation tools to assist surgeons and improve patient safety. As an unobtrusive, always available source of information in the operating room (OR), this research proposes the use of surgical video for extracting useful information during surgical operations. Methodology proposed includes tools' tracking algorithm and 3D reconstruction of the surgical field. The motivation for these solutions is the augmentation of the laparoscopic view in order to provide orientation aids, optimal surgical path visualization, or preoperative virtual models overlay

Conceptual framework for the design, implementation and validation of psychomotor skills’ assessment systems in minimally invasive surgery

Las técnicas de cirugía de mínima invasión (CMI) se están consolidando hoy en día como alternativa a la cirugía tradicional, debido a sus numerosos beneficios para los pacientes. Este cambio de paradigma implica que los cirujanos deben aprender una serie de habilidades distintas de aquellas requeridas en cirugía abierta. El entrenamiento y evaluación de estas habilidades se ha convertido en una de las mayores preocupaciones en los programas de formación de cirujanos, debido en gran parte a la presión de una sociedad que exige cirujanos bien preparados y una reducción en el número de errores médicos. Por tanto, se está prestando especial atención a la definición de nuevos programas que permitan el entrenamiento y la evaluación de las habilidades psicomotoras en entornos seguros antes de que los nuevos cirujanos puedan operar sobre pacientes reales. Para tal fin, hospitales y centros de formación están gradualmente incorporando instalaciones de entrenamiento donde los residentes puedan practicar y aprender sin riesgos. Es cada vez más común que estos laboratorios dispongan de simuladores virtuales o simuladores físicos capaces de registrar los movimientos del instrumental de cada residente. Estos simuladores ofrecen una gran variedad de tareas de entrenamiento y evaluación, así como la posibilidad de obtener información objetiva de los ejercicios. Los diferentes estudios de validación llevados a cabo dan muestra de su utilidad; pese a todo, los niveles de evidencia presentados son en muchas ocasiones insuficientes. Lo que es más importante, no existe un consenso claro a la hora de definir qué métricas son más útiles para caracterizar la pericia quirúrgica. El objetivo de esta tesis doctoral es diseñar y validar un marco de trabajo conceptual para la definición y validación de entornos para la evaluación de habilidades en CMI, en base a un modelo en tres fases: pedagógica (tareas y métricas a emplear), tecnológica (tecnologías de adquisición de métricas) y analítica (interpretación de la competencia en base a las métricas). Para tal fin, se describe la implementación práctica de un entorno basado en (1) un sistema de seguimiento de instrumental fundamentado en el análisis del vídeo laparoscópico; y (2) la determinación de la pericia en base a métricas de movimiento del instrumental. Para la fase pedagógica se diseñó e implementó un conjunto de tareas para la evaluación de habilidades psicomotoras básicas, así como una serie de métricas de movimiento. La validación de construcción llevada a cabo sobre ellas mostró buenos resultados para tiempo, camino recorrido, profundidad, velocidad media, aceleración media, economía de área y economía de volumen. Adicionalmente, los resultados obtenidos en la validación de apariencia fueron en general positivos en todos los grupos considerados (noveles, residentes, expertos). Para la fase tecnológica, se introdujo el EVA Tracking System, una solución para el seguimiento del instrumental quirúrgico basado en el análisis del vídeo endoscópico. La precisión del sistema se evaluó a 16,33ppRMS para el seguimiento 2D de la herramienta en la imagen; y a 13mmRMS para el seguimiento espacial de la misma. La validación de construcción con una de las tareas de evaluación mostró buenos resultados para tiempo, camino recorrido, profundidad, velocidad media, aceleración media, economía de área y economía de volumen. La validación concurrente con el TrEndo® Tracking System por su parte presentó valores altos de correlación para 8 de las 9 métricas analizadas. Finalmente, para la fase analítica se comparó el comportamiento de tres clasificadores supervisados a la hora de determinar automáticamente la pericia quirúrgica en base a la información de movimiento del instrumental, basados en aproximaciones lineales (análisis lineal discriminante, LDA), no lineales (máquinas de soporte vectorial, SVM) y difusas (sistemas adaptativos de inferencia neurodifusa, ANFIS). Los resultados muestran que en media SVM presenta un comportamiento ligeramente superior: 78,2% frente a los 71% y 71,7% obtenidos por ANFIS y LDA respectivamente. Sin embargo las diferencias estadísticas medidas entre los tres no fueron demostradas significativas. En general, esta tesis doctoral corrobora las hipótesis de investigación postuladas relativas a la definición de sistemas de evaluación de habilidades para cirugía de mínima invasión, a la utilidad del análisis de vídeo como fuente de información y a la importancia de la información de movimiento de instrumental a la hora de caracterizar la pericia quirúrgica. Basándose en estos cimientos, se han de abrir nuevos campos de investigación que contribuyan a la definición de programas de formación estructurados y objetivos, que puedan garantizar la acreditación de cirujanos sobradamente preparados y promocionen la seguridad del paciente en el quirófano. Abstract Minimally invasive surgery (MIS) techniques have become a standard in many surgical sub-specialties, due to their many benefits for patients. However, this shift in paradigm implies that surgeons must acquire a complete different set of skills than those normally attributed to open surgery. Training and assessment of these skills has become a major concern in surgical learning programmes, especially considering the social demand for better-prepared professionals and for the decrease of medical errors. Therefore, much effort is being put in the definition of structured MIS learning programmes, where practice with real patients in the operating room (OR) can be delayed until the resident can attest for a minimum level of psychomotor competence. To this end, skills’ laboratory settings are being introduced in hospitals and training centres where residents may practice and be assessed on their psychomotor skills. Technological advances in the field of tracking technologies and virtual reality (VR) have enabled the creation of new learning systems such as VR simulators or enhanced box trainers. These systems offer a wide range of tasks, as well as the capability of registering objective data on the trainees’ performance. Validation studies give proof of their usefulness; however, levels of evidence reported are in many cases low. More importantly, there is still no clear consensus on topics such as the optimal metrics that must be used to assess competence, the validity of VR simulation, the portability of tracking technologies into real surgeries (for advanced assessment) or the degree to which the skills measured and obtained in laboratory environments transfer to the OR. The purpose of this PhD is to design and validate a conceptual framework for the definition and validation of MIS assessment environments based on a three-pillared model defining three main stages: pedagogical (tasks and metrics to employ), technological (metric acquisition technologies) and analytical (interpretation of competence based on metrics). To this end, a practical implementation of the framework is presented, focused on (1) a video-based tracking system and (2) the determination of surgical competence based on the laparoscopic instruments’ motionrelated data. The pedagogical stage’s results led to the design and implementation of a set of basic tasks for MIS psychomotor skills’ assessment, as well as the definition of motion analysis parameters (MAPs) to measure performance on said tasks. Validation yielded good construct results for parameters such as time, path length, depth, average speed, average acceleration, economy of area and economy of volume. Additionally, face validation results showed positive acceptance on behalf of the experts, residents and novices. For the technological stage the EVA Tracking System is introduced. EVA provides a solution for tracking laparoscopic instruments from the analysis of the monoscopic video image. Accuracy tests for the system are presented, which yielded an average RMSE of 16.33pp for 2D tracking of the instrument on the image and of 13mm for 3D spatial tracking. A validation experiment was conducted using one of the tasks and the most relevant MAPs. Construct validation showed significant differences for time, path length, depth, average speed, average acceleration, economy of area and economy of volume; especially between novices and residents/experts. More importantly, concurrent validation with the TrEndo® Tracking System presented high correlation values (>0.7) for 8 of the 9 MAPs proposed. Finally, the analytical stage allowed comparing the performance of three different supervised classification strategies in the determination of surgical competence based on motion-related information. The three classifiers were based on linear (linear discriminant analysis, LDA), non-linear (support vector machines, SVM) and fuzzy (adaptive neuro fuzzy inference systems, ANFIS) approaches. Results for SVM show slightly better performance than the other two classifiers: on average, accuracy for LDA, SVM and ANFIS was of 71.7%, 78.2% and 71% respectively. However, when confronted, no statistical significance was found between any of the three. Overall, this PhD corroborates the investigated research hypotheses regarding the definition of MIS assessment systems, the use of endoscopic video analysis as the main source of information and the relevance of motion analysis in the determination of surgical competence. New research fields in the training and assessment of MIS surgeons can be proposed based on these foundations, in order to contribute to the definition of structured and objective learning programmes that guarantee the accreditation of well-prepared professionals and the promotion of patient safety in the OR.

Relevance of motion-related assessment metrics in laparoscopic surgery

INTRODUCTION: Motion metrics have become an important source of information when addressing the assessment of surgical expertise. However, their direct relationship with the different surgical skills has not been fully explored. The purpose of this study is to investigate the relevance of motion-related metrics in the evaluation processes of basic psychomotor laparoscopic skills, as well as their correlation with the different abilities sought to measure. METHODS: A framework for task definition and metric analysis is proposed. An explorative survey was first conducted with a board of experts to identify metrics to assess basic psychomotor skills. Based on the output of that survey, three novel tasks for surgical assessment were designed. Face and construct validation study was performed, with focus on motion-related metrics. Tasks were performed by 42 participants (16 novices, 22 residents and 4 experts). Movements of the laparoscopic instruments were registered with the TrEndo tracking system and analyzed. RESULTS: Time, path length and depth showed construct validity for all three tasks. Motion smoothness and idle time also showed validity for tasks involving bi-manual coordination and tasks requiring a more tactical approach respectively. Additionally, motion smoothness and average speed showed a high internal consistency, proving them to be the most task-independent of all the metrics analyzed. CONCLUSION: Motion metrics are complementary and valid for assessing basic psychomotor skills, and their relevance depends on the skill being evaluated. A larger clinical implementation, combined with quality performance information, will give more insight on the relevance of the results shown in this study.

Learning curves of basic laparoscopic psychomotor skills in SINERGIA VR simulator

Purpose: Surgical simulators are currently essential within any laparoscopic training program because they provide a low-stakes, reproducible and reliable environment to acquire basic skills. The purpose of this study is to determine the training learning curve based on different metrics corresponding to five tasks included in SINERGIA laparoscopic virtual reality simulator. Methods: Thirty medical students without surgical experience participated in the study. Five tasks of SINERGIA were included: Coordination, Navigation, Navigation and touch, Accurate grasping and Coordinated pulling. Each participant was trained in SINERGIA. This training consisted of eight sessions (R1–R8) of the five mentioned tasks and was carried out in two consecutive days with four sessions per day. A statistical analysis was made, and the results of R1, R4 and R8 were pair-wise compared with Wilcoxon signed-rank test. Significance is considered at P value <0.005. Results: In total, 84.38% of the metrics provided by SINERGIA and included in this study show significant differences when comparing R1 and R8. Metrics are mostly improved in the first session of training (75.00% when R1 and R4 are compared vs. 37.50% when R4 and R8 are compared). In tasks Coordination and Navigation and touch, all metrics are improved. On the other hand, Navigation just improves 60% of the analyzed metrics. Most learning curves show an improvement with better results in the fulfillment of the different tasks. Conclusions: Learning curves of metrics that assess the basic psychomotor laparoscopic skills acquired in SINERGIA virtual reality simulator show a faster learning rate during the first part of the training. Nevertheless, eight repetitions of the tasks are not enough to acquire all psychomotor skills that can be trained in SINERGIA. Therefore, and based on these results together with previous works, SINERGIA could be used as training tool with a properly designed training program.

Aircraft flight trajectory deviation in the standard instrument departures. Operators penalties.

In the recent years many problems are emerging due to the aircraft noise on the airport surrounding areas. The solution to this problem is not easy considering that the neighbourhood asks for the reduction of the number of aircraft operations and the airlines ask for a growing demand in the number of operations in the major airports. So the airport and regulatory authorities try to get a solution imposing a fine to the aircraft which its actual trajectory differs from the nominal one more than a lateral deviation. But, which is the value of this deviation?. The current situation is that many operators have to pay a lot of money for exceeding a deviation which has been established without operational criteria. This paper presents the results of a research program which is being carried out by the authors which aims to determine the "delta" deviation to be used for this purpose. In addition it is proposed a customized method per SID and per airport to be used for determining the maximum allowed lateral deviation by which if the aircraft is within it, then none fine will be imposed.

Instrument for testing the impact response of different materials. Application to apples and pears

Based on two research projects, a device for testing the response to-impact of fruits and related materials has been designed and tested during the last three years. As it is not related directly to potatoes, this contribution focuses mainly on the principles of impact and static loading and on the description of the device, and the type of results obtained up to now in different fruits.

Preliminary approach to neutron instrument selecction at ESSB-Bilbao based on experience at ISIS molecular spectroscopy group

Collaborative efforts between the Neutronics and Target Design Group at the Instituto de Fusión Nuclear and the Molecular Spectroscopy Group at the ISIS Pulsed Neutron and Muon Source date back to 2012 in the context of the ESS-Bilbao project. The rationale for these joint activities was twofold, namely: to assess the realm of applicability of the low-energy neutron source proposed by ESS-Bilbao - for details; and to explore instrument capabilities for pulsed-neutron techniques in the range 0.05-3 ms, a time range where ESS-Bilbao and ISIS could offer a significant degree of synergy and complementarity. As part of this collaboration, J.P. de Vicente has spent a three-month period within the ISIS Molecular Spectroscopy Group, to gain hands-on experience on the practical aspects of neutron-instrument design and the requisite neutron-transport simulations. To date, these activities have resulted in a joint MEng thesis as well as a number of publications and contributions to national and international conferences. Building upon these previous works, the primary aim of this report is to provide a self-contained discussion of general criteria for instrument selection at ESS-Bilbao, the first accelerator-driven, low-energy neutron source designed in Spain. To this end, Chapter 1 provides a brief overview of the current design parameters of the accelerator and target station. Neutron moderation is covered in Chapter 2, where we take a closer look at two possible target-moderator-reflector configurations and pay special attention to the spectral and temporal characteristics of the resulting neutron pulses. This discussion provides a necessary starting point to assess the operation of ESSB in short- and long-pulse modes. These considerations are further explored in Chapter 3, dealing with the primary characteristics of ESS-Bilbao as a short- or long-pulse facility in terms of accessible dynamic range and spectral resolution. Other practical aspects including background suppression and the use of fast choppers are also discussed. The guiding principles introduced in the first three chapters are put to use in Chapter 4 where we analyse in some detail the capabilities of a small-angle scattering instrument, as well as how specific scientific requirements can be mapped onto the optimal use of ESS-Bilbao for condensed-matter research. Part 2 of the report contains additional supporting documentation, including a description of the ESSB McStas component, a detailed characterisation of moderator response and neutron pulses, and estimates ofparameters associated with the design and operation of neutron choppers. In closing this brief foreword, we wish to thank both ESS-Bilbao and ISIS for their continuing encouragement and support along the way.

Correlation study between motion analysis of laparoscopic instruments and a suturing checklist

Purpose: To compare assessment capabilities of a motion analysis tool against a validated checklist during laparoscopic training.

Supervised classification of minimally invasive surgery psychomotor competence based on instruments motion analysis

Background Objective assessment of psychomotor skills has become an important challenge in the training of minimally invasive surgical (MIS) techniques. Currently, no gold standard defining surgical competence exists for classifying residents according to their surgical skills. Supervised classification has been proposed as a means for objectively establishing competence thresholds in psychomotor skills evaluation. This report presents a study comparing three classification methods for establishing their validity in a set of tasks for basic skills’ assessment. Methods Linear discriminant analysis (LDA), support vector machines (SVM), and adaptive neuro-fuzzy inference systems (ANFIS) were used. A total of 42 participants, divided into an experienced group (4 expert surgeons and 14 residents with >10 laparoscopic surgeries performed) and a nonexperienced group (16 students and 8 residents with <10 laparoscopic surgeries performed), performed three box trainer tasks validated for assessment of MIS psychomotor skills. Instrument movements were captured using the TrEndo tracking system, and nine motion analysis parameters (MAPs) were analyzed. The performance of the classifiers was measured by leave-one-out cross-validation using the scores obtained by the participants. Results The mean accuracy performances of the classifiers were 71 % (LDA), 78.2 % (SVM), and 71.7 % (ANFIS). No statistically significant differences in the performance were identified between the classifiers. Conclusions The three proposed classifiers showed good performance in the discrimination of skills, especially when information from all MAPs and tasks combined were considered. A correlation between the surgeons’ previous experience and their execution of the tasks could be ascertained from results. However, misclassifications across all the classifiers could imply the existence of other factors influencing psychomotor competence.

The cup anemometer, a fundamental meteorological instrument for the wind energy industry

The cup anemometer has been used widely by the wind energy industry since its early beginning, covering two fundamental aspects: wind mill performance control and wind energy production forecast. Furthermore, despite modern technological advances such as LIDAR and SODAR, the cup anemometer remains clearly the most used instrument by the wind energy industry. Together with the major advantages of this instrument (precision, robustness), some issues must be taken into account by scientists and researchers when using it. Overspeeding, interaction with stream wakes due to allocation on masts and wind- mills, loss of performance due to wear and tear, change of performance due to different climatic conditions, checking of the maintenance status and recalibration, etc. In the present work a review of the research campaigns carried out at the IDR/UPM Institute to analyze cup anemometer performance is included. Several aspects of this instrument are examined: the calibration process, the loss of performances due to aging and wear and tear, the effect of changes of air density, the rotor aerodynamics, and the harmonic terms contained in the anemometer output signal and their possible relation to the anemometer performances.

A novel scanning lens instrument for evaluating Fresnel lens performance: equipment development and initial results

A system dedicated to the optical transmittance characterization of Fresnel lenses has been developed at NREL, in collaboration with the UPM. The system quantifies the optical efficiency of the lens by generating a performance map. The shape of the focused spot may also be analyzed to understand change in the lens performance. The primary instrument components (lasers and CCD detector) have been characterized to confirm their capability for performing optical transmittance measurements. Measurements performed on SoG and PMMA lenses subject to a variety of indoor conditions (e.g., UV and damp heat) identified differences in the optical efficiency of the evaluated lenses, demonstrating the ability of the Scanning Lens Instrument (SLI) to distinguish between the aged lenses.

The Cup Anemometer, a Fundamental Meteorological Instrument for the Wind Energy Industry. Research at the IDR/UPM Institute

The results of several research campaigns investigating cup anemometer performance carried out since 2008 at the IDR/UPM Institute are included in the present paper. Several analysis of large series of calibrations were done by studying the effect of the rotor’s geometry, climatic conditions during calibration, and anemometers’ ageing. More specific testing campaigns were done regarding the cup anemometer rotor aerodynamics, and the anemometer signals. The effect of the rotor’s geometry on the cup anemometer transfer function has been investigated experimentally and analytically. The analysis of the anemometer’s output signal as a way of monitoring the anemometer status is revealed as a promising procedure for detecting anomalies.

Validation of an instrument for injury data collection in strength training

Purpose: To provide for the basis for collecting strength training data using a rigorously validated injury report form. Methods: A group of specialist designed a questionnaire of 45 item grouped into 4 dimensions. Six stages were used to assess face, content, and criterion validity of the weight training injury report form. A 13 members panel assessed the form for face validity, and an expert panel assessed it for content and criterion validity. Panel members were consulted until consensus was reached. A yardstick developed by an expert panel using Intraclass correlation technique was used to assess the reability of the form. Test-retest reliability was assessed with the intraclass correlation coefficient (ICC).The strength training injury report form was developed, and the face, content, and criterion validity successfully assessed. A six step protocol to create a yardstick was also developed to assist in the validation process. Both inter-rater and intra rater reliability results indicated a 98% agreement. Inter-rater reliability agreement of 98% for three injuries. Results: The Cronbach?s alpha of the questionnaire was 0.944 (pmenor que0.01) and the ICC of the entire questionnaire was 0.894 (pmenor que0.01). Conclusion: The questionnaire gathers together enough psychometric properties to be considered a valid and reliable tool for register injury data in strength training, and providing researchers with a basis for future studies in this area. Key Words: data collection; validation; injury prevention; strength training