Conceptual framework for the design, implementation and validation of psychomotor skills’ assessment systems in minimally invasive surgery

Las técnicas de cirugía de mínima invasión (CMI) se están consolidando hoy en día como alternativa a la cirugía tradicional, debido a sus numerosos beneficios para los pacientes. Este cambio de paradigma implica que los cirujanos deben aprender una serie de habilidades distintas de aquellas requeridas en cirugía abierta. El entrenamiento y evaluación de estas habilidades se ha convertido en una de las mayores preocupaciones en los programas de formación de cirujanos, debido en gran parte a la presión de una sociedad que exige cirujanos bien preparados y una reducción en el número de errores médicos. Por tanto, se está prestando especial atención a la definición de nuevos programas que permitan el entrenamiento y la evaluación de las habilidades psicomotoras en entornos seguros antes de que los nuevos cirujanos puedan operar sobre pacientes reales. Para tal fin, hospitales y centros de formación están gradualmente incorporando instalaciones de entrenamiento donde los residentes puedan practicar y aprender sin riesgos. Es cada vez más común que estos laboratorios dispongan de simuladores virtuales o simuladores físicos capaces de registrar los movimientos del instrumental de cada residente. Estos simuladores ofrecen una gran variedad de tareas de entrenamiento y evaluación, así como la posibilidad de obtener información objetiva de los ejercicios. Los diferentes estudios de validación llevados a cabo dan muestra de su utilidad; pese a todo, los niveles de evidencia presentados son en muchas ocasiones insuficientes. Lo que es más importante, no existe un consenso claro a la hora de definir qué métricas son más útiles para caracterizar la pericia quirúrgica. El objetivo de esta tesis doctoral es diseñar y validar un marco de trabajo conceptual para la definición y validación de entornos para la evaluación de habilidades en CMI, en base a un modelo en tres fases: pedagógica (tareas y métricas a emplear), tecnológica (tecnologías de adquisición de métricas) y analítica (interpretación de la competencia en base a las métricas). Para tal fin, se describe la implementación práctica de un entorno basado en (1) un sistema de seguimiento de instrumental fundamentado en el análisis del vídeo laparoscópico; y (2) la determinación de la pericia en base a métricas de movimiento del instrumental. Para la fase pedagógica se diseñó e implementó un conjunto de tareas para la evaluación de habilidades psicomotoras básicas, así como una serie de métricas de movimiento. La validación de construcción llevada a cabo sobre ellas mostró buenos resultados para tiempo, camino recorrido, profundidad, velocidad media, aceleración media, economía de área y economía de volumen. Adicionalmente, los resultados obtenidos en la validación de apariencia fueron en general positivos en todos los grupos considerados (noveles, residentes, expertos). Para la fase tecnológica, se introdujo el EVA Tracking System, una solución para el seguimiento del instrumental quirúrgico basado en el análisis del vídeo endoscópico. La precisión del sistema se evaluó a 16,33ppRMS para el seguimiento 2D de la herramienta en la imagen; y a 13mmRMS para el seguimiento espacial de la misma. La validación de construcción con una de las tareas de evaluación mostró buenos resultados para tiempo, camino recorrido, profundidad, velocidad media, aceleración media, economía de área y economía de volumen. La validación concurrente con el TrEndo® Tracking System por su parte presentó valores altos de correlación para 8 de las 9 métricas analizadas. Finalmente, para la fase analítica se comparó el comportamiento de tres clasificadores supervisados a la hora de determinar automáticamente la pericia quirúrgica en base a la información de movimiento del instrumental, basados en aproximaciones lineales (análisis lineal discriminante, LDA), no lineales (máquinas de soporte vectorial, SVM) y difusas (sistemas adaptativos de inferencia neurodifusa, ANFIS). Los resultados muestran que en media SVM presenta un comportamiento ligeramente superior: 78,2% frente a los 71% y 71,7% obtenidos por ANFIS y LDA respectivamente. Sin embargo las diferencias estadísticas medidas entre los tres no fueron demostradas significativas. En general, esta tesis doctoral corrobora las hipótesis de investigación postuladas relativas a la definición de sistemas de evaluación de habilidades para cirugía de mínima invasión, a la utilidad del análisis de vídeo como fuente de información y a la importancia de la información de movimiento de instrumental a la hora de caracterizar la pericia quirúrgica. Basándose en estos cimientos, se han de abrir nuevos campos de investigación que contribuyan a la definición de programas de formación estructurados y objetivos, que puedan garantizar la acreditación de cirujanos sobradamente preparados y promocionen la seguridad del paciente en el quirófano. Abstract Minimally invasive surgery (MIS) techniques have become a standard in many surgical sub-specialties, due to their many benefits for patients. However, this shift in paradigm implies that surgeons must acquire a complete different set of skills than those normally attributed to open surgery. Training and assessment of these skills has become a major concern in surgical learning programmes, especially considering the social demand for better-prepared professionals and for the decrease of medical errors. Therefore, much effort is being put in the definition of structured MIS learning programmes, where practice with real patients in the operating room (OR) can be delayed until the resident can attest for a minimum level of psychomotor competence. To this end, skills’ laboratory settings are being introduced in hospitals and training centres where residents may practice and be assessed on their psychomotor skills. Technological advances in the field of tracking technologies and virtual reality (VR) have enabled the creation of new learning systems such as VR simulators or enhanced box trainers. These systems offer a wide range of tasks, as well as the capability of registering objective data on the trainees’ performance. Validation studies give proof of their usefulness; however, levels of evidence reported are in many cases low. More importantly, there is still no clear consensus on topics such as the optimal metrics that must be used to assess competence, the validity of VR simulation, the portability of tracking technologies into real surgeries (for advanced assessment) or the degree to which the skills measured and obtained in laboratory environments transfer to the OR. The purpose of this PhD is to design and validate a conceptual framework for the definition and validation of MIS assessment environments based on a three-pillared model defining three main stages: pedagogical (tasks and metrics to employ), technological (metric acquisition technologies) and analytical (interpretation of competence based on metrics). To this end, a practical implementation of the framework is presented, focused on (1) a video-based tracking system and (2) the determination of surgical competence based on the laparoscopic instruments’ motionrelated data. The pedagogical stage’s results led to the design and implementation of a set of basic tasks for MIS psychomotor skills’ assessment, as well as the definition of motion analysis parameters (MAPs) to measure performance on said tasks. Validation yielded good construct results for parameters such as time, path length, depth, average speed, average acceleration, economy of area and economy of volume. Additionally, face validation results showed positive acceptance on behalf of the experts, residents and novices. For the technological stage the EVA Tracking System is introduced. EVA provides a solution for tracking laparoscopic instruments from the analysis of the monoscopic video image. Accuracy tests for the system are presented, which yielded an average RMSE of 16.33pp for 2D tracking of the instrument on the image and of 13mm for 3D spatial tracking. A validation experiment was conducted using one of the tasks and the most relevant MAPs. Construct validation showed significant differences for time, path length, depth, average speed, average acceleration, economy of area and economy of volume; especially between novices and residents/experts. More importantly, concurrent validation with the TrEndo® Tracking System presented high correlation values (>0.7) for 8 of the 9 MAPs proposed. Finally, the analytical stage allowed comparing the performance of three different supervised classification strategies in the determination of surgical competence based on motion-related information. The three classifiers were based on linear (linear discriminant analysis, LDA), non-linear (support vector machines, SVM) and fuzzy (adaptive neuro fuzzy inference systems, ANFIS) approaches. Results for SVM show slightly better performance than the other two classifiers: on average, accuracy for LDA, SVM and ANFIS was of 71.7%, 78.2% and 71% respectively. However, when confronted, no statistical significance was found between any of the three. Overall, this PhD corroborates the investigated research hypotheses regarding the definition of MIS assessment systems, the use of endoscopic video analysis as the main source of information and the relevance of motion analysis in the determination of surgical competence. New research fields in the training and assessment of MIS surgeons can be proposed based on these foundations, in order to contribute to the definition of structured and objective learning programmes that guarantee the accreditation of well-prepared professionals and the promotion of patient safety in the OR.

EVA: Laparoscopic instrument tracking based on endoscopic video analysis for psychomotor skills assessment

INTRODUCTION: The EVA (Endoscopic Video Analysis) tracking system a new tracking system for extracting motions of laparoscopic instruments based on non-obtrusive video tracking was developed. The feasibility of using EVA in laparoscopic settings has been tested in a box trainer setup. METHODS: EVA makes use of an algorithm that employs information of the laparoscopic instrument's shaft edges in the image, the instrument's insertion point, and the camera's optical centre to track the 3D position of the instrument tip. A validation study of EVA comprised a comparison of the measurements achieved with EVA and the TrEndo tracking system. To this end, 42 participants (16 novices, 22 residents, and 4 experts) were asked to perform a peg transfer task in a box trainer. Ten motion-based metrics were used to assess their performance. RESULTS: Construct validation of the EVA has been obtained for seven motion-based metrics. Concurrent validation revealed that there is a strong correlation between the results obtained by EVA and the TrEndo for metrics such as path length (p=0,97), average speed (p=0,94) or economy of volume (p=0,85), proving the viability of EVA. CONCLUSIONS: EVA has been successfully used in the training setup showing potential of endoscopic video analysis to assess laparoscopic psychomotor skills. The results encourage further implementation of video tracking in training setups and in image guided surgery.

Relevance of motion-related assessment metrics in laparoscopic surgery

INTRODUCTION: Motion metrics have become an important source of information when addressing the assessment of surgical expertise. However, their direct relationship with the different surgical skills has not been fully explored. The purpose of this study is to investigate the relevance of motion-related metrics in the evaluation processes of basic psychomotor laparoscopic skills, as well as their correlation with the different abilities sought to measure. METHODS: A framework for task definition and metric analysis is proposed. An explorative survey was first conducted with a board of experts to identify metrics to assess basic psychomotor skills. Based on the output of that survey, three novel tasks for surgical assessment were designed. Face and construct validation study was performed, with focus on motion-related metrics. Tasks were performed by 42 participants (16 novices, 22 residents and 4 experts). Movements of the laparoscopic instruments were registered with the TrEndo tracking system and analyzed. RESULTS: Time, path length and depth showed construct validity for all three tasks. Motion smoothness and idle time also showed validity for tasks involving bi-manual coordination and tasks requiring a more tactical approach respectively. Additionally, motion smoothness and average speed showed a high internal consistency, proving them to be the most task-independent of all the metrics analyzed. CONCLUSION: Motion metrics are complementary and valid for assessing basic psychomotor skills, and their relevance depends on the skill being evaluated. A larger clinical implementation, combined with quality performance information, will give more insight on the relevance of the results shown in this study.

The Current Landscape of Pitfalls in Ontologies

A growing number of ontologies are already available thanks to development initiatives in many different fields. In such ontology developments, developers must tackle a wide range of difficulties and handicaps, which can result in the appearance of anomalies in the resulting ontologies. Therefore, ontology evaluation plays a key role in ontology development projects. OOPS! is an on-line tool that automatically detects pitfalls, considered as potential errors or problems, and thus may help ontology developers to improve their ontologies. To gain insight in the existence of pitfalls and to assess whether there are differences among ontologies developed by novices, a random set of already scanned ontologies, and existing well-known ones, data of 406 OWL ontologies were analysed on OOPS!’s 21 pitfalls, of which 24 ontologies were also examined manually on the detected pitfalls. The various analyses performed show only minor differences between the three sets of ontologies, therewith providing a general landscape of pitfalls in ontologies.

Computer analysis and design of concrete shell roofs

This paper is a preliminary version of Chapter 3 of a State-of-the-Art Report by the IASS Working Group 5: Concrete Shell Roofs. The intention of this chapter is to set forth for those who intend to design concrete shell roofs information and advice about the selection, verification and utilization of commercial computer tools for analysis and design tasks.The computer analysis and design steps for a concrete shell roof are described. Advice follows on the aspects to be considered in the application of commercial finite element (FE)computer programs to concrete shell analysis, starting with recommendations on how novices can gain confidence and competence in the use of software. To establish vocabulary and provide background references, brief surveys are presented of, first,element types and formulations for shells and, second, challenges presented by advanced analyses of shells. The final section of the chapter indicates what capabilities to seek in selecting commercial FE software for the analysis and design of concrete shell roofs. Brief concluding remarks summarize advice regarding judicious use of computer analysis in design practice.

Estudio empírico de la influencia de la experiencia y del conocimiento del dominio del analista en la efectividad de la educción de requisitos

Contexto: La presente tesis doctoral se enmarca en la actividad de educción de los requisitos. La educción de requisitos es generalmente aceptada como una de las actividades más importantes dentro del proceso de Ingeniería de Requisitos, y tiene un impacto directo en la calidad del software. Es una actividad donde la comunicación entre los involucrados (analistas, clientes, usuarios) es primordial. La efectividad y eficacia del analista en la compresión de las necesidades de clientes y usuarios es un factor crítico para el éxito del desarrollo de software. La literatura se ha centrado principalmente en estudiar y comprender un conjunto específico de capacidades o habilidades personales que debe poseer el analista para realizar de forma efectiva la actividad de educción. Sin embargo, existen muy pocos trabajos que han estudiado dichas capacidades o habilidades empíricamente. Objetivo: La presente investigación tiene por objetivo estudiar el efecto de la experiencia, el conocimiento acerca del dominio y la titulación académica que poseen los analistas en la efectividad del proceso de educción de los requisitos, durante los primeros contactos del analista con el cliente. Método de Investigación: Hemos ejecutado 8 estudios empíricos entre cuasi-experimentos (4) y experimentos controlados (4). Un total de 110 sujetos experimentales han participado en los estudios, entre estudiantes de post-grado de la Escuela Técnica Superior de Ingenieros Informáticos de la Universidad Politécnica de Madrid y profesionales. La tarea experimental consistió en realizar sesiones de educción de requisitos sobre uno o más dominios de problemas (de carácter conocido y desconocido para los sujetos). Las sesiones de educción se realizaron empleando la entrevista abierta. Finalizada la entrevista, los sujetos reportaron por escrito toda la información adquirida. Resultados: Para dominios desconocidos, la experiencia (entrevistas, requisitos, desarrollo y profesional) del analista no influye en su efectividad. En dominios conocidos, la experiencia en entrevistas (r = 0.34, p-valor = 0.080) y la experiencia en requisitos (r = 0.22, p-valor = 0.279), ejercen un efecto positivo. Esto es, los analistas con más años de experiencia en entrevistas y/o requisitos tienden a alcanzar mejores efectividades. Por el contrario, la experiencia en desarrollo (r = -0.06, p-valor = 0.765) y la experiencia profesional (r = -0.35, p-valor = 0.077), tienden a ejercer un efecto nulo y negativo, respectivamente. En lo que respecta al conocimiento acerca del dominio del problema que poseen los analistas, ejerce un moderado efecto positivo (r=0.31), estadísticamente significativo (p-valor = 0.029) en la efectividad de la actividad de educción. Esto es, los analistas con conocimiento tienden a ser más efectivos en los dominios de problema conocidos. En lo que respecta a la titulación académica, por falta de diversidad en las titulaciones académicas de los sujetos experimentales no es posible alcanzar una conclusión. Hemos podido explorar el efecto de la titulación académica en sólo dos cuasi-experimentos, sin embargo, nuestros resultados arrojan efectos contradictorios (r = 0.694, p-valor = 0.51 y r = -0.266, p-valor = 0.383). Además de las variables estudiadas indicadas anteriormente, hemos confirmado la existencia de variables moderadoras que afectan a la actividad de educción, tales como el entrevistado o la formación. Nuestros datos experimentales confirman que el entrevistado es un factor clave en la actividad de educción. Estadísticamente ejerce una influencia significativa en la efectividad de los analistas (p-valor= 0.000). La diferencia entre entrevistar a uno u otro entrevistado, en unidades naturales, varía entre un 18% - 23% en efectividad. Por otro lado, la formación en requisitos aumenta considerablemente la efectividad de los analistas. Los sujetos que realizaron la educción de requisitos después de recibir una formación específica en requisitos tienden a ser entre un 12% y 20% más efectivos que aquellos que no la recibieron. El efecto es significativo (p-valor = 0.000). Finalmente, hemos observado tres hechos que podrían influir en los resultados de esta investigación. En primer lugar, la efectividad de los analistas es diferencial dependiendo del tipo de elemento del dominio. En dominios conocidos, los analistas con experiencia tienden a adquirir más conceptos que los analistas noveles. En los dominios desconocidos, son los procesos los que se adquieren de forma prominente. En segundo lugar, los analistas llegan a una especie de “techo de cristal” que no les permite adquirir más información. Es decir, el analista sólo reconoce (parte de) los elementos del dominio del problema mencionado. Este hecho se observa tanto en el dominio de problema desconocido como en el conocido, y parece estar relacionado con el modo en que los analistas exploran el dominio del problema. En tercer lugar, aunque los años de experiencia no parecen predecir cuán efectivo será un analista, sí parecen asegurar que un analista con cierta experiencia, en general, tendrá una efectividad mínima que será superior a la efectividad mínima de los analistas con menos experiencia. Conclusiones: Los resultados obtenidos muestran que en dominios desconocidos, la experiencia por sí misma no determina la efectividad de los analistas de requisitos. En dominios conocidos, la efectividad de los analistas se ve influenciada por su experiencia en entrevistas y requisitos, aunque sólo parcialmente. Otras variables influyen en la efectividad de los analistas, como podrían ser las habilidades débiles. El conocimiento del dominio del problema por parte del analista ejerce un efecto positivo en la efectividad de los analistas, e interacciona positivamente con la experiencia incrementando aún más la efectividad de los analistas. Si bien no fue posible obtener conclusiones sólidas respecto al efecto de la titulación académica, si parece claro que la formación específica en requisitos ejerce una importante influencia positiva en la efectividad de los analistas. Finalmente, el analista no es el único factor relevante en la actividad de educción. Los clientes/usuarios (entrevistados) también juegan un rol importante en el proceso de generación de información. ABSTRACT Context: This PhD dissertation addresses requirements elicitation activity. Requirements elicitation is generally acknowledged as one of the most important activities of the requirements process, having a direct impact in the software quality. It is an activity where the communication among stakeholders (analysts, customers, users) is paramount. The analyst’s ability to effectively understand customers/users’ needs represents a critical factor for the success of software development. The literature has focused on studying and comprehending a specific set of personal skills that the analyst must have to perform requirements elicitation effectively. However, few studies have explored those skills from an empirical viewpoint. Goal: This research aims to study the effects of experience, domain knowledge and academic qualifications on the analysts’ effectiveness when performing requirements elicitation, during the first stages of analyst-customer interaction. Research method: We have conducted eight empirical studies, quasi-experiments (four) and controlled experiments (four). 110 experimental subjects participated, including: graduate students with the Escuela Técnica Superior de Ingenieros Informáticos of the Universidad Politécnica de Madrid, as well as researchers and professionals. The experimental tasks consisted in elicitation sessions about one or several problem domains (ignorant and/or aware for the subjects). Elicitation sessions were conducted using unstructured interviews. After each interview, the subjects reported in written all collected information. Results: In ignorant domains, the analyst’s experience (interviews, requirements, development and professional) does not influence her effectiveness. In aware domains, interviewing experience (r = 0.34, p-value = 0.080) and requirements experience (r = 0.22, p-value = 0.279), make a positive effect, i.e.: the analysts with more years of interviewing/requirements experience tend to achieve higher effectiveness. On the other hand, development experience (r = -0.06, p-value = 0.765) and professional experience (r = -0.35, p-value = 0.077) tend to make a null and negative effect, respectively. On what regards the analyst’s problem domain knowledge, it makes a modest positive effect (r=0.31), statistically significant (p-value = 0.029) on the effectiveness of the elicitation activity, i.e.: the analysts with tend to be more effective in problem domains they are aware of. On what regards academic qualification, due to the lack of diversity in the subjects’ academic degrees, we cannot come to a conclusion. We have been able to explore the effect of academic qualifications in two experiments; however, our results show opposed effects (r = 0.694, p-value = 0.51 y r = -0.266, p-value = 0.383). Besides the variables mentioned above, we have confirmed the existence of moderator variables influencing the elicitation activity, such as the interviewee and the training. Our data confirm that the interviewee is a key factor in the elicitation activity; it makes statistically significant effect on analysts’ effectiveness (p-value = 0.000). Interviewing one or another interviewee represents a difference in effectiveness of 18% - 23%, in natural units. On the other hand, requirements training increases to a large extent the analysts’ effectiveness. Those subjects who performed requirements elicitation after specific training tend to be 12% - 20% more effective than those who did not receive training. The effect is statistically significant (p-value = 0.000). Finally, we have observed three phenomena that could have an influence on the results of this research. First, the analysts’ effectiveness differs depending on domain element types. In aware domains, experienced analysts tend to capture more concepts than novices. In ignorant domains, processes are identified more frequently. Second, analysts get to a “glass ceiling” that prevents them to acquire more information, i.e.: analysts only identify (part of) the elements of the problem domain. This fact can be observed in both the ignorant and aware domains. Third, experience years do not look like a good predictor of how effective an analyst will be; however, they seem to guarantee that an analyst with some experience years will have a higher minimum effectiveness than the minimum effectiveness of analysts with fewer experience years. Conclusions: Our results point out that experience alone does not explain analysts’ effectiveness in ignorant domains. In aware domains, analysts’ effectiveness is influenced the experience in interviews and requirements, albeit partially. Other variables influence analysts’ effectiveness, e.g.: soft skills. The analysts’ problem domain knowledge makes a positive effect in analysts’ effectiveness; it positively interacts with the experience, increasing even further analysts’ effectiveness. Although we could not obtain solid conclusions on the effect of the academic qualifications, it is plain clear that specific requirements training makes a rather positive effect on analysts’ effectiveness. Finally, the analyst is not the only relevant factor in the elicitation activity. The customers/users (interviewees) play also an important role in the information generation process.