Integration of Real-Time Simulator, Motion Platform and Haptics

This thesis describes the process of the integration of a real-time simulator environment with a motion platform and a haptic device as a part of the Kvalive project. Several programs running on two computers were made to control the different devices of the environment. User tests were made to obtain information of needed improvements to make the simulator more realistic. Also new ideas for improving the simulator and directions of further research were obtained with the help of this research.

Perception and haptics

For people with motion impairments, access to and independent control of a computer can be essential. Symptoms such as tremor and spasm, however, can make the typical keyboard and mouse arrangement for computer interaction difficult or even impossible to use. This paper describes three approaches to improving computer input effectivness for people with motion impairments. The three approaches are: (1) to increase the number of interaction channels, (2) to enhance commonly existing interaction channels, and (3) to make more effective use of all the available information in an existing input channel. Experiments in multimodal input, haptic feedback, user modelling, and cursor control are discussed in the context of the three approaches. A haptically enhanced keyboard emulator with perceptive capability is proposed, combining approaches in a way that improves computer access for motion impaired users.

Haptics in medicine and clinical skill acquisition

THE clinical skills of medical professionals rely strongly on the sense of touch, combined with anatomical and diagnostic knowledge. Haptic exploratory procedures allow the expert to detect anomalies via gross and fine palpation, squeezing, and contour following. Haptic feedback is also key to medical interventions, for example when an anaesthetist inserts an epidural needle, a surgeon makes an incision, a dental surgeon drills into a carious lesion, or a veterinarian sutures a wound. Yet, current trends in medical technology and training methods involve less haptic feedback to clinicians and trainees. For example, minimally invasive surgery removes the direct contact between the patient and clinician that gives rise to natural haptic feedback, and furthermore introduces scaling and rotational transforms that confuse the relationship between movements of the hand and the surgical site. Similarly, it is thought that computer-based medical simulation and training systems require high-resolution and realistic haptic feedback to the trainee for significant training transfer to occur. The science and technology of haptics thus has great potential to affect the performance of medical procedures and learning of clinical skills. This special section is about understanding

Haptics in forensics: the possibilities and advantages in using the haptic device for reconstruction approaches in forensic science

Non-invasive documentation methods such as surface scanning and radiological imaging are gaining in importance in the forensic field. These three-dimensional technologies provide digital 3D data, which are processed and handled in the computer. However, the sense of touch gets lost using the virtual approach. The haptic device enables the use of the sense of touch to handle and feel digital 3D data. The multifunctional application of a haptic device for forensic approaches is evaluated and illustrated in three different cases: the representation of bone fractures of the lower extremities, by traffic accidents, in a non-invasive manner; the comparison of bone injuries with the presumed injury-inflicting instrument; and in a gunshot case, the identification of the gun by the muzzle imprint, and the reconstruction of the holding position of the gun. The 3D models of the bones are generated from the Computed Tomography (CT) images. The 3D models of the exterior injuries, the injury-inflicting tools and the bone injuries, where a higher resolution is necessary, are created by the optical surface scan. The haptic device is used in combination with the software FreeForm Modelling Plus for touching the surface of the 3D models to feel the minute injuries and the surface of tools, to reposition displaced bone parts and to compare an injury-causing instrument with an injury. The repositioning of 3D models in a reconstruction is easier, faster and more precisely executed by means of using the sense of touch and with the user-friendly movement in the 3D space. For representation purposes, the fracture lines of bones are coloured. This work demonstrates that the haptic device is a suitable and efficient application in forensic science. The haptic device offers a new way in the handling of digital data in the virtual 3D space.

Bimanual anterior segment revision surgery for anterior capsule contraction syndrome associated with anterior flexion of intraocular lens haptics

PURPOSE To report the incidence of anterior capsule contraction syndrome (ACCS) and to present a novel minimally invasive bimanual technique for anterior segment revision surgery associated with ACCS with anterior flexion of the intraocular lens haptics. METHODS A consecutive cohort of 268 eyes of 161 patients undergoing phacoemulsification and implantation of the same type of hydrophilic acrylic aspheric intraocular lens cohort were analysed and a novel technique of minimally invasive bimanual technique for anterior segment revision surgery is described. RESULTS We identified four eyes (1.5%) of three patients with advanced ACCS. Successful restoration of a clear visual axis with minimal induction of astigmatism and rapid visual rehabilitation was achieved in all four cases. CONCLUSION This technique is a safe and minimally invasive alternative to laser or vitrector-cut capsulotomy to restore a clear visual axis. In cases of advanced ACCS, it offers the option for haptic reposition or amputation.

Volume Haptics and Dendritic Spine Editing

Esta tesis se ha desarrollado en el contexto del proyecto Cajal Blue Brain, una iniciativa europea dedicada al estudio del cerebro. Uno de los objetivos de esta iniciativa es desarrollar nuevos métodos y nuevas tecnologías que simplifiquen el análisis de datos en el campo neurocientífico. El presente trabajo se ha centrado en diseñar herramientas que combinen información proveniente de distintos canales sensoriales con el fin de acelerar la interacción y análisis de imágenes neurocientíficas. En concreto se estudiará la posibilidad de combinar información visual con información háptica. Las espinas dendríticas son pequeñas protuberancias que recubren la superficie dendrítica de muchas neuronas del cerebro. A día de hoy, se cree que tienen un papel clave en la transmisión de señales neuronales. Motivo por el cual, el interés por parte de la comunidad científica por estas estructuras ha ido en aumento a medida que las técnicas de adquisición de imágenes mejoraban hasta alcanzar una calidad suficiente para analizar dichas estructuras. A menudo, los neurocientíficos utilizan técnicas de microscopía con luz para obtener los datos que les permitan analizar estructuras neuronales tales como neuronas, dendritas y espinas dendríticas. A pesar de que estas técnicas ofrezcan ciertas ventajas frente a su equivalente electrónico, las técnicas basadas en luz permiten una menor resolución. En particular, estructuras pequeñas como las espinas dendríticas pueden capturarse de forma incorrecta en las imágenes obtenidas, impidiendo su análisis. En este trabajo, se presenta una nueva técnica, que permite editar imágenes volumétricas, mediante un dispositivo háptico, con el fin de reconstruir de los cuellos de las espinas dendríticas. Con este objetivo, en un primer momento se desarrolló un algoritmo que proporciona retroalimentación háptica en datos volumétricos, completando la información que provine del canal visual. Dicho algoritmo de renderizado háptico permite a los usuarios tocar y percibir una isosuperficie en el volumen de datos. El algoritmo asegura un renderizado robusto y eficiente. Se utiliza un método basado en las técnicas de “marching tetrahedra” para la extracción local de una isosuperficie continua, lineal y definida por intervalos. La robustez deriva tanto de una etapa de detección de colisiones continua de la isosuperficie extraída, como del uso de técnicas eficientes de renderizado basadas en un proxy puntual. El método de “marching tetrahedra” propuesto garantiza que la topología de la isosuperficie extraída coincida con la topología de una isosuperficie equivalente determinada utilizando una interpolación trilineal. Además, con el objetivo de mejorar la coherencia entre la información háptica y la información visual, el algoritmo de renderizado háptico calcula un segundo proxy en la isosuperficie pintada en la pantalla. En este trabajo se demuestra experimentalmente las mejoras en, primero, la etapa de extracción de isosuperficie, segundo, la robustez a la hora de mantener el proxy en la isosuperficie deseada y finalmente la eficiencia del algoritmo. En segundo lugar, a partir del algoritmo de renderizado háptico propuesto, se desarrolló un procedimiento, en cuatro etapas, para la reconstrucción de espinas dendríticas. Este procedimiento, se puede integrar en los cauces de segmentación automática y semiautomática existentes como una etapa de pre-proceso previa. El procedimiento está diseñando para que tanto la navegación como el proceso de edición en sí mismo estén controlados utilizando un dispositivo háptico. Se han diseñado dos experimentos para evaluar esta técnica. El primero evalúa la aportación de la retroalimentación háptica y el segundo se centra en evaluar la idoneidad del uso de un háptico como dispositivo de entrada. En ambos casos, los resultados demuestran que nuestro procedimiento mejora la precisión de la reconstrucción. En este trabajo se describen también dos casos de uso de nuestro procedimiento en el ámbito de la neurociencia: el primero aplicado a neuronas situadas en la corteza cerebral humana y el segundo aplicado a espinas dendríticas situadas a lo largo de neuronas piramidales de la corteza del cerebro de una rata. Por último, presentamos el programa, Neuro Haptic Editor, desarrollado a lo largo de esta tesis junto con los diferentes algoritmos ya mencionados. ABSTRACT This thesis took place within the Cajal Blue Brain project, a European initiative dedicated to the study of the brain. One of the main goals of this project is the development of new methods and technologies simplifying data analysis in neuroscience. This thesis focused on the development of tools combining information originating from distinct sensory channels with the aim of accelerating both the interaction with neuroscience images and their analysis. In concrete terms, the objective is to study the possibility of combining visual information with haptic information. Dendritic spines are thin protrusions that cover the dendritic surface of numerous neurons in the brain and whose function seems to play a key role in neural circuits. The interest of the neuroscience community toward those structures kept increasing as and when acquisition methods improved, eventually to the point that the produced datasets enabled their analysis. Quite often, neuroscientists use light microscopy techniques to produce the dataset that will allow them to analyse neuronal structures such as neurons, dendrites and dendritic spines. While offering some advantages compared to their electronic counterpart, light microscopy techniques achieve lower resolutions. Particularly, small structures such as dendritic spines might suffer from a very low level of fluorescence in the final dataset, preventing further analysis. This thesis introduces a new technique enabling the edition of volumetric datasets in order to recreate dendritic spine necks using a haptic device. In order to fulfil this objective, we first presented an algorithm to provide haptic feedback directly from volumetric datasets, as an aid to regular visualization. The haptic rendering algorithm lets users perceive isosurfaces in volumetric datasets, and it relies on several design features that ensure a robust and efficient rendering. A marching tetrahedra approach enables the dynamic extraction of a piecewise linear continuous isosurface. Robustness is derived using a Continuous Collision Detection step coupled with acknowledged proxy-based rendering methods over the extracted isosurface. The introduced marching tetrahedra approach guarantees that the extracted isosurface will match the topology of an equivalent isosurface computed using trilinear interpolation. The proposed haptic rendering algorithm improves the coherence between haptic and visual cues computing a second proxy on the isosurface displayed on screen. Three experiments demonstrate the improvements on the isosurface extraction stage as well as the robustness and the efficiency of the complete algorithm. We then introduce our four-steps procedure for the complete reconstruction of dendritic spines. Based on our haptic rendering algorithm, this procedure is intended to work as an image processing stage before the automatic segmentation step giving the final representation of the dendritic spines. The procedure is designed to allow both the navigation and the volume image editing to be carried out using a haptic device. We evaluated our procedure through two experiments. The first experiment concerns the benefits of the force feedback and the second checks the suitability of the use of a haptic device as input. In both cases, the results shows that the procedure improves the editing accuracy. We also report two concrete cases where our procedure was employed in the neuroscience field, the first one concerning dendritic spines in the human cortex, the second one referring to an ongoing experiment studying dendritic spines along dendrites of mouse cortical pyramidal neurons. Finally, we present the software program, Neuro Haptic Editor, that was built along the development of the different algorithms implemented during this thesis, and used by neuroscientists to use our procedure.

Simulador em Realidade Virtual para Aprendizagem da Física

A forma como aprendemos depende do contexto tecnológico e sociocultural que nos rodeia, actualmente a inclusão de tecnologia recente na sala de aula não é mais considerada opcional, mas sim uma necessidade pois a forma como o aluno aprende está em constante evolução. Tendo em atenção esta necessidade, foi desenvolvido no decorrer desta tese um simulador em realidade virtual que utiliza comandos/interfaces hápticos. O objectivo deste simulador é ensinar conceitos de física de forma interactiva. Os dispositivos hápticos permitem adicionar o sentido táctil ou de toque à interacção entre homem e máquina, permitindo assim aceder a novas sensações relativas ao seu uso nomeadamente com objectivos de aprendizagem. O simulador desenvolvido designado por “Forces of Physics” aborda três tipos de forças da física: forças de atrito, forças gravitacionais e forças aerodinâmicas. Cada tipo de força corresponde a um módulo do simulador contendo uma simulação individual em que são explicados conceitos específicos dessa força num ambiente visual estimulante e com uma interacção mais realista devido à inclusão do dispositivo háptico Novint Falcon. O simulador foi apresentado a vários utilizadores bem como á comunidade científica através de apresentações em conferências. A avaliação foi realizada com recurso a um questionário com dez perguntas, cinco de sobre aprendizagem e cinco sobre a utilização, tendo sido preenchido por 14 utilizadores. O simulador obteve uma boa recepção por parte dos utilizadores, tendo vários utilizadores expressado as suas opiniões sobre estado actual do simulador, do futuro do mesmo e da respectiva validade para uso na sala de aula.

Real-Time Simulation of Mobile and Industrial Machines Using the Multibody Simulation Approach

This thesis introduces a real-time simulation environment based on the multibody simulation approach. The environment consists of components that are used in conventional product development, including computer aided drawing, visualization, dynamic simulation and finite element software architecture, data transfer and haptics. These components are combined to perform as a coupled system on one platform. The environment is used to simulate mobile and industrial machines at different stages of a product life time. Consequently, the demands of the simulated scenarios vary. In this thesis, a real-time simulation environment based on the multibody approach is used to study a reel mechanism of a paper machine and a gantry crane. These case systems are used to demonstrate the usability of the real-time simulation environment for fault detection purposes and in the context of a training simulator. In order to describe the dynamical performance of a mobile or industrial machine, the nonlinear equations of motion must be defined. In this thesis, the dynamical behaviour of machines is modelled using the multibody simulation approach. A multibody system may consist of rigid and flexible bodies which are joined using kinematic joint constraints while force components are used to describe the actuators. The strength of multibody dynamics relies upon its ability to describe nonlinearities arising from wearing of the components, friction, large rotations or contact forces in a systematic manner. For this reason, the interfaces between subsystems such as mechanics, hydraulics and control systems of the mechatronic machine can be defined and analyzed in a straightforward manner.

Virtual technology and haptic interface solutions for design and control of mobile working machines

Commercially available haptic interfaces are usable for many purposes. However, as generic devices they are not the most suitable for the control of heavy duty mobile working machines like mining machines, container handling equipment and excavators. Alternative mechanical constructions for a haptic controller are presented and analysed. A virtual reality environment (VRE) was built to test the proposed haptic controller mechanisms. Verification of an electric motor emulating a hydraulic pump in the electro-hydraulic system of a mobile working machine is carried out. A real-time simulator using multi-body-dynamics based software with hardware-in-loop (HIL) setup was used for the tests. Recommendations for further development of a haptic controller and emulator electric motor are given.

High bandwidth, large workspace haptic interaction: flying phantoms

It is well understood that for haptic interaction: free motion performance and closed-loop constrained motion performance have conflicting requirements. The difficulties for both conditions are compounded when increased workspace is required as most solutions result in a reduction of achievable impedance and bandwidth. A method of chaining devices together to increase workspace without adverse effect on performance is described and analysed. The method is then applied to a prototype, colloquially known as 'The Flying Phantom', and shown to provide high-bandwidth, low impedance interaction over the full range of horizontal movement across the front of a human user.

Perceptual cues for orientation in a two finger haptic grasp task

Single point interaction haptic devices do not provide the natural grasp and manipulations found in the real world, as afforded by multi-fingered haptics. The present study investigates a two-fingered grasp manipulation involving rotation with and without force feedback. There were three visual cue conditions: monocular, binocular and projective lighting. Performance metrics of time and positional accuracy were assessed. The results indicate that adding haptics to an object manipulation task increases the positional accuracy but slightly increases the overall time taken.

Force shading and bump mapping using the friction cone algorithm

Previous work has presented the friction cone algorithm, a generalised method to resolve forces on a simulated haptic object when two or more fingers are in contact. Two extensions to the friction cone algorithm are presented: force shading and bump mapping. Force shading removes the discontinuities that are present when transitioning from one face to the next, whilst bump mapping provides a mechanism for rendering haptic textures on polygonal surfaces. Both these extensions can be combined whilst still maintaining the friction cone algorithm's intrinsic ability to simulate arbitrarily complex friction models.