Design and control of multi-finger haptic devices for dexterous manipulation

En la interacción con el entorno que nos rodea durante nuestra vida diaria (utilizar un cepillo de dientes, abrir puertas, utilizar el teléfono móvil, etc.) y en situaciones profesionales (intervenciones médicas, procesos de producción, etc.), típicamente realizamos manipulaciones avanzadas que incluyen la utilización de los dedos de ambas manos. De esta forma el desarrollo de métodos de interacción háptica multi-dedo dan lugar a interfaces hombre-máquina más naturales y realistas. No obstante, la mayoría de interfaces hápticas disponibles en el mercado están basadas en interacciones con un solo punto de contacto; esto puede ser suficiente para la exploración o palpación del entorno pero no permite la realización de tareas más avanzadas como agarres. En esta tesis, se investiga el diseño mecánico, control y aplicaciones de dispositivos hápticos modulares con capacidad de reflexión de fuerzas en los dedos índice, corazón y pulgar del usuario. El diseño mecánico de la interfaz diseñada, ha sido optimizado con funciones multi-objetivo para conseguir una baja inercia, un amplio espacio de trabajo, alta manipulabilidad y reflexión de fuerzas superiores a 3 N en el espacio de trabajo. El ancho de banda y la rigidez del dispositivo se han evaluado mediante simulación y experimentación real. Una de las áreas más importantes en el diseño de estos dispositivos es el efector final, ya que es la parte que está en contacto con el usuario. Durante este trabajo se ha diseñado un dedal de bajo peso, adaptable a diferentes usuarios que, mediante la incorporación de sensores de contacto, permite estimar fuerzas normales y tangenciales durante la interacción con entornos reales y virtuales. Para el diseño de la arquitectura de control, se estudiaron los principales requisitos para estos dispositivos. Entre estos, cabe destacar la adquisición, procesado e intercambio a través de internet de numerosas señales de control e instrumentación; la computación de equaciones matemáticas incluyendo la cinemática directa e inversa, jacobiana, algoritmos de detección de agarres, etc. Todos estos componentes deben calcularse en tiempo real garantizando una frecuencia mínima de 1 KHz. Además, se describen sistemas para manipulación de precisión virtual y remota; así como el diseño de un método denominado "desacoplo cinemático iterativo" para computar la cinemática inversa de robots y la comparación con otros métodos actuales. Para entender la importancia de la interacción multimodal, se ha llevado a cabo un estudio para comprobar qué estímulos sensoriales se correlacionan con tiempos de respuesta más rápidos y de mayor precisión. Estos experimentos se desarrollaron en colaboración con neurocientíficos del instituto Technion Israel Institute of Technology. Comparando los tiempos de respuesta en la interacción unimodal (auditiva, visual y háptica) con combinaciones bimodales y trimodales de los mismos, se demuestra que el movimiento sincronizado de los dedos para generar respuestas de agarre se basa principalmente en la percepción háptica. La ventaja en el tiempo de procesamiento de los estímulos hápticos, sugiere que los entornos virtuales que incluyen esta componente sensorial generan mejores contingencias motoras y mejoran la credibilidad de los eventos. Se concluye que, los sistemas que incluyen percepción háptica dotan a los usuarios de más tiempo en las etapas cognitivas para rellenar información de forma creativa y formar una experiencia más rica. Una aplicación interesante de los dispositivos hápticos es el diseño de nuevos simuladores que permitan entrenar habilidades manuales en el sector médico. En colaboración con fisioterapeutas de Griffith University en Australia, se desarrolló un simulador que permite realizar ejercicios de rehabilitación de la mano. Las propiedades de rigidez no lineales de la articulación metacarpofalange del dedo índice se estimaron mediante la utilización del efector final diseñado. Estos parámetros, se han implementado en un escenario que simula el comportamiento de la mano humana y que permite la interacción háptica a través de esta interfaz. Las aplicaciones potenciales de este simulador están relacionadas con entrenamiento y educación de estudiantes de fisioterapia. En esta tesis, se han desarrollado nuevos métodos que permiten el control simultáneo de robots y manos robóticas en la interacción con entornos reales. El espacio de trabajo alcanzable por el dispositivo háptico, se extiende mediante el cambio de modo de control automático entre posición y velocidad. Además, estos métodos permiten reconocer el gesto del usuario durante las primeras etapas de aproximación al objeto para su agarre. Mediante experimentos de manipulación avanzada de objetos con un manipulador y diferentes manos robóticas, se muestra que el tiempo en realizar una tarea se reduce y que el sistema permite la realización de la tarea con precisión. Este trabajo, es el resultado de una colaboración con investigadores de Harvard BioRobotics Laboratory. ABSTRACT When we interact with the environment in our daily life (using a toothbrush, opening doors, using cell-phones, etc.), or in professional situations (medical interventions, manufacturing processes, etc.) we typically perform dexterous manipulations that involve multiple fingers and palm for both hands. Therefore, multi-Finger haptic methods can provide a realistic and natural human-machine interface to enhance immersion when interacting with simulated or remote environments. Most commercial devices allow haptic interaction with only one contact point, which may be sufficient for some exploration or palpation tasks but are not enough to perform advanced object manipulations such as grasping. In this thesis, I investigate the mechanical design, control and applications of a modular haptic device that can provide force feedback to the index, thumb and middle fingers of the user. The designed mechanical device is optimized with a multi-objective design function to achieve a low inertia, a large workspace, manipulability, and force-feedback of up to 3 N within the workspace; the bandwidth and rigidity for the device is assessed through simulation and real experimentation. One of the most important areas when designing haptic devices is the end-effector, since it is in contact with the user. In this thesis the design and evaluation of a thimble-like, lightweight, user-adaptable, and cost-effective device that incorporates four contact force sensors is described. This design allows estimation of the forces applied by a user during manipulation of virtual and real objects. The design of a real-time, modular control architecture for multi-finger haptic interaction is described. Requirements for control of multi-finger haptic devices are explored. Moreover, a large number of signals have to be acquired, processed, sent over the network and mathematical computations such as device direct and inverse kinematics, jacobian, grasp detection algorithms, etc. have to be calculated in Real Time to assure the required high fidelity for the haptic interaction. The Hardware control architecture has different modules and consists of an FPGA for the low-level controller and a RT controller for managing all the complex calculations (jacobian, kinematics, etc.); this provides a compact and scalable solution for the required high computation capabilities assuring a correct frequency rate for the control loop of 1 kHz. A set-up for dexterous virtual and real manipulation is described. Moreover, a new algorithm named the iterative kinematic decoupling method was implemented to solve the inverse kinematics of a robotic manipulator. In order to understand the importance of multi-modal interaction including haptics, a subject study was carried out to look for sensory stimuli that correlate with fast response time and enhanced accuracy. This experiment was carried out in collaboration with neuro-scientists from Technion Israel Institute of Technology. By comparing the grasping response times in unimodal (auditory, visual, and haptic) events with the response times in events with bimodal and trimodal combinations. It is concluded that in grasping tasks the synchronized motion of the fingers to generate the grasping response relies on haptic cues. This processing-speed advantage of haptic cues suggests that multimodalhaptic virtual environments are superior in generating motor contingencies, enhancing the plausibility of events. Applications that include haptics provide users with more time at the cognitive stages to fill in missing information creatively and form a richer experience. A major application of haptic devices is the design of new simulators to train manual skills for the medical sector. In collaboration with physical therapists from Griffith University in Australia, we developed a simulator to allow hand rehabilitation manipulations. First, the non-linear stiffness properties of the metacarpophalangeal joint of the index finger were estimated by using the designed end-effector; these parameters are implemented in a scenario that simulates the behavior of the human hand and that allows haptic interaction through the designed haptic device. The potential application of this work is related to educational and medical training purposes. In this thesis, new methods to simultaneously control the position and orientation of a robotic manipulator and the grasp of a robotic hand when interacting with large real environments are studied. The reachable workspace is extended by automatically switching between rate and position control modes. Moreover, the human hand gesture is recognized by reading the relative movements of the index, thumb and middle fingers of the user during the early stages of the approximation-to-the-object phase and then mapped to the robotic hand actuators. These methods are validated to perform dexterous manipulation of objects with a robotic manipulator, and different robotic hands. This work is the result of a research collaboration with researchers from the Harvard BioRobotics Laboratory. The developed experiments show that the overall task time is reduced and that the developed methods allow for full dexterity and correct completion of dexterous manipulations.

Implementación de una API para la interacción del guante P5 con entornos de realidad virtual desarrollados en Java y Java 3D

Este artículo presenta el proceso de implementación de una API (Application Programming Interface) que permite la interacción del guante P5 de Essential Reality1 con un entorno virtual desarrollado en el lenguaje de programación Java y su librería Java 3D.2 Por otra parte, se describe un ejemplo implementado, haciendo uso de la API en cuestión. Con base en este ejemplo se presentan los resultados de la ejecución de pruebas de requerimientos de recursos físicos como la CPU y memoria física. Finalmente, se especifican las conclusiones y resultados obtenidos.

Amygdala activation in response to 2D and 3D emotion-inducing stimuli

Studying changes in brain activation according to the valence of emotion-inducing stimuli is essential in the research on emotions. Due to the ecological potential of virtual reality, it is also important to examine whether brain activation in response to emotional stimuli can be modulated by the three-dimensional (3D) properties of the images. This study uses functional Magnetic Resonance Imaging to compare differences between 3D and standard (2D) visual stimuli in the activation of emotion-related brain areas. The stimuli were organized in three virtual-reality scenarios, each with a different emotional valence (pleasant, unpleasant and neutral). The scenarios were presented in a pseudo-randomized order in the two visualization modes to twelve healthy males. Data were analyzed through a GLM-based fixed effects procedure. Unpleasant and neutral stimuli activated the right amygdala more strongly when presented in 3D than in 2D. These results suggest that 3D stimuli, when used as “building blocks” for virtual environments, can induce increased emotional loading, as shown here through neuroimaging.

Avatar modeling: a telepresence study with natural user interface

Dissertação para obtenção do Grau de Mestre em Engenharia Electrotécnica e de Computadores

Immersiveness of ubiquitous computing environments prototypes: A case study

The development of ubiquitous computing (ubicomp) environments raises several challenges in terms of their evaluation. Ubicomp virtual reality prototyping tools enable users to experience the system to be developed and are of great help to face those challenges, as they support developers in assessing the consequences of a design decision in the early phases of development. Given the situated nature of ubicomp environments, a particular issue to consider is the level of realism provided by the prototypes. This work presents a case study where two ubicomp prototypes, featuring different levels of immersion (desktop-based versus CAVE-based), were developed and compared. The goal was to determine the cost/benefits relation of both solutions, which provided better user experience results, and whether or not simpler solutions provide the same user experience results as more elaborate one.

Body Image in Eating Disorders: The Influence of Exposure to Virtual-Reality Environments.

The aim of this article was to study the effect of virtual-reality exposure to situations that are emotionally significant for patients with eating disorders (ED) on the stability of body-image distortion and body-image dissatisfaction. A total of 85 ED patients and 108 non-ED students were randomly exposed to four experimental virtual environments: a kitchen with low-calorie food, a kitchen with high-calorie food, a restaurant with low-calorie food, and a restaurant with high-calorie food. In the interval between the presentation of each situation, body-image distortion and body-image dissatisfaction were assessed. Several 2 x 2 x 2 repeated measures analyses of variance (high-calorie vs. low-calorie food x presence vs. absence of people x ED group vs. control group) showed that ED participants had significantly higher levels of body-image distortion and body dissatisfaction after eating high-calorie food than after eating low-calorie food, while control participants reported a similar body image in all situations. The results suggest that body-image distortion and body-image dissatisfaction show both trait and state features. On the one hand, ED patients show a general predisposition to overestimate their body size and to feel more dissatisfied with their body image than controls. On the other hand, these body-image disturbances fluctuate when participants are exposed to virtual situations that are emotionally relevant for them.

Generating Aesthetically Pleasing Images in a Virtual Environment using Particle Swarm Optimization

This research focuses on generating aesthetically pleasing images in virtual environments using the particle swarm optimization (PSO) algorithm. The PSO is a stochastic population based search algorithm that is inspired by the flocking behavior of birds. In this research, we implement swarms of cameras flying through a virtual world in search of an image that is aesthetically pleasing. Virtual world exploration using particle swarm optimization is considered to be a new research area and is of interest to both the scientific and artistic communities. Aesthetic rules such as rule of thirds, subject matter, colour similarity and horizon line are all analyzed together as a multi-objective problem to analyze and solve with rendered images. A new multi-objective PSO algorithm, the sum of ranks PSO, is introduced. It is empirically compared to other single-objective and multi-objective swarm algorithms. An advantage of the sum of ranks PSO is that it is useful for solving high-dimensional problems within the context of this research. Throughout many experiments, we show that our approach is capable of automatically producing images satisfying a variety of supplied aesthetic criteria.

Use of auditory cues for wayfinding assistance in virtual environment: music aids route decision

This paper addresses the crucial problem of wayfinding assistance in the Virtual Environments (VEs). A number of navigation aids such as maps, agents, trails and acoustic landmarks are available to support the user for navigation in VEs, however it is evident that most of the aids are visually dominated. This work-in-progress describes a sound based approach that intends to assist the task of 'route decision' during navigation in a VE using music. Furthermore, with use of musical sounds it aims to reduce the cognitive load associated with other visually as well as physically dominated tasks. To achieve these goals, the approach exploits the benefits provided by music to ease and enhance the task of wayfinding, whilst making the user experience in the VE smooth and enjoyable.

Identifying likely successful users of virtual reality systems

This paper suggests a method for identifying individuals who are most suited to using virtual reality (VR) systems. The aim is to help both an individual or employer to decide where that individual's skills and abilities would be best deployed. By considering a potential user's competence and temperament, a graphical representation is introduced that may then be used to crudely delineate a high-aptitude participant against those with lesser capabilities. By introducing standard tests for competence and a standard classifier for temperament, and by further weighting each measure with respect to the technology currently available and the application, a detailed representation of the effectiveness of different users is developed.

The causal surface and its effect on distribution transparency in a distributed virtual environment

One goal in the development of distributed virtual environments (DVEs) is to create a system such that users are unaware of the distribution-the distribution should be transparent. The paper begins by discussing the general issues in DVEs that might make this possible, and a system that allows some level of distribution transparency is described. The system described suffers from effects of inconsistency, which in turn cause undesirable visual effects. The causal surface is introduced as a solution that removes these visual effects. The paper then introduces two determining factors of distribution transparency relating to user perception and performance. With regard to these factors, two hypotheses are stated relating to the causal surface. A user-trial on forty-five subjects is used to validate the hypotheses. A discussion of the results of the trial concludes that the causal surface solution does significantly improve the distribution transparency in a DVE.

Maximising concurrency and scalability in a consistent, causal, distributed virtual reality system, whilst minimising the effect of network delays

The development of large scale virtual reality and simulation systems have been mostly driven by the DIS and HLA standards community. A number of issues are coming to light about the applicability of these standards, in their present state, to the support of general multi-user VR systems. This paper pinpoints four issues that must be readdressed before large scale virtual reality systems become accessible to a larger commercial and public domain: a reduction in the effects of network delays; scalable causal event delivery; update control; and scalable reliable communication. Each of these issues is tackled through a common theme of combining wall clock and causal time-related entity behaviour, knowledge of network delays and prediction of entity behaviour, that together overcome many of the effects of network delay.

Maximising concurrency and scalability in a consistent, causal, distributed virtual reality system, whilst minimising the effect of network delays

The development of large scale virtual reality and simulation systems have been mostly driven by the DIS and HLA standards community. A number of issues are coming to light about the applicability of these standards, in their present state, to the support of general multi-user VR systems. This paper pinpoints four issues that must be readdressed before large scale virtual reality systems become accessible to a larger commercial and public domain: a reduction in the effects of network delays; scalable causal event delivery; update control; and scalable reliable communication. Each of these issues is tackled through a common theme of combining wall clock and causal time-related entity behaviour, knowledge of network delays and prediction of entity behaviour, that together overcome many of the effects of network delays.

H-N2N : uma arquitetura hierárquica escalável para ambientes de realidade mista de grande porte

In this work, we propose a solution to solve the scalability problem found in collaborative, virtual and mixed reality environments of large scale, that use the hierarchical client-server model. Basically, we use a hierarchy of servers. When the capacity of a server is reached, a new server is created as a sun of the first one, and the system load is distributed between them (father and sun). We propose efficient tools and techniques for solving problems inherent to client-server model, as the definition of clusters of users, distribution and redistribution of users through the servers, and some mixing and filtering operations, that are necessary to reduce flow between servers. The new model was tested, in simulation, emulation and in interactive applications that were implemented. The results of these experimentations show enhancements in the traditional, previous models indicating the usability of the proposed in problems of all-to-all communications. This is the case of interactive games and other applications devoted to Internet (including multi-user environments) and interactive applications of the Brazilian Digital Television System, to be developed by the research group. Keywords: large scale virtual environments, interactive digital tv, distributed

Avaliando o impacto da utilização de renderizações panorâmicas para a navegação em ambientes virtuais

Panoramic rendering is the visualization of three-dimensional objects in a virtual environment through a wide viewing angle. This work investigated if the use of panoramas is able to promote faster searches in a virtual environment. Panoramas allow the presentation of space through less need to change the orientation of the camera, especially for the case of projections spanning 360º surrounding the user, which can benefit searching. However, the larger the angle, more distorted is the visualization of the environment, causing confusion in navigation. The distortion is even bigger when the user changes the pitch of the camera, by looking up or down. In this work we developed a technique to eliminate specifically the distortions caused by changes in pitch, which was called hemispheric projection. Experiments were done to evaluate the performance of search navigation through perspective, cylindrical and hemispherical projections. The results indicate that navigating with perspective projection is superior than navigating with panoramic projections, possibly due to factors such as (i) lack of experience of the participants in understanding the scenes displayed as panoramas, (ii) the inherent presence of distortion in panoramic projections and (iii) a lower display resolution because the objects are presented in smaller sizes in panoramic projections, making the perception of details more difficult. However, the hemispherical projection was better than the cylindrical, indicating that the developed technique provides benefits for navigation compared to current techniques of panoramic projection. The hemispheric projection also provided the least amount of changes of camera orientation, which is an indication that the hemispheric projections may be particularly useful in situations where there are restrictions on the ease to change the orientation. Future research will investigate the performance of cameras interactions on slower devices, such as using only keyboard, or brain-machine interfaces

A multiprojeção de ambientes virtuais gerados por motores de jogo: o histórico e o design de uma solução genérica aplicado no motor de jogo Unity

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)