Enabling multi-point haptic grasping in virtual environments

Haptic interaction has received increasing research interest in recent years. Currently, most commercially available haptic devices provide the user with a single point of interaction. Multi-point haptic devices present a logical progression in device design and enable the operator to experience a far wider range of haptic interactions, particularly the ability to grasp via multiple fingers. This is highly desirable for various haptically enabled applications including virtual training, telesurgery and telemanipulation. This paper presents a gripper attachment which utilises two low-cost commercially available haptic devices to facilitate multi-point haptic grasping. It provides the ability to render forces to the user's fingers independently and using Phantom Omni haptic devices offers several benefits over more complex approaches such as low-cost, reliability, and ease of programming. The workspace of the gripper attachment is considered and in order to haptically render the desired forces to the user's fingers, kinematic analysis is discussed and necessary formulations presented. The integrated multi-point haptic platform is presented and exploration of a virtual environment using CHAI 3D is demonstrated.

Virtual haptic cell model for operator training

Microrobotic cell injection is an area of growing research interest. Typically, operators rely on visual feedback to perceive the microscale environment and are subject to lengthy training times and low success rates. Haptic interaction offers the ability to utilise the operator’s haptic modality and to enhance operator performance. Our earlier work presented a haptically enabled system for assisting the operator with certain aspects of the cell injection task. The system aimed to enhance the operator’s controllability of the micropipette through a logical mapping between the haptic device and microrobot, as well as introducing virtual fixtures for haptic guidance. The system was also designed in such a way that given the availability of appropriate force sensors, haptic display of the cell penetration force is straightforward. This work presents our progress towards a virtual replication of the system, aimed at facilitating offline operator training. It is suggested that operators can use the virtual system to train offline and later transfer their skills to the physical system. In order to achieve the necessary representation of the cell within the virtual system, methods based on a particle-based cell model are utilised. In addition to providing the necessary visual representation, the cell model provides the ability to estimate cell penetration forces and haptically display them to the operator. Two different approaches to achieving the virtual system are discussed.

A haptic platform to enable fingertip grasping and manipulation

Haptic technologies allow human users to haptically interact with virtual environments. Haptics has been employed in many application domains including operator training, virtual exploration and teleoperation. Currently, most commercially available haptic devices focus on a single point of haptic interaction. While single-point haptics have been successfully employed in many applications, they remain limited to particular types of haptic interaction. Multi-point haptic devices are a logical progression and facilitate a far wider range of interactions including object grasping, multi-finger object manipulation and size discrimination. The ability to effectively achieve such interactions offers significant benefits for many applications including virtual training, telesurgery and telemanipulation. In such applications, the ability to use multi-point haptic interactions can provide far more effective user interaction as well improved perception of the virtual environment.

Function-based approach to mixed haptic effects rendering

Commonly, surface and solid haptic effects are defined in such a way that they hardly can be rendered together. We propose a method for defining mixed haptic effects including surface, solid, and force fields. These haptic effects can be applied to virtual scenes containing various objects, including polygon meshes, point clouds, impostors, and layered textures, voxel models as well as function-based shapes. Accordingly, we propose a way how to identify location of the haptic tool in such virtual scenes as well as consistently and seamlessly determine haptic effects when the haptic tool moves in the scenes with objects having different sizes, locations, and mutual penetrations. To provide for an efficient and flexible rendering of haptic effects, we propose to concurrently use explicit, implicit and parametric functions, and algorithmic procedures.

Multipoint haptic mediator interface for robotic teleoperation

Despite recent advances in artificial intelligence and autonomous robotics, teleoperation can provide distinct benefits in applications requiring real-time human judgement and intuition. However, as robotic systems are increasingly becoming sophisticated and are performing more complex tasks, realizing these benefits requires new approaches to teleoperation. This paper introduces a novel haptic mediator interface for teleoperating mobile robotic platforms that have a variety of manipulators and functions. Identical master-slave bilateral teleoperation of the robotic manipulators is achieved by representing them in virtual reality and by allowing the operator to interact with them using a multipoint haptic device. The operator is also able to command motions to the mobile platform by using a novel haptic interaction metaphor rather than a separate dedicated input device. The presented interaction techniques enable the operator to perform a wide range of control functions and achieve functionality similar to that of conventional teleoperation schemes that use a single haptic interface. The mediator interface is presented, and important considerations such as workspace mapping and scaling are discussed. © 2015 IEEE.

Multipoint haptic guidance for micrograsping systems

The ability to perform accurate micromanipulation offers wide-reaching benefits and is of increasing interest to researchers. Recent research into microgripper, microtweezer, and microforcep systems contributes toward accurate micrograsping and manipulation. Despite these efforts, achieving adequate operator control remains a distinct research challenge. Haptic interfaces interact with the human's haptic modality and offer the ability to enhance the operator's controllability of micromanipulation systems. Our previous work introduced single-point haptic guidance to assist the operator during intracellular microinjection. This paper extends the approach to propose multipoint haptic guidance for micrograsping tasks. Accurate micrograsping is valuable in many applications, including microassembly and biomanipulation. A multipoint haptic gripper facilitates haptic interaction, and haptic guidance assists the operator in controlling systems suitable for micrograsping. Force fields are used to guide the operator to suitable grasp points on micrometer-sized objects and consist of attractive and repulsive forces. The ability of the force field to effectively assist the operator in grasping the cell is evaluated using a virtual environment. Evaluation results demonstrate the ability of the approach to significantly reduce participants' average grasping error.

Design and control of multi-finger haptic devices for dexterous manipulation

En la interacción con el entorno que nos rodea durante nuestra vida diaria (utilizar un cepillo de dientes, abrir puertas, utilizar el teléfono móvil, etc.) y en situaciones profesionales (intervenciones médicas, procesos de producción, etc.), típicamente realizamos manipulaciones avanzadas que incluyen la utilización de los dedos de ambas manos. De esta forma el desarrollo de métodos de interacción háptica multi-dedo dan lugar a interfaces hombre-máquina más naturales y realistas. No obstante, la mayoría de interfaces hápticas disponibles en el mercado están basadas en interacciones con un solo punto de contacto; esto puede ser suficiente para la exploración o palpación del entorno pero no permite la realización de tareas más avanzadas como agarres. En esta tesis, se investiga el diseño mecánico, control y aplicaciones de dispositivos hápticos modulares con capacidad de reflexión de fuerzas en los dedos índice, corazón y pulgar del usuario. El diseño mecánico de la interfaz diseñada, ha sido optimizado con funciones multi-objetivo para conseguir una baja inercia, un amplio espacio de trabajo, alta manipulabilidad y reflexión de fuerzas superiores a 3 N en el espacio de trabajo. El ancho de banda y la rigidez del dispositivo se han evaluado mediante simulación y experimentación real. Una de las áreas más importantes en el diseño de estos dispositivos es el efector final, ya que es la parte que está en contacto con el usuario. Durante este trabajo se ha diseñado un dedal de bajo peso, adaptable a diferentes usuarios que, mediante la incorporación de sensores de contacto, permite estimar fuerzas normales y tangenciales durante la interacción con entornos reales y virtuales. Para el diseño de la arquitectura de control, se estudiaron los principales requisitos para estos dispositivos. Entre estos, cabe destacar la adquisición, procesado e intercambio a través de internet de numerosas señales de control e instrumentación; la computación de equaciones matemáticas incluyendo la cinemática directa e inversa, jacobiana, algoritmos de detección de agarres, etc. Todos estos componentes deben calcularse en tiempo real garantizando una frecuencia mínima de 1 KHz. Además, se describen sistemas para manipulación de precisión virtual y remota; así como el diseño de un método denominado "desacoplo cinemático iterativo" para computar la cinemática inversa de robots y la comparación con otros métodos actuales. Para entender la importancia de la interacción multimodal, se ha llevado a cabo un estudio para comprobar qué estímulos sensoriales se correlacionan con tiempos de respuesta más rápidos y de mayor precisión. Estos experimentos se desarrollaron en colaboración con neurocientíficos del instituto Technion Israel Institute of Technology. Comparando los tiempos de respuesta en la interacción unimodal (auditiva, visual y háptica) con combinaciones bimodales y trimodales de los mismos, se demuestra que el movimiento sincronizado de los dedos para generar respuestas de agarre se basa principalmente en la percepción háptica. La ventaja en el tiempo de procesamiento de los estímulos hápticos, sugiere que los entornos virtuales que incluyen esta componente sensorial generan mejores contingencias motoras y mejoran la credibilidad de los eventos. Se concluye que, los sistemas que incluyen percepción háptica dotan a los usuarios de más tiempo en las etapas cognitivas para rellenar información de forma creativa y formar una experiencia más rica. Una aplicación interesante de los dispositivos hápticos es el diseño de nuevos simuladores que permitan entrenar habilidades manuales en el sector médico. En colaboración con fisioterapeutas de Griffith University en Australia, se desarrolló un simulador que permite realizar ejercicios de rehabilitación de la mano. Las propiedades de rigidez no lineales de la articulación metacarpofalange del dedo índice se estimaron mediante la utilización del efector final diseñado. Estos parámetros, se han implementado en un escenario que simula el comportamiento de la mano humana y que permite la interacción háptica a través de esta interfaz. Las aplicaciones potenciales de este simulador están relacionadas con entrenamiento y educación de estudiantes de fisioterapia. En esta tesis, se han desarrollado nuevos métodos que permiten el control simultáneo de robots y manos robóticas en la interacción con entornos reales. El espacio de trabajo alcanzable por el dispositivo háptico, se extiende mediante el cambio de modo de control automático entre posición y velocidad. Además, estos métodos permiten reconocer el gesto del usuario durante las primeras etapas de aproximación al objeto para su agarre. Mediante experimentos de manipulación avanzada de objetos con un manipulador y diferentes manos robóticas, se muestra que el tiempo en realizar una tarea se reduce y que el sistema permite la realización de la tarea con precisión. Este trabajo, es el resultado de una colaboración con investigadores de Harvard BioRobotics Laboratory. ABSTRACT When we interact with the environment in our daily life (using a toothbrush, opening doors, using cell-phones, etc.), or in professional situations (medical interventions, manufacturing processes, etc.) we typically perform dexterous manipulations that involve multiple fingers and palm for both hands. Therefore, multi-Finger haptic methods can provide a realistic and natural human-machine interface to enhance immersion when interacting with simulated or remote environments. Most commercial devices allow haptic interaction with only one contact point, which may be sufficient for some exploration or palpation tasks but are not enough to perform advanced object manipulations such as grasping. In this thesis, I investigate the mechanical design, control and applications of a modular haptic device that can provide force feedback to the index, thumb and middle fingers of the user. The designed mechanical device is optimized with a multi-objective design function to achieve a low inertia, a large workspace, manipulability, and force-feedback of up to 3 N within the workspace; the bandwidth and rigidity for the device is assessed through simulation and real experimentation. One of the most important areas when designing haptic devices is the end-effector, since it is in contact with the user. In this thesis the design and evaluation of a thimble-like, lightweight, user-adaptable, and cost-effective device that incorporates four contact force sensors is described. This design allows estimation of the forces applied by a user during manipulation of virtual and real objects. The design of a real-time, modular control architecture for multi-finger haptic interaction is described. Requirements for control of multi-finger haptic devices are explored. Moreover, a large number of signals have to be acquired, processed, sent over the network and mathematical computations such as device direct and inverse kinematics, jacobian, grasp detection algorithms, etc. have to be calculated in Real Time to assure the required high fidelity for the haptic interaction. The Hardware control architecture has different modules and consists of an FPGA for the low-level controller and a RT controller for managing all the complex calculations (jacobian, kinematics, etc.); this provides a compact and scalable solution for the required high computation capabilities assuring a correct frequency rate for the control loop of 1 kHz. A set-up for dexterous virtual and real manipulation is described. Moreover, a new algorithm named the iterative kinematic decoupling method was implemented to solve the inverse kinematics of a robotic manipulator. In order to understand the importance of multi-modal interaction including haptics, a subject study was carried out to look for sensory stimuli that correlate with fast response time and enhanced accuracy. This experiment was carried out in collaboration with neuro-scientists from Technion Israel Institute of Technology. By comparing the grasping response times in unimodal (auditory, visual, and haptic) events with the response times in events with bimodal and trimodal combinations. It is concluded that in grasping tasks the synchronized motion of the fingers to generate the grasping response relies on haptic cues. This processing-speed advantage of haptic cues suggests that multimodalhaptic virtual environments are superior in generating motor contingencies, enhancing the plausibility of events. Applications that include haptics provide users with more time at the cognitive stages to fill in missing information creatively and form a richer experience. A major application of haptic devices is the design of new simulators to train manual skills for the medical sector. In collaboration with physical therapists from Griffith University in Australia, we developed a simulator to allow hand rehabilitation manipulations. First, the non-linear stiffness properties of the metacarpophalangeal joint of the index finger were estimated by using the designed end-effector; these parameters are implemented in a scenario that simulates the behavior of the human hand and that allows haptic interaction through the designed haptic device. The potential application of this work is related to educational and medical training purposes. In this thesis, new methods to simultaneously control the position and orientation of a robotic manipulator and the grasp of a robotic hand when interacting with large real environments are studied. The reachable workspace is extended by automatically switching between rate and position control modes. Moreover, the human hand gesture is recognized by reading the relative movements of the index, thumb and middle fingers of the user during the early stages of the approximation-to-the-object phase and then mapped to the robotic hand actuators. These methods are validated to perform dexterous manipulation of objects with a robotic manipulator, and different robotic hands. This work is the result of a research collaboration with researchers from the Harvard BioRobotics Laboratory. The developed experiments show that the overall task time is reduced and that the developed methods allow for full dexterity and correct completion of dexterous manipulations.

Pulse Gauntlets

To feel another person’s pulse is an intimate and physical interaction. In these prototypes we use near field communications to extend the tangible reach of our heart beat, so another person can feel our heart beat at a distance. The work is an initial experiment in near field haptic interaction, and is used to explore the quality of interactions resulting from feeling another persons pulse. The work takes the form of two feathered white gauntlets, to be worn on the fore arm. Each of the gauntlets contain a pulse sensor, radio transmitter and vibrator. The pulse of the wearer is transmitted to the other feathered gauntlet and transformed into haptic feedback. When there are two wearers, their heart beats are exchanged. To be felt by of each other without physical contact.

Grasping virtual objects with multi-point haptics

The majority of commercially available haptic devices offer a single point of haptic interaction. These devices are limited when it is desirable to grasp with multiple fingers in applications including virtual training, telesurgery and telemanipulation. Multipoint haptic devices serve to facilitate a greater range of interactions. This paper presents a gripper attachment to enable multi-point haptic grasping in virtual environments. The approach employs two Phantom Omni haptic devices to independently render forces to the user's thumb and other fingers. Compared with more complex approaches to multi-point haptics, this approach provides a number of advantages including low-cost, reliability and ease of programming. The ability of the integrated multi-point haptic platform to interact within a CHAI 3D virtual environment is also presented.

Integrating kinect and haptics for interactive STEM education in local and distributed environments

Skill shortage is a realistic social problem that Australia is currently facing, especially in the fields of Science, Technology, Engineering and Mathematics (STEM). Various approaches have been proposed to soften this issue. By now the most successful approach is to attract pre-university youth and university freshmen into those fields before they make a decision on future subjects by introducing them with interactive, modifiable and inspiring virtual environments, which incorporates most essential knowledge of STEM. We propose to design a comprehensive virtual reality platform with immersive interactions, pluggable components and flexible configurations. It also involves haptics, motion capture and gesture recognition, and could be deployed in both local and distributed environments. The platform utilizes off the shelf low cost haptics and motion capture products, however the fidelity can be maintained at a good level. The proposed platform has been implemented with different configurations and has been tested on a group of users. Preliminary test results show that the interactivity, flexibility and fidelity of the platform are highly appreciated by users. User surveys also indicate that the proposed platform could help pre-university students and university freshmen build an overview of various aspects of STEM education. Besides, users are also positive on the fact that the platform enabled them to identify the challenges for higher education in STEM by providing them opportunities to interactively modify system configurations and instantly experience the corresponding results both visually and haptically.

Interaction of visual and haptic information in simulated environments: texture perception

This paper describes experiments relating to the perception of the roughness of simulated surfaces via the haptic and visual senses. Subjects used a magnitude estimation technique to judge the roughness of “virtual gratings” presented via a PHANToM haptic interface device, and a standard visual display unit. It was shown that under haptic perception, subjects tended to perceive roughness as decreasing with increased grating period, though this relationship was not always statistically significant. Under visual exploration, the exact relationship between spatial period and perceived roughness was less well defined, though linear regressions provided a reliable approximation to individual subjects’ estimates.

Human-Computer interaction methodologies applied in the evaluation of haptic digital musical instruments

Recent developments in interactive technologies have seen major changes in the manner in which artists, performers, and creative individuals interact with digital music technology; this is due to the increasing variety of interactive technologies that are readily available today. Digital Musical Instruments (DMIs) present musicians with performance challenges that are unique to this form of computer music. One of the most significant deviations from conventional acoustic musical instruments is the level of physical feedback conveyed by the instrument to the user. Currently, new interfaces for musical expression are not designed to be as physically communicative as acoustic instruments. Specifically, DMIs are often void of haptic feedback and therefore lack the ability to impart important performance information to the user. Moreover, there currently is no standardised way to measure the effect of this lack of physical feedback. Best practice would expect that there should be a set of methods to effectively, repeatedly, and quantifiably evaluate the functionality, usability, and user experience of DMIs. Earlier theoretical and technological applications of haptics have tried to address device performance issues associated with the lack of feedback in DMI designs and it has been argued that the level of haptic feedback presented to a user can significantly affect the user’s overall emotive feeling towards a musical device. The outcome of the investigations contained within this thesis are intended to inform new haptic interface.