Multiple haptic targets for motion-impaired computer users

Although a number of studies have reported that force feedback gravity wells can improve performance in "point-and-click" tasks, there have been few studies addressing issues surrounding the use of gravity wells for multiple on-screen targets. This paper investigates the performance of users, both with and without motion-impairments, in a "point-and-click" task when an undesired haptic distractor is present. The importance of distractor location is studied explicitly. Results showed that gravity wells can still improve times and error rates, even on occasions when the cursor is pulled into a distractor. The greatest improvement is seen for the most impaired users. In addition to traditional measures such as time and errors, performance is studied in terms of measures of cursor movement along a path. Two cursor measures, angular distribution and temporal components, are proposed and their ability to explain performance differences is explored.

The use of cursor measures for motion-impaired computer users

“Point and click” interactions remain one of the key features of graphical user interfaces (GUIs). People with motion-impairments, however, can often have difficulty with accurate control of standard pointing devices. This paper discusses work that aims to reveal the nature of these difficulties through analyses that consider the cursor’s path of movement. A range of cursor measures was applied, and a number of them were found to be significant in capturing the differences between able-bodied users and motion-impaired users, as well as the differences between a haptic force feedback condition and a control condition. The cursor measures found in the literature, however, do not make up a comprehensive list, but provide a starting point for analysing cursor movements more completely. Six new cursor characteristics for motion-impaired users are introduced to capture aspects of cursor movement different from those already proposed.

Haptic control of multi-axis robotic systems

Control of tele-operated remote robot’s is nothing new; the public was introduced to this 'new' field in 1986 when the Chernobyl cleanup began. Pictures of weird and wonderful robotic workers pouring concrete or moving rubble flooded the world. Integration of force feedback or 'haptics' to remote robot's is a new development and one that is likely to make a big difference in man-machine interaction. Development of haptic capable tele-operation schema is a challenge. Often platform specific software is developed for one off tasks. This research focussed on the development of an open software platform for haptic control of multiple remote robotic platforms. The software utilises efficient server/client architecture for low data latency, while efficiently performing required kinematic transforms and data manipulation in real time. A description of the algorithm, software interface and hardware is presented in this paper. Preliminary results are encouraging as haptic control has been shown to greatly enhances remote positioning tasks.

A haptically enabled low-cost reconnaissance platform for law enforcement

Traditionally, the control system of a modern teleoperated mobile robot consists of one or more two-dimensional joysticks placed on a control interface. While this simplistic interface allows an operator to remotely drive the platform, feedback is limited to visual information supplied by on-board cameras. Significant advances in the field of haptics have the potential to meaningfully enhance situational awareness of a remote robot. The focus of this research is the augmentation of Deakin University's OzBot trade MkIV mobile platform to include haptic control methodologies. Utilising the platform's inertial measurement unit, a remote operator has the ability to gain knowledge of the vehicle's operating performance and terrain while supplying a finer level of control to the drive motors. Our development of a generic multi-platform ActiveX allows the easy implementation of haptic force feedback to many computer based robot controllers. Furthermore, development of communication protocols has progressed with Joint Architecture for Unmanned Systems (JAUS) compliance in mind. The haptic force control algorithms are presented along with results highlighting the benefits of haptic operator feedback on the MklV OzBot trade chassis.

Haptically simulating soft tissue using spherical voxel and reactive agents

This thesis describes technology developed by the author enabling trainee surgeons to perform needle insertion procedures with force feedback (haptics) on a virtual patient. Addition of the sense of touch to medical simulation is arguably the most important step forward in the evolution of haptic technology to this day.

Ozbot and haptics : remote surveillance to physical presence

This paper reports on robotic and haptic technologies and capabilities developed for the law enforcement and defence community within Australia by the Centre for Intelligent Systems Research (CISR). The OzBot series of small and medium surveillance robots have been designed in Australia and evaluated by law enforcement and defence personnel to determine suitability and ruggedness in a variety of environments. Using custom developed digital electronics and featuring expandable data busses including RS485, I2C, RS232, video and Ethernet, the robots can be directly connected to many off the shelf payloads such as gas sensors, x-ray sources and camera systems including thermal and night vision. Differentiating the OzBot platform from its peers is its ability to be integrated directly with haptic technology or the 'haptic bubble' developed by CISR. Haptic interfaces allow an operator to physically 'feel' remote environments through position-force control and experience realistic force feedback. By adding the capability to remotely grasp an object, feel its weight, texture and other physical properties in real-time from the remote ground control unit, an operator's situational awareness is greatly improved through Haptic augmentation in an environment where remote-system feedback is often limited.

Haptics enabled offline AFM image analysis

Current advancements in nanotechnology are dependent on the capabilities that can enable nano-scientists to extend their eyes and hands into the nano-world. For this purpose, a haptics (devices capable of recreating tactile or force sensations) based system for AFM (Atomic Force Microscope) is proposed. The system enables the nano-scientists to touch and feel the sample surfaces, viewed through AFM, in order to provide them with better understanding of the physical properties of the surface, such as roughness, stiffness and shape of molecular architecture. At this stage, the proposed work uses of ine images produced using AFM and perform image analysis to create virtual surfaces suitable for haptics force analysis. The research work is in the process of extension from of ine to online process where interaction will be done directly on the material surface for realistic analysis.

Haptics and virtual reality in modern manufacturing

Product assembly is one of the most studied processes in modern manufacturing. In recent years a number of computer-based virtual reality systems have been proposed, developed and adopted by the manufacturing industries. Such systems have major advantages over conventional training practices for product assembly. Significant cost savings can be realized due to shorter training-scenario development times and reuse of existing engineering math models. In addition, the time span from the product design to full production can be shortened due to non-reliance on actual components and subsystems for training. Such training systems are effective if the knowledge required to be transferred is just process sequence such as assembly sequence. However, knowledge transfer for procedural and cognitive learning as well as skills development is very limited, due to the lack of user interactivity and immersion.

This talk will focus on a research technology platform where haptics and virtual reality are integrated to create an effective environment for production assembly operators’ training. In this system virtual reality provides the grounds for realistic visualization, as well as immersion, whereas haptics enforces physical constraints within the virtual world generating the feelings of realistic interaction, making it accessible for formal learning and better understanding during task performance.

The developed research technology platform imitates real physical training scenarios by providing comprehensive user interaction, constrained within the physical limitations of the real world. Through the utilization of a haptics device, providing realistic force feedback, users are able to engage in product assembly training with a stronger sense of ‘reality’.

Emerging technology for remote IED render safe using the OzBot

The Centre for Intelligent Systems Research (CISR) based within Deakin University’s Geelong campus has been developing technology specifically for remote render-safe of IED since being awarded a CTD contact in 2006. During this time, research engineers have worked with key defence and industry stakeholders to develop a series of robotic platforms tasked with immersing a soldier in his or her remote environment. Utilising Haptics (force feedback technology), stereovision (binocular video stream for depth perception) and intuitive user controls, the robots have been engineered to deliver maximum effectiveness while allowing minimal training liability. In Victoria, CISR’s OzBot series of mobile platforms have been used by the Victorian Police in a first-responder capacity, exploiting the 30-sec system boot-up and man-portable design to get eyes-on-target at the soonest possible moment. The CISR robotics group has been working on technologies that reduce operator fatigue, minimise training liability and maintenance, developing simulation technologies for increased training availability and develop mobile platforms with increased range, payload, manipulator reach and capability. This paper describes some of the technologies, methods and systems developed by CISR in the field of IED neutralisation with the aim of increasing military awareness of Australian capability.

3D particle-based cell modelling for haptic microrobotic cell injection

Introducing haptic interface to conduct microrobotic intracellular injection has many beneficial implications. In particular, the haptic device provides force feedback to the bio-operator's hand. This paper introduces a 3D particle-based model to simulate the deformation of the cell membrane and corresponding cellular forces during microrobotic cell injection. The model is based on the kinematic and dynamic of spring – damper multi particle joints considering visco-elastic fluidic properties. It simulates the indentation force feedback as well as cell visual deformation during the microinjection. The model is verified using experimental data of zebrafish embryo microinjection. The results demonstrate that the developed cell model is capable of estimating zebrafish embryo deformation and force feedback accurately.

Next generation robotic counter IED technology : A step change in capability

Improvised Explosive Devices (IEDs) are reported as the number one cause of injury and death for allied troops in the current theater of operation. Deakin University’s Centre for Intelligent Systems Research (CISR) is working on next-generation technology to combat the threat. In 2006 CISR was awarded funding through the Capability and Technology Demonstrator (CTD) Program managed by the Australian Defence Force. The objective was to investigate the use of haptics or force feedback technology for Counter-IED (CIED) tasks. Over the past six years, engineers from CISR have worked alongside Defence stakeholders to develop a series of robotic platforms designed to immerse a soldier in the remote environment. Utilising a natural user interface, haptic force feedback and stereovision, the technology has undergone initial trials in Sydney, Canberra, Woomera and at the CISR testing facility in Geelong, Australia. The technology has proved popular among operators allowing them increased fidelity and manipulation speed while significantly reducing required training. CISR has a history of rapidly delivering technology to meet the needs of police and law enforcement in Australia. The OzBot™ series of robots developed in conjunction with the Victorian Police is currently in service and used extensively for hostage negotiation and first responder roles. The CISR robotics group works on technologies that reduce operator fatigue, minimise training liability and maintenance. Over 55 engineers develop simulation environments for increased training availability and continuous improvement to the current range of mobile platforms, including communications range, payload, manipulator reach and capability. This paper describes a number of the technologies, methods and systems developed by CISR for IED neutralisation, with the aim to increasing military awareness of Australian capability.

Formulation and simulation of a 3D mechanical model of embryos for microinjection

The understanding of cell manipulation, for example in microinjection, requires an accurate model of the cells. Motivated by this important requirement, a 3D particlebased mechanical model is derived for simulating the deformation of the fish egg membrane and the corresponding cellular forces during microrobotic cell injection. The model is formulated based on the kinematic and dynamic of spring- damper configuration with multi-particle joints considering the visco-elastic fluidic properties. It simulates the indentation force feedback as well as cell visual deformation during microinjection. A preliminary simulation study is conducted with different parameter configurations. The results indicate that the proposed particle-based model is able to provide similar deformation profiles as observed from a real microinjection experiment of the zebrafish embryo published in the literature. As a generic modelling approach is adopted, the proposed model also has the potential in applications with different types of manipulation such as micropipette cell aspiration.

Countering Improvised explosive devices through a multi-point haptic teleoperation system

Improvised Explosive Devices (IEDs) are reported as the number one cause of injury and death for allied troops in the current theater of operation. Current stand-off technologies for Counter IED (CIED) tasks rely on robotic platforms that have not improved in capability over the past decade to combat the ever increasing threat of IEDs. While they provide operational capability, the effectiveness of these platforms is limited. This is because they primarily utilise video and audio feedback, and require extensive training and specialist operators. Recent operational experience has demonstrated the need for robotic systems that are highly capable, yet easily operable for high fidelity manipulation. Force feedback provides an operator with more intuitive control of a robotic system. This sense of touch allows an operator to obtain a sense of feel from a stand-off location of what the robot touches or grasps through a human-robot interface. This paper reports the design and development of a Haptically-Enabled Counter IED robotic system that was funded by the Australian Defence Force. The presented work focuses on the design methodology for the system, and provides the results of the manipulator analysis and trial outcomes.

Feasibility of Wii Fit Training to Improve Clinical Measures of Balance in Older Adults

Background and purpose: Numerous interventions have been proposed to improve balance in older adults with varying degrees of success. A novel approach may be to use an off-the-shelf video game system utilizing real-time force feedback to train older adults. The purpose of this study is to investigate the feasibility of using Nintendo's Wii Fit for training to improve clinical measures of balance in older adults and to retain the improvements after a period of time. Methods: Twelve healthy older adults (aged >70 years) were randomly divided into two groups. The experimental group completed training using Nintendo's Wii Fit game three times a week for 3 weeks while the control group continued with normal activities. Four clinical measures of balance were assessed before training, 1 week after training, and 1 month after training: Berg Balance Scale (BBS), Fullerton Advanced Balance (FAB) scale, Functional Reach (FR), and Timed Up and Go (TUG). Friedman two-way analysis of variance by ranks was conducted on the control and experimental group to determine if training using the Wii Balance Board with Wii Fit had an influence on clinical measures of balance. Results: Nine older adults completed the study (experimental group n = 4, control group n = 5). The experimental group significantly increased their BBS after training while the control group did not. There was no significant change for either group with FAB, FR, and TUG. Conclusion: Balance training with Nintendo's Wii Fit may be a novel way for older adults to improve balance as measured by the BBS.

Design and control of multi-finger haptic devices for dexterous manipulation

En la interacción con el entorno que nos rodea durante nuestra vida diaria (utilizar un cepillo de dientes, abrir puertas, utilizar el teléfono móvil, etc.) y en situaciones profesionales (intervenciones médicas, procesos de producción, etc.), típicamente realizamos manipulaciones avanzadas que incluyen la utilización de los dedos de ambas manos. De esta forma el desarrollo de métodos de interacción háptica multi-dedo dan lugar a interfaces hombre-máquina más naturales y realistas. No obstante, la mayoría de interfaces hápticas disponibles en el mercado están basadas en interacciones con un solo punto de contacto; esto puede ser suficiente para la exploración o palpación del entorno pero no permite la realización de tareas más avanzadas como agarres. En esta tesis, se investiga el diseño mecánico, control y aplicaciones de dispositivos hápticos modulares con capacidad de reflexión de fuerzas en los dedos índice, corazón y pulgar del usuario. El diseño mecánico de la interfaz diseñada, ha sido optimizado con funciones multi-objetivo para conseguir una baja inercia, un amplio espacio de trabajo, alta manipulabilidad y reflexión de fuerzas superiores a 3 N en el espacio de trabajo. El ancho de banda y la rigidez del dispositivo se han evaluado mediante simulación y experimentación real. Una de las áreas más importantes en el diseño de estos dispositivos es el efector final, ya que es la parte que está en contacto con el usuario. Durante este trabajo se ha diseñado un dedal de bajo peso, adaptable a diferentes usuarios que, mediante la incorporación de sensores de contacto, permite estimar fuerzas normales y tangenciales durante la interacción con entornos reales y virtuales. Para el diseño de la arquitectura de control, se estudiaron los principales requisitos para estos dispositivos. Entre estos, cabe destacar la adquisición, procesado e intercambio a través de internet de numerosas señales de control e instrumentación; la computación de equaciones matemáticas incluyendo la cinemática directa e inversa, jacobiana, algoritmos de detección de agarres, etc. Todos estos componentes deben calcularse en tiempo real garantizando una frecuencia mínima de 1 KHz. Además, se describen sistemas para manipulación de precisión virtual y remota; así como el diseño de un método denominado "desacoplo cinemático iterativo" para computar la cinemática inversa de robots y la comparación con otros métodos actuales. Para entender la importancia de la interacción multimodal, se ha llevado a cabo un estudio para comprobar qué estímulos sensoriales se correlacionan con tiempos de respuesta más rápidos y de mayor precisión. Estos experimentos se desarrollaron en colaboración con neurocientíficos del instituto Technion Israel Institute of Technology. Comparando los tiempos de respuesta en la interacción unimodal (auditiva, visual y háptica) con combinaciones bimodales y trimodales de los mismos, se demuestra que el movimiento sincronizado de los dedos para generar respuestas de agarre se basa principalmente en la percepción háptica. La ventaja en el tiempo de procesamiento de los estímulos hápticos, sugiere que los entornos virtuales que incluyen esta componente sensorial generan mejores contingencias motoras y mejoran la credibilidad de los eventos. Se concluye que, los sistemas que incluyen percepción háptica dotan a los usuarios de más tiempo en las etapas cognitivas para rellenar información de forma creativa y formar una experiencia más rica. Una aplicación interesante de los dispositivos hápticos es el diseño de nuevos simuladores que permitan entrenar habilidades manuales en el sector médico. En colaboración con fisioterapeutas de Griffith University en Australia, se desarrolló un simulador que permite realizar ejercicios de rehabilitación de la mano. Las propiedades de rigidez no lineales de la articulación metacarpofalange del dedo índice se estimaron mediante la utilización del efector final diseñado. Estos parámetros, se han implementado en un escenario que simula el comportamiento de la mano humana y que permite la interacción háptica a través de esta interfaz. Las aplicaciones potenciales de este simulador están relacionadas con entrenamiento y educación de estudiantes de fisioterapia. En esta tesis, se han desarrollado nuevos métodos que permiten el control simultáneo de robots y manos robóticas en la interacción con entornos reales. El espacio de trabajo alcanzable por el dispositivo háptico, se extiende mediante el cambio de modo de control automático entre posición y velocidad. Además, estos métodos permiten reconocer el gesto del usuario durante las primeras etapas de aproximación al objeto para su agarre. Mediante experimentos de manipulación avanzada de objetos con un manipulador y diferentes manos robóticas, se muestra que el tiempo en realizar una tarea se reduce y que el sistema permite la realización de la tarea con precisión. Este trabajo, es el resultado de una colaboración con investigadores de Harvard BioRobotics Laboratory. ABSTRACT When we interact with the environment in our daily life (using a toothbrush, opening doors, using cell-phones, etc.), or in professional situations (medical interventions, manufacturing processes, etc.) we typically perform dexterous manipulations that involve multiple fingers and palm for both hands. Therefore, multi-Finger haptic methods can provide a realistic and natural human-machine interface to enhance immersion when interacting with simulated or remote environments. Most commercial devices allow haptic interaction with only one contact point, which may be sufficient for some exploration or palpation tasks but are not enough to perform advanced object manipulations such as grasping. In this thesis, I investigate the mechanical design, control and applications of a modular haptic device that can provide force feedback to the index, thumb and middle fingers of the user. The designed mechanical device is optimized with a multi-objective design function to achieve a low inertia, a large workspace, manipulability, and force-feedback of up to 3 N within the workspace; the bandwidth and rigidity for the device is assessed through simulation and real experimentation. One of the most important areas when designing haptic devices is the end-effector, since it is in contact with the user. In this thesis the design and evaluation of a thimble-like, lightweight, user-adaptable, and cost-effective device that incorporates four contact force sensors is described. This design allows estimation of the forces applied by a user during manipulation of virtual and real objects. The design of a real-time, modular control architecture for multi-finger haptic interaction is described. Requirements for control of multi-finger haptic devices are explored. Moreover, a large number of signals have to be acquired, processed, sent over the network and mathematical computations such as device direct and inverse kinematics, jacobian, grasp detection algorithms, etc. have to be calculated in Real Time to assure the required high fidelity for the haptic interaction. The Hardware control architecture has different modules and consists of an FPGA for the low-level controller and a RT controller for managing all the complex calculations (jacobian, kinematics, etc.); this provides a compact and scalable solution for the required high computation capabilities assuring a correct frequency rate for the control loop of 1 kHz. A set-up for dexterous virtual and real manipulation is described. Moreover, a new algorithm named the iterative kinematic decoupling method was implemented to solve the inverse kinematics of a robotic manipulator. In order to understand the importance of multi-modal interaction including haptics, a subject study was carried out to look for sensory stimuli that correlate with fast response time and enhanced accuracy. This experiment was carried out in collaboration with neuro-scientists from Technion Israel Institute of Technology. By comparing the grasping response times in unimodal (auditory, visual, and haptic) events with the response times in events with bimodal and trimodal combinations. It is concluded that in grasping tasks the synchronized motion of the fingers to generate the grasping response relies on haptic cues. This processing-speed advantage of haptic cues suggests that multimodalhaptic virtual environments are superior in generating motor contingencies, enhancing the plausibility of events. Applications that include haptics provide users with more time at the cognitive stages to fill in missing information creatively and form a richer experience. A major application of haptic devices is the design of new simulators to train manual skills for the medical sector. In collaboration with physical therapists from Griffith University in Australia, we developed a simulator to allow hand rehabilitation manipulations. First, the non-linear stiffness properties of the metacarpophalangeal joint of the index finger were estimated by using the designed end-effector; these parameters are implemented in a scenario that simulates the behavior of the human hand and that allows haptic interaction through the designed haptic device. The potential application of this work is related to educational and medical training purposes. In this thesis, new methods to simultaneously control the position and orientation of a robotic manipulator and the grasp of a robotic hand when interacting with large real environments are studied. The reachable workspace is extended by automatically switching between rate and position control modes. Moreover, the human hand gesture is recognized by reading the relative movements of the index, thumb and middle fingers of the user during the early stages of the approximation-to-the-object phase and then mapped to the robotic hand actuators. These methods are validated to perform dexterous manipulation of objects with a robotic manipulator, and different robotic hands. This work is the result of a research collaboration with researchers from the Harvard BioRobotics Laboratory. The developed experiments show that the overall task time is reduced and that the developed methods allow for full dexterity and correct completion of dexterous manipulations.