Human-robot interaction for telemanipulation in large workspaces

Independientemente de la existencia de técnicas altamente sofisticadas y capacidades de cómputo cada vez más elevadas, los problemas asociados a los robots que interactúan con entornos no estructurados siguen siendo un desafío abierto en robótica. A pesar de los grandes avances de los sistemas robóticos autónomos, hay algunas situaciones en las que una persona en el bucle sigue siendo necesaria. Ejemplos de esto son, tareas en entornos de fusión nuclear, misiones espaciales, operaciones submarinas y cirugía robótica. Esta necesidad se debe a que las tecnologías actuales no pueden realizar de forma fiable y autónoma cualquier tipo de tarea. Esta tesis presenta métodos para la teleoperación de robots abarcando distintos niveles de abstracción que van desde el control supervisado, en el que un operador da instrucciones de alto nivel en la forma de acciones, hasta el control bilateral, donde los comandos toman la forma de señales de control de bajo nivel. En primer lugar, se presenta un enfoque para llevar a cabo la teleoperación supervisada de robots humanoides. El objetivo es controlar robots terrestres capaces de ejecutar tareas complejas en entornos de búsqueda y rescate utilizando enlaces de comunicación limitados. Esta propuesta incorpora comportamientos autónomos que el operador puede utilizar para realizar tareas de navegación y manipulación mientras se permite cubrir grandes áreas de entornos remotos diseñados para el acceso de personas. Los resultados experimentales demuestran la eficacia de los métodos propuestos. En segundo lugar, se investiga el uso de dispositivos rentables para telemanipulación guiada. Se presenta una aplicación que involucra un robot humanoide bimanual y un traje de captura de movimiento basado en sensores inerciales. En esta aplicación, se estudian las capacidades de adaptación introducidas por el factor humano y cómo estas pueden compensar la falta de sistemas robóticos de alta precisión. Este trabajo es el resultado de una colaboración entre investigadores del Biorobotics Laboratory de la Universidad de Harvard y el Centro de Automática y Robótica UPM-CSIC. En tercer lugar, se presenta un nuevo controlador háptico que combina velocidad y posición. Este controlador bilateral híbrido hace frente a los problemas relacionados con la teleoperación de un robot esclavo con un gran espacio de trabajo usando un dispositivo háptico pequeño como maestro. Se pueden cubrir amplias áreas de trabajo al cambiar automáticamente entre los modos de control de velocidad y posición. Este controlador háptico es ideal para sistemas maestro-esclavo con cinemáticas diferentes, donde los comandos se transmiten en el espacio de la tarea del entorno remoto. El método es validado para realizar telemanipulación hábil de objetos con un robot industrial. Por último, se introducen dos contribuciones en el campo de la manipulación robótica. Por un lado, se presenta un nuevo algoritmo de cinemática inversa, llamado método iterativo de desacoplamiento cinemático. Este método se ha desarrollado para resolver el problema cinemático inverso de un tipo de robot de seis grados de libertad donde una solución cerrada no está disponible. La eficacia del método se compara con métodos numéricos convencionales. Además, se ha diseñado una taxonomía robusta de agarres que permite controlar diferentes manos robóticas utilizando una correspondencia, basada en gestos, entre los espacios de trabajo de la mano humana y de la mano robótica. El gesto de la mano humana se identifica mediante la lectura de los movimientos relativos del índice, el pulgar y el dedo medio del usuario durante las primeras etapas del agarre. ABSTRACT Regardless of the availability of highly sophisticated techniques and ever increasing computing capabilities, the problems associated with robots interacting with unstructured environments remains an open challenge. Despite great advances in autonomous robotics, there are some situations where a humanin- the-loop is still required, such as, nuclear, space, subsea and robotic surgery operations. This is because the current technologies cannot reliably perform all kinds of task autonomously. This thesis presents methods for robot teleoperation strategies at different levels of abstraction ranging from supervisory control, where the operator gives high-level task actions, to bilateral teleoperation, where the commands take the form of low-level control inputs. These strategies contribute to improve the current human-robot interfaces specially in the case of slave robots deployed at large workspaces. First, an approach to perform supervisory teleoperation of humanoid robots is presented. The goal is to control ground robots capable of executing complex tasks in disaster relief environments under constrained communication links. This proposal incorporates autonomous behaviors that the operator can use to perform navigation and manipulation tasks which allow covering large human engineered areas of the remote environment. The experimental results demonstrate the efficiency of the proposed methods. Second, the use of cost-effective devices for guided telemanipulation is investigated. A case study involving a bimanual humanoid robot and an Inertial Measurement Unit (IMU) Motion Capture (MoCap) suit is introduced. Herein, it is corroborated how the adaptation capabilities offered by the human-in-the-loop factor can compensate for the lack of high-precision robotic systems. This work is the result of collaboration between researchers from the Harvard Biorobotics Laboratory and the Centre for Automation and Robotics UPM-CSIC. Thirdly, a new haptic rate-position controller is presented. This hybrid bilateral controller copes with the problems related to the teleoperation of a slave robot with large workspace using a small haptic device as master. Large workspaces can be covered by automatically switching between rate and position control modes. This haptic controller is ideal to couple kinematic dissimilar master-slave systems where the commands are transmitted in the task space of the remote environment. The method is validated to perform dexterous telemanipulation of objects with a robotic manipulator. Finally, two contributions for robotic manipulation are introduced. First, a new algorithm, the Iterative Kinematic Decoupling method, is presented. It is a numeric method developed to solve the Inverse Kinematics (IK) problem of a type of six-DoF robotic arms where a close-form solution is not available. The effectiveness of this IK method is compared against conventional numerical methods. Second, a robust grasp mapping has been conceived. It allows to control a wide range of different robotic hands using a gesture based correspondence between the human hand space and the robotic hand space. The human hand gesture is identified by reading the relative movements of the index, thumb and middle fingers of the user during the early stages of grasping.

Context modelling for natural Human Computer Interaction applications in e-health

The conception of IoT (Internet of Things) is accepted as the future tendency of Internet among academia and industry. It will enable people and things to be connected at anytime and anyplace, with anything and anyone. IoT has been proposed to be applied into many areas such as Healthcare, Transportation，Logistics, and Smart environment etc. However, this thesis emphasizes on the home healthcare area as it is the potential healthcare model to solve many problems such as the limited medical resources, the increasing demands for healthcare from elderly and chronic patients which the traditional model is not capable of. A remarkable change in IoT in semantic oriented vision is that vast sensors or devices are involved which could generate enormous data. Methods to manage the data including acquiring, interpreting, processing and storing data need to be implemented. Apart from this, other abilities that IoT is not capable of are concluded, namely, interoperation, context awareness and security & privacy. Context awareness is an emerging technology to manage and take advantage of context to enable any type of system to provide personalized services. The aim of this thesis is to explore ways to facilitate context awareness in IoT. In order to realize this objective, a preliminary research is carried out in this thesis. The most basic premise to realize context awareness is to collect, model, understand, reason and make use of context. A complete literature review for the existing context modelling and context reasoning techniques is conducted. The conclusion is that the ontology-based context modelling and ontology-based context reasoning are the most promising and efficient techniques to manage context. In order to fuse ontology into IoT, a specific ontology-based context awareness framework is proposed for IoT applications. In general, the framework is composed of eight components which are hardware, UI (User Interface), Context modelling, Context fusion, Context reasoning, Context repository, Security unit and Context dissemination. Moreover, on the basis of TOVE (Toronto Virtual Enterprise), a formal ontology developing methodology is proposed and illustrated which consists of four stages: Specification & Conceptualization, Competency Formulation, Implementation and Validation & Documentation. In addition, a home healthcare scenario is elaborated by listing its well-defined functionalities. Aiming at representing this specific scenario, the proposed ontology developing methodology is applied and the ontology-based model is developed in a free and open-source ontology editor called Protégé. Finally, the accuracy and completeness of the proposed ontology are validated to show that this proposed ontology is able to accurately represent the scenario of interest.

Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns

A more natural, intuitive, user-friendly, and less intrusive Human–Computer interface for controlling an application by executing hand gestures is presented. For this purpose, a robust vision-based hand-gesture recognition system has been developed, and a new database has been created to test it. The system is divided into three stages: detection, tracking, and recognition. The detection stage searches in every frame of a video sequence potential hand poses using a binary Support Vector Machine classifier and Local Binary Patterns as feature vectors. These detections are employed as input of a tracker to generate a spatio-temporal trajectory of hand poses. Finally, the recognition stage segments a spatio-temporal volume of data using the obtained trajectories, and compute a video descriptor called Volumetric Spatiograms of Local Binary Patterns (VS-LBP), which is delivered to a bank of SVM classifiers to perform the gesture recognition. The VS-LBP is a novel video descriptor that constitutes one of the most important contributions of the paper, which is able to provide much richer spatio-temporal information than other existing approaches in the state of the art with a manageable computational cost. Excellent results have been obtained outperforming other approaches of the state of the art.

Development of a hand-gesture recognition system for human-computer interaction

The aim of this Master Thesis is the analysis, design and development of a robust and reliable Human-Computer Interaction interface, based on visual hand-gesture recognition. The implementation of the required functions is oriented to the simulation of a classical hardware interaction device: the mouse, by recognizing a specific hand-gesture vocabulary in color video sequences. For this purpose, a prototype of a hand-gesture recognition system has been designed and implemented, which is composed of three stages: detection, tracking and recognition. This system is based on machine learning methods and pattern recognition techniques, which have been integrated together with other image processing approaches to get a high recognition accuracy and a low computational cost. Regarding pattern recongition techniques, several algorithms and strategies have been designed and implemented, which are applicable to color images and video sequences. The design of these algorithms has the purpose of extracting spatial and spatio-temporal features from static and dynamic hand gestures, in order to identify them in a robust and reliable way. Finally, a visual database containing the necessary vocabulary of gestures for interacting with the computer has been created.

Human-computer interaction for optical music recognition tasks

The need to digitise music scores has led to the development of Optical Music Recognition (OMR) tools. Unfortunately, the performance of these systems is still far from providing acceptable results. This situation forces the user to be involved in the process due to the need of correcting the mistakes made during recognition. However, this correction is performed over the output of the system, so these interventions are not exploited to improve the performance of the recognition. This work sets the scenario in which human and machine interact to accurately complete the OMR task with the least possible effort for the user.

Quantifying epistemic actions in human-computer interaction

As digital systems move away from traditional desktop setups, new interaction paradigms are emerging that better integrate with users’ realworld surroundings, and better support users’ individual needs. While promising, these modern interaction paradigms also present new challenges, such as a lack of paradigm-specific tools to systematically evaluate and fully understand their use. This dissertation tackles this issue by framing empirical studies of three novel digital systems in embodied cognition – an exciting new perspective in cognitive science where the body and its interactions with the physical world take a central role in human cognition. This is achieved by first, focusing the design of all these systems on a contemporary interaction paradigm that emphasizes physical interaction on tangible interaction, a contemporary interaction paradigm; and second, by comprehensively studying user performance in these systems through a set of novel performance metrics grounded on epistemic actions, a relatively well established and studied construct in the literature on embodied cognition. The first system presented in this dissertation is an augmented Four-in-a-row board game. Three different versions of the game were developed, based on three different interaction paradigms (tangible, touch and mouse), and a repeated measures study involving 36 participants measured the occurrence of three simple epistemic actions across these three interfaces. The results highlight the relevance of epistemic actions in such a task and suggest that the different interaction paradigms afford instantiation of these actions in different ways. Additionally, the tangible version of the system supports the most rapid execution of these actions, providing novel quantitative insights into the real benefits of tangible systems. The second system presented in this dissertation is a tangible tabletop scheduling application. Two studies with single and paired users provide several insights into the impact of epistemic actions on the user experience when these are performed outside of a system’s sensing boundaries. These insights are clustered by the form, size and location of ideal interface areas for such offline epistemic actions to occur, as well as how can physical tokens be designed to better support them. Finally, and based on the results obtained to this point, the last study presented in this dissertation directly addresses the lack of empirical tools to formally evaluate tangible interaction. It presents a video-coding framework grounded on a systematic literature review of 78 papers, and evaluates its value as metric through a 60 participant study performed across three different research laboratories. The results highlight the usefulness and power of epistemic actions as a performance metric for tangible systems. In sum, through the use of such novel metrics in each of the three studies presented, this dissertation provides a better understanding of the real impact and benefits of designing and developing systems that feature tangible interaction.

An investigation of the effectiveness of existing bridge design methodology in providing adequate structural resistance to seismic disturbances : Phase III, Nonlinear soil-structure interaction of skew highway bridges : final report : prepared for Federal Highway Administration, Offices of Research and Development /

Bibliography: p. 46-48.

Dynamic soil-structure interaction /

Mode of access: Internet.

Formal analysis of human-computer interaction using model checking

Experiments with simulators allow psychologists to better understand the causes of human errors and build models of cognitive processes to be used in human reliability assessment (HRA). This paper investigates an approach to task failure analysis based on patterns of behaviour, by contrast to more traditional event-based approaches. It considers, as a case study, a formal model of an air traffic control (ATC) system which incorporates controller behaviour. The cognitive model is formalised in the CSP process algebra. Patterns of behaviour are expressed as temporal logic properties. Then a model-checking technique is used to verify whether the decomposition of the operator's behaviour into patterns is sound and complete with respect to the cognitive model. The decomposition is shown to be incomplete and a new behavioural pattern is identified, which appears to have been overlooked in the analysis of the data provided by the experiments with the simulator. This illustrates how formal analysis of operator models can yield fresh insights into how failures may arise in interactive systems.

Individual differences in human-computer interaction

This thesis initially presents an 'assay' of the literature pertaining to individual differences in human-computer interaction. A series of experiments is then reported, designed to investigate the association between a variety of individual characteristics and various computer task and interface factors. Predictor variables included age, computer expertise, and psychometric tests of spatial visualisation, spatial memory, logical reasoning, associative memory, and verbal ability. These were studied in relation to a variety of computer-based tacks, including: (1) word processing and its component elements; (ii) the location of target words within passages of text; (iii) the navigation of networks and menus; (iv) command generation using menus and command line interfaces; (v) the search and selection of icons and text labels; (vi) information retrieval. A measure of self-report workload was also included in several of these experiments. The main experimental findings included: (i) an interaction between spatial ability and the manipulation of semantic but not spatial interface content; (ii) verbal ability being only predictive of certain task components of word processing; (iii) age differences in word processing and information retrieval speed but not accuracy; (iv) evidence of compensatory strategies being employed by older subjects; (v) evidence of performance strategy differences which disadvantaged high spatial subjects in conditions of low spatial information content; (vi) interactive effects of associative memory, expertise and command strategy; (vii) an association between logical reasoning and word processing but not information retrieval; (viii) an interaction between expertise and cognitive demand; and (ix) a stronger association between cognitive ability and novice performance than expert performance.

Human-computer interaction and innovation in handheld, mobile and wearable technologies

Handheld and mobile technologies have witnessed significant advances in functionality, leading to their widespread use as both business and social networking tools. Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies reviews concepts relating to the design, development, evaluation, and application of mobile technologies. Studies on mobile user interfaces, mobile learning, and mobile commerce contribute to the growing body of knowledge on this expanding discipline.

Finite element model calibration of an instrumented RC building based on seismic excitation including non-structural components and soil-structure- interaction

Peer reviewed

A New Method for Modeling Free Surface Flows and Fluid-structure Interaction with Ocean Applications

The computational modeling of ocean waves and ocean-faring devices poses numerous challenges. Among these are the need to stably and accurately represent both the fluid-fluid interface between water and air as well as the fluid-structure interfaces arising between solid devices and one or more fluids. As techniques are developed to stably and accurately balance the interactions between fluid and structural solvers at these boundaries, a similarly pressing challenge is the development of algorithms that are massively scalable and capable of performing large-scale three-dimensional simulations on reasonable time scales. This dissertation introduces two separate methods for approaching this problem, with the first focusing on the development of sophisticated fluid-fluid interface representations and the second focusing primarily on scalability and extensibility to higher-order methods.

We begin by introducing the narrow-band gradient-augmented level set method (GALSM) for incompressible multiphase Navier-Stokes flow. This is the first use of the high-order GALSM for a fluid flow application, and its reliability and accuracy in modeling ocean environments is tested extensively. The method demonstrates numerous advantages over the traditional level set method, among these a heightened conservation of fluid volume and the representation of subgrid structures.

Next, we present a finite-volume algorithm for solving the incompressible Euler equations in two and three dimensions in the presence of a flow-driven free surface and a dynamic rigid body. In this development, the chief concerns are efficiency, scalability, and extensibility (to higher-order and truly conservative methods). These priorities informed a number of important choices: The air phase is substituted by a pressure boundary condition in order to greatly reduce the size of the computational domain, a cut-cell finite-volume approach is chosen in order to minimize fluid volume loss and open the door to higher-order methods, and adaptive mesh refinement (AMR) is employed to focus computational effort and make large-scale 3D simulations possible. This algorithm is shown to produce robust and accurate results that are well-suited for the study of ocean waves and the development of wave energy conversion (WEC) devices.

Nonlinear soil-structure interaction for buried pressure pipes under differential ground motion

Pipelines extend thousands of kilometers across wide geographic areas as a network to provide essential services for modern life. It is inevitable that pipelines must pass through unfavorable ground conditions, which are susceptible to natural disasters. This thesis investigates the behaviour of buried pressure pipelines experiencing ground distortions induced by normal faulting. A recent large database of physical modelling observations on buried pipes of different stiffness relative to the surrounding soil subjected to normal faults provided a unique opportunity to calibrate numerical tools. Three-dimensional finite element models were developed to enable the complex soil-structure interaction phenomena to be further understood, especially on the subjects of gap formation beneath the pipe and the trench effect associated with the interaction between backfill and native soils. Benchmarked numerical tools were then used to perform parametric analysis regarding project geometry, backfill material, relative pipe-soil stiffness and pipe diameter. Seismic loading produces a soil displacement profile that can be expressed by isoil, the distance between the peak curvature and the point of contraflexure. A simplified design framework based on this length scale (i.e., the Kappa method) was developed, which features estimates of longitudinal bending moments of buried pipes using a characteristic length, ipipe, the distance from peak to zero curvature. Recent studies indicated that empirical soil springs that were calibrated against rigid pipes are not suitable for analyzing flexible pipes, since they lead to excessive conservatism (for design). A large-scale split-box normal fault simulator was therefore assembled to produce experimental data for flexible PVC pipe responses to a normal fault. Digital image correlation (DIC) was employed to analyze the soil displacement field, and both optical fibres and conventional strain gauges were used to measure pipe strains. A refinement to the Kappa method was introduced to enable the calculation of axial strains as a function of pipe elongation induced by flexure and an approximation of the longitudinal ground deformations. A closed-form Winkler solution of flexural response was also derived to account for the distributed normal fault pattern. Finally, these two analytical solutions were evaluated against the pipe responses observed in the large-scale laboratory tests.

CASIMIRO: a robot head for human-computer interaction

The physical appearance and behavior of a robot is an important asset in terms of Human-Computer Interaction. Multimodality is also fundamental, as we humans usually expect to interact in a natural way with voice, gestures, etc. People approach complex interaction devices with stances similar to those used in their interaction with other people. In this paper we describe a robot head, currently under development, that aims to be a multimodal (vision, voice, gestures,...) perceptual user interface.