Dynamic Human Robot Interaction Framework Using Deep Learning and Robot Operating System (ROS): a practical approach

Trying to explain to a robot what to do is a difficult undertaking, and only specific types of people have been able to do so far, such as programmers or operators who have learned how to use controllers to communicate with a robot. My internship's goal was to create and develop a framework that would make that easier. The system uses deep learning techniques to recognize a set of hand gestures, both static and dynamic. Then, based on the gesture, it sends a command to a robot. To be as generic as feasible, the communication is implemented using Robot Operating System (ROS). Furthermore, users can add new recognizable gestures and link them to new robot actions; a finite state automaton enforces the users' input verification and correct action sequence. Finally, the users can create and utilize a macro to describe a sequence of actions performable by a robot.

La piattaforma ros per lo sviluppo di applicazioni robotiche

Lo sviluppo hardware nel campo della robotica ha raggiunto negli ultimi anni livelli impressionanti ed è in continua crescita, e di pari passo si è espansa l’eterogeneità delle forme che può assumere, dalle tipologie basate su movimento a terra ai droni volanti, fino a forme più sofisticate di robot umanoidi che cercano di emularne il comportamento. Se da un lato ora possiamo disporre di hardware sempre più potente ed efficiente a costi sempre minori, dall’altro programmare il comportamento che un robot deve tenere nelle svariate circostanze in cui può imbattersi, nel poter portare a compimento il proprio obbiettivo, risulta essere sempre più complesso. Dopo una breve introduzione alla robotica e alle difficoltà che deve affrontare e una panoramica sui robot, cosa siano e come siano strutturati, fulcro della tesi sarà l’esposizione delle caratteristiche principali di ROS, Robot Operating System, come piattaforma di sviluppo software nel campo della robotica, e si concluderà con un semplice caso di studio in cui ne verrà messo in mostra concretamente l’utilizzo.

Sviluppo di un pacchetto ROS per il controllo di un cobot addetto alla movimentazione di un pannello di protezione di una macchina automatica

L’Industria 4.0 richiede sempre più tecnologie con un notevole grado di flessibilità, in modo da garantire il più alto livello di integrazione uomo-macchina e macchina- macchina. In quest’ottica, l’avvento della robotica collaborativa, ha agevolato il pro- cesso. I robot collaborativi (cobot) possono essere facilmente installati all’interno del- le linee di assemblaggio/produzione, senza necessità di barriere che vietino l’accesso agli operatori. Tra i tanti compiti a cui possono asservire i cobot, ci sono quelli dedica- ti all’ispezione delle varie macchine (e.g. microfermate di emergenza), in cui di solito, l’apertura del pannello di protezione è affidata ad un cobot montato a bordo di un AGV. È in questo contesto che si inserisce l’elaborato di tesi, volto al controllo in forza di un robot collaborativo per la movimentazione di un pannello di protezione di una mac- china automatica. In particolare, per ragioni logistiche e di ingombro, un simulacro del pannello di protezione è stato realizzato in scala, mentre il controllo real-time del cobot è stato implementato utilizzando ROS (Robot Operating System), piattaforma disponibile in modalità open-source.

Robotics and neurodevelopmental disorder: design and implementation of a system of engagement metrics in human-robot interactions.

In the recent decades, robotics has become firmly embedded in areas such as education, teaching, medicine, psychology and many others. We focus here on social robotics; social robots are designed to interact with people in a natural and interpersonal way, often to achieve positive results in different applications. To interact and cooperate with humans in their daily-life activities, robots should exhibit human-like intelligence. The rapid expansion of social robotics and the existence of various kinds of robots on the market have allowed research groups to carry out multiple experiments. The experiments carried out have led to the collections of various kinds of data, which can be used or processed for psychological studies, and studies in other fields. However, there are no tools available in which data can be stored, processed and shared with other research groups. This thesis proposes the design and implementation of visual tool for organizing dataflows in Human Robot Interaction (HRI).

Development of a robotic system for switchgear cabling

Industrial robots are an inalienable part of modern automated production. Typical applications of robots include welding, painting, (dis)assembly, packaging, labeling, palletizing, pick and place and others. Many of that applications includes object manipulation. If the shape and position of the object are known in advance, it is possible to design the trajectory of the robot’s end-effector to take and place. Such a strategy is applicable for rigid objects and widely used in the manufacturing field. But flexible (deformable) objects can change their shape and position upon contact with the robot’s end-effector or environment. That is the reason why the general approach is unacceptable. It means that the robot can fail to grasp such an object and can’t place it in the desired position. This thesis has addressed the problem of cable manipulation by bilateral robotic setup for the industrial manufacturing of electrical switchgear. The considered solution is based on the idea of tensioned cable. If the cable was grasped by the ends and tensioned, it has a line shape. Since the position of the robot’s end-effectors known, the position of the cable is known as well. Such an approach is capable to place cable in cable ducts of switchgear. The considered solution has been tested experimentally on a real bilateral robotic setup.

Integrating an Emotion Recognition Model for the Flobi System

This thesis investigates if emotional states of users interacting with a virtual robot can be recognized reliably and if specific interaction strategy can change the users’ emotional state and affect users’ risk decision. For this investigation, the OpenFace [1] emotion recognition model was intended to be integrated into the Flobi [2] system, to allow the agent to be aware of the current emotional state of the user and to react appropriately. There was an open source ROS [3] bridge available online to integrate OpenFace to the Flobi simulation but it was not consistent with some other projects in Flobi distribution. Then due to technical reasons DeepFace was selected. In a human-agent interaction, the system is compared to a system without using emotion recognition. Evaluation could happen at different levels: evaluation of emotion recognition model, evaluation of the interaction strategy, and evaluation of effect of interaction on user decision. The results showed that the happy emotion induction was 58% and fear emotion induction 77% successful. Risk decision results show that: in happy induction after interaction 16.6% of participants switched to a lower risk decision and 75% of them did not change their decision and the remaining switched to a higher risk decision. In fear inducted participants 33.3% decreased risk 66.6 % did not change their decision The emotion recognition accuracy was and had bias to. The sensitivity and specificity is calculated for each emotion class. The emotion recognition model classifies happy emotions as neutral in most of the time.

Applicazione ROS 2 di un robot antropomorfo in esecuzione su container VxWorks in architettura IMA

L’elaborato di tesi discute del progetto di integrazione tra ROS 2, framework open-source per lo sviluppo di applicazioni robotiche, e VxWorks, sistema operativo in tempo reale (RTOS), attraverso l’utilizzo di container OCI compliant su VxWorks. L’integrazione è stata svolta all’interno dello stack software di IMA (Industria Macchine Automatiche). Il progetto ha dunque integrato ROS 2 Humble e VxWorks 7 permettendo l’utilizzo di costrutti software di ROS 2 su dei container in esecuzione a livello User su VxWorks. Successivamente è stata creata una applicazione di pick and place con un robot antropomorfo (Universal Robots Ur5e) avvalendosi di ROS 2 Control, framework per l’introduzione e gestione di hardware e controllori, e MoveIt 2, framework per incorporare algoritmi di motion-planning, cinematica, controllo e navigazione. Una volta progettata l’applicazione, il sistema è stato integrato all’interno dell’architettura di controllo di IMA. L’architettura a container VxWorks di IMA è stata estesa per il caso ROS 2, la comunicazione tra campo e applicazione ROS 2 è passata tramite il master EtherCAT e il modulo WebServer presenti nell’architettura IMA. Una volta eseguito il container ROS 2 posizione e velocità dei servo motori sono stati inviati tramite al WebServer di IMA sfruttando la comunicazione VLAN interna. Una volta ricevuto il messaggio, il WebServer si è occupato di trasferirlo al master EtherCAT che in aggiunta si è occupato anche di ottenere le informazioni sullo stato attuale del robot. L’intero progetto è stato sviluppato in prima battuta in ambiente di simulazione per validarne l’architettura. Successivamente si è passati all’installazione in ambiente embedded grazie all’ausilio di IPC sui quali è stato testato l’effettivo funzionamento dell’integrazione all’interno dell’architettura IMA.

Structure learning of graphical models for task-oriented robot grasping

In the collective imaginaries a robot is a human like machine as any androids in science fiction. However the type of robots that you will encounter most frequently are machinery that do work that is too dangerous, boring or onerous. Most of the robots in the world are of this type. They can be found in auto, medical, manufacturing and space industries. Therefore a robot is a system that contains sensors, control systems, manipulators, power supplies and software all working together to perform a task. The development and use of such a system is an active area of research and one of the main problems is the development of interaction skills with the surrounding environment, which include the ability to grasp objects. To perform this task the robot needs to sense the environment and acquire the object informations, physical attributes that may influence a grasp. Humans can solve this grasping problem easily due to their past experiences, that is why many researchers are approaching it from a machine learning perspective finding grasp of an object using information of already known objects. But humans can select the best grasp amongst a vast repertoire not only considering the physical attributes of the object to grasp but even to obtain a certain effect. This is why in our case the study in the area of robot manipulation is focused on grasping and integrating symbolic tasks with data gained through sensors. The learning model is based on Bayesian Network to encode the statistical dependencies between the data collected by the sensors and the symbolic task. This data representation has several advantages. It allows to take into account the uncertainty of the real world, allowing to deal with sensor noise, encodes notion of causality and provides an unified network for learning. Since the network is actually implemented and based on the human expert knowledge, it is very interesting to implement an automated method to learn the structure as in the future more tasks and object features can be introduced and a complex network design based only on human expert knowledge can become unreliable. Since structure learning algorithms presents some weaknesses, the goal of this thesis is to analyze real data used in the network modeled by the human expert, implement a feasible structure learning approach and compare the results with the network designed by the expert in order to possibly enhance it.

Night Vision Imaging System (NVIS) certification requirements analysis of an Airbus Helicopters H135

The safe operation of nighttime flight missions would be enhanced using Night Vision Imaging Systems (NVIS) equipment. This has been clear to the military since 1970s and to the civil helicopters since 1990s. In these last months, even Italian Emergency Medical Service (EMS) operators require Night Vision Goggles (NVG) devices that therefore amplify the ambient light. In order to fly with this technology, helicopters have to be NVIS-approved. The author have supported a company, to quantify the potentiality of undertaking the certification activity, through a feasibility study. Even before, NVG description and working principles have been done, then specifications analysis about the processes to make a helicopter NVIS-approved has been addressed. The noteworthy difference between military specifications and the civilian ones highlights non-irrevelant lacks in the latter. The activity of NVIS certification could be a good investment because the following targets have been achieved: Reductions of the certification cost, of the operating time and of the number of non-compliance.

Un’applicazione di elaborazione del linguaggio naturale per il gioco di "Indovina chi?" su robot umanoide

La tesi è stata incentrata sul gioco «Indovina chi?» per l’identificazione da parte del robot Nao di un personaggio tramite la sua descrizione. In particolare la descrizione avviene tramite domande e risposte L’obiettivo della tesi è la progettazione di un sistema in grado di capire ed elaborare dei dati comunicati usando un sottoinsieme del linguaggio naturale, estrapolarne le informazioni chiave e ottenere un riscontro con informazioni date in precedenza. Si è quindi programmato il robot Nao in modo che sia in grado di giocare una partita di «Indovina chi?» contro un umano comunicando tramite il linguaggio naturale. Sono state implementate regole di estrazione e categorizzazione per la comprensione del testo utilizzando Cogito, una tecnologia brevettata dall'azienda Expert System. In questo modo il robot è in grado di capire le risposte e rispondere alle domande formulate dall'umano mediante il linguaggio naturale. Per il riconoscimento vocale è stata utilizzata l'API di Google e PyAudio per l'utilizzo del microfono. Il programma è stato implementato in Python e i dati dei personaggi sono memorizzati in un database che viene interrogato e modificato dal robot. L'algoritmo del gioco si basa su calcoli probabilistici di vittoria del robot e sulla scelta delle domande da proporre in base alle risposte precedentemente ricevute dall'umano. Le regole semantiche realizzate danno la possibilità al giocatore di formulare frasi utilizzando il linguaggio naturale, inoltre il robot è in grado di distinguere le informazioni che riguardano il personaggio da indovinare senza farsi ingannare. La percentuale di vittoria del robot ottenuta giocando 20 partite è stata del 50%. Il data base è stato sviluppato in modo da poter realizzare un identikit completo di una persona, oltre a quello dei personaggi del gioco. È quindi possibile ampliare il progetto per altri scopi, oltre a quello del gioco, nel campo dell'identificazione.

A ROS-based Workspace Control and Trajectory Planner for a seven degrees of freedom Robotic arm

In this Bachelor Thesis I want to provide readers with tools and scripts for the control of a 7DOF manipulator, backed up by some theory of Robotics and Computer Science, in order to better contextualize the work done. In practice, we will see most common software, and developing environments, used to cope with our task: these include ROS, along with visual simulation by VREP and RVIZ, and an almost "stand-alone" ROS extension called MoveIt!, a very complete programming interface for trajectory planning and obstacle avoidance. As we will better appreciate and understand in the introduction chapter, the capability of detecting collision objects through a camera sensor, and re-plan to the desired end-effector pose, are not enough. In fact, this work is implemented in a more complex system, where recognition of particular objects is needed. Through a package of ROS and customized scripts, a detailed procedure will be provided on how to distinguish a particular object, retrieve its reference frame with respect to a known one, and then allow navigation to that target. Together with technical details, the aim is also to report working scripts and a specific appendix (A) you can refer to, if desiring to put things together.

Feasibility study and flow scheduling optimization of a multi-product pipeline system

The first goal of this study is to analyse a real-world multiproduct onshore pipeline system in order to verify its hydraulic configuration and operational feasibility by constructing a simulation model step by step from its elementary building blocks that permits to copy the operation of the real system as precisely as possible. The second goal is to develop this simulation model into a user-friendly tool that one could use to find an “optimal” or “best” product batch schedule for a one year time period. Such a batch schedule could change dynamically as perturbations occur during operation that influence the behaviour of the entire system. The result of the simulation, the ‘best’ batch schedule is the one that minimizes the operational costs in the system. The costs involved in the simulation are inventory costs, interface costs, pumping costs, and penalty costs assigned to any unforeseen situations. The key factor to determine the performance of the simulation model is the way time is represented. In our model an event based discrete time representation is selected as most appropriate for our purposes. This means that the time horizon is divided into intervals of unequal lengths based on events that change the state of the system. These events are the arrival/departure of the tanker ships, the openings and closures of loading/unloading valves of storage tanks at both terminals, and the arrivals/departures of trains/trucks at the Delivery Terminal. In the feasibility study we analyse the system’s operational performance with different Head Terminal storage capacity configurations. For these alternative configurations we evaluated the effect of different tanker ship delay magnitudes on the number of critical events and product interfaces generated, on the duration of pipeline stoppages, the satisfaction of the product demand and on the operative costs. Based on the results and the bottlenecks identified, we propose modifications in the original setup.

Real-time PMU monitoring of an IEEE 5-Bus Network implemented on OPAL system

This thesis studies the state-of-the-art of phasor measurement units (PMUs) as well as their metrological requirements stated in the IEEE C37.118.1 and C37.118.2 Standards for guaranteeing correct measurement performances. Communication systems among PMUs and their possible applicability in the field of power quality (PQ) assessment are also investigated. This preliminary study is followed by an analysis of the working principle of real-time (RT) simulators and the importance of hardware-in-the-loop (HIL) implementation, examining the possible case studies specific for PMUs, including compliance tests which are one of the most important parts. The core of the thesis is focused on the implementation of a PMU model in the IEEE 5-bus network in Simulink and in the validation of the results using OPAL RT-4510 as a real-time simulator. An initial check allows one to get an idea about the goodness of the results in Simulink, comparing the PMU data with respect to the load-flow steady-state information. In this part, accuracy indices are also calculated for both voltage and current synchrophasors. The following part consists in the implementation of the same code in OPAL-RT 4510 simulator, after which an initial analysis is carried out in a qualitative way in order to get a sense of the goodness of the outcomes. Finally, the confirmation of the results is based on an examination of the attained voltage and current synchrophasors and accuracy indices coming from Simulink models and from OPAL system, using a Matlab script. This work also proposes suggestions for an upcoming operation of PMUs in a more complex system as the Digital Twin (DT) in order to improve the performances of the already-existing protection devices of the distribution system operator (DSO) for a future enhancement of power systems reliability.

Rapid Control Prototyping of an L2+ Highway Assist System

Advanced Driver Assistance Systems (ADAS) are proving to have huge potential in road safety, comfort, and efficiency. In recent years, car manufacturers have equipped their high-end vehicles with Level 2 ADAS, which are, according to SAE International, systems that combine both longitudinal and lateral active motion control. These automated driving features, while only available in highway scenarios, appear to be very promising towards the introduction of hands-free driving. However, as they rely only on an on-board sensor suite, their continuative operation may be affected by the current environmental conditions: this prevents certain functionalities such as the automated lane change, other than requiring the driver to keep constantly the hands on the steering wheel. The enabling factor for hands-free highway driving proposed by Mobileye is the integration of high-definition maps, thus leading to the so-called Level 2+. This thesis was carried out during an internship in Maserati's Virtual Engineering team. The activity consisted of the design of an L2+ Highway Assist System following the Rapid Control Prototyping approach, starting from the definition of the requirements up to the real-time implementation and testing on a simulator of the brand new compact SUV Maserati Grecale. The objective was to enhance the current Level 2 highway driving assistance system with hands-free driving capability; for this purpose an Autonomous Lane Change functionality has been designed, proposing a Model Predictive Control-based decision-maker, in charge of assessing both the feasibility and convenience of performing a lane-change maneuver. The result is a Highway Assist System capable of driving the vehicle in a traffic scenario safely and efficiently, never requiring driver intervention.

Integration of the computer controlled screwdriver for Robotic manufacturing in the ROS environment

The computer controlled screwdriver is a modern technique to perform automatic screwing/unscrewing operations.The main focus is to study the integration of the computer controlled screwdriver for Robotic manufacturing in the ROS environment.This thesis describes a concept of automatic screwing mechanism composed by universal robots, in which one arm of the robot is for inserting cables and the other is for screwing the cables on the control panel switch gear box. So far this mechanism is carried out by human operators and is a fairly complex one to perform, due to the multiple cables and connections involved. It's for this reason that an automatic cabling and screwing process would be highly preferred within automotive/automation industries. A study is carried out to analyze the difficulties currently faced and a controller based algorithm is developed to replace the manual human efforts using universal robots, thereby allowing robot arms to insert the cables and screw them onto the control panel switch gear box. Experiments were conducted to evaluate the insertion and screwing strategy, which shows the result of inserting and screwing cables on the control panel switch gearbox precisely.