Hand gesture recognition system based in computer vision and machine learning: Applications on human-machine interaction

Tese de Doutoramento em Engenharia de Eletrónica e de Computadores

Generic system for human-computer gesture interaction: applications on sign language recognition and robotic soccer refereeing

Hand gestures are a powerful way for human communication, with lots of potential applications in the area of human computer interaction. Vision-based hand gesture recognition techniques have many proven advantages compared with traditional devices, giving users a simpler and more natural way to communicate with electronic devices. This work proposes a generic system architecture based in computer vision and machine learning, able to be used with any interface for human-computer interaction. The proposed solution is mainly composed of three modules: a pre-processing and hand segmentation module, a static gesture interface module and a dynamic gesture interface module. The experiments showed that the core of visionbased interaction systems could be the same for all applications and thus facilitate the implementation. For hand posture recognition, a SVM (Support Vector Machine) model was trained and used, able to achieve a final accuracy of 99.4%. For dynamic gestures, an HMM (Hidden Markov Model) model was trained for each gesture that the system could recognize with a final average accuracy of 93.7%. The proposed solution as the advantage of being generic enough with the trained models able to work in real-time, allowing its application in a wide range of human-machine applications. To validate the proposed framework two applications were implemented. The first one is a real-time system able to interpret the Portuguese Sign Language. The second one is an online system able to help a robotic soccer game referee judge a game in real time.

Vision-based portuguese sign language recognition system

Vision-based hand gesture recognition is an area of active current research in computer vision and machine learning. Being a natural way of human interaction, it is an area where many researchers are working on, with the goal of making human computer interaction (HCI) easier and natural, without the need for any extra devices. So, the primary goal of gesture recognition research is to create systems, which can identify specific human gestures and use them, for example, to convey information. For that, vision-based hand gesture interfaces require fast and extremely robust hand detection, and gesture recognition in real time. Hand gestures are a powerful human communication modality with lots of potential applications and in this context we have sign language recognition, the communication method of deaf people. Sign lan- guages are not standard and universal and the grammars differ from country to coun- try. In this paper, a real-time system able to interpret the Portuguese Sign Language is presented and described. Experiments showed that the system was able to reliably recognize the vowels in real-time, with an accuracy of 99.4% with one dataset of fea- tures and an accuracy of 99.6% with a second dataset of features. Although the im- plemented solution was only trained to recognize the vowels, it is easily extended to recognize the rest of the alphabet, being a solid foundation for the development of any vision-based sign language recognition user interface system.

IEA EBC Annex56 vision for cost effective energy and carbon emissions optimization in building renovation

Building sector has become an important target for carbon emissions reduction, energy consumption and resources depletion. Due to low rates of replacement of the existing buildings, their low energy performances are a major concern. Most of the current regulations are focused on new buildings and do not account with the several technical, functional and economic constraints that have to be faced in the renovation of existing buildings. Thus, a new methodology is proposed to be used in the decision making process for energy related building renovation, allowing finding a cost-effective balance between energy consumption, carbon emissions and overall added value.

A demonstration case on the derivation of process-level logical architectures for ambient assisted living ecosystems

When representing the requirements for an intended software solution during the development process, a logical architecture is a model that provides an organized vision of how functionalities behave regardless of the technologies to be implemented. If the logical architecture represents an ambient assisted living (AAL) ecosystem, such representation is a complex task due to the existence of interrelated multidomains, which, most of the time, results in incomplete and incoherent user requirements. In this chap- ter, we present the results obtained when applying process-level modeling techniques to the derivation of the logical architecture for a real industrial AAL project. We adopt a V-Model–based approach that expresses the AAL requirements in a process-level perspec- tive, instead of the traditional product-level view. Additionally, we ensure compliance of the derived logical architecture with the National Institute of Standards and Technology (NIST) reference architecture as nonfunctional requirements to support the implementa- tion of the AAL architecture in cloud contexts.

Influence of climate on clinical diagnostic dry eye tests: pilot study

Purpose. To analyze dry eye disease (DED) tests and their consistency in similar nonsymptomatic population samples living in two geographic locations with different climates (Continental vs. Atlantic). Methods. This is a pilot study including 14 nonsymptomatic residents from Valladolid (Continental climate, Spain) and 14 sex-matched and similarly aged residents from Braga (Atlantic climate, Portugal); they were assessed during the same season (spring) of two consecutive years. Phenol red thread test, conjunctival hyperemia, fluorescein tear breakup time, corneal and conjunctival staining, and Schirmer test were evaluated on three different consecutive visits. Reliability was assessed using the intraclass correlation coefficient and weighted kappa (J) coefficient for quantitative and ordinal variables, respectively. Results. Fourteen subjects were recruited in each city with a mean (TSD) age of 63.0 (T1.7) and 59.1 (T0.9) years (p = 0.08) in Valladolid and Braga, respectively. Intraclass correlation coefficient and J values of the tests performed were below 0.69 and 0.61, respectively, for both samples, thus showing moderate to poor reliability. Subsequently, comparisons were made between the results corresponding to the middle and higher outdoor relative humidity (RH) visit in each location as there were no differences in mean temperature (p Q 0.75) despite RH values significantly differing (p e 0.005). Significant (p e 0.05) differences were observed between Valladolid and Braga samples on tear breakup time (middle RH visit, 2.76 T 0.60 vs. 5.26 T 0.64 seconds; higher RH visit, 2.61 T 0.32 vs. 5.78 T 0.88 seconds) and corneal (middle RH, 0.64 T 0.17 vs. 0.14 T 0.10; higher RH, 0.60 T 0.22 vs. 0.0 T 0.0) and conjunctival staining (middle RH, 0.61 T 0.17 vs. 0.14 T 0.08; higher RH, 0.57 T 0.15 vs. 0.18 T 0.09). Conclusions. This pilot study provides initial evidence to support that DED test outcomes assessing the ocular surface integrity and tear stability are climate dependent. Future large-sample studies should support these outcomes also in DED patients. This knowledge is fundamental for multicenter clinical trials. Lack of consistency in diagnostic clinical tests for DED was also corroborated. (Optom Vis Sci 2015;92:e284Ye289)

Three-dimensional technology facilitates surgical performance of novice laparoscopy surgeons: a quantitative assessment on a porcine kidney model

Objective To determine whether the use of 3-dimensional (3D) imaging translates into a better surgical performance of naïve urologic laparoscopic surgeons during pyeloplasty (PY) and partial nephrectomy (PN) procedures. Materials and Methods Eighteen surgeons without any previous laparoscopic experience were randomly assigned to perform PY and PN in a porcine model using initially 2-dimensional (2D) and 3D laparoscopy. A surgical performance score was rated by an "expert" tutor through a modified 5-item global rating scale contemplating operative field view, bimanual dexterity, efficiency, tissue handling, and autonomy. Overall surgical time, complications, subjective perception of participating surgeons, and inconveniences related to the 3D vision were recorded. Results No difference in terms if operative time was found between 2D or 3D laparoscopy for both the PY (P =.51) and the PN (P =.28) procedures. A better rate in terms of surgical performance score was noted by the tutors when the study participants were using 3D vs 2D, for both PY (3.6 [0.8] vs 3.0 [0.4]; P =.034) and PN (3.6 [0.51] vs 3.15 [0.63]; P =.001). No complications occurred in any of the procedures. Most (77.2%) of the participating na??ve laparoscopic surgeons had the perception that 3D laparoscopy was overall easier than 2D. Headache (18.1%), nausea (18.1%), and visual disturbance (18.1%) were the most common issues reported by the surgeons during 3D procedures. Conclusion Despite the absence of translation in a shorter operative time, the use of 3D technology seems to facilitate the surgical performance of naive surgeons during laparoscopic kidney procedures on a porcine model.