Hand gesture recognition system based in computer vision and machine learning

"Lecture notes in computational vision and biomechanics series, ISSN 2212-9391, vol. 19"

Hand gesture recognition system based in computer vision and machine learning: Applications on human-machine interaction

Tese de Doutoramento em Engenharia de Eletrónica e de Computadores

Hand gesture recognition for human computer interaction : a comparative study of different image features

Hand gesture recognition for human computer interaction, being a natural way of human computer interaction, is an area of active research in computer vision and machine learning. This is an area with many different possible applications, giving users a simpler and more natural way to communicate with robots/systems interfaces, without the need for extra devices. So, the primary goal of gesture recognition research is to create systems, which can identify specific human gestures and use them to convey information or for device control. For that, vision-based hand gesture interfaces require fast and extremely robust hand detection, and gesture recognition in real time. In this study we try to identify hand features that, isolated, respond better in various situations in human-computer interaction. The extracted features are used to train a set of classifiers with the help of RapidMiner in order to find the best learner. A dataset with our own gesture vocabulary consisted of 10 gestures, recorded from 20 users was created for later processing. Experimental results show that the radial signature and the centroid distance are the features that when used separately obtain better results, with an accuracy of 91% and 90,1% respectively obtained with a Neural Network classifier. These to methods have also the advantage of being simple in terms of computational complexity, which make them good candidates for real-time hand gesture recognition.

Generic system for human-computer gesture interaction

Hand gestures are a powerful way for human communication, with lots of potential applications in the area of human computer interaction. Vision-based hand gesture recognition techniques have many proven advantages compared with traditional devices, giving users a simpler and more natural way to communicate with electronic devices. This work proposes a generic system architecture based in computer vision and machine learning, able to be used with any interface for humancomputer interaction. The proposed solution is mainly composed of three modules: a pre-processing and hand segmentation module, a static gesture interface module and a dynamic gesture interface module. The experiments showed that the core of vision-based interaction systems can be the same for all applications and thus facilitate the implementation. In order to test the proposed solutions, three prototypes were implemented. For hand posture recognition, a SVM model was trained and used, able to achieve a final accuracy of 99.4%. For dynamic gestures, an HMM model was trained for each gesture that the system could recognize with a final average accuracy of 93.7%. The proposed solution as the advantage of being generic enough with the trained models able to work in real-time, allowing its application in a wide range of human-machine applications.

Modulation of the secretome of hBMSCs by tailoring the macromolecular gradient in hydrogels to generate tissue-to-tissue interfaces

Tissue-to-tissue interfaces are commonly present in all tissues exhibiting structural, biological and chemical gradients serving a wide range of physiological functions. These interfaces are responsible for mediation of load transfer between two adjacent tissues. They are also important structures in sustaining the cellular communications to retain tissueâ s functional integration and homeostasis. [1] All cells have the capacity to sense and respond to physical and chemical stimulus and when cultured in three-dimensional (3D) environments they tend to perform their function better than in two-dimensional (2D) environments. Spatial and temporal 3D gradient hydrogels better resemble the natural environment of cells in mimicking their extracellular matrix. [2] In this study we hypothesize that differential functional properties can be engineered by modulation of macromolecule gradients in a cell seeded threedimensional hydrogel system. Specifically, differential paracrine secretory profiles can be engineered using human Bone Marrow Stem Cells (hBMSCâ s). Hence, the specific objectives of this study are to: assemble the macromolecular gradient hydrogels to evaluate the suitablity for hBMSCâ s encapsulation by cellular viability and biofunctionality by assessing the paracrine secretion of hBMSCâ s over time. The gradient hydrogels solutions were prepared by blend of macromolecules in one solution such as hyaluronic (HA) acid and collagen (Col) at different ratios. The gradient hydrogels were fabricated into cylindrical silicon moulds with higher ratio solutions assembled at the bottom of the mould and adding the two solutions consecutively on top of each other. The labelling of the macromolecules was performed to confirm the gradient through fluorescence microscopy. Additionally, AFM was conducted to assess the gradient hydrogels stiffness. Gradient hydrogels characterization was performed by HA and Col degradation assay, degree of crosslinking and stability. hBMSCâ s at P3 were encapsulated into each batch solution at 106 cells/ml solution and gradient hydrogels were produced as previously described. The hBMSCâ s were observed under confocal microscopy to assess viability by Live/Dead® staining. Cellular behaviour concerning proliferation and matrix deposition was also performed. Secretory cytokine measurement for pro-inflammatory and angiogenesis factors was carried out using ELISA. At genomic level, qPCR was carried out. The 3D gradient hydrogels platform made of different macromolecules showed to be a suitable environment for hBMSCâ s. The hBMSCâ s gradient hydrogels supported high cell survival and exhibited biofunctionality. Besides, the 3D gradient hydrogels demonstrated differentially secretion of pro-inflammatory and angiogenic factors by the encapsulated hBMSCâ s. References: 1. Mikos, AG. et al., Engineering complex tissues. Tissue Engineering 12,3307, 2006 2. Phillips, JE. et al., Proc Natl Acad Sci USA, 26:12170-5, 2008

Experimental analysis of new interfaces for connections by adhesion, interlocking and friction.

The authors thank the federal agency CAPES and the Foundation for Research Support of the state of Sao Paulo, Brazil (FAPESP) for providing a PhD scholarship, and the University of Minho, in Portugal, for the international collaboration.

Generic system for human-computer gesture interaction: applications on sign language recognition and robotic soccer refereeing

Hand gestures are a powerful way for human communication, with lots of potential applications in the area of human computer interaction. Vision-based hand gesture recognition techniques have many proven advantages compared with traditional devices, giving users a simpler and more natural way to communicate with electronic devices. This work proposes a generic system architecture based in computer vision and machine learning, able to be used with any interface for human-computer interaction. The proposed solution is mainly composed of three modules: a pre-processing and hand segmentation module, a static gesture interface module and a dynamic gesture interface module. The experiments showed that the core of visionbased interaction systems could be the same for all applications and thus facilitate the implementation. For hand posture recognition, a SVM (Support Vector Machine) model was trained and used, able to achieve a final accuracy of 99.4%. For dynamic gestures, an HMM (Hidden Markov Model) model was trained for each gesture that the system could recognize with a final average accuracy of 93.7%. The proposed solution as the advantage of being generic enough with the trained models able to work in real-time, allowing its application in a wide range of human-machine applications. To validate the proposed framework two applications were implemented. The first one is a real-time system able to interpret the Portuguese Sign Language. The second one is an online system able to help a robotic soccer game referee judge a game in real time.

Vision-based portuguese sign language recognition system

Vision-based hand gesture recognition is an area of active current research in computer vision and machine learning. Being a natural way of human interaction, it is an area where many researchers are working on, with the goal of making human computer interaction (HCI) easier and natural, without the need for any extra devices. So, the primary goal of gesture recognition research is to create systems, which can identify specific human gestures and use them, for example, to convey information. For that, vision-based hand gesture interfaces require fast and extremely robust hand detection, and gesture recognition in real time. Hand gestures are a powerful human communication modality with lots of potential applications and in this context we have sign language recognition, the communication method of deaf people. Sign lan- guages are not standard and universal and the grammars differ from country to coun- try. In this paper, a real-time system able to interpret the Portuguese Sign Language is presented and described. Experiments showed that the system was able to reliably recognize the vowels in real-time, with an accuracy of 99.4% with one dataset of fea- tures and an accuracy of 99.6% with a second dataset of features. Although the im- plemented solution was only trained to recognize the vowels, it is easily extended to recognize the rest of the alphabet, being a solid foundation for the development of any vision-based sign language recognition user interface system.

Dynamic group formation in mobile computer supported collaborative learning environment

Forming suitable learning groups is one of the factors that determine the efficiency of collaborative learning activities. However, only a few studies were carried out to address this problem in the mobile learning environments. In this paper, we propose a new approach for an automatic, customized, and dynamic group formation in Mobile Computer Supported Collaborative Learning (MCSCL) contexts. The proposed solution is based on the combination of three types of grouping criteria: learner’s personal characteristics, learner’s behaviours, and context information. The instructors can freely select the type, the number, and the weight of grouping criteria, together with other settings such as the number, the size, and the type of learning groups (homogeneous or heterogeneous). Apart from a grouping mechanism, the proposed approach represents a flexible tool to control each learner, and to manage the learning processes from the beginning to the end of collaborative learning activities. In order to evaluate the quality of the implemented group formation algorithm, we compare its Average Intra-cluster Distance (AID) with the one of a random group formation method. The results show a higher effectiveness of the proposed algorithm in forming homogenous and heterogeneous groups compared to the random method.

Digital manipulatives as scaffolds for preschoolers’ language development

The study reported here aims at contributing to a deeper understanding of the educational possibilities offered by digital manipulatives in preschool contexts. It presents a study carried with a digital manipulative to enhance the development of lexical knowledge and language awareness, which are relevant language abilities for formal literacy learning. The study took place in a Portuguese preschool, with a class of 20 five-year-olds in collaboration with the teacher. The digital manipulative supported the construction of multiple fictional worlds, motivating children's verbal interactions, and the playing of words and sound games, thus contextualizing the learning of an extensive collection of vocabulary and language awareness abilities. The degree of engagement and involvement that the manipulative provided in supporting children’s imaginative play as well as the imitation, in their own play, of the playful pedagogical interventions that the teacher had designed, shows the importance of well- designed materials that support a child-centered learning model. As such, it sustains a discussion on the potential of digital manipulatives to enhance fundamental language development in the preschool years. Further, the study highlights the importance of multidisciplinary teams in the creation of innovative pedagogical materials.

t-­stories: improving and expanding t-­words

versão acessível em http://ace2015.info/wp-content/uploads/2015/11/ACE_2015_submission_148.pdf

From the periphery to the brain: Lipocalin-2, a friend or foe?

Lipocalin-2 (LCN2) is an acute-phase protein that, by binding to iron-loaded siderophores, acts as a potent bacteriostatic agent in the iron-depletion strategy of the immune system to control pathogens. The recent identification of a mammalian siderophore also suggests a physiological role for LCN2 in iron homeostasis, specifically in iron delivery to cells via a transferrin-independent mechanism. LCN2 participates, as well, in a variety of cellular processes, including cell proliferation, cell differentiation and apoptosis, and has been mostly found up-regulated in various tissues and under inflammatory states, being its expression regulated by several inducers. In the central nervous system less is known about the processes involving LCN2, namely by which cells it is produced/secreted, and its impact on cell proliferation and death, or in neuronal plasticity and behaviour. Importantly, LCN2 recently emerged as a potential clinical biomarker in multiple sclerosis and in ageing-related cognitive decline. Still, there are conflicting views on the role of LCN2 in pathophysiological processes, with some studies pointing to its neurodeleterious effects, while others indicate neuroprotection. Herein, these various perspectives are reviewed and a comprehensive and cohesive view of the general function of LCN2, particularly in the brain, is provided.

Record, play, go…

This workshop aims at stimulating children’s oral language skills by involving them on playing and creating different language games or activities using the t-stories interface. The interface allows recording and playing audio on the stories’ modules, as well as recording and playing based on identification with NFC tags that can be used as sticker on objects, paper, or other materials and placed in different locations. After the presentation of t-stories by the workshop facilitators, children will have the opportunity to explore the interface on their own, then they will be asked to participate in different language games, whereby they actively create their own content. Afterwards children will be challenged to imagine and create activities for their peers.

Assessing the application of polypropylene multifilament yarns in brain phantoms

The currently available clinical imaging methods do not provide highly detailed information about location and severity of axonal injury or the expected recovery time of patients with traumatic brain injury [1]. High-Definition Fiber Tractography (HDFT) is a novel imaging modality that allows visualizing and quantifying, directly, the degree of axons damage, predicting functional deficits due to traumatic axonal injury and loss of cortical projections. This imaging modality is based on diffusion technology [2]. The inexistence of a phantom able to mimic properly the human brain hinders the possibility of testing, calibrating and validating these medical imaging techniques. Most research done in this area fails in key points, such as the size limit reproduced of the brain fibers and the quick and easy reproducibility of phantoms [3]. For that reason, it is necessary to develop similar structures matching the micron scale of axon tubes. Flexible textiles can play an important role since they allow producing controlled packing densities and crossing structures that match closely the human crossing patterns of the brain. To build a brain phantom, several parameters must be taken into account in what concerns to the materials selection, like hydrophobicity, density and fiber diameter, since these factors influence directly the values of fractional anisotropy. Fiber cross-section shape is other important parameter. Earlier studies showed that synthetic fibrous materials are a good choice for building a brain phantom [4]. The present work is integrated in a broader project that aims to develop a brain phantom made by fibrous materials to validate and calibrate HDFT. Due to the similarity between thousands of hollow multifilaments in a fibrous arrangement, like a yarn, and the axons, low twist polypropylene multifilament yarns were selected for this development. In this sense, extruded hollow filaments were analysed in scanning electron microscope to characterize their main dimensions and shape. In order to approximate the dimensional scale to human axons, five types of polypropylene yarns with different linear density (denier) were used, aiming to understand the effect of linear density on the filament inner and outer areas. Moreover, in order to achieve the required dimensions, the polypropylene filaments cross-section was diminished in a drawing stage of a filament extrusion line. Subsequently, tensile tests were performed to characterize the mechanical behaviour of hollow filaments and to evaluate the differences between stretched and non-stretched filaments. In general, an increase of the linear density causes the increase in the size of the filament cross section. With the increase of structure orientation of filaments, induced by stretching, breaking tenacity increases and elongation at break decreases. The production of hollow fibers, with the required characteristics, is one of the key steps to create a brain phantom that properly mimics the human brain that may be used for the validation and calibration of HDFT, an imaging approach that is expected to contribute significantly to the areas of brain related research.