Hand gesture recognition system based in computer vision and machine learning

"Lecture notes in computational vision and biomechanics series, ISSN 2212-9391, vol. 19"

Generic system for human-computer gesture interaction: applications on sign language recognition and robotic soccer refereeing

Hand gestures are a powerful way for human communication, with lots of potential applications in the area of human computer interaction. Vision-based hand gesture recognition techniques have many proven advantages compared with traditional devices, giving users a simpler and more natural way to communicate with electronic devices. This work proposes a generic system architecture based in computer vision and machine learning, able to be used with any interface for human-computer interaction. The proposed solution is mainly composed of three modules: a pre-processing and hand segmentation module, a static gesture interface module and a dynamic gesture interface module. The experiments showed that the core of visionbased interaction systems could be the same for all applications and thus facilitate the implementation. For hand posture recognition, a SVM (Support Vector Machine) model was trained and used, able to achieve a final accuracy of 99.4%. For dynamic gestures, an HMM (Hidden Markov Model) model was trained for each gesture that the system could recognize with a final average accuracy of 93.7%. The proposed solution as the advantage of being generic enough with the trained models able to work in real-time, allowing its application in a wide range of human-machine applications. To validate the proposed framework two applications were implemented. The first one is a real-time system able to interpret the Portuguese Sign Language. The second one is an online system able to help a robotic soccer game referee judge a game in real time.

Vision-based portuguese sign language recognition system

Vision-based hand gesture recognition is an area of active current research in computer vision and machine learning. Being a natural way of human interaction, it is an area where many researchers are working on, with the goal of making human computer interaction (HCI) easier and natural, without the need for any extra devices. So, the primary goal of gesture recognition research is to create systems, which can identify specific human gestures and use them, for example, to convey information. For that, vision-based hand gesture interfaces require fast and extremely robust hand detection, and gesture recognition in real time. Hand gestures are a powerful human communication modality with lots of potential applications and in this context we have sign language recognition, the communication method of deaf people. Sign lan- guages are not standard and universal and the grammars differ from country to coun- try. In this paper, a real-time system able to interpret the Portuguese Sign Language is presented and described. Experiments showed that the system was able to reliably recognize the vowels in real-time, with an accuracy of 99.4% with one dataset of fea- tures and an accuracy of 99.6% with a second dataset of features. Although the im- plemented solution was only trained to recognize the vowels, it is easily extended to recognize the rest of the alphabet, being a solid foundation for the development of any vision-based sign language recognition user interface system.

Discrimination of Brazilian propolis according to the seasoning using chemometrics and machine learning based on UV-Vis scanning data

Propolis is a chemically complex biomass produced by honeybees (Apis mellifera) from plant resins added of salivary enzymes, beeswax, and pollen. The biological activities described for propolis were also identified for donor plants resin, but a big challenge for the standardization of the chemical composition and biological effects of propolis remains on a better understanding of the influence of seasonality on the chemical constituents of that raw material. Since propolis quality depends, among other variables, on the local flora which is strongly influenced by (a)biotic factors over the seasons, to unravel the harvest season effect on the propolis chemical profile is an issue of recognized importance. For that, fast, cheap, and robust analytical techniques seem to be the best choice for large scale quality control processes in the most demanding markets, e.g., human health applications. For that, UV-Visible (UV-Vis) scanning spectrophotometry of hydroalcoholic extracts (HE) of seventy-three propolis samples, collected over the seasons in 2014 (summer, spring, autumn, and winter) and 2015 (summer and autumn) in Southern Brazil was adopted. Further machine learning and chemometrics techniques were applied to the UV-Vis dataset aiming to gain insights as to the seasonality effect on the claimed chemical heterogeneity of propolis samples determined by changes in the flora of the geographic region under study. Descriptive and classification models were built following a chemometric approach, i.e. principal component analysis (PCA) and hierarchical clustering analysis (HCA) supported by scripts written in the R language. The UV-Vis profiles associated with chemometric analysis allowed identifying a typical pattern in propolis samples collected in the summer. Importantly, the discrimination based on PCA could be improved by using the dataset of the fingerprint region of phenolic compounds ( = 280-400m), suggesting that besides the biological activities of those secondary metabolites, they also play a relevant role for the discrimination and classification of that complex matrix through bioinformatics tools. Finally, a series of machine learning approaches, e.g., partial least square-discriminant analysis (PLS-DA), k-Nearest Neighbors (kNN), and Decision Trees showed to be complementary to PCA and HCA, allowing to obtain relevant information as to the sample discrimination.

Digital manipulatives as scaffolds for preschoolers’ language development

The study reported here aims at contributing to a deeper understanding of the educational possibilities offered by digital manipulatives in preschool contexts. It presents a study carried with a digital manipulative to enhance the development of lexical knowledge and language awareness, which are relevant language abilities for formal literacy learning. The study took place in a Portuguese preschool, with a class of 20 five-year-olds in collaboration with the teacher. The digital manipulative supported the construction of multiple fictional worlds, motivating children's verbal interactions, and the playing of words and sound games, thus contextualizing the learning of an extensive collection of vocabulary and language awareness abilities. The degree of engagement and involvement that the manipulative provided in supporting children’s imaginative play as well as the imitation, in their own play, of the playful pedagogical interventions that the teacher had designed, shows the importance of well- designed materials that support a child-centered learning model. As such, it sustains a discussion on the potential of digital manipulatives to enhance fundamental language development in the preschool years. Further, the study highlights the importance of multidisciplinary teams in the creation of innovative pedagogical materials.

Hand gesture recognition system based in computer vision and machine learning: Applications on human-machine interaction

Tese de Doutoramento em Engenharia de Eletrónica e de Computadores

What your baby says about learning to talk

This leaflet was developed for parents and carers of babies aged up to one year old who need help learning to talk.

What your toddler says about learning to talk

This leaflet was developed for parents and carers of babies aged between two and three years old who need help learning to talk.

Analysis and evaluation of techniques for the extraction of classes in the ontology learning process

This paper analyzes and evaluates, in the context of Ontology learning, some techniques to identify and extract candidate terms to classes of a taxonomy. Besides, this work points out some inconsistencies that may be occurring in the preprocessing of text corpus, and proposes techniques to obtain good terms candidate to classes of a taxonomy.

Reusing IMS-LD formalized best practices in collaborative learning structuring

Designs of CSCL (Computer Supported Collaborative Learning)activities should be flexible, effective and customizable toparticular learning situations. On the other hand, structureddesigns aim to create favourable conditions for learning. Thus,this paper proposes the collection of representative and broadlyaccepted (best practices) structuring techniques in collaborative learning. With the aim of establishing a conceptual common ground among collaborative learning practitioners and softwaredevelopers, and reusing the expertise that best practicesrepresent, the paper also proposes the formulation of these techniques as patterns: the so-called CLFPs (CollaborativeLearning Flow Patterns). To formalize these patterns, we havechosen the educational modelling language IMS Learning Design (IMS-LD). IMS-LD has the capability to specify many of the collaborative characteristics of the CLFPs. Nevertheless, the language bears limited capability for describing the services that mediate interactions within a learning activity and the specification of temporal or rotated roles. This analysis isdiscussed in the paper, as well as our approaches towards thedevelopment of a system capable of integrating tools using IMSLDscripts and a CLFP-based Learning Design authoring tool.

From a pattern-language to a pattern ontology approach for CSCL script design

Collaborative activities, in which students actively interact with each other, have proved to provide significant learning benefits. In Computer-Supported Collaborative Learning (CSCL), these collaborative activities are assisted by technologies. However, the use of computers does not guarantee collaboration, as free collaboration does not necessary lead to fruitful learning. Therefore, practitioners need to design CSCL scripts that structure the collaborative settings so that they promote learning. However, not all teachers have the technical and pedagogical background needed to design such scripts. With the aim of assisting teachers in designing effective CSCL scripts, we propose a model to support the selection of reusable good practices (formulated as patterns) so that they can be used as a starting point for their own designs. This model is based on a pattern ontology that computationally represents the knowledge captured on a pattern language for the design of CSCL scripts. A preliminary evaluation of the proposed approach is provided with two examples based on a set of meaningful interrelated patters computationally represented with the pattern ontology, and a paper prototyping experience carried out with two teaches. The results offer interesting insights towards the implementation of the pattern ontology in software tools.

Language acquisition during a stay abroad period following formal instruction: temporal effects on oral fluency development

This study investigates the development of fluency in 30 advanced L2 learners of English over a period of 15 months. In order to measure fluency, several temporal variables and hesitation phenomena are analyzed and compared. Oral competence is assessed by means of an oral interview carried out by the learners. Data collection takes place at three different times: before (T1) and after (T2) a six-month period of FI (80 hours) in the home university, and after a three-month SA term (T3). The data is analyzed quantitatively. Developmental gains in fluency are measured for the whole period, adopting a view of complementarity between the two learning contexts. From these results, a group of high fluency speakers is identified. Correlations between fluency gains and individual and contextual variables are executed and a more qualitative analysis is performed for high fluency speakers' performance and behavior. Results show an overall development of students' oral fluency during a period of 15 months favored by the combination of a period of FI at home followed by a 3-months SA.

Measuring Instruments In Cardiology Adapted Into Portuguese Language Of Brazil: A Systematic Review

This was a systematic review aimed at identifying and characterizing measuring instruments, developed in the context of cardiology, which were adapted into Portuguese language of Brazil. Systematic searches were performed in six databases. Information extracted included cultural adaptation process and measurement properties. To assess the methodological quality of studies, criteria based on international guidelines for cultural adaptation of instruments were used. Among the 114 articles found, 14 were eligible for review. Of these, most evaluated quality of life (35.7%) and health knowledge/learning (28.6%). Most studies followed all stages of the adaptation process recommended in the literature. With respect to measurement properties, internal consistency, verified by Cronbach’s alpha, was the property reported in the majority of the studies, as well as construct and criterion validity. This study is expected to provide to the scientific community a critical evaluation of adapted questionnaires available in the context of cardiology.


Interference of Native Language in English Language Acquisition in Santa Cruz and Calheta Secondary schools

This study analyzes the interference of native language when students are learning a foreign language, in this case English. According to this study, interferences from the native language such as lexical-semantic and phonological are not infrequent. It is one of the main barriers that create difficulties when learning English. The outcome of this study is presented at the end of the Monografia: to provide English-language teachers and students the most significant linguistic interferences that occur during the learning process of English language. Furthermore, some teaching strategies are discussed to avoid the biases that appear due to the interferences of the native language. As a result of this study, I hope to contribute to the learning success among English-language students.

Understanding the Use of the Students’ Common Language (Creole/Portuguese) in the 7th Grade Capeverdean English Classroom

This thesis is a pedagogical and methodological work related to the Teacher’s use of the students’ common language in 7th grade (beginners-level 1) Capeverdean English classroom. It discusses the importance of a limited and judicious use of the students’ common language (Creole/Portuguese) as a teaching technique to assist in the teaching and learning process. This thesis contains four chapters. The first chapter defines and shows the difference between mother tongue, second language and foreign language, talks about the methods and approaches (classroom procedures) to teach English as a foreign language, the different opinions about the teacher’s use of the students’ first language in the EFL classroom, and presents two studies already conducted on the use of the students’ mother tongue in the English classroom in two different EFL context. The second Chapter describes the methodology of research to conduct a study on the use of the students’ common language (Creole/Portuguese) in the EFL Capeverdean context with 7th grade students. The third chapter is the presentation of the Results and Analyses of the field research. And finally the fourth chapter is the recommendations and conclusions.