Grounding human vocabulary in robot perception through interaction

This thesis addresses the problem of word learning in computational agents. The motivation behind this work lies in the need to support language-based communication between service robots and their human users, as well as grounded reasoning using symbols relevant for the assigned tasks. The research focuses on the problem of grounding human vocabulary in robotic agent’s sensori-motor perception. Words have to be grounded in bodily experiences, which emphasizes the role of appropriate embodiments. On the other hand, language is a cultural product created and acquired through social interactions. This emphasizes the role of society as a source of linguistic input. Taking these aspects into account, an experimental scenario is set up where a human instructor teaches a robotic agent the names of the objects present in a visually shared environment. The agent grounds the names of these objects in visual perception. Word learning is an open-ended problem. Therefore, the learning architecture of the agent will have to be able to acquire words and categories in an openended manner. In this work, four learning architectures were designed that can be used by robotic agents for long-term and open-ended word and category acquisition. The learning methods used in these architectures are designed for incrementally scaling-up to larger sets of words and categories. A novel experimental evaluation methodology, that takes into account the openended nature of word learning, is proposed and applied. This methodology is based on the realization that a robot’s vocabulary will be limited by its discriminatory capacity which, in turn, depends on its sensors and perceptual capabilities. An extensive set of systematic experiments, in multiple experimental settings, was carried out to thoroughly evaluate the described learning approaches. The results indicate that all approaches were able to incrementally acquire new words and categories. Although some of the approaches could not scale-up to larger vocabularies, one approach was shown to learn up to 293 categories, with potential for learning many more.

Movement Sonification: Effects on Motor Learning beyond Rhythmic Adjustments

Motor learning is based on motor perception and emergent perceptual-motor representations. A lot of behavioral research is related to single perceptual modalities but during last two decades the contribution of multimodal perception on motor behavior was discovered more and more. A growing number of studies indicates an enhanced impact of multimodal stimuli on motor perception, motor control and motor learning in terms of better precision and higher reliability of the related actions. Behavioral research is supported by neurophysiological data, revealing that multisensory integration supports motor control and learning. But the overwhelming part of both research lines is dedicated to basic research. Besides research in the domains of music, dance and motor rehabilitation, there is almost no evidence for enhanced effectiveness of multisensory information on learning of gross motor skills. To reduce this gap, movement sonification is used here in applied research on motor learning in sports. Based on the current knowledge on the multimodal organization of the perceptual system, we generate additional real-time movement information being suitable for integration with perceptual feedback streams of visual and proprioceptive modality. With ongoing training, synchronously processed auditory information should be initially integrated into the emerging internal models, enhancing the efficacy of motor learning. This is achieved by a direct mapping of kinematic and dynamic motion parameters to electronic sounds, resulting in continuous auditory and convergent audiovisual or audio-proprioceptive stimulus arrays. In sharp contrast to other approaches using acoustic information as error-feedback in motor learning settings, we try to generate additional movement information suitable for acceleration and enhancement of adequate sensorimotor representations and processible below the level of consciousness. In the experimental setting, participants were asked to learn a closed motor skill (technique acquisition of indoor rowing). One group was treated with visual information and two groups with audiovisual information (sonification vs. natural sounds). For all three groups learning became evident and remained stable. Participants treated with additional movement sonification showed better performance compared to both other groups. Results indicate that movement sonification enhances motor learning of a complex gross motor skill-even exceeding usually expected acoustic rhythmic effects on motor learning.

Elaboração de programa de intervenção com as habilidades percepto-viso-motoras em escolares com dislexia do desenvolvimento

Pós-graduação em Fonoaudiologia - FFC

Percepção viso-motora de escolares com Transtorno do Déficit de Atenção com Hiperatividade

PURPOSE: The aim of this study was to characterize and to compare the visual-motor perception of students with Attention Deficit with Hyperactivity Disorder (ADHD) with students with good academic performance. METHODS: Forty students from 2nd to 5th grades of an elementary public school, male gender (100%), aged between 7 and 10 years and 8 months old participated, divided into: GI (20 students with ADHD) and GII (20 students with good academic performance), paired according to age, schooling and gender with GI. The students were submitted to Developmental Test of Visual Perception (DTVP-2). RESULTS: The students of GI presented low performance in spatial position and visual closure (reduced motor) and inferior age equivalent in reduced motor perception, when compared to GII. CONCLUSION: The difficulties in visual-motor perception presented by students of GI cannot be attributed to a primary deficit, but to a secondary phenomenon of inattention that interferes directly in their visual-motor performance.

Near optimal combination of sensory and motor uncertainty in time during a naturalistic perception-action task.

Most behavioral tasks have time constraints for successful completion, such as catching a ball in flight. Many of these tasks require trading off the time allocated to perception and action, especially when only one of the two is possible at any time. In general, the longer we perceive, the smaller the uncertainty in perceptual estimates. However, a longer perception phase leaves less time for action, which results in less precise movements. Here we examine subjects catching a virtual ball. Critically, as soon as subjects began to move, the ball became invisible. We study how subjects trade-off sensory and movement uncertainty by deciding when to initiate their actions. We formulate this task in a probabilistic framework and show that subjects' decisions when to start moving are statistically near optimal given their individual sensory and motor uncertainties. Moreover, we accurately predict individual subject's task performance. Thus we show that subjects in a natural task are quantitatively aware of how sensory and motor variability depend on time and act so as to minimize overall task variability.

Dissociations between motor timing, motor coordination, and time perception after the administration of alcohol or caffeine

The impacts of psychoactive drugs on timing have usefully informed theories of timing and its substrates.

The Perception of Natural, Cell Phone, and Computer-Synthesized Speech During The Performance Of Simultaneous Visual-Motor Tasks

This study investigated the influence of top-down and bottom-up information on speech perception in complex listening environments. Specifically, the effects of listening to different types of processed speech were examined on intelligibility and on simultaneous visual-motor performance. The goal was to extend the generalizability of results in speech perception to environments outside of the laboratory. The effect of bottom-up information was evaluated with natural, cell phone and synthetic speech. The effect of simultaneous tasks was evaluated with concurrent visual-motor and memory tasks. Earlier works on the perception of speech during simultaneous visual-motor tasks have shown inconsistent results (Choi, 2004; Strayer & Johnston, 2001). In the present experiments, two dual-task paradigms were constructed in order to mimic non-laboratory listening environments. In the first two experiments, an auditory word repetition task was the primary task and a visual-motor task was the secondary task. Participants were presented with different kinds of speech in a background of multi-speaker babble and were asked to repeat the last word of every sentence while doing the simultaneous tracking task. Word accuracy and visual-motor task performance were measured. Taken together, the results of Experiments 1 and 2 showed that the intelligibility of natural speech was better than synthetic speech and that synthetic speech was better perceived than cell phone speech. The visual-motor methodology was found to demonstrate independent and supplemental information and provided a better understanding of the entire speech perception process. Experiment 3 was conducted to determine whether the automaticity of the tasks (Schneider & Shiffrin, 1977) helped to explain the results of the first two experiments. It was found that cell phone speech allowed better simultaneous pursuit rotor performance only at low intelligibility levels when participants ignored the listening task. Also, simultaneous task performance improved dramatically for natural speech when intelligibility was good. Overall, it could be concluded that knowledge of intelligibility alone is insufficient to characterize processing of different speech sources. Additional measures such as attentional demands and performance of simultaneous tasks were also important in characterizing the perception of different kinds of speech in complex listening environments.

Comparison of parental perception and therapist interpretation of child's performance of the Peabody Fine Motor Scale

Public Law 102-119 (Individuals with Disabilities Education Act of 1991), mandates that family members, if they wish, participate in developing a plan of treatment for their child. Traditionally, therapist have not relied on parental assessments based upon the assumption that parents overestimate their child's abilities. The present study compared parental perceptions about the developmental status of their child's fine motor abilities to the therapist's interpretation of a standardized assessment using the Peabody Developmental Motor Scale (Fine Motor). Thirty seven children, enrolled in an early intervention program, and their parents were recruited for the study. The results indicated that the parents and the therapist estimates were highly correlated and showed no significant differences when paired t-tests were computed for developmental ages and scaled scores. However, analyses of variances were significantly correlated for gender and number of siblings.

Risk perception and correlates of alcohol use among out-of-school youth in motor parks in Lagos State, Nigeria

Background and Aim: The prevalence of alcohol use has increased globally. Out-of-school youth are a vulnerable group who might have missed opportunities for learning healthy behaviours in a formal school environment. The purpose of this study was to determine the risk perception, pattern of use, and correlates of alcohol use among out-of-school youth in Lagos, Nigeria. Methods: A cross-sectional study was conducted among 380 out-of-school youth in motor parks in Lagos State, Nigeria, using interviewer administered questionnaires. Results: The lifetime prevalence of alcohol use was 61.1%, while 55.5% were current drinkers. Beer (57.3%) was the most consumed type of alcohol, followed by distilled spirits (29.8%). Using the CAGE scoring system, more than half (57.8%) of the current drinkers had a drinking problem. Almost three quarters (70.1%) had experienced at least one episode of alcohol intoxication within the past month. A considerable number of current drinkers (63.5%) desired to reduce their alcohol intake or stop drinking, while 45.5% had made unsuccessful attempts to do so within the past year. Only 28.9% had received assistance to quit or reduce their drinking and of these less than half (39.3%) received assistance from a professional or healthcare worker. Males were more likely to be current drinkers and to have experienced episodes of alcohol intoxication. Parental and peer drinking were associated with alcohol use but not with intoxication. Conclusions: It is important to design specific programmes to reduce alcohol use among out-of-school youth in these settings.

Vision, functional and cognitive determinants of motor vehicle incidents in older drivers

Background: The proportion of older individuals in the driving population is predicted to increase in the next 50 years. This has important implications for driving safety as abilities which are important for safe driving, such as vision (which accounts for the majority of the sensory input required for driving), processing ability and cognition have been shown to decline with age. The current methods employed for screening older drivers upon re-licensure are also vision based. This study, which investigated social, behavioural and professional aspects involved with older drivers, aimed to determine: (i) if the current visual standards in place for testing upon re-licensure are effective in reducing the older driver fatality rate in Australia; (ii) if the recommended visual standards are actually implemented as part of the testing procedures by Australian optometrists; and (iii) if there are other non-standardised tests which may be better at predicting the on-road incident-risk (including near misses and minor incidents) in older drivers than those tests recommended in the standards. Methods: For the first phase of the study, state-based age- and gender-stratified numbers of older driver fatalities for 2000-2003 were obtained from the Australian Transportation Safety Bureau database. Poisson regression analyses of fatality rates were considered by renewal frequency and jurisdiction (as separate models), adjusting for possible confounding variables of age, gender and year. For the second phase, all practising optometrists in Australia were surveyed on the vision tests they conduct in consultations relating to driving and their knowledge of vision requirements for older drivers. Finally, for the third phase of the study to investigate determinants of on-road incident risk, a stratified random sample of 600 Brisbane residents aged 60 years and were selected and invited to participate using an introductory letter explaining the project requirements. In order to capture the number and type of road incidents which occurred for each participant over 12 months (including near misses and minor incidents), an important component of the prospective research study was the development and validation of a driving diary. The diary was a tool in which incidents that occurred could be logged at that time (or very close in time to which they occurred) and thus, in comparison with relying on participant memory over time, recall bias of incident occurrence was minimised. Association between all visual tests, cognition and scores obtained for non-standard functional tests with retrospective and prospective incident occurrence was investigated. Results: In the first phase,rivers aged 60-69 years had a 33% lower fatality risk (Rate Ratio [RR] = 0.75, 95% CI 0.32-1.77) in states with vision testing upon re-licensure compared with states with no vision testing upon re-licensure, however, because the CIs are wide, crossing 1.00, this result should be regarded with caution. However, overall fatality rates and fatality rates for those aged 70 years and older (RR=1.17, CI 0.64-2.13) did not differ between states with and without license renewal procedures, indicating no apparent benefit in vision testing legislation. For the second phase of the study, nearly all optometrists measured visual acuity (VA) as part of a vision assessment for re-licensing, however, 20% of optometrists did not perform any visual field (VF) testing and only 20% routinely performed automated VF on older drivers, despite the standards for licensing advocating automated VF as part of the vision standard. This demonstrates the need for more effective communication between the policy makers and those responsible for carrying out the standards. It may also indicate that the overall higher driver fatality rate in jurisdictions with vision testing requirements is resultant as the tests recommended by the standards are only partially being conducted by optometrists. Hence a standardised protocol for the screening of older drivers for re-licensure across the nation must be established. The opinions of Australian optometrists with regard to the responsibility of reporting older drivers who fail to meet the licensing standards highlighted the conflict between maintaining patient confidentiality or upholding public safety. Mandatory reporting requirements of those drivers who fail to reach the standards necessary for driving would minimise potential conflict between the patient and their practitioner, and help maintain patient trust and goodwill. The final phase of the PhD program investigated the efficacy of vision, functional and cognitive tests to discriminate between at-risk and safe older drivers. Nearly 80% of the participants experienced an incident of some form over the prospective 12 months, with the total incident rate being 4.65/10 000 km. Sixty-three percent reported having a near miss and 28% had a minor incident. The results from the prospective diary study indicate that the current vision screening tests (VA and VF) used for re-licensure do not accurately predict older drivers who are at increased odds of having an on-road incident. However, the variation in visual measurements of the cohort was narrow, also affecting the results seen with the visual functon questionnaires. Hence a larger cohort with greater variability should be considered for a future study. A slightly lower cognitive level (as measured with the Mini-Mental State Examination [MMSE]) did show an association with incident involvement as did slower reaction time (RT), however the Useful-Field-of-View (UFOV) provided the most compelling results of the study. Cut-off values of UFOV processing (>23.3ms), divided attention (>113ms), selective attention (>258ms) and overall score (moderate/ high/ very high risk) were effective in determining older drivers at increased odds of having any on-road incident and the occurrence of minor incidents. Discussion: The results have shown that for the 60-69 year age-group, there is a potential benefit in testing vision upon licence renewal. However, overall fatality rates and fatality rates for those aged 70 years and older indicated no benefit in vision testing legislation and suggests a need for inclusion of screening tests which better predict on-road incidents. Although VA is routinely performed by Australian optometrists on older drivers renewing their licence, VF is not. Therefore there is a need for a protocol to be developed and administered which would result in standardised methods conducted throughout the nation for the screening of older drivers upon re-licensure. Communication between the community, policy makers and those conducting the protocol should be maximised. By implementing a standardised screening protocol which incorporates a level of mandatory reporting by the practitioner, the ethical dilemma of breaching patient confidentiality would also be resolved. The tests which should be included in this screening protocol, however, cannot solely be ones which have been implemented in the past. In this investigation, RT, MMSE and UFOV were shown to be better determinants of on-road incidents in older drivers than VA and VF, however, as previously mentioned, there was a lack of variability in visual status within the cohort. Nevertheless, it is the recommendation from this investigation, that subject to appropriate sensitivity and specificity being demonstrated in the future using a cohort with wider variation in vision, functional performance and cognition, these tests of cognition and information processing should be added to the current protocol for the screening of older drivers which may be conducted at licensing centres across the nation.

A comparison of the hazard perception ability of matched groups of healthy drivers aged 35 to 55, 65 to 74, and 75 to 84 years

We examined differences in response latencies obtained during a validated video-based hazard perception driving test between three healthy, community-dwelling groups: 22 mid-aged (35-55 years), 34 young-old (65-74 years), and 23 old-old (75-84 years) current drivers, matched for gender, education level, and vocabulary. We found no significant difference in performance between mid-aged and young-old groups, but the old-old group was significantly slower than the other two groups. The differences between the old-old group and the other groups combined were independently mediated by useful field of view (UFOV), contrast sensitivity, and simple reaction time measures. Given that hazard perception latency has been linked with increased crash risk, these results are consistent with the idea that increased crash risk in older adults could be a function of poorer hazard perception, though this decline does not appear to manifest until age 75+ in healthy drivers.

The effect of Amblyopia on motor and psychosocial skills in children

Background/Aims: In an investigation of the functional impact of amblyopia on children, the fine motor skills, perceived self-esteem and eye movements of amblyopic children were compared with that of age-matched controls. The influence of amblyogenic condition or treatment factors that might predict any decrement in outcome measures was investigated. The relationship between indirect measures of eye movements that are used clinically and eye movement characteristics recorded during reading was examined and the relevance of proficiency in fine motor skills to performance on standardised educational tests was explored in a sub-group of the control children. Methods: Children with amblyopia (n=82; age 8.2 ± 1.3 years) from differing causes (infantile esotropia n=17, acquired strabismus n=28, anisometropia n=15, mixed n=13 and deprivation n=9), and a control group of children (n=106; age 9.5 ± 1.2 years) participated in this study. Measures of visual function included monocular logMAR visual acuity (VA) and stereopsis assessed with the Randot Preschool Stereoacuity test, while fine motor skills were measured using the Visual-Motor Control (VMC) and Upper Limb Speed and Dexterity (ULSD) subtests of the Brunicks-Oseretsky Test of Motor Proficiency. Perceived self esteem was assessed for those children from grade 3 school level with the Harter Self Perception Profile for Children and for those in younger grades (preschool to grade 2) with the Pictorial Scale of Perceived Competence and Acceptance for Young Children. A clinical measure of eye movements was made with the Developmental Eye Movement (DEM) test for those children aged eight years and above. For appropriate case-control comparison of data, the results from amblyopic children were compared with age-matched sub-samples drawn from the group of children with normal vision who completed the tests. Eye movements during reading for comprehension were recorded by the Visagraph infra-red recording system and results of standardised tests of educational performance were also obtained for a sub-set of the control group. Results Amblyopic children (n=82; age 8.2 ± 1.7 years) performed significantly poorer than age-matched control children (n=37; age 8.3 ± 1.3 years) on 9 of 16 fine motor skills sub-items and for the overall age-standardised scores for both VMC and ULSD items (p<0.05); differences were most evident on timed manual dexterity tasks. The underlying aetiology of amblyopia and level of stereoacuity significantly affected fine motor skill performance on both items. However, when examined in a multiple regression model that took into account the inter-correlation between visual characteristics, poorer fine motor skills performance was only associated with strabismus (F1,75 = 5.428; p =0. 022), and not with the level of stereoacuity, refractive error or visual acuity in either eye. Amblyopic children from grade 3 school level and above (n=47; age 9.2 ± 1.3 years), particularly those with acquired strabismus, had significantly lower social acceptance scores than age-matched control children (n=52; age 9.4 ± 0.5 years) (F(5,93) = 3.14; p = 0.012). However, the scores of the amblyopic children were not significantly different to controls for other areas related to self-esteem, including scholastic competence, physical appearance, athletic competence, behavioural conduct and global self worth. A lower social acceptance score was independently associated with a history of treatment with patching but not with a history of strabismus or wearing glasses. Amblyopic children from pre-school to grade 2 school level (n=29; age = 6.6 ± 0.6 years) had similar self-perception scores to their age-matched peers (n=20; age = 6.4 ± 0.5 years). There were no significant differences between the amblyopic (n=39; age 9.1 ± 0.9 years) and age-matched control (n = 42; age = 9.3 ± 0.38 years) groups for any of the DEM outcome measures (Vertical Time, Horizontal Time, Number of Errors and Ratio (Horizontal time/Vertical time)). Performance on the DEM did not significantly relate to measures of VA in either eye, level of binocular function, history of strabismus or refractive error. Developmental Eye Movement test outcome measures Horizontal Time and Vertical Time were significantly correlated with reading rates measured by the Visagraph for both reading for comprehension and naming numbers (r>0.5). Some moderate correlations were also seen between the DEM Ratio and word reading rates as recorded by Visagraph (r=0.37). In children with normal vision, academic scores in mathematics, spelling and reading were associated with measures of fine motor skills. Strongest effect sizes were seen with the timed manual dexterity domain, Upper Limb Speed and Dexterity. Conclusions Amblyopia may have a negative impact on a child’s fine motor skills and an older child’s sense of acceptance by their peers may be influenced by treatment that includes eye patching. Clinical measures of eye movements were not affected in amblyopic children. A number of the outcome measures of the DEM are associated with objective recordings of reading rates, supporting its clinical use for identification of children with slower reading rates. In children with normal vision, proficiency on clinical measures of fine motor skill are associated with outcomes on standardised measures of educational performance. Scores on timed manual dexterity tasks had the strongest association with educational performance. Collectively, the results of this study indicate that, in addition to the reduction in visual acuity and binocular function that define the condition, amblyopes have functional impairment in childhood development skills that underlie proficiency in everyday activities. The study provides support for strategies aimed at early identification and remediation of amblyopia and the co-morbidities that arise from abnormal visual neurodevelopment.

Perception of affordances in multi-scale dynamics as an alternative explanation for equivalence of analogical and inferential reasoning in animals and humans

Recent claims of equivalence of animal and human reasoning are evaluated and a study of avian cognition serves as an exemplar of weaknesses in these arguments. It is argued that current research into neurobiological cognition lacks theoretical breadth to substantiate comparative analyses of cognitive function. Evaluation of a greater range of theoretical explanations is needed to verify claims of equivalence in animal and human cognition. We conclude by exemplifying how the notion of affordances in multi-scale dynamics can capture behavior attributed to processes of analogical and inferential reasoning in animals and humans.