Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

The effect of perturbation warning on attention switching abilities during dual-task performance in young and older adults

This study examined whether providing an auditory warning would facilitate attention switching abilities in older adults during dual-tasking. Fifteen young and 16 older adults performed a tracking task while recovering their balance from a support surface translation. For half of the trials, an auditory warning was presented to inform participants of the upcoming translation. Performance was quantified through electromyographic (EMG) recordings of the lower limb muscles, while the ability to switch attention between tasks was determined by tracking task error. Providing warning of an upcoming loss of balance resulted in both young and older adults increasing their leg EMG activity by 10-165% (p<0.05) in preparation for the upcoming translation. However, no differences in the timing of attention switching were observed with or without the warning (p=0.424). Together, these findings suggest that providing a perturbation warning has minimal benefits in improving attention switching abilities for balance recovery in healthy older adults.

Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

Detecting single-target changes in multiple object tracking: The case of peripheral vision

In sports games, it is often necessary to perceive a large number of moving objects (e.g., the ball and players). In this context, the role of peripheral vision for processing motion information in the periphery is often discussed especially when motor responses are required. In an attempt to test the capability of using peripheral vision in those sports-games situations, a Multiple-Object-Tracking task that requires to track a certain number of targets amidst distractors, was chosen to determine the sensitivity of detecting target changes with peripheral vision only. Participants’ primary task was to recall four targets (out of 10 rectangular stimuli) after six seconds of quasi-random motion. As a second task, a button had to be pressed if a target change occurred (Exp 1: stop vs. form change to a diamond for 0.5 s; Exp 2: stop vs. slowdown for 0.5 s). Eccentricities of changes (5-10° vs. 15-20°) were manipulated, decision accuracy (recall and button press correct), motor response time and saccadic reaction time (change onset to saccade onset) were calculated and eye-movements were recorded. Results show that participants indeed used peripheral vision to detect changes, because either no or very late saccades to the changed target were executed in correct trials. Moreover, a saccade was more often executed when eccentricities were small. Response accuracies were higher and response times were lower in the stop conditions of both experiments while larger eccentricities led to higher response times in all conditions. Summing up, it could be shown that monitoring targets and detecting changes can be processed by peripheral vision only and that a monitoring strategy on the basis of peripheral vision may be the optimal one as saccades may be afflicted with certain costs. Further research is planned to address the question whether this functionality is also evident in sports tasks.

Active Visual Features Based on Events to Guide Robot Manipulators in Tracking Tasks

Traditional visual servoing systems do not deal with the topic of moving objects tracking. When these systems are employed to track a moving object, depending on the object velocity, visual features can go out of the image, causing the fail of the tracking task. This occurs specially when the object and the robot are both stopped and then the object starts the movement. In this work, we have employed a retina camera based on Address Event Representation (AER) in order to use events as input in the visual servoing system. The events launched by the camera indicate a pixel movement. Event visual information is processed only at the moment it occurs, reducing the response time of visual servoing systems when they are used to track moving objects.

Attentional blink reflex modulation in a continuous performance task is modality specific

Four experiments investigated the attentional modulation of acoustic blinks during continuous spatial tracking tasks. Experiment 1 found blink magnitude inhibition in a visual tracking task. Experiment 2 replicated this finding and also found blink latency slowing. Experiment 3 varied the difficulty of the task and found larger blink inhibition in the easy condition. Blink latency slowing did not differ and was significant at both difficulty levels. Experiment 4 employed less difficult visual and acoustic tracking tasks at two levels of task load. Blink magnitude inhibition during the visual and facilitation during the acoustic task was significant during high load in both modality groups. Blink latency was slowed in all visual task conditions and shortened in the difficult acoustic task. These results indicate that attentional blink modulation in a continuous spatial tracking task is modality specific.

On waveform analysis, the adaptive task technique, and the effect of vibration on manual control skill

A re-examination of fundamental concepts and a formal structuring of the waveform analysis problem is presented in Part I. eg. the nature of frequency is examined and a novel alternative to the classical methods of detection proposed and implemented which has the advantage of speed and independence from amplitude. Waveform analysis provides the link between Parts I and II. Part II is devoted to Human Factors and the Adaptive Task Technique. The Historical, Technical and Intellectual development of the technique is traced in a review which examines the evidence of its advantages relative to non-adaptive fixed task methods of training, skill assessment and man-machine optimisation. A second review examines research evidence on the effect of vibration on manual control ability. Findings are presented in terms of percentage increment or decrement in performance relative to performance without vibration in the range 0-0.6Rms'g'. Primary task performance was found to vary by as much as 90% between tasks at the same Rms'g'. Differences in task difficulty accounted for this difference. Within tasks vibration-added-difficulty accounted for the effects of vibration intensity. Secondary tasks were found to be largely insensitive to vibration except secondaries which involved fine manual adjustment of minor controls. Three experiments are reported next in which an adaptive technique was used to measure the % task difficulty added by vertical random and sinusoidal vibration to a 'Critical Compensatory Tracking task. At vibration intensities between 0 - 0.09 Rms 'g' it was found that random vibration added (24.5 x Rms'g')/7.4 x 100% to the difficulty of the control task. An equivalence relationship between Random and Sinusoidal vibration effects was established based upon added task difficulty. Waveform Analyses which were applied to the experimental data served to validate Phase Plane analysis and uncovered the development of a control and possibly a vibration isolation strategy. The submission ends with an appraisal of subjects mentioned in the thesis title.

The effect of imposed and self-selected computer monitor height on posture and gaze angle

Objective: The objectives were to determine the postural consequences of varying computer monitor height and to describe self-selected monitor heights and postures. Design: The design involved experimental manipulation of computer monitor height, description of self-selected heights, and measurement of posture and gaze angles. Background. Disagreement exists with regard to the appropriate height of computer monitors. It is known that users alter both head orientation and gaze angle in response to changes in monitor height; however the relative contribution of atlanto-occipital and cervical flexion to the change in head rotation is unknown. No information is available with regard to self-selected monitor heights. Methods. Twelve students performed a tracking task with the monitor placed at three different heights. The subjects then completed eight trials in which monitor height was first self-selected. Sagittal postural and gaze angle data were determined by digitizing markers defining a two-dimensional three-link model of the trunk, cervical spine and head. Results. The 27 degrees change in monitor height imposed was, on average, accommodated by 18 degrees of head inclination and a 9 degrees change in gaze angle relative to the head. The change in head inclination was achieved by a 6 degrees change in trunk inclination, a 4 degrees change in cervical flexion, and a 7 degrees change in atlanto-occipital flexion. The self-selected height varied depending on the initial monitor height and inclination. Conclusions. Self-selected monitor heights were lower than current 'eye-level' recommendations. Lower monitor heights are likely to reduce both visual and musculoskeletal discomfort. Relevance Musculoskeletal and visual discomfort may be reduced by placing computer monitors lower than currently recommended. (C) 1998 Elsevier Science Ltd. All rights reserved.

Fear of heights: cognitive performance and postural control

Fear of heights, or acrophobia, is one of the most frequent subtypes of specific phobia frequently associated to depression and other anxiety disorders. Previous evidence suggests a correlation between acrophobia and abnormalities in balance control, particularly involving the use of visual information to keep postural stability. This study investigates the hypotheses that (1) abnormalities in balance control are more frequent in individuals with acrophobia even when not exposed to heights, that (2) acrophobic symptoms are associated to abnormalities in visual perception of movement; and that (3) individuals with acrophobia are more sensitive to balance-cognition interactions. Thirty-one individuals with specific phobia of heights and thirty one non-phobic controls were compared using dynamic posturography and a manual tracking task. Acrophobics had poorer performance in both tasks, especially when carried out simultaneously. Previously described interference between posture control and cognitive activity seems to play a major role in these individuals. The presence of physiologic abnormalities is compatible with the hypothesis of a non-associative acquisition of fear of heights, i.e., not associated to previous traumatic events or other learning experiences. Clinically, this preliminary study corroborates the hypothesis that vestibular physical therapy can be particularly useful in treating individuals with fear of heights.

Effect of varying levels of mental workload on startle eyeblink modulation

Previous research using punctuate reaction time and counting tasks has found that the startle eyeblink reflex is sensitive to attentional demands. The present experiment explored whether startle eyeblink is also modulated during a complex continuous task and is sensitive to different levels of mental workload. Participants (N=14) performed a visual horizontal tracking task either alone (single-task condition) or in combination with a visual gauge monitoring task (multiple-task condition) for three minutes. On some task trials, the startle eyeblink reflex was elicited by a noise burst. Results showed that startle eyeblink was attenuated during both tasks and that the attenuation was greater during the multiple-task condition than during the single-task condition. Subjective ratings, endogenous eyeblink rate, heart period, and heart period variability provided convergent validity of the workload manipulations. The findings suggest that the startle eyeblink is sensitive to the workload demands associated with a continuous visual task. The application of startle eyeblink modulation as a workload metric and the possibility that it may be diagnostic of workload demands in different stimulus modalities is discussed.

Impact of cannabis inhalation on driving skills in occasional smokers

Objectives: Our aim was to study the brain regions involved in a divided attention tracking task related to driving in occasional cannabis smokers. In addition we assessed the relationship between THC levels in whole blood and changes in brain activity, behavioural and psychomotor performances. Methods: Twenty-one smokers participated to two independent cross-over fMRI experiments before and after smoking cannabis and a placebo. The paradigm was based on a visuo-motor tracking task, alternating active tracking blocks with passive tracking viewing and rest condition. Half of the active tracking conditions included randomly presented traffic lights as distractors. Blood samples were taken at regular intervals to determine the time-profiles of the major cannabinoids. Their levels during the fMRI experiments were interpolated from concentrations measured by GCMS/ MS just before and after brain imaging. Results: Behavioural data, such as the discard between target and cursor, the time of correct tracking and the reaction time during traffic lights appearance showed a statistical significant impairment of subject s skills due to THC intoxication. Highest THC blood concentrations were measured soon after smoking and ranged between 28.8 and 167.9 ng/ml. These concentrations reached values of a few ng/ml during the fMRI. fMRI results pointed out that under the effect of THC, high order visual areas (V3d) and Intraparietal sulcus (IPS) showed an higher activation compared to the control condition. The opposite comparison showed a decrease of activation during the THC condition in the anterior cingulate gyrus and orbitofrontal areas. In these locations, the BOLD showed a negative correlation with the THC level. Conclusion: Acute cannabis smoking significantly impairs performances and brain activity during active tracking tasks, partly reorganizing the recruitment of brain areas of the attention network. Neural activity in the anterior cingulate might be responsible of the changes in the cognitive controls required in our divided attention task.

Acute Effects of Cannabis Smoking on Skills Related to Driving : an fMRI Study

Introduction : Driving is a complex everyday task requiring mechanisms of perception, attention, learning, memory, decision making and action control, thus indicating that involves numerous and varied brain networks. If many data have been accumulated over time about the effects of alcohol consumption on driving capability, much less is known about the role of other psychoactive substances, such as cannabis (Chang et al.2007, Ramaekers et al, 2006). Indeed, the solicited brain areas during safe driving which could be affected by cannabis exposure have not yet been clearly identified. Our aim is to study these brain regions during a tracking task related to driving skills and to evaluate the modulation due to the tolerance of cannabis effects. Methods : Eight non-smoker control subjects participated to an fMRI experiment based on a visuo-motor tracking task, alternating active tracking blocks with passive tracking viewing and rest condition. Half of the active tracking conditions included randomly presented traffic lights as distractors. Subjects were asked to track with a joystick with their right hand and to press a button with their left index at each appearance of a distractor. Four smoking subjects participated to the same fMRI sessions once before and once after smoking cannabis and a placebo in two independent cross-over experiments. We quantified the performance of the subjects by measuring the precision of the behavioural responses (i.e. percentage of time of correct tracking and reaction times to distractors). Functional MRI data were acquired using on a 3.0T Siemens Trio system equipped with a 32-channel head coil. BOLD signals will be obtained with a gradient-echo EPI sequence (TR=2s, TE=30ms, FoV=216mm, FA=90°, matrix size 72×72, 32 slices, thickness 3mm). Preprocessing, single subject analysis and group statistics were conducted on SPM8b. Results were thresholded at p<0.05 (FWE corrected) and at k>30 for spatial extent. Results : Behavioural results showed a significant impairment in task and cognitive test performance of the subjects after cannabis inhalation when comparing their tracking accuracy either to the controls subjects or to their performances before the inhalation or after the placebo inhalation (p<0.001 corrected). In controls, fMRI BOLD analysis of the active tracking condition compared to the passive one revealed networks of polymodal areas in superior frontal and parietal cortex dealing with attention and visuo-spatial coordination. In accordance to what is known of the visual and sensory motor networks we found activations in V4, frontal eye-field, right middle frontal gyrus, intra-parietal sulcus, temporo-parietal junction, premotor and sensory-motor cortex. The presence of distractors added a significant activation in the precuneus. Preliminary results on cannabis smokers in the acute phase, compared either to themselves before the cannabis inhalation or to control subjects, showed a decreased activation in large portions of the frontal and parietal attention network during the simple tracking task, but greater involvement of precuneus, of the superior part of intraparietal sulcus and middle frontal gyrus bilaterally when distractors were present in the task. Conclusions : Our preliminary results suggest that acute cannabis smoking alters performances and brain activity during active tracking tasks, partly reorganizing the recruitment of brain areas of the attention network.

Weed or wheel! FMRI, behavioural, and toxicological investigations of how cannabis smoking affects skills necessary for driving.

Marijuana is the most widely used illicit drug, however its effects on cognitive functions underling safe driving remain mostly unexplored. Our goal was to evaluate the impact of cannabis on the driving ability of occasional smokers, by investigating changes in the brain network involved in a tracking task. The subject characteristics, the percentage of Δ(9)-Tetrahydrocannabinol in the joint, and the inhaled dose were in accordance with real-life conditions. Thirty-one male volunteers were enrolled in this study that includes clinical and toxicological aspects together with functional magnetic resonance imaging of the brain and measurements of psychomotor skills. The fMRI paradigm was based on a visuo-motor tracking task, alternating active tracking blocks with passive tracking viewing and rest condition. We show that cannabis smoking, even at low Δ(9)-Tetrahydrocannabinol blood concentrations, decreases psychomotor skills and alters the activity of the brain networks involved in cognition. The relative decrease of Blood Oxygen Level Dependent response (BOLD) after cannabis smoking in the anterior insula, dorsomedial thalamus, and striatum compared to placebo smoking suggests an alteration of the network involved in saliency detection. In addition, the decrease of BOLD response in the right superior parietal cortex and in the dorsolateral prefrontal cortex indicates the involvement of the Control Executive network known to operate once the saliencies are identified. Furthermore, cannabis increases activity in the rostral anterior cingulate cortex and ventromedial prefrontal cortices, suggesting an increase in self-oriented mental activity. Subjects are more attracted by intrapersonal stimuli ("self") and fail to attend to task performance, leading to an insufficient allocation of task-oriented resources and to sub-optimal performance. These effects correlate with the subjective feeling of confusion rather than with the blood level of Δ(9)-Tetrahydrocannabinol. These findings bolster the zero-tolerance policy adopted in several countries that prohibits the presence of any amount of drugs in blood while driving.

Multimodal MRI analysis of structural and functional networks in the human brain : application to two clinical studies : driving abilities under THC intoxication and early white matter changes in FXTAS

Les approches multimodales dans l'imagerie cérébrale non invasive sont de plus en plus considérées comme un outil indispensable pour la compréhension des différents aspects de la structure et de la fonction cérébrale. Grâce aux progrès des techniques d'acquisition des images de Resonance Magnetique et aux nouveaux outils pour le traitement des données, il est désormais possible de mesurer plusieurs paramètres sensibles aux différentes caractéristiques des tissues cérébraux. Ces progrès permettent, par exemple, d'étudier les substrats anatomiques qui sont à la base des processus cognitifs ou de discerner au niveau purement structurel les phénomènes dégénératifs et développementaux. Cette thèse met en évidence l'importance de l'utilisation d'une approche multimodale pour étudier les différents aspects de la dynamique cérébrale grâce à l'application de cette approche à deux études cliniques: l'évaluation structurelle et fonctionnelle des effets aigus du cannabis fumé chez des consommateurs réguliers et occasionnels, et l'évaluation de l'intégrité de la substance grise et blanche chez des jeunes porteurs de la prémutations du gène FMR1 à risque de développer le FXTAS (Fragile-X Tremor Ataxia Syndrome). Nous avons montré que chez les fumeurs occasionnels de cannabis, même à faible concentration du principal composant psychoactif (THC) dans le sang, la performance lors d'une tâche visuo-motrice est fortement diminuée, et qu'il y a des changements dans l'activité des trois réseaux cérébraux impliqués dans les processus cognitifs: le réseau de saillance, le réseau du contrôle exécutif, et le réseau actif par défaut (Default Mode). Les sujets ne sont pas en mesure de saisir les saillances dans l'environnement et de focaliser leur attention sur la tâche. L'augmentation de la réponse hémodynamique dans le cortex cingulaire antérieur suggère une augmentation de l'activité introspective. Une investigation des ef¬fets au niveau cérébral d'une exposition prolongée au cannabis, montre des changements persistants de la substance grise dans les régions associées à la mémoire et au traitement des émotions. Le niveau d'atrophie dans ces structures corrèle avec la consommation de cannabis au cours des trois mois précédant l'étude. Dans la deuxième étude, nous démontrons des altérations structurelles des décennies avant l'apparition du syndrome FXTAS chez des sujets jeunes, asymptomatiques, et porteurs de la prémutation du gène FMR1. Les modifications trouvées peuvent être liées à deux mécanismes différents. Les altérations dans le réseau moteur du cervelet et dans la fimbria de l'hippocampe, suggèrent un effet développemental de la prémutation. Elles incluent aussi une atrophie de la substance grise du lobule VI du cervelet et l'altération des propriétés tissulaires de la substance blanche des projections afférentes correspondantes aux pédoncules cérébelleux moyens. Les lésions diffuses de la substance blanche cérébrale peu¬vent être un marquer précoce du développement de la maladie, car elles sont liées à un phénomène dégénératif qui précède l'apparition des symptômes du FXTAS. - Multimodal brain imaging is becoming a leading tool for understanding different aspects of brain structure and function. Thanks to the advances in Magnetic Resonance imaging (MRI) acquisition schemes and data processing techniques, it is now possible to measure different parameters sensitive to different tissue characteristics. This allows for example to investigate anatomical substrates underlying cognitive processing, or to disentangle, at a pure structural level degeneration and developmental processes. This thesis highlights the importance of using a multimodal approach for investigating different aspects of brain dynamics by applying this approach to two clinical studies: functional and structural assessment of the acute effects of cannabis smoking in regular and occasional users, and grey and white matter assessment in young FMR1 premutation carriers at risk of developing FXTAS. We demonstrate that in occasional smokers cannabis smoking, even at low concentration of the main psychoactive component (THC) in the blood, strongly decrease subjects' performance on a visuo-motor tracking task, and globally alters the activity of the three brain networks involved in cognitive processing: the Salience, the Control Executive, and the Default Mode networks. Subjects are unable to capture saliences in the environment and to orient attention to the task; the increase in Hemodynamic Response in the Anterior Cingulate Cortex suggests an increase in self-oriented mental activity. A further investigation on long term exposure to cannabis, shows a persistent grey matter modification in brain regions associated with memory and affective processing. The degree of atrophy in these structures also correlates with the estimation of drug use in the three months prior the participation to the study. In the second study we demonstrate structural changes in young asymptomatic premutation carriers decades before the onset of FXTAS that might be related to two different mechanisms. Alteration of the cerebellar motor network and of the hippocampal fimbria/ fornix, may reflect a potential neurodevelopmental effect of the premutation. These include grey matter atrophy in lobule VI and modification of white matter tissue property in the corresponding afferent projections through the Middle Cerebellar Peduncles. Diffuse hemispheric white matter lesions that seem to appear closer to the onset of FXTAS and be related to a neurodegenerative phenomenon may mark the imminent onset of FXTAS.

Gradient-based reinforcement learning techniques for underwater robotics behavior learning

Darrerament, l'interès pel desenvolupament d'aplicacions amb robots submarins autònoms (AUV) ha crescut de forma considerable. Els AUVs són atractius gràcies al seu tamany i el fet que no necessiten un operador humà per pilotar-los. Tot i això, és impossible comparar, en termes d'eficiència i flexibilitat, l'habilitat d'un pilot humà amb les escasses capacitats operatives que ofereixen els AUVs actuals. L'utilització de AUVs per cobrir grans àrees implica resoldre problemes complexos, especialment si es desitja que el nostre robot reaccioni en temps real a canvis sobtats en les condicions de treball. Per aquestes raons, el desenvolupament de sistemes de control autònom amb l'objectiu de millorar aquestes capacitats ha esdevingut una prioritat. Aquesta tesi tracta sobre el problema de la presa de decisions utilizant AUVs. El treball presentat es centra en l'estudi, disseny i aplicació de comportaments per a AUVs utilitzant tècniques d'aprenentatge per reforç (RL). La contribució principal d'aquesta tesi consisteix en l'aplicació de diverses tècniques de RL per tal de millorar l'autonomia dels robots submarins, amb l'objectiu final de demostrar la viabilitat d'aquests algoritmes per aprendre tasques submarines autònomes en temps real. En RL, el robot intenta maximitzar un reforç escalar obtingut com a conseqüència de la seva interacció amb l'entorn. L'objectiu és trobar una política òptima que relaciona tots els estats possibles amb les accions a executar per a cada estat que maximitzen la suma de reforços totals. Així, aquesta tesi investiga principalment dues tipologies d'algoritmes basats en RL: mètodes basats en funcions de valor (VF) i mètodes basats en el gradient (PG). Els resultats experimentals finals mostren el robot submarí Ictineu en una tasca autònoma real de seguiment de cables submarins. Per portar-la a terme, s'ha dissenyat un algoritme anomenat mètode d'Actor i Crític (AC), fruit de la fusió de mètodes VF amb tècniques de PG.