EEG alpha activity reflects motor preparation rather than the mode of action selection

Alpha-band activity (8-13 Hz) is not only suppressed by sensory stimulation and movements, but also modulated by attention, working memory and mental tasks, and could be sensitive to higher motor control functions. The aim of the present study was to examine alpha oscillatory activity during the preparation of simple left or right finger movements, contrasting the external and internal mode of action selection. Three preparation conditions were examined using a precueing paradigm with S1 as the preparatory and S2 as the imperative cue: Full, laterality instructed by S1; Free, laterality freely selected and None, laterality instructed by S2. Time-frequency (TF) analysis was performed in the alpha frequency range during the S1-S2 interval, and alpha motor-related amplitude asymmetries (MRAA) were also calculated. The significant MRAA during the Full and Free conditions indicated effective external and internal motor response preparation. In the absence of specific motor preparation (None), a posterior alpha event-related desynchronization (ERD) dominated, reflecting the main engagement of attentional resources. In Full and Free motor preparation, posterior alpha ERD was accompanied by a midparietal alpha event-related synchronization (ERS), suggesting a concomitant inhibition of task-irrelevant visual activity. In both Full and Free motor preparation, analysis of alpha power according to MRAA amplitude revealed two types of functional activation patterns: (1) a motor alpha pattern, with predominantly midparietal alpha ERS and large MRAA corresponding to lateralized motor activation/visual inhibition and (2) an attentional alpha pattern, with dominating right posterior alpha ERD and small MRAA reflecting visuospatial attention. The present results suggest that alpha oscillatory patterns do not resolve the selection mode of action, but rather distinguish separate functional strategies of motor preparation.

Code-specific learning rules improve action selection by populations of spiking neurons

Population coding is widely regarded as a key mechanism for achieving reliable behavioral decisions. We previously introduced reinforcement learning for population-based decision making by spiking neurons. Here we generalize population reinforcement learning to spike-based plasticity rules that take account of the postsynaptic neural code. We consider spike/no-spike, spike count and spike latency codes. The multi-valued and continuous-valued features in the postsynaptic code allow for a generalization of binary decision making to multi-valued decision making and continuous-valued action selection. We show that code-specific learning rules speed up learning both for the discrete classification and the continuous regression tasks. The suggested learning rules also speed up with increasing population size as opposed to standard reinforcement learning rules. Continuous action selection is further shown to explain realistic learning speeds in the Morris water maze. Finally, we introduce the concept of action perturbation as opposed to the classical weight- or node-perturbation as an exploration mechanism underlying reinforcement learning. Exploration in the action space greatly increases the speed of learning as compared to exploration in the neuron or weight space.

Consciousness, Action Selection, Meaning and Phenomenic Anticipation

Phenomenal states are generally considered the ultimate sources of intrinsic motivation for autonomous biological agents. In this article, we will address the issue of the necessity of exploiting these states for the design and implementation of robust goal-directed artificial systems. We will provide an analysis of consciousness in terms of a precise definition of how an agent "understands" the informational flows entering the agent and its very own action possibilities. This abstract model of consciousness and understanding will be based in the analysis and evaluation of phenomenal states along potential future trajectories in the state space of the agents. This implies that a potential strategy to follow in order to build autonomous but still customer-useful systems is to embed them with the particular, ad hoc phenomenality that captures the system-external requirements that define the system usefulness from a customer-based, requirements-strict engineering viewpoint.

Aging and motor programming: differential effects according to the selection of action

There are controversial reports about the effect of aging on movement preparation, and it is unclear to which extent cognitive and/or motor related cerebral processes may be affected. This study examines the age effects on electro-cortical oscillatory patterns during various motor programming tasks, in order to assess potential differences according to the mode of action selection. Twenty elderly (EP, 60-84 years) and 20 young (YP, 20-29 years) participants with normal cognition underwent 3 pre-cued response tasks (S1-S2 paradigm). S1 carried either complete information on response side (Full; stimulus-driven motor preparation), no information (None; general motor alertness), or required free response side selection (Free; internally-driven motor preparation). Electroencephalogram (EEG) was recorded using 64 surface electrodes. Alpha (8-12 Hz) desynchronization (ERD)/synchronization (ERS) and motor-related amplitude asymmetries (MRAA) were analyzed during the S1-S2 interval. Reaction times (RTs) to S2 were slower in EP than YP, and in None than in the other 2 tasks. There was an Age x Task interaction due to increased RTs in Free compared to Full in EP only. Central bilateral and midline activation (alpha ERD) was smaller in EP than YP in None. In Full just before S2, readiness to move was reflected by posterior midline inhibition (alpha ERS) in both groups. In Free, such inhibition was present only in YP. Moreover, MRAA showed motor activity lateralization in both groups in Full, but only in YP in Free. The results indicate reduced recruitment of motor regions for motor alertness in the elderly. They further show less efficient cerebral processes subtending free selection of movement in elders, suggesting reduced capacity for internally-driven action with age.

Accelerating autonomous learning by using heuristic selection of actions

This paper investigates how to make improved action selection for online policy learning in robotic scenarios using reinforcement learning (RL) algorithms. Since finding control policies using any RL algorithm can be very time consuming, we propose to combine RL algorithms with heuristic functions for selecting promising actions during the learning process. With this aim, we investigate the use of heuristics for increasing the rate of convergence of RL algorithms and contribute with a new learning algorithm, Heuristically Accelerated Q-learning (HAQL), which incorporates heuristics for action selection to the Q-Learning algorithm. Experimental results on robot navigation show that the use of even very simple heuristic functions results in significant performance enhancement of the learning rate.

Role of the monkey ventrolateral prefrontal cortex in the integration of contextual information for the selection and understanding of goal-directed actions

Action selection and organization are very complex processes that need to exploit contextual information and the retrieval of previously memorized information, as well as the integration of these different types of data. On the basis of anatomical connection with premotor and parietal areas involved in action goal coding, and on the data about the literature it seems appropriate to suppose that one of the most candidate involved in the selection of neuronal pools for the selection and organization of intentional actions is the prefrontal cortex. We recorded single ventrolateral prefrontal (VLPF) neurons activity while monkeys performed simple and complex manipulative actions aimed at distinct final goals, by employing a modified and more strictly controlled version of the grasp-to-eat(a food pellet)/grasp-to-place(an object) paradigm used in previous studies on parietal (Fogassi et al., 2005) and premotor neurons (Bonini et al., 2010). With this task we have been able both to evaluate the processing and integration of distinct (visual and auditory) contextual sequentially presented information in order to select the forthcoming action to perform and to examine the possible presence of goal-related activity in this portion of cortex. Moreover, we performed an observation task to clarify the possible contribution of VLPF neurons to the understanding of others’ goal-directed actions. Simple Visuo Motor Task (sVMT). We found four main types of neurons: unimodal sensory-driven, motor-related, unimodal sensory-and-motor, and multisensory neurons. We found a substantial number of VLPF neurons showing both a motor-related discharge and a visual presentation response (sensory-and-motor neurons), with remarkable visuo-motor congruence for the preferred target. Interestingly the discharge of multisensory neurons reflected a behavioural decision independently from the sensory modality of the stimulus allowing the monkey to make it: some encoded a decision to act/refraining from acting (the majority), while others specified one among the four behavioural alternatives. Complex Visuo Motor Task (cVMT). The cVMT was similar to the sVMT, but included a further grasping motor act (grasping a lid in order to remove it, before grasping the target) and was run in two modalities: randomized and in blocks. Substantially, motor-related and sensory-and-motor neurons tested in the cVMTrandomized were activated already during the first grasping motor act, but the selectivity for one of the two graspable targets emerged only during the execution of the second grasping. In contrast, when the cVMT was run in block, almost all these neurons not only discharged during the first grasping motor act, but also displayed the same target selectivity showed in correspondence of the hand contact with the target. Observation Task (OT). A great part of the neurons active during the OT showed a firing rate modulation in correspondence with the action performed by the experimenter. Among them, we found neurons significantly activated during the observation of the experimenter’s action (action observation-related neurons) and neurons responding not only to the action observation, but also to the presented cue stimuli (sensory-and-action observation-related neurons. Among the neurons of the first set, almost the half displayed a target selectivity, with a not clear difference between the two presented targets; Concerning to the second neuronal set, sensory-and-action related neurons, we found a low target selectivity and a not strictly congruence between the selectivity exhibited in the visual response and in the action observation.

On the interaction between perceptual and response selection: neuropsychological evidence

We examined the relations between selection for perception and selection for action in a patient FK, with bilateral damage to his temporal and medial frontal cortices. The task required a simple grasp response to a common object (a cup) in the presence of a distractor (another cup). The target was cued by colour or location, and FK made manual responses. We examined the effects on performance of cued and uncued dimensions of both the target and the distractor. FK was impaired at perceptually selecting the target when cued by colour, when the target colour but not its location changed on successive trials. The effect was sensitive to the relative orientations of targets and distractors, indicating an effect of action selection on perceptual selection, when perceptual selection was weakly instantiated. The dimension-specific carry-over effect on reaching was enhanced when there was a temporal delay between a cue and the response, and it disappeared when there was a between-trial delay. The results indicate that perceptual and action selection systems interact to determine the efficiency with which actions are selected to particular objects.

L'influence des oscillations en delta sur la sélection d'une action

Résumé : La capacité de décider parmi plusieurs possibilités d'actions, grâce à l'information sensorielle disponible, est essentielle à un organisme interagissant avec un environnement complexe. Les modèles actuels de sélection d'une action soutiennent que le cerveau traite continuellement l'information sensorielle afin de planifier plusieurs possibilités d'action en parallèle. Dans cette perspective, ces représentations motrices, associées à chaque possibilité d'action, sont en constante compétition entre elles. Afin qu'une alternative puisse être sélectionnée pour le mouvement, une valeur de pondération, intégrant une multitude de facteurs, doit être associée à chacun des plans moteurs afin de venir moduler la compétition. Plusieurs études se sont intéressées aux différents facteurs modulant la sélection de l'action, tels que la disposition de l'environnement, le coût des actions, le niveau de récompense, etc. Par contre, il semble qu'aucune étude n'ait rapporté ce qu'il advient lorsque la valeur de pondération de chacune des actions possibles est identique. Dans ce contexte, quel est l'élément permettant de venir moduler la sélection de l'action? De ce fait, l'objectif principal de mon projet de maitrise est d'investiguer le facteur permettant au cerveau de sélectionner une action lorsque tous les facteurs rapportés dans la littérature sont contrôlés. De récentes données ont montré que les oscillations corticales lentes dans la bande delta peuvent servir d'instrument de sélection attentionnelle en modulant l'amplitude de la réponse neuronale. Ainsi, les stimuli arrivant dans le cortex pendant une phase en delta de forte excitabilité sont amplifiés, tandis que ceux arrivant lors d'une phase en delta de faible excitabilité sont atténués. Ceci dit, il est possible que la phase en delta dans laquelle se trouve le cerveau au moment d'effectuer la sélection d'une action puisse influencer la décision. Utilisant une tâche de sélection de main, cette étude teste l'hypothèse que la sélection de la main est associée à la phase en delta des ensembles neuronaux codant le mouvement de chacune des mains, lorsque tous les facteurs connus influençant la décision sont contrôlés. L'électroencéphalographie (EEG) fut utilisée afin d'enregistrer les signaux corticaux pendant que les participants effectuaient une tâche de sélection de main dans laquelle ils devaient, à chaque essai, atteindre une cible visuelle aussi rapidement que possible en utilisant la main de leur choix. La tâche fut conçue de façon à ce que les facteurs spatiaux et biomécaniques soient contrôlés. Ceci fut réalisé enidentifiant premièrement, sur une base individuelle, l'emplacement de la cible pour laquelle les mains droite et gauche avaient une probabilité équivalente d'être choisies (point d'égalité subjective, PSE). Ensuite, dans l'expérience principale, les participants effectuaient plusieurs mouvements d'atteinte vers des cibles positionnées près et loin du PSE, toujours avec la main de leur choix. L'utilisation de cinq cibles très près du PSE a permis de collecter de nombreux essais dans lesquels la main droite et la main gauche furent sélectionnées en réponse à un même stimulus visuel. Ceci a ainsi permis d'analyser les signaux des deux cortex dans des conditions d'utilisation de la main droite et gauche, tout en contrôlant pour les autres facteurs pouvant moduler la sélection de la main. Les résultats de cette recherche révèlent que l'hémisphère cortical se trouvant dans la phase la plus excitable en delta (près du pic négatif), lors de l'apparition du stimulus, est associé à la fois à la main qui sera sélectionnée ainsi qu'au temps de réaction. Ces résultats montrent que l'excitabilité corticale momentanée (phase du signal) pourrait agir comme un facteur modulant la sélection d'une action. Dans cette optique, ces données élargissent considérablement les modèles actuels en montrant que la sélection d'une action est en partie déterminée par l'état du cerveau au moment d'effectuer un choix, d'une manière qui est indépendante de toutes les variables de décision connues.

Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

Autonomous underwater vehicle control using reinforcement learning policy search methods

Autonomous underwater vehicles (AUV) represent a challenging control problem with complex, noisy, dynamics. Nowadays, not only the continuous scientific advances in underwater robotics but the increasing number of subsea missions and its complexity ask for an automatization of submarine processes. This paper proposes a high-level control system for solving the action selection problem of an autonomous robot. The system is characterized by the use of reinforcement learning direct policy search methods (RLDPS) for learning the internal state/action mapping of some behaviors. We demonstrate its feasibility with simulated experiments using the model of our underwater robot URIS in a target following task

Towards Direct Policy Search Reinforcement Learning for Robot Control

This paper proposes a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot. Although the dominant approach, when using RL, has been to apply value function based algorithms, the system here detailed is characterized by the use of direct policy search methods. Rather than approximating a value function, these methodologies approximate a policy using an independent function approximator with its own parameters, trying to maximize the future expected reward. The policy based algorithm presented in this paper is used for learning the internal state/action mapping of a behavior. In this preliminary work, we demonstrate its feasibility with simulated experiments using the underwater robot GARBI in a target reaching task

Aging and the lateralization of oscillatory activities related to external and internal motor preparation

Selection of action may rely on external guidance or be motivated internally, engaging partially distinct cerebral networks. With age, there is an increased allocation of sensorimotor processing resources, accompanied by a reduced differentiation between the two networks of action selection. The present study examines the age effects on the motor-related oscillatory patterns related to the preparation of externally and internally guided movements. Thirty-two older and 30 younger adults underwent three delayed motor tasks with S1 as preparatory and S2 as imperative cue: Full, laterality instructed by S1 (external guidance); Free, laterality freely selected (internal guidance); None, laterality instructed by S2 (no preparation). Electroencephalogram (EEG) was recorded using 64 surface electrodes. Motor-Related Amplitude Asymmetries (MRAA), indexing the lateralization of oscillatory activities, were analyzed within the S1-S2 interval in the mu (9-12 Hz) and low beta (15-20 Hz) motor-related frequency bands. Reaction times to S2 were slower in older than younger subjects, and slower in the Free than in the Full condition in older subjects only. In the Full condition, there were significant mu MRAA in both age groups, and significant low beta MRAA only in older adults. The Free condition was associated with large mu MRAA in younger adults and limited low beta MRAA in older adults. In younger subjects, the lateralization of mu activity in both Full and Free conditions indicated effective external and internal motor preparation. In older subjects, external motor preparation was associated with lateralization of low beta in addition with mu activity, compatible with an increase of motor-related resources. In contrast, absence of mu and limited low beta lateralization in internal motor preparation was concomitant with reaction time slowing and suggested less efficient cerebral processes subtending free movement selection in older adults, indicating reduced capacity for internally driven action with age.

Fluctuation-Driven Neural Dynamics Reproduce Drosophila Locomotor Patterns.

The neural mechanisms determining the timing of even simple actions, such as when to walk or rest, are largely mysterious. One intriguing, but untested, hypothesis posits a role for ongoing activity fluctuations in neurons of central action selection circuits that drive animal behavior from moment to moment. To examine how fluctuating activity can contribute to action timing, we paired high-resolution measurements of freely walking Drosophila melanogaster with data-driven neural network modeling and dynamical systems analysis. We generated fluctuation-driven network models whose outputs-locomotor bouts-matched those measured from sensory-deprived Drosophila. From these models, we identified those that could also reproduce a second, unrelated dataset: the complex time-course of odor-evoked walking for genetically diverse Drosophila strains. Dynamical models that best reproduced both Drosophila basal and odor-evoked locomotor patterns exhibited specific characteristics. First, ongoing fluctuations were required. In a stochastic resonance-like manner, these fluctuations allowed neural activity to escape stable equilibria and to exceed a threshold for locomotion. Second, odor-induced shifts of equilibria in these models caused a depression in locomotor frequency following olfactory stimulation. Our models predict that activity fluctuations in action selection circuits cause behavioral output to more closely match sensory drive and may therefore enhance navigation in complex sensory environments. Together these data reveal how simple neural dynamics, when coupled with activity fluctuations, can give rise to complex patterns of animal behavior.

Neural correlates of affordance competition in dorsal premotor cortex

Le travail présenté dans cette thèse porte sur le rôle du cortex prémoteur dorsal (PMd) au sujet de la prise de décision (sélection d’une action parmis nombreux choix) et l'orientation visuelle des mouvements du bras. L’ouvrage décrit des expériences électrophysiologiques chez le singe éveillé (Macaca mulatta) permettant d’adresser une fraction importante des prédictions proposées par l'hypothèse des affordances concurrentes (Cisek, 2006; Cisek, 2007a). Cette hypothèse suggère que le choix de toute action est l’issue d'une concurrence entre les représentations internes des exigences et des atouts de chacune des options présentées (affordances; Gibson, 1979). Un intérêt particulier est donné au traitement de l'information spatiale et la valeur des options (expected value, EV) dans la prise de décisions. La première étude (article 1) explore la façon dont PMd reflète ces deux paramètres dans la période délai ainsi que de leur intéraction. La deuxième étude (article 2) explore le mécanisme de décision de façon plus détaillée et étend les résultats au cortex prémoteur ventral (PMv). Cette étude porte également sur la représentation spatiale et l’EV dans une perspective d'apprentissage. Dans un environnement nouveau les paramètres spatiaux des actions semblent être présents en tout temps dans PMd, malgré que la représentation de l’EV apparaît uniquement lorsque les animaux commencent à prendre des décisions éclairées au sujet de la valeur des options disponibles. La troisième étude (article 3) explore la façon dont PMd est impliqué aux “changements d'esprit“ dans un procès de décision. Cette étude décrit comment la sélection d’une action est mise à jour à la suite d'une instruction de mouvement (GO signal). I II Les résultats principaux des études sont reproduits par un modèle computationnel (Cisek, 2006) suggérant que la prise de décision entre plusieurs actions alternatives peux se faire par voie d’un mécanisme de concurrence (biased competition) qui aurait lieu dans la même région qui spécifie les actions.

Contribution du globus pallidus lors de la locomotion sous guidage visuel.

Les nombreuses études cliniques sur les maladies affectant les noyaux gris centraux (NGC) ont montré l’importance des déficits moteurs engendrés par un dysfonctionnement de ces structures. La locomotion est particulièrement affectée chez ces patients, ce qui suggère un rôle important des NGC dans le contrôle de celle-ci. En revanche, très peu d’études fondamentales ont enregistré l’activité des neurones des NGC durant la locomotion chez des modèles intacts. Le globus pallidus, en tant qu’une des principales structures de sortie des NGC, constitue une cible idéale pour étudier le signal transmis par les NGC au thalamus et au tronc cérébral. L’objectif de notre étude est de mieux comprendre la contribution que joue le segment externe du globus pallidus (GPe) dans le contrôle de la locomotion sous guidage visuel. Cette étude constitue le premier rapport d’enregistrements de l’activité des neurones du GPe lors de modifications volontaires de la marche sous guidage visuel. Notre hypothèse générale est que le GPe contribue au contrôle de la locomotion, et particulièrement lors des modifications de la marche. Nous avons enregistré l’activité des neurones du GPe de chats au repos, puis marchant sur un tapis et enjambant des obstacles. Nos résultats ont révélé que la plupart des cellules montraient peu de changements lors de la locomotion non-obstruée par rapport au repos. En revanche, la moitié des neurones enregistrés présentaient une modulation significative de leur décharge durant l’enjambement de l’obstacle. Nous avons observé une majorité de ces cellules qui diminuaient leur fréquence et un tiers de neurones qui l’augmentaient pendant le pas modifié, suggérant que le GPe exerce sur le mouvement un contrôle dépendant du contexte, permettant sélectivement l’augmentation ou la diminution de l’activité motrice. De plus, presque toutes les cellules déchargeaient durant la totalité de la phase de balancement du membre antérieur controlatéral, ce qui semble indiquer une contribution du GPe dans le timing et la durée de cette phase. Finalement, la moitié des neurones présentaient un changement réciproque entre la condition de lead et la condition de trail. Ce résultat suggère que le GPe participerait à la sélection d’action du membre lors des modifications de la marche. Nos résultats illustrent la complexité de l’activité du GPe lors de la locomotion sous guidage visuel et ils sont compatibles avec une contribution plus générale au mouvement que celle exercée par le cortex moteur.