Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

Autonomous underwater vehicle control using reinforcement learning policy search methods

Autonomous underwater vehicles (AUV) represent a challenging control problem with complex, noisy, dynamics. Nowadays, not only the continuous scientific advances in underwater robotics but the increasing number of subsea missions and its complexity ask for an automatization of submarine processes. This paper proposes a high-level control system for solving the action selection problem of an autonomous robot. The system is characterized by the use of reinforcement learning direct policy search methods (RLDPS) for learning the internal state/action mapping of some behaviors. We demonstrate its feasibility with simulated experiments using the model of our underwater robot URIS in a target following task

Towards Direct Policy Search Reinforcement Learning for Robot Control

This paper proposes a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot. Although the dominant approach, when using RL, has been to apply value function based algorithms, the system here detailed is characterized by the use of direct policy search methods. Rather than approximating a value function, these methodologies approximate a policy using an independent function approximator with its own parameters, trying to maximize the future expected reward. The policy based algorithm presented in this paper is used for learning the internal state/action mapping of a behavior. In this preliminary work, we demonstrate its feasibility with simulated experiments using the underwater robot GARBI in a target reaching task

Visuo-motor and cognitive procedural learning in children with basal ganglia pathology.

We investigated procedural learning in 18 children with basal ganglia (BG) lesions or dysfunctions of various aetiologies, using a visuo-motor learning test, the Serial Reaction Time (SRT) task, and a cognitive learning test, the Probabilistic Classification Learning (PCL) task. We compared patients with early (<1 year old, n=9), later onset (>6 years old, n=7) or progressive disorder (idiopathic dystonia, n=2). All patients showed deficits in both visuo-motor and cognitive domains, except those with idiopathic dystonia, who displayed preserved classification learning skills. Impairments seem to be independent from the age of onset of pathology. As far as we know, this study is the first to investigate motor and cognitive procedural learning in children with BG damage. Procedural impairments were documented whatever the aetiology of the BG damage/dysfunction and time of pathology onset, thus supporting the claim of very early skill learning development and lack of plasticity in case of damage.

Adaptive intelligent agent approach to guide the web navigation on the PLAN-G distance learning platform

Hypermedia systems based on the Web for open distance education are becoming increasinglypopular as tools for user-driven access learning information. Adaptive hypermedia is a new direction in research within the area of user-adaptive systems, to increase its functionality by making it personalized [Eklu 961. This paper sketches a general agents architecture to include navigationaladaptability and user-friendly processes which would guide and accompany the student during hislher learning on the PLAN-G hypermedia system (New Generation Telematics Platform to Support Open and Distance Learning), with the aid of computer networks and specifically WWW technology [Marz 98-1] [Marz 98-2]. The PLAN-G actual prototype is successfully used with some informatics courses (the current version has no agents yet). The propased multi-agent system, contains two different types of adaptive autonomous software agents: Personal Digital Agents {Interface), to interacl directly with the student when necessary; and Information Agents (Intermediaries), to filtrate and discover information to learn and to adapt navigation space to a specific student

Immune Microenvironment in Colorectal Cancer: A New Hallmark to Change Old Paradigms

Impact of immune microenvironment in prognosis of solid tumors has been extensively studied in the last few years. Specifically in colorectal carcinoma, increased knowledge of the immune events around these tumors and their relation with clinical outcomes have led to consider immune microenvironment as one of the most important prognostic factors in this disease. In this review we will summarize and update the current knowledge with respect to this intriguing and complex new hallmark of cancer, paying special attention to infiltration by T-infiltrating lymphocytes and their subtypes in colorectal cancer, as well as its eventual clinical translation in terms of long-term prognosis. Finally, we suggest some possible investigational approaches based on combinatorial strategies to trigger and boost immune reaction against tumor cells.

Environnement et capacités cognitives chez le rat : compétition entre les facteurs qui influencent les stratégies spatiales

RESUME Ce travail se propose de discuter des résultats comportementaux observés chez des rats obtenus dans trois paradigmes expérimentaux différents : le bassin de Morris (Morris Water Maze, Morris, 1984) ; la table à trous (Homing Board, Schenk, 1989) et le labyrinthe radial (Radial Arm Maze, Olton et Samuelson, 1976). Les deux premières tâches sont spatiales et permettent un apprentissage de place en environnements contrôlés, et la troisième est une tâche comportementale qui différencie deux habiletés particulières, celle d'élimination (basée sur la mémoire de travail) et celle de sélection (basée sur la mémoire de référence). La discussion des résultats porte sur les stratégies de navigation utilisées par les animaux pour résoudre les tâches et plus précisément sur les facteurs qui peuvent influencer le choix de ces stratégies. Le facteur environnemental (environnement contrôlé) et le facteur cognitif (vieillissement) représentent les variables étudiées ici. C'est ainsi que certaines hypothèses communément acceptées ont été malmenées par nos résultats. Or si l'espace est habituellement supposé homogène (toutes les positions spatiales présentent le même degré de difficulté lors d'un apprentissage en champ ouvert), ce travail établit qu'une position associée -sans contiguïté - à l'un des trois indices visuels situés dans la périphérie de l'environnement est plus difficile à apprendre qu'une position située entre deux des trois indices. Deuxièmement, alors qu'il est admis que l'apprentissage d'une place dans un environnement riche requiert le même type d'information. dans la bassin de Morris (tâche nagée) que sur la table à trous (tâche marchée), nous avons montré que la discrimination spatiale en bassin ne peut être assurée par les trois indices visuels périphériques et nécessite la présence d'au moins un élément supplémentaire. Enfin, l'étude du vieillissement a souvent montré que l'âge réduit les capacités cognitives nécessaires à la navigation spatiale, conduisant à un déficit général des performances d'un animal sénescent, alors que dans notre travail, nous avons trouvé les animaux âgés plus performants et plus efficaces que les adultes dans une tâche particulière de collecte de nourriture. Ces expériences s'inscrivent dans une étude générale qui met à l'épreuve le modèle théorique proposé pax Schenk et Jacobs (2003), selon lequel l'encodage de la carte cognitive (Tolman, 1948 ; O'Keefe et Nadel, 1978) se ferait dans l'hippocampe par l'activité de deux modules complémentaires :d'une part le CA3 - Gyrus Denté pour le traitement d'une trame spatiale basée sur des éléments directionnels et Jou distribués en gradient (bearing map) et d'autre part le CAl - Subiculum pour le traitement des représentations locales basées sur les positions relatives des éléments fixes de l'environnement (sketch map). SUMMARY This work proposes to talk about behavioural results observed in three different experimental paradigms with rats: the Morris Water Maze (Morris, 1984); the Homing Board (Schenk, 1989) and the Radial Arm Maze (Olton and Samuelson, 1976). The two first tasks are spatial ones and allow place learning in controlled environments. The third one is a behavioural task which contrasts two particular skills, the elimination (based on working memory) and the selection one (based on reference memory). The topic of the discussion will be the navigation strategies used by animals to solve the different tasks, and more precisely the factors which can bias this strategies' choice. The environmental (controlled) and the cognitive (aging) factors are the variables studied here. Thus, some hypotheses usually accepted were manhandled by our results. Indeed, if space is habitually homogenously considered (all spatial positions present the same degree of difficulty in an open field learning), this work establishes that an associated position -without being adjacent - to one of the three visual cues localised in the environmental periphery is more difficult to learn than a configurationnel position (situated between two of the three cues). Secondly, if it is received that place learning in a rich environment requires the same information in the Morris water maze (swimming task) that on the Homing board (walking task), we showed that spatial discrimination in the water maze can't be provided by the three peripheral cue cards and needs the presence of a supplementary cue. At last, aging studies often showed that oldness decreases cognitive skills in spatial navigation, leading to a general deficit in performances. But, in our work, we found that senescent rats were more efficient than adult ones in a special food collecting task. These experiments come within the scope of a general study which tests the theoretical model proposed by Jacobs and Schenk (2003), according to which the cognitive map's encoding (Tolman, 1948, O'Keefe and Nadel, 1978) should take place in the hippocampus by two complementary modules, first the DG-CA3 should encode directional and/or gradients references (the bearing map), and secondly the Subiculum-CAl should process locale elements (the sketch map).

Aversive Stimuli and Functional Networks of Fear Learning in PTSD : 734

Background: One characteristic of post traumatic stress disorder is an inability to adapt to a safe environment i.e. to change behavior when predictions of adverse outcomes are not met. Recent studies have also indicated that PTSD patients have altered pain processing, with hyperactivation of the putamen and insula to aversive stimuli (Geuze et al, 2007). The present study examined neuronal responses to aversive and predicted aversive events. Methods: Twenty-four trauma exposed non-PTSD controls and nineteen subjects with PTSD underwent fMRI imaging during a partial reinforcement fear conditioning paradigm, with a mild electric shock as the unconditioned stimuli (UCS). Three conditions were analyzed: actual presentations of the UCS, events when a UCS was expected, but omitted (CS+), and events when the UCS was neither expected nor delivered (CS-). Results: The UCS evoked significant alterations in the pain matrix consisting of the brainstem, the midbrain, the thalamus, the insula, the anterior and middle cingulate and the contralateral somatosensory cortex. PTSD subjects displayed bilaterally elevated putamen activity to the electric shock, as compared to controls. In trials when USC was expected, but omitted, significant activations were observed in the brainstem, the midbrain, the anterior insula and the anterior cingulate. PTSD subjects displayed similar activations, but also elevated activations in the amygdala and the posterior insula. Conclusions: These results indicate altered fear and safety learning in PTSD, and neuronal activations are further explored in terms of functional connectivity using psychophysiological interaction analyses.

Spatial learning by rats across visually disconnected environments

Experiments were designed to examine some properties of spatial representations in rats. Adult subjects were trained to escape through a hole at a fixed position in a large circular arena (see Schenk 1989). The experiments were conducted in the dark, with a limited number of controlled visual light cues in order to assess the minimal cue requirement for place learning. Three identical light cues (shape, height and distance from the table) were used. Depending on the condition, they were either permanently on, or alternatively on or off, depending on the position of the rat in the field. Two questions were asked: a) how many identical visual cues were necessary for spatial discrimination in the dark, and b) could rats integrate the relative positions of separate cues, under conditions in which the rat was never allowed to perceive all three cues simultaneously. The results suggest that rats are able to achieve a place discrimination task even if the three cues necessary for efficient orientation can never be seen simultaneously. A dissociation between the discrimination of the spatial position of the goal and the capacity to reach it by a direct path suggests that a reduced number of cues might require prolonged locomotion to allow an accurate orientation in the environment.

Indirect effects of peri-and postnatal choline treatment on place-learning abilities in rat.

This work was aimed at analyzing the effects of perinatal choline supplementation on the development of spatial abilities and upon adult performance. Choline supplementation (3.5 g/L in 0.02 M saccharin solution in tap water) was maintained for two weeks before birth and for up to four weeks postnatally. Additional supplementation was maintained from the fifth to the tenth week postnatally. Spatial-learning capacities were studied at the ages of 26, 65, or 80 days in a circular swimming pool (Morris place-navigation task) and at the age of 7 months in a homing arena. Treatment effects were found in both juvenile and adult rats, and thus persisted for several months after the cessation of the supplementation. The choline supplementation improved the performance in the water maze in a very selective manner. The most consistent effect was a reduction in the latency to reach a cued platform at a fixed position in space, whereas the improvement was limited when the platform was invisible and had to be located relative to distant cues only. However, after removal of the goal cue, the treated rats showed a better retention of the training position than did the control rats. A similar effect was observed in a dry-land task conducted in the homing arena. The choline supplementation thus induced a significant improvement of spatial memory. But since this effect was only evident following training with a salient cue, it might be regarded as an indirect effect promoted by an optimal combination of cue guidance with a place strategy.

The interactional space in CLIL (Content and Language Integrated Learning) and EFL (English as a Foreign Language) classrooms

Aquest estudi pretén investigar els intercanvis verbals mestre/a – aprenent(s) en dos contextos d'instrucció diferents: classes amb un enfocament AICLE (Aprenentatge Integrat de Continguts Curriculars i Llengua Estrangera) on s’aprenen continguts no lingüístics a través de l’anglès, per una banda, i classes 'tradicionals' d'anglès com a llengua estrangera, on l’anglès és alhora objecte d’estudi i vehicle de comunicació, per una altra banda. Més concretament, les preguntes que formula el/la mestre/a, la producció oral dels aprenents i el 'feedback' del/de la mestre/a en els episodis d’atenció a la forma s’han estudiat a la llum de les principals teories provinents del camp de l’Adquisició de Segones Llengües (SLA) per tal de demostrar el seu paper en l’aprenentatge de l’anglès. El corpus de dades prové de l’enregistrament de 7 sessions AICLE i d'11 sessions EFL enregistrades en format àudio i vídeo en dos centres públics d’Educació Primària (EP) de Catalunya. A cadascuna de les escoles, el/la mateix/a mestre/a és l’encarregat/da dels dos tipus d’instrucció amb el mateix grup d’aprenents (10-11 anys d’edat), fet que permet eliminar variables individuals com l'aptitud dels aprenents o l'estil del/de la mestre/a.Els resultats mostren un cert nombre de similituds discursives entre AICLE i EFL donat que ambdós enfocaments tenen lloc en el context-classe amb unes característiques ben definides. Tal com apunta la recerca realitzada en aquest camp, la instrucció AICLE reuneix un seguit de condicions idònies per un major desenvolupament dels nivells de llengua anglesa més enllà de les classes ‘tradicionals’ d’anglès. Malgrat això, aquest estudi sembla indicar que el potencial d'AICLE pel que fa a facilitar una exposició rica a l’anglès i una producció oral significativa no s’explota degudament. En aquest sentit, els resultats d’aquest estudi poden contribuir a la formació dels futurs professors d'AICLE si es busca l’assoliment d’una complementarietat d’ambdós contextos amb l’objectiu últim de millorar els nivells de domini de la llengua anglesa.

MENTAL IMAGERY IN PERCEPTUAL LEARNING

L'imagerie mentale est définie comme une expérience similaire à la perception mais se déroulant en l'absence d'une stimulation physique. Des recherches antérieures ont montré que l'imagerie mentale améliore la performance dans certains domaines, comme par exemple le domaine moteur. Cependant, son rôle dans l'apprentissage perceptif n'a pas encore été étudié. L'apprentissage perceptif correspond à l'amélioration permanente des performances suite à la répétition de la même tâche. Cette thèse présente une série des résultats empiriques qui montrent que l'apprentissage perceptif peut aussi être achevé en l'absence des stimuli physiques. En effet, imaginer des stimuli visuels amène à une meilleure performance avec les stimuli réels. Donc, les processus sous-jacents l'apprentissage perceptif ne sont pas uniquement déclenchés par les stimuli sensoriels, mais également par des signaux internes. En plus, l'apprentissage perceptif à travers l'imagerie mentale ne se réalise que seule-ment quand les stimuli ne sont pas (complètement) présents, mais gaiement quand les stimuli montrés ne sont pas utiles quant à la résolution de la tâche. - Mental imagery is described as an experience that resembles pereeptnal ex-perience but which occurs in the absence ef a physical stimulation. Despite its beneficial effects in, among others, motor performance, the role of mental imagery m perceptual learning has not yet been addressed. Here we focus on a specific sensory modality: vision. Perceptual learning is the ability to improve perception in a stable way through the repetition of a given task Here I demonstrate by a series of empirical results that a perceptual improve¬ment can also occur in the absence of a stimulation. Imagining visual stimuli is sufficient for successful perceptual learning. Hence, processes underlying perceptual learning are not only stimulus-driven but can also be driven by internally generated signals. Moreover, I also show that perceptual learning via mental imagery can occur not only when physical stimuli are (partially) absent, but also in conditions where stimuli are uninformative with respect to the task that has to be learned.

Diseño de una herramienta de e-learning para el área de cirugía traumatológica

Este artículo trata, en primer lugar, de analizar el estado actual de las herramientas de e-learning relacionadas y aplicadas en el área de cirugía traumatológica, presentando las herramientas disponibles en la actualidad como vídeos, audios, simuladores de realidad virtual, pacientes virtuales, LMS, entre otras; para, a continuación, describir el diseño de una herramienta en la que los componentes cumplan con los criterios de integración, interactividad, estandarización y asegure la reutilización. Como conclusión, se valora positivamente el diseño de una herramienta totalmente de código abierto que incorpora componentes de LMCS, repositorios de objetos, pacientes virtuales, simuladores hápticos de realidad virtual y objetos educativos, entre otros. Finalmente se recomienda implementar y comprobar la utilidad de la herramienta propuesta en la formación y entrenamiento de cirujanos traumatólogos.

Aplicación web para informes de asignaturas de trabajo en grupo

Este Proyecto Fin de Carrera tiene como principal objetivo el desarrollo de una aplicación web que permita el registro de profesores y usuarios dentro de ella, con el fin de que los profesores elaboren cuestionarios con preguntas de diversa índole (preguntas con más de una respuesta correcta, con varias respuestas entre las que se deberá elegir una de ellas, respuestas textuales, etc.) y los alumnos responder a ellas.

Electronic voting system for computer supported collaborative learning

The EVS4CSCL project starts in the context of a Computer Supported Collaborative Learning environment (CSCL). Previous UOC projects created a CSCL generic platform (CLPL) to facilitate the development of CSCL applications. A discussion forum (DF) was the first application developed over the framework. This discussion forum was different from other products on the marketplace because of its focus on the learning process. The DF carried out the specification and elaboration phases from the discussion learning process but there was a lack in the consensus phase. The consensus phase in a learning environment is not something to be achieved but tested. Common tests are done by Electronic Voting System (EVS) tools, but consensus test is not an assessment test. We are not evaluating our students by their answers but by their discussion activity. Our educational EVS would be used as a discussion catalyst proposing a discussion about the results after an initial query or it would be used after a discussion period in order to manifest how the discussion changed the students mind (consensus). It should be also used by the teacher as a quick way to know where the student needs some reinforcement. That is important in a distance-learning environment where there is no direct contact between the teacher and the student and it is difficult to detect the learning lacks. In an educational environment, assessment it is a must and the EVS will provide direct assessment by peer usefulness evaluation, teacher marks on every query created and indirect assessment from statistics regarding the user activity.