A behavior-based scheme using reinforcement learning for autonomous underwater vehicles

This paper presents a hybrid behavior-based scheme using reinforcement learning for high-level control of autonomous underwater vehicles (AUVs). Two main features of the presented approach are hybrid behavior coordination and semi on-line neural-Q_learning (SONQL). Hybrid behavior coordination takes advantages of robustness and modularity in the competitive approach as well as efficient trajectories in the cooperative approach. SONQL, a new continuous approach of the Q_learning algorithm with a multilayer neural network is used to learn behavior state/action mapping online. Experimental results show the feasibility of the presented approach for AUVs

Gradient-based reinforcement learning techniques for underwater robotics behavior learning

Darrerament, l'interès pel desenvolupament d'aplicacions amb robots submarins autònoms (AUV) ha crescut de forma considerable. Els AUVs són atractius gràcies al seu tamany i el fet que no necessiten un operador humà per pilotar-los. Tot i això, és impossible comparar, en termes d'eficiència i flexibilitat, l'habilitat d'un pilot humà amb les escasses capacitats operatives que ofereixen els AUVs actuals. L'utilització de AUVs per cobrir grans àrees implica resoldre problemes complexos, especialment si es desitja que el nostre robot reaccioni en temps real a canvis sobtats en les condicions de treball. Per aquestes raons, el desenvolupament de sistemes de control autònom amb l'objectiu de millorar aquestes capacitats ha esdevingut una prioritat. Aquesta tesi tracta sobre el problema de la presa de decisions utilizant AUVs. El treball presentat es centra en l'estudi, disseny i aplicació de comportaments per a AUVs utilitzant tècniques d'aprenentatge per reforç (RL). La contribució principal d'aquesta tesi consisteix en l'aplicació de diverses tècniques de RL per tal de millorar l'autonomia dels robots submarins, amb l'objectiu final de demostrar la viabilitat d'aquests algoritmes per aprendre tasques submarines autònomes en temps real. En RL, el robot intenta maximitzar un reforç escalar obtingut com a conseqüència de la seva interacció amb l'entorn. L'objectiu és trobar una política òptima que relaciona tots els estats possibles amb les accions a executar per a cada estat que maximitzen la suma de reforços totals. Així, aquesta tesi investiga principalment dues tipologies d'algoritmes basats en RL: mètodes basats en funcions de valor (VF) i mètodes basats en el gradient (PG). Els resultats experimentals finals mostren el robot submarí Ictineu en una tasca autònoma real de seguiment de cables submarins. Per portar-la a terme, s'ha dissenyat un algoritme anomenat mètode d'Actor i Crític (AC), fruit de la fusió de mètodes VF amb tècniques de PG.

A proposal of a behavior-based control architecture with reinforcement learning for an autonomous underwater robot

Aquesta tesi proposa l'ús d'un seguit de tècniques pel control a alt nivell d'un robot autònom i també per l'aprenentatge automàtic de comportaments. L'objectiu principal de la tesis fou el de dotar d'intel·ligència als robots autònoms que han d'acomplir unes missions determinades en entorns desconeguts i no estructurats. Una de les premisses tingudes en compte en tots els passos d'aquesta tesis va ser la selecció d'aquelles tècniques que poguessin ésser aplicades en temps real, i demostrar-ne el seu funcionament amb experiments reals. El camp d'aplicació de tots els experiments es la robòtica submarina. En una primera part, la tesis es centra en el disseny d'una arquitectura de control que ha de permetre l'assoliment d'una missió prèviament definida. En particular, la tesis proposa l'ús de les arquitectures de control basades en comportaments per a l'assoliment de cada una de les tasques que composen la totalitat de la missió. Una arquitectura d'aquest tipus està formada per un conjunt independent de comportaments, els quals representen diferents intencions del robot (ex.: "anar a una posició", "evitar obstacles",...). Es presenta una recerca bibliogràfica sobre aquest camp i alhora es mostren els resultats d'aplicar quatre de les arquitectures basades en comportaments més representatives a una tasca concreta. De l'anàlisi dels resultats se'n deriva que un dels factors que més influeixen en el rendiment d'aquestes arquitectures, és la metodologia emprada per coordinar les respostes dels comportaments. Per una banda, la coordinació competitiva és aquella en que només un dels comportaments controla el robot. Per altra banda, en la coordinació cooperativa el control del robot és realitza a partir d'una fusió de totes les respostes dels comportaments actius. La tesis, proposa un esquema híbrid d'arquitectura capaç de beneficiar-se dels principals avantatges d'ambdues metodologies. En una segona part, la tesis proposa la utilització de l'aprenentatge per reforç per aprendre l'estructura interna dels comportaments. Aquest tipus d'aprenentatge és adequat per entorns desconeguts i el procés d'aprenentatge es realitza al mateix temps que el robot està explorant l'entorn. La tesis presenta també un estat de l'art d'aquest camp, en el que es detallen els principals problemes que apareixen en utilitzar els algoritmes d'aprenentatge per reforç en aplicacions reals, com la robòtica. El problema de la generalització és un dels que més influeix i consisteix en permetre l'ús de variables continues sense augmentar substancialment el temps de convergència. Després de descriure breument les principals metodologies per generalitzar, la tesis proposa l'ús d'una xarxa neural combinada amb l'algoritme d'aprenentatge per reforç Q_learning. Aquesta combinació proporciona una gran capacitat de generalització i una molt bona disposició per aprendre en tasques de robòtica amb exigències de temps real. No obstant, les xarxes neurals són aproximadors de funcions no-locals, el que significa que en treballar amb un conjunt de dades no homogeni es produeix una interferència: aprendre en un subconjunt de l'espai significa desaprendre en la resta de l'espai. El problema de la interferència afecta de manera directa en robòtica, ja que l'exploració de l'espai es realitza sempre localment. L'algoritme proposat en la tesi té en compte aquest problema i manté una base de dades representativa de totes les zones explorades. Així doncs, totes les mostres de la base de dades s'utilitzen per actualitzar la xarxa neural, i per tant, l'aprenentatge és homogeni. Finalment, la tesi presenta els resultats obtinguts amb la arquitectura de control basada en comportaments i l'algoritme d'aprenentatge per reforç. Els experiments es realitzen amb el robot URIS, desenvolupat a la Universitat de Girona, i el comportament après és el seguiment d'un objecte mitjançant visió per computador. La tesi detalla tots els dispositius desenvolupats pels experiments així com les característiques del propi robot submarí. Els resultats obtinguts demostren la idoneïtat de les propostes en permetre l'aprenentatge del comportament en temps real. En un segon apartat de resultats es demostra la capacitat de generalització de l'algoritme d'aprenentatge mitjançant el "benchmark" del "cotxe i la muntanya". Els resultats obtinguts en aquest problema milloren els resultats d'altres metodologies, demostrant la millor capacitat de generalització de les xarxes neurals.

Improving Student Engagement and Verbal Behavior Through Cooperative Learning

In this action research study of my classroom of 10th grade Algebra II students, I investigated three related areas. First, I looked at how heterogeneous cooperative groups, where students in the group are responsible to present material, increase the number of students on task and the time on task when compared to individual practice. I noticed that their time on task might have been about the same, but they were communicating with each other mathematically. The second area I examined was the effect heterogeneous cooperative groups had on the teacher’s and the students’ verbal and nonverbal problem solving skills and understanding when compared to individual practice. At the end of the action research, students were questioning each other, and the instructor was answering questions only when the entire group had a question. The third area of data collection focused on what effect heterogeneous cooperative groups had on students’ listening skills when compared to individual practice. In the research I implemented individual quizzes and individual presentations. Both of these had a positive effect on listing in the groups. As a result of this research, I plan to continue implementing the round robin style of in- class practice with heterogeneous grouping and randomly selected individual presentations. For individual accountability I will continue the practice of individual quizzes one to two times a week.

Improving Student Engagement and Verbal Behavior Through Cooperative Learning

In this action research study of my classroom of 10th grade Algebra II students, I investigated three related areas. First, I looked at how heterogeneous cooperative groups, where students in the group are responsible to present material, increase the number of students on task and the time on task when compared to individual practice. I noticed that their time on task might have been about the same, but they were communicating with each other mathematically. The second area I examined was the effect heterogeneous cooperative groups had on the teacher’s and the students’ verbal and nonverbal problem solving skills and understanding when compared to individual practice. At the end of the action research, students were questioning each other, and the instructor was answering questions only when the entire group had a question. The third area of data collection focused on what effect heterogeneous cooperative groups had on students’ listening skills when compared to individual practice. In the research I implemented individual quizzes and individual presentations. Both of these had a positive effect on listing in the groups. As a result of this research, I plan to continue implementing the round robin style of in- class practice with heterogeneous grouping and randomly selected individual presentations. For individual accountability I will continue the practice of individual quizzes one to two times a week.

Critical behavior in a cross-situational lexicon learning scenario

The associationist account for early word learning is based on the co-occurrence between referents and words. Here we introduce a noisy cross-situational learning scenario in which the referent of the uttered word is eliminated from the context with probability gamma, thus modeling the noise produced by out-of-context words. We examine the performance of a simple associative learning algorithm and find a critical value of the noise parameter gamma(c) above which learning is impossible. We use finite-size scaling to show that the sharpness of the transition persists across a region of order tau(-1/2) about gamma(c), where tau is the number of learning trials, as well as to obtain the learning error (scaling function) in the critical region. In addition, we show that the distribution of durations of periods when the learning error is zero is a power law with exponent -3/2 at the critical point. Copyright (C) EPLA, 2012

Behavior competence development through e-learning: experience at the undergraduate level in the context of Aula a Distancia Abierta (ADA) Madrid, Spain

This communication presents the results of an innovative approach for competencedevelopment suggesting a new methodology for the integration of these elements in professional development within the ADA initiative (AulaaDistanciaAbierta, Distance and Open Classroom) of the Community of Madrid. The main objective of this initiative is to promote the use of Information and Communication Technologies (ICTs) for educational activities by creating a new learning environment structured on the premises of commitment to self–learning, individual work, communication and virtual interaction, and self and continuous assessment. Results from this experience showed that conceptualization is a positive contribution to learning, as students added names and characteristics to competences and abilities that were previously unknown or underestimated. Also, the diversity of participants’ disciplines indicated multidimensional interest in this idea and supported the theory that this approach to competencedevelopment could be successful in all knowledge areas.

JAABA: interactive machine learning for automatic annotation of animal behavior

We present a machine learning-based system for automatically computing interpretable, quantitative measures of animal behavior. Through our interactive system, users encode their intuition about behavior by annotating a small set of video frames. These manual labels are converted into classifiers that can automatically annotate behaviors in screen-scale data sets. Our general-purpose system can create a variety of accurate individual and social behavior classifiers for different organisms, including mice and adult and larval Drosophila.

The Impact of Motor Learning on Motor Behavior and Cortical Dynamics in a Complex Stressful Social Environment

An economy of effort is a core characteristic of highly skilled motor performance often described as being effortless or automatic. Electroencephalographic (EEG) evaluation of cortical activity in elite performers has consistently revealed a reduction in extraneous associative cortical activity and an enhancement of task-relevant cortical processes. However, this has only been demonstrated under what are essentially practice-like conditions. Recently it has been shown that cerebral cortical activity becomes less efficient when performance occurs in a stressful, complex social environment. This dissertation examines the impact of motor skill training or practice on the EEG cortical dynamics that underlie performance in a stressful, complex social environment. Sixteen ROTC cadets participated in head-to-head pistol shooting competitions before and after completing nine sessions of skill training over three weeks. Spectral power increased in the theta frequency band and decreased in the low alpha frequency band after skill training. EEG Coherence increased in the left frontal region and decreased in the left temporal region after the practice intervention. These suggest a refinement of cerebral cortical dynamics with a reduction of task extraneous processing in the left frontal region and an enhancement of task related processing in the left temporal region consistent with the skill level reached by participants. Partitioning performance into ‘best’ and ‘worst’ based on shot score revealed that deliberate practice appears to optimize cerebral cortical activity of ‘best’ performances which are accompanied by a reduction in task-specific processes reflected by increased high-alpha power, while ‘worst’ performances are characterized by an inappropriate reduction in task-specific processing resulting in a loss of focus reflected by higher high-alpha power after training when compared to ‘best’ performances. Together, these studies demonstrate the power of experience afforded by practice, as a controllable factor, to promote resilience of cerebral cortical efficiency in complex environments.

Learning processes and the neural analysis of conditioning

Classical and operant conditioning principles, such as the behavioral discrepancy-derived assumption that reinforcement always selects antecedent stimulus and response relations, have been studied at the neural level, mainly by observing the strengthening of neuronal responses or synaptic connections. A review of the literature on the neural basis of behavior provided extensive scientific data that indicate a synthesis between the two conditioning processes based mainly on stimulus control in learning tasks. The resulting analysis revealed the following aspects. Dopamine acts as a behavioral discrepancy signal in the midbrain pathway of positive reinforcement, leading toward the nucleus accumbens. Dopamine modulates both types of conditioning in the Aplysia mollusk and in mammals. In vivo and in vitro mollusk preparations show convergence of both types of conditioning in the same motor neuron. Frontal cortical neurons are involved in behavioral discrimination in reversal and extinction procedures, and these neurons preferentially deliver glutamate through conditioned stimulus or discriminative stimulus pathways. Discriminative neural responses can reliably precede operant movements and can also be common to stimuli that share complex symbolic relations. The present article discusses convergent and divergent points between conditioning paradigms at the neural level of analysis to advance our knowledge on reinforcement.

Surface dental enamel lead levels and antisocial behavior in Brazilian adolescents

Lead poisoning has been reportedly linked to a high risk of learning disabilities, aggression and criminal offenses. To study the association between lead exposure and antisocial/delinquent behavior, a cross-sectional study was conducted with 173 Brazilian youths aged 14\201318 and their parents (n = 93), living in impoverished neighborhoods of Bauru-SP, with high criminality indices. Self-Reported Delinquency (SRD) and Child Behavior Checklist (CBCL) questionnaires were used to evaluate delinquent/antisocial behavior. Body lead burdens were evaluated in surface dental enamel acid microbiopsies. The dental enamel lead levels (DELL) were quantified by graphite furnace atomic absorption spectrometry (GFAAS) and phosphorus content was measured using inductively coupled plasma optical emission spectrometry (ICP-OES). Logistic regression was used to identify associations between DELL and each scale defined by CBCL and SRD scores. Odd ratios adjusted for familial and social covariates, considering a group of youths exposed to high lead levels (\2265 75 percentile), indicated that high DELL is associated with increased risk of exceeding the clinical score for somatic complaints, social problems, rule-breaking behavior and externalizing problems (CI 95 per cent). High DELL was not found to be associated with elevated SRD scores. In conclusion, our data support the hypothesis that high-level lead exposure can trigger antisocial behavior, which calls for public policies to prevent lead poisoning

Evaluation of gastrointestinal motility in awake rats: a learning exercise for undergraduate biomedical students

Souza MA, Souza MH, Palheta RC Jr, Cruz PR, Medeiros BA, Rola FH, Magalhaes PJ, Troncon LE, Santos AA. Evaluation of gastrointestinal motility in awake rats: a learning exercise for undergraduate biomedical students. Adv Physiol Educ 33: 343-348, 2009; doi: 10.1152/advan.90176.2008.-Current medical curricula devote scarce time for practical activities on digestive physiology, despite frequent misconceptions about dyspepsia and dysmotility phenomena. Thus, we designed a hands-on activity followed by a small-group discussion on gut motility. Male awake rats were randomly submitted to insulin, control, or hypertonic protocols. Insulin and control rats were gavage fed with 5% glucose solution, whereas hypertonic-fed rats were gavage fed with 50% glucose solution. Insulin treatment was performed 30 min before a meal. All meals (1.5 ml) contained an equal mass of phenol red dye. After 10, 15, or 20 min of meal gavage, rats were euthanized. Each subset consisted of six to eight rats. Dye recovery in the stomach and proximal, middle, and distal small intestine was measured by spectrophotometry, a safe and reliable method that can be performed by minimally trained students. In a separate group of rats, we used the same protocols except that the test meal contained (99m)Tc as a marker. Compared with control, the hypertonic meal delayed gastric emptying and gastrointestinal transit, whereas insulinic hypoglycemia accelerated them. The session helped engage our undergraduate students in observing and analyzing gut motor behavior. In conclusion, the fractional dye retention test can be used as a teaching tool to strengthen the understanding of basic physiopathological features of gastrointestinal motility.