Does temporal discounting explain unhealthy behavior? A systematic review and reinforcement learning perspective.

The tendency to make unhealthy choices is hypothesized to be related to an individual's temporal discount rate, the theoretical rate at which they devalue delayed rewards. Furthermore, a particular form of temporal discounting, hyperbolic discounting, has been proposed to explain why unhealthy behavior can occur despite healthy intentions. We examine these two hypotheses in turn. We first systematically review studies which investigate whether discount rates can predict unhealthy behavior. These studies reveal that high discount rates for money (and in some instances food or drug rewards) are associated with several unhealthy behaviors and markers of health status, establishing discounting as a promising predictive measure. We secondly examine whether intention-incongruent unhealthy actions are consistent with hyperbolic discounting. We conclude that intention-incongruent actions are often triggered by environmental cues or changes in motivational state, whose effects are not parameterized by hyperbolic discounting. We propose a framework for understanding these state-based effects in terms of the interplay of two distinct reinforcement learning mechanisms: a "model-based" (or goal-directed) system and a "model-free" (or habitual) system. Under this framework, while discounting of delayed health may contribute to the initiation of unhealthy behavior, with repetition, many unhealthy behaviors become habitual; if health goals then change, habitual behavior can still arise in response to environmental cues. We propose that the burgeoning development of computational models of these processes will permit further identification of health decision-making phenotypes.

Behavior learning of a face robot based on the characteristics of human instruction

Computer simulation experiments were performed to examine the effectiveness of OR- and comparative-reinforcement learning algorithms. In the simulation, human rewards were given as +1 and -1. Two models of human instruction that determine which reward is to be given in every step of a human instruction were used. Results show that human instruction may have a possibility of including both model-A and model-B characteristics, and it can be expected that the comparative-reinforcement learning algorithm is more effective for learning by human instructions.

Learning and Reuse of Experience in Behavior-Based Service Robots

M.H.Lee, Q. Meng and H. Holstein, ?Learning and Reuse of Experience in Behavior-Based Service Robots?, Seventh International Conference on Control, Automation, Robotics and Vision (ICARCV2002), pp1019-24, December 2002, Singapore

Did children’s perceptions of an after-school social learning program predict change in their behavior?

This research explored the influence of children’s perceptions of a pro-social behavior after-school program on actual change in the children’s behavioral outcomes over the program’s duration. Children’s perceptions of three program processes were collected as well as self-reported pro-social and anti-social behavior before and after the program. Statistical models showed that: Positive perceptions of the program facilitators’ dispositions significantly predicted reductions in anti-social behavior; and positive perceptions with the program activities significantly predicted gains in pro-social behavior. The children’s perceptions of their peers’ behavior in the sessions were not found to a significant predictor of behavioral change. The two significant perceptual indicators predicted a small percentage of the change in the behavioral outcomes. However, as after-school social learning programs have a research history of problematic implementation children’s perceptions should be considered in future program design, evaluation and monitoring.

Can Future Managers and Business Executives be Influenced to Behave more Ethically in the Workplace? The Impact of Approaches to Learning on Business Students’ Cheating Behavior

This study considers the potential for influencing business students to become ethical managers by directing their undergraduate learning environment. In particular, the relationship between business students’ academic cheating, as a predictor of workplace ethical behavior, and their approaches to learning is explored. The three approaches to learning identified from the students’ approaches to learning literature are deep approach, represented by an intrinsic interest in and a desire to understand the subject, surface approach, characterized by rote learning and memorization without understanding, and strategic approach, associated with competitive students whose motivation is the achievement of good grades by adopting either a surface or deep approach. Consistent with the hypothesized theoretical model, structural equation modeling revealed that the surface approach is associated with higher levels of cheating, while the deep approach is related to lower levels. The strategic approach was also associated with less cheating and had a statistically stronger influence than the deep approach. Further, a significantly positive relationship reported between deep and strategic approaches suggests that cheating is reduced when deep and strategic approaches are paired. These findings suggest that future managers and business executives can be influenced to behave more ethically in the workplace by directing their learning approaches. It is hoped that the evidence presented may encourage those involved in the design of business programs to implement educational strategies which optimize students’ approaches to learning towards deep and strategic characteristics, thereby equipping tomorrow’s managers and business executives with skills to recognize and respond appropriately to workplace ethical dilemmas.

A behavior-based scheme using reinforcement learning for autonomous underwater vehicles

This paper presents a hybrid behavior-based scheme using reinforcement learning for high-level control of autonomous underwater vehicles (AUVs). Two main features of the presented approach are hybrid behavior coordination and semi on-line neural-Q_learning (SONQL). Hybrid behavior coordination takes advantages of robustness and modularity in the competitive approach as well as efficient trajectories in the cooperative approach. SONQL, a new continuous approach of the Q_learning algorithm with a multilayer neural network is used to learn behavior state/action mapping online. Experimental results show the feasibility of the presented approach for AUVs

Gradient-based reinforcement learning techniques for underwater robotics behavior learning

Darrerament, l'interès pel desenvolupament d'aplicacions amb robots submarins autònoms (AUV) ha crescut de forma considerable. Els AUVs són atractius gràcies al seu tamany i el fet que no necessiten un operador humà per pilotar-los. Tot i això, és impossible comparar, en termes d'eficiència i flexibilitat, l'habilitat d'un pilot humà amb les escasses capacitats operatives que ofereixen els AUVs actuals. L'utilització de AUVs per cobrir grans àrees implica resoldre problemes complexos, especialment si es desitja que el nostre robot reaccioni en temps real a canvis sobtats en les condicions de treball. Per aquestes raons, el desenvolupament de sistemes de control autònom amb l'objectiu de millorar aquestes capacitats ha esdevingut una prioritat. Aquesta tesi tracta sobre el problema de la presa de decisions utilizant AUVs. El treball presentat es centra en l'estudi, disseny i aplicació de comportaments per a AUVs utilitzant tècniques d'aprenentatge per reforç (RL). La contribució principal d'aquesta tesi consisteix en l'aplicació de diverses tècniques de RL per tal de millorar l'autonomia dels robots submarins, amb l'objectiu final de demostrar la viabilitat d'aquests algoritmes per aprendre tasques submarines autònomes en temps real. En RL, el robot intenta maximitzar un reforç escalar obtingut com a conseqüència de la seva interacció amb l'entorn. L'objectiu és trobar una política òptima que relaciona tots els estats possibles amb les accions a executar per a cada estat que maximitzen la suma de reforços totals. Així, aquesta tesi investiga principalment dues tipologies d'algoritmes basats en RL: mètodes basats en funcions de valor (VF) i mètodes basats en el gradient (PG). Els resultats experimentals finals mostren el robot submarí Ictineu en una tasca autònoma real de seguiment de cables submarins. Per portar-la a terme, s'ha dissenyat un algoritme anomenat mètode d'Actor i Crític (AC), fruit de la fusió de mètodes VF amb tècniques de PG.

A proposal of a behavior-based control architecture with reinforcement learning for an autonomous underwater robot

Aquesta tesi proposa l'ús d'un seguit de tècniques pel control a alt nivell d'un robot autònom i també per l'aprenentatge automàtic de comportaments. L'objectiu principal de la tesis fou el de dotar d'intel·ligència als robots autònoms que han d'acomplir unes missions determinades en entorns desconeguts i no estructurats. Una de les premisses tingudes en compte en tots els passos d'aquesta tesis va ser la selecció d'aquelles tècniques que poguessin ésser aplicades en temps real, i demostrar-ne el seu funcionament amb experiments reals. El camp d'aplicació de tots els experiments es la robòtica submarina. En una primera part, la tesis es centra en el disseny d'una arquitectura de control que ha de permetre l'assoliment d'una missió prèviament definida. En particular, la tesis proposa l'ús de les arquitectures de control basades en comportaments per a l'assoliment de cada una de les tasques que composen la totalitat de la missió. Una arquitectura d'aquest tipus està formada per un conjunt independent de comportaments, els quals representen diferents intencions del robot (ex.: "anar a una posició", "evitar obstacles",...). Es presenta una recerca bibliogràfica sobre aquest camp i alhora es mostren els resultats d'aplicar quatre de les arquitectures basades en comportaments més representatives a una tasca concreta. De l'anàlisi dels resultats se'n deriva que un dels factors que més influeixen en el rendiment d'aquestes arquitectures, és la metodologia emprada per coordinar les respostes dels comportaments. Per una banda, la coordinació competitiva és aquella en que només un dels comportaments controla el robot. Per altra banda, en la coordinació cooperativa el control del robot és realitza a partir d'una fusió de totes les respostes dels comportaments actius. La tesis, proposa un esquema híbrid d'arquitectura capaç de beneficiar-se dels principals avantatges d'ambdues metodologies. En una segona part, la tesis proposa la utilització de l'aprenentatge per reforç per aprendre l'estructura interna dels comportaments. Aquest tipus d'aprenentatge és adequat per entorns desconeguts i el procés d'aprenentatge es realitza al mateix temps que el robot està explorant l'entorn. La tesis presenta també un estat de l'art d'aquest camp, en el que es detallen els principals problemes que apareixen en utilitzar els algoritmes d'aprenentatge per reforç en aplicacions reals, com la robòtica. El problema de la generalització és un dels que més influeix i consisteix en permetre l'ús de variables continues sense augmentar substancialment el temps de convergència. Després de descriure breument les principals metodologies per generalitzar, la tesis proposa l'ús d'una xarxa neural combinada amb l'algoritme d'aprenentatge per reforç Q_learning. Aquesta combinació proporciona una gran capacitat de generalització i una molt bona disposició per aprendre en tasques de robòtica amb exigències de temps real. No obstant, les xarxes neurals són aproximadors de funcions no-locals, el que significa que en treballar amb un conjunt de dades no homogeni es produeix una interferència: aprendre en un subconjunt de l'espai significa desaprendre en la resta de l'espai. El problema de la interferència afecta de manera directa en robòtica, ja que l'exploració de l'espai es realitza sempre localment. L'algoritme proposat en la tesi té en compte aquest problema i manté una base de dades representativa de totes les zones explorades. Així doncs, totes les mostres de la base de dades s'utilitzen per actualitzar la xarxa neural, i per tant, l'aprenentatge és homogeni. Finalment, la tesi presenta els resultats obtinguts amb la arquitectura de control basada en comportaments i l'algoritme d'aprenentatge per reforç. Els experiments es realitzen amb el robot URIS, desenvolupat a la Universitat de Girona, i el comportament après és el seguiment d'un objecte mitjançant visió per computador. La tesi detalla tots els dispositius desenvolupats pels experiments així com les característiques del propi robot submarí. Els resultats obtinguts demostren la idoneïtat de les propostes en permetre l'aprenentatge del comportament en temps real. En un segon apartat de resultats es demostra la capacitat de generalització de l'algoritme d'aprenentatge mitjançant el "benchmark" del "cotxe i la muntanya". Els resultats obtinguts en aquest problema milloren els resultats d'altres metodologies, demostrant la millor capacitat de generalització de les xarxes neurals.

Improving Student Engagement and Verbal Behavior Through Cooperative Learning

In this action research study of my classroom of 10th grade Algebra II students, I investigated three related areas. First, I looked at how heterogeneous cooperative groups, where students in the group are responsible to present material, increase the number of students on task and the time on task when compared to individual practice. I noticed that their time on task might have been about the same, but they were communicating with each other mathematically. The second area I examined was the effect heterogeneous cooperative groups had on the teacher’s and the students’ verbal and nonverbal problem solving skills and understanding when compared to individual practice. At the end of the action research, students were questioning each other, and the instructor was answering questions only when the entire group had a question. The third area of data collection focused on what effect heterogeneous cooperative groups had on students’ listening skills when compared to individual practice. In the research I implemented individual quizzes and individual presentations. Both of these had a positive effect on listing in the groups. As a result of this research, I plan to continue implementing the round robin style of in- class practice with heterogeneous grouping and randomly selected individual presentations. For individual accountability I will continue the practice of individual quizzes one to two times a week.

Improving Student Engagement and Verbal Behavior Through Cooperative Learning

In this action research study of my classroom of 10th grade Algebra II students, I investigated three related areas. First, I looked at how heterogeneous cooperative groups, where students in the group are responsible to present material, increase the number of students on task and the time on task when compared to individual practice. I noticed that their time on task might have been about the same, but they were communicating with each other mathematically. The second area I examined was the effect heterogeneous cooperative groups had on the teacher’s and the students’ verbal and nonverbal problem solving skills and understanding when compared to individual practice. At the end of the action research, students were questioning each other, and the instructor was answering questions only when the entire group had a question. The third area of data collection focused on what effect heterogeneous cooperative groups had on students’ listening skills when compared to individual practice. In the research I implemented individual quizzes and individual presentations. Both of these had a positive effect on listing in the groups. As a result of this research, I plan to continue implementing the round robin style of in- class practice with heterogeneous grouping and randomly selected individual presentations. For individual accountability I will continue the practice of individual quizzes one to two times a week.

Critical behavior in a cross-situational lexicon learning scenario

The associationist account for early word learning is based on the co-occurrence between referents and words. Here we introduce a noisy cross-situational learning scenario in which the referent of the uttered word is eliminated from the context with probability gamma, thus modeling the noise produced by out-of-context words. We examine the performance of a simple associative learning algorithm and find a critical value of the noise parameter gamma(c) above which learning is impossible. We use finite-size scaling to show that the sharpness of the transition persists across a region of order tau(-1/2) about gamma(c), where tau is the number of learning trials, as well as to obtain the learning error (scaling function) in the critical region. In addition, we show that the distribution of durations of periods when the learning error is zero is a power law with exponent -3/2 at the critical point. Copyright (C) EPLA, 2012

Behavior competence development through e-learning: experience at the undergraduate level in the context of Aula a Distancia Abierta (ADA) Madrid, Spain

This communication presents the results of an innovative approach for competencedevelopment suggesting a new methodology for the integration of these elements in professional development within the ADA initiative (AulaaDistanciaAbierta, Distance and Open Classroom) of the Community of Madrid. The main objective of this initiative is to promote the use of Information and Communication Technologies (ICTs) for educational activities by creating a new learning environment structured on the premises of commitment to self–learning, individual work, communication and virtual interaction, and self and continuous assessment. Results from this experience showed that conceptualization is a positive contribution to learning, as students added names and characteristics to competences and abilities that were previously unknown or underestimated. Also, the diversity of participants’ disciplines indicated multidimensional interest in this idea and supported the theory that this approach to competencedevelopment could be successful in all knowledge areas.

JAABA: interactive machine learning for automatic annotation of animal behavior

We present a machine learning-based system for automatically computing interpretable, quantitative measures of animal behavior. Through our interactive system, users encode their intuition about behavior by annotating a small set of video frames. These manual labels are converted into classifiers that can automatically annotate behaviors in screen-scale data sets. Our general-purpose system can create a variety of accurate individual and social behavior classifiers for different organisms, including mice and adult and larval Drosophila.