SMILE: Simulator for Maryland Imitation Learning Environment

As robot imitation learning is beginning to replace conventional hand-coded approaches in programming robot behaviors, much work is focusing on learning from the actions of demonstrators. We hypothesize that in many situations, procedural tasks can be learned more effectively by observing object behaviors while completely ignoring the demonstrator's motions. To support studying this hypothesis and robot imitation learning in general, we built a software system named SMILE that is a simulated 3D environment. In this virtual environment, both a simulated robot and a user-controlled demonstrator can manipulate various objects on a tabletop. The demonstrator is not embodied in SMILE, and therefore a recorded demonstration appears as if the objects move on their own. In addition to recording demonstrations, SMILE also allows programing the simulated robot via Matlab scripts, as well as creating highly customizable objects for task scenarios via XML. This report describes the features and usages of SMILE.

Learning Robot Motions from Visual Sensing

Learning from demonstration becomes increasingly popular as an efficient way of robot programming. Not only a scientific interest acts as an inspiration in this case but also the possibility of producing the machines that would find application in different areas of life: robots helping with daily routine at home, high performance automata in industries or friendly toys for children. One way to teach a robot to fulfill complex tasks is to start with simple training exercises, combining them to form more difficult behavior. The objective of the Master’s thesis work was to study robot programming with visual input. Dynamic movement primitives (DMPs) were chosen as a tool for motion learning and generation. Assuming a movement to be a spring system influenced by an external force, making this system move, DMPs represent the motion as a set of non-linear differential equations. During the experiments the properties of DMP, such as temporal and spacial invariance, were examined. The effect of the DMP parameters, including spring coefficient, damping factor, temporal scaling, on the trajectory generated were studied.

Learning Two-Person Interaction Models for Responsive Synthetic Humanoids

Imitation learning is a promising approach for generating life-like behaviors of virtual humans and humanoid robots. So far, however, imitation learning has been mostly restricted to single agent settings where observed motions are adapted to new environment conditions but not to the dynamic behavior of interaction partners. In this paper, we introduce a new imitation learning approach that is based on the simultaneous motion capture of two human interaction partners. From the observed interactions, low-dimensional motion models are extracted and a mapping between these motion models is learned. This interaction model allows the real-time generation of agent behaviors that are responsive to the body movements of an interaction partner. The interaction model can be applied both to the animation of virtual characters as well as to the behavior generation for humanoid robots.

Estratégias baseadas em aprendizado para coordenação de uma frota de robôs em tarefas cooperativas

In multi-robot systems, both control architecture and work strategy represent a challenge for researchers. It is important to have a robust architecture that can be easily adapted to requirement changes. It is also important that work strategy allows robots to complete tasks efficiently, considering that robots interact directly in environments with humans. In this context, this work explores two approaches for robot soccer team coordination for cooperative tasks development. Both approaches are based on a combination of imitation learning and reinforcement learning. Thus, in the first approach was developed a control architecture, a fuzzy inference engine for recognizing situations in robot soccer games, a software for narration of robot soccer games based on the inference engine and the implementation of learning by imitation from observation and analysis of others robotic teams. Moreover, state abstraction was efficiently implemented in reinforcement learning applied to the robot soccer standard problem. Finally, reinforcement learning was implemented in a form where actions are explored only in some states (for example, states where an specialist robot system used them) differently to the traditional form, where actions have to be tested in all states. In the second approach reinforcement learning was implemented with function approximation, for which an algorithm called RBF-Sarsa($lambda$) was created. In both approaches batch reinforcement learning algorithms were implemented and imitation learning was used as a seed for reinforcement learning. Moreover, learning from robotic teams controlled by humans was explored. The proposal in this work had revealed efficient in the robot soccer standard problem and, when implemented in other robotics systems, they will allow that these robotics systems can efficiently and effectively develop assigned tasks. These approaches will give high adaptation capabilities to requirements and environment changes.

Studio longitudinale sulle capacità cognitive del cane: discriminazione di quantità ed apprendimento per imitazione in relazione all'attaccamento

Il presente studio ha indagato e valutato alcune abilità cognitive del cane: la capacità di discriminare quantità e le capacità di apprendimento mediante imitazione; quest’ultima è poi stata messa in relazione con l’attaccamento nei confronti del proprietario. Per l’esecuzione della prima indagine sono stati messi appunto due test: il primo si è basato esclusivamente sulla presentazione di uno stimolo visivo: diversi quantitativi di cibo, differenti tra loro del 50%, sono stati presentati al cane; la scelta effettuata dai soggetti testati è stata premiata con differenti tipi di rinforzo differenziale o non differenziale. Il secondo test è stato diviso in due parti: sono stati presentati al cane diversi quantitativi di cibo sempre differenti tra loro del 50% ma nella prima parte del test l’input sensoriale per il cane è stato esclusivamente uditivo mentre nella seconda parte è stato sia uditivo che visivo. Ove è stato possibile è stato applicato ai cani un cardiofrequenzimetro al fine di eseguire una valutazione delle variazioni della frequenza cardiaca nel corso del test. Lo scopo è stato quello di valutare se i soggetti testati erano in grado di discriminare la quantità maggiore. La seconda indagine ha analizzato le capacità di apprendimento di 36 soggetti che sono stati suddivisi in cani da lavoro e pet. I soggetti protagonisti dello studio hanno eseguito il Mirror Test per la valutazione dell’apprendimento per imitazione. I soggetti presi in considerazione, sono stati sottoposti a scansione termografica all’inizio ed al termine del test ed è stata rilevata la loro frequenza respiratoria nella fase iniziale e finale del test. In 11 soggetti che hanno eseguito il precedente test è stato possibile eseguire anche il Strange Situation Test per la valutazione dell’attaccamento al proprietario; i test in questione sono stati videoregistrati ed analizzati per mezzo di un software preposto (OBSERVER XT 10).

SEQUENTIAL DECISIONS AND PREDICTIONS IN NATURAL LANGUAGE PROCESSING

Natural language processing has achieved great success in a wide range of ap- plications, producing both commercial language services and open-source language tools. However, most methods take a static or batch approach, assuming that the model has all information it needs and makes a one-time prediction. In this disser- tation, we study dynamic problems where the input comes in a sequence instead of all at once, and the output must be produced while the input is arriving. In these problems, predictions are often made based only on partial information. We see this dynamic setting in many real-time, interactive applications. These problems usually involve a trade-off between the amount of input received (cost) and the quality of the output prediction (accuracy). Therefore, the evaluation considers both objectives (e.g., plotting a Pareto curve). Our goal is to develop a formal understanding of sequential prediction and decision-making problems in natural language processing and to propose efficient solutions. Toward this end, we present meta-algorithms that take an existent batch model and produce a dynamic model to handle sequential inputs and outputs. Webuild our framework upon theories of Markov Decision Process (MDP), which allows learning to trade off competing objectives in a principled way. The main machine learning techniques we use are from imitation learning and reinforcement learning, and we advance current techniques to tackle problems arising in our settings. We evaluate our algorithm on a variety of applications, including dependency parsing, machine translation, and question answering. We show that our approach achieves a better cost-accuracy trade-off than the batch approach and heuristic-based decision- making approaches. We first propose a general framework for cost-sensitive prediction, where dif- ferent parts of the input come at different costs. We formulate a decision-making process that selects pieces of the input sequentially, and the selection is adaptive to each instance. Our approach is evaluated on both standard classification tasks and a structured prediction task (dependency parsing). We show that it achieves similar prediction quality to methods that use all input, while inducing a much smaller cost. Next, we extend the framework to problems where the input is revealed incremen- tally in a fixed order. We study two applications: simultaneous machine translation and quiz bowl (incremental text classification). We discuss challenges in this set- ting and show that adding domain knowledge eases the decision-making problem. A central theme throughout the chapters is an MDP formulation of a challenging problem with sequential input/output and trade-off decisions, accompanied by a learning algorithm that solves the MDP.

Exact Combinatorial Optimization with Graph Convolutional Neural Networks

Combinatorial optimization problems are typically tackled by the branch-and-bound paradigm. We propose to learn a variable selection policy for branch-and-bound in mixed-integer linear programming, by imitation learning on a diversified variant of the strong branching expert rule. We encode states as bipartite graphs and parameterize the policy as a graph convolutional neural network. Experiments on a series of synthetic problems demonstrate that our approach produces policies that can improve upon expert-designed branching rules on large problems, and generalize to instances significantly larger than seen during training.

Practical lessons in Welsh in imitation of the natural method of learning to speak a language.

Mode of access: Internet.

Seeing How it Sounds: Observation, Imitation and Improved Learning in Piano Playing

Research in various fields has shown that students benefit from teacher action demonstrations during instruction, establishing the need to better understand the effectiveness of different demonstration types across student proficiency levels. This study centres upon a piano learning and teaching environment in which beginners and intermediate piano students (N=48) learning to perform a specific type of staccato were submitted to three different (group exclusive) teaching conditions: audio-only demonstration of the musical task; observation of the teacher's action demonstration followed by student imitation (blockedobservation); and observation of the teacher's action demonstration whilst alternating imitation of the task with the teacher's performance (interleaved-observation). Learning was measured in relation to students' range of wrist amplitude (RWA) and ratio of sound and inter-sound duration (SIDR) before, during and after training. Observation and imitation of the teacher’s action demonstrations had a beneficial effect on students' staccato knowledge retention at different times after training: students submitted to interleaved-observation presented significantly shorter note duration and larger wrist rotation, and as such, were more proficient at the learned technique in each of the lesson and retention tests than students in the other learning conditions. There were no significant differences in performance or retention for students of different proficiency levels. These findings have relevant implications for instrumental music pedagogy and other contexts where embodied action is an essential aspect of the learning process.

Emergence of joint attention: Relationships between gaze following, social referencing, imitation, and naming in infancy

The authors investigated the extent to which the joint-attention behaviors of gaze following, social referencing, and object-directed imitation were related to each other and to infants vocabulary development in a sample of 60 infants between the ages of 8 and 14 months. Joint-attention skills and vocabulary development were assessed in a laboratory setting. Split-half reliability analyses on the joint-attention measures indicated that the tasks reliably assessed infants' capabilities. In the main analysis, no significant correlations were found among the joint-attention behaviors except for a significant relationship between gaze following and the number of names in infants' productive vocabularies. The overall pattern of results did not replicate results of previous studies (e.g., M. Carpenter, K. Nagell, & M. Tomasello, 1998) that found relationships between various emerging joint-attention behaviors.

Evolutionarily stable learning schedules and cumulative culture in discrete generation models.

Individual learning (e.g., trial-and-error) and social learning (e.g., imitation) are alternative ways of acquiring and expressing the appropriate phenotype in an environment. The optimal choice between using individual learning and/or social learning may be dictated by the life-stage or age of an organism. Of special interest is a learning schedule in which social learning precedes individual learning, because such a schedule is apparently a necessary condition for cumulative culture. Assuming two obligatory learning stages per discrete generation, we obtain the evolutionarily stable learning schedules for the three situations where the environment is constant, fluctuates between generations, or fluctuates within generations. During each learning stage, we assume that an organism may target the optimal phenotype in the current environment by individual learning, and/or the mature phenotype of the previous generation by oblique social learning. In the absence of exogenous costs to learning, the evolutionarily stable learning schedules are predicted to be either pure social learning followed by pure individual learning ("bang-bang" control) or pure individual learning at both stages ("flat" control). Moreover, we find for each situation that the evolutionarily stable learning schedule is also the one that optimizes the learned phenotype at equilibrium.

Acquisition of automatic imitation is sensitive to sensorimotor contingency

The associative sequence learning model proposes that the development of the mirror system depends on the same mechanisms of associative learning that mediate Pavlovian and instrumental conditioning. To test this model, two experiments used the reduction of automatic imitation through incompatible sensorimotor training to assess whether mirror system plasticity is sensitive to contingency (i.e., the extent to which activation of one representation predicts activation of another). In Experiment 1, residual automatic imitation was measured following incompatible training in which the action stimulus was a perfect predictor of the response (contingent) or not at all predictive of the response (noncontingent). A contingency effect was observed: There was less automatic imitation indicative of more learning in the contingent group. Experiment 2 replicated this contingency effect and showed that, as predicted by associative learning theory, it can be abolished by signaling trials in which the response occurs in the absence of an action stimulus. These findings support the view that mirror system development depends on associative learning and indicate that this learning is not purely Hebbian. If this is correct, associative learning theory could be used to explain, predict, and intervene in mirror system development.

Robotic movement elicits automatic imitation

Recent behavioural and neuroimaging studies have found that observation of human movement, but not of robotic movement, gives rise to visuomotor priming. This implies that the 'mirror neuron' or 'action observation–execution matching' system in the premotor and parietal cortices is entirely unresponsive to robotic movement. The present study investigated this hypothesis using an 'automatic imitation' stimulus–response compatibility procedure. Participants were required to perform a prespecified movement (e.g. opening their hand) on presentation of a human or robotic hand in the terminal posture of a compatible movement (opened) or an incompatible movement (closed). Both the human and the robotic stimuli elicited automatic imitation; the prespecified action was initiated faster when it was cued by the compatible movement stimulus than when it was cued by the incompatible movement stimulus. However, even when the human and robotic stimuli were of comparable size, colour and brightness, the human hand had a stronger effect on performance. These results suggest that effector shape is sufficient to allow the action observation–matching system to distinguish human from robotic movement. They also indicate, as one would expect if this system develops through learning, that to varying degrees both human and robotic action can be 'simulated' by the premotor and parietal cortices.

Learning processes in student teaching: Analyses from a study using learning diaries

The present study seeks to obtain deeper insight into the learning processes in practical training in primary teacher education in Upper Austria. Based on the offer-and-use model of instruction, 230 diary entries of 46 student teachers (28 students in their third semester, 18 students in their fifth semester) were analysed with legard to the learning topics, learning sourcesJ and Ìealning processes involved in practical training. The results show a variety of learning forms, ranging from the unreflective imitation of school mentors' practices to active knowledge construction. In addition, they illustrate that the available learning offers were suboptimally utilized by stuclent teachers who failed to work systernatically and continuously on their professional development.

A preference for own-subspecies' song guides vocal learning in a song bird

In many song birds, males develop their songs as adults by imitating the songs of one or more tutors, memorized previously during a sensitive phase early in life. Previous work using two assays, the production of imitations by adult males and playback-induced calling by young birds during the sensitive phase for memorization, has shown that song birds can discriminate between their own and other species' songs. Herein I use both assays to show that male mountain white-crowned sparrows, Zonotrichia leucophrys oriantha, must learn to sing but have a genetic predisposition to memorize and learn the songs of their own subspecies. Playback tests to young naive birds before they even begin to sing reveal that birds give begging calls more in response to oriantha song than to songs of another species. After 10 days of tutoring with songs of either their own or another subspecies, birds continue to give stronger call responses to songs of their own subspecies, irrespective of whether they were tutored with them, and are more discriminating in distinguishing between different dialects of their own subspecies. The memory processes that facilitate recognition and discrimination of own-subspecies' song may also mediate the preferential imitation of song of a bird's own subspecies. Such perceptual biases could constrain the direction and rate of cultural evolution of learned songs.