Learning Robot Motions from Visual Sensing

Learning from demonstration becomes increasingly popular as an efficient way of robot programming. Not only a scientific interest acts as an inspiration in this case but also the possibility of producing the machines that would find application in different areas of life: robots helping with daily routine at home, high performance automata in industries or friendly toys for children. One way to teach a robot to fulfill complex tasks is to start with simple training exercises, combining them to form more difficult behavior. The objective of the Master’s thesis work was to study robot programming with visual input. Dynamic movement primitives (DMPs) were chosen as a tool for motion learning and generation. Assuming a movement to be a spring system influenced by an external force, making this system move, DMPs represent the motion as a set of non-linear differential equations. During the experiments the properties of DMP, such as temporal and spacial invariance, were examined. The effect of the DMP parameters, including spring coefficient, damping factor, temporal scaling, on the trajectory generated were studied.

Fetal alcohol spectrum disorders in Finnish children and adolescents : diagnosis, cognition, behavior, adaptation and brain metabolic alterations

Alcohol consumption during pregnancy can potentially affect the developing fetus in devastating ways, leading to a range of physical, neurological, and behavioral alterations most accurately termed Fetal Alcohol Spectrum Disorders (FASD). Despite the fact that it is a preventable disorder, prenatal alcohol exposure today constitutes a leading cause of intellectual disability in the Western world. In Western countries where prevalence studies have been performed the rates of FASD exceed, for example, autism spectrum disorders, Down’s syndrome and cerebral palsy. In addition to the direct effects of alcohol, children and adolescents with FASD are often exposed to a double burden in life, as their neurological sequelae are accompanied by adverse living surroundings exposing them to further environmental risk. However, children with FASD today remain remarkably underdiagnosed by the health care system. This thesis forms part of a larger multinational research project, The Collaborative Initiative on Fetal Alcohol Spectrum Disorders (the CIFASD), initiated by the National Institute of Alcohol Abuse and Alcoholism (NIAAA) in the U.S.A. The general aim of the present thesis was to examine a cohort of children and adolescents growing up with fetal alcohol-related damage in Finland. The thesis consists of five studies with a broad focus on diagnosis, cognition, behavior, adaptation and brain metabolic alterations in children and adolescents with FASD. The participants consisted of four different groups: one group with histories of prenatal exposure to alcohol, the FASD group; one IQ matched contrast group mostly consisting of children with specific learning disorder (SLD); and two typically-developing control groups (CON1 and CON2). Participants were identified through medical records, random sampling from the Finnish national population registry and email alerts to students. Importantly, the participants in the present studies comprise a group of very carefully clinically characterized children with FASD as the studies were performed in close collaboration with leading experts in the field (Prof. Edward Riley and Prof. Sarah Mattson, Center for Behavioral Teratology, San Diego State University, U.S.A; Prof. Eugene Hoyme, Sanford School of Medicine, University of South Dakota, U.S.A.). In the present thesis, the revised Institute of Medicine diagnostic criteria for FASD were tested on a Finnish population and found to be a reliable tool for differentiating among the subgroups of FASD. A weighted dysmorphology scoring system proved to be a valuable additional adjunct in quantification of growth deficits and dysmorphic features in children with FASD (Study 1). The purpose of Study 2 was to clarify the relationship between alcohol-related dysmorphic features and general cognitive capacity. Results showed a significant correlation between dysmorphic features and cognitive capacity, suggesting that children with more severe growth deficiency and dysmorphic features have more cognitive limitations. This association was, however, only moderate, indicating that physical markers and cognitive capacity not always go hand in hand in individuals with FASD. Behavioral problems in the FASD group proved substantial compared to the typically developing control group. In Study 3 risk and protective factors associated with behavioral problems in the FASD group were explored further focusing on diagnostic and environmental factors. Two groups with elevated risks for behavioral problems emerged: length of time spent in residential care and a low dysmorphology score proved to be the most pervasive risk factor for behavioral problems. The results underscore the clinical importance of appropriate services and care for less visibly alcohol affected children and highlight the need to attend to children with FASD being raised in institutions. With their background of early biological and psychological impairment compounded with less opportunity for a close and continuous caregiver relationship, such children seem to run an especially great risk of adverse life outcomes. Study 4 focused on adaptive abilities such as communication, daily living skills and social skills, in other words skills that are important for gradually enabling an independent life, maintain social relationships and allow the individual to become integrated into society. The results showed that adaptive abilities of children and adolescents growing up with FASD were significantly compromised compared to both typically-developing peers and IQ-matched children with SLD. Clearly different adaptive profiles were revealed where the FASD group performed worse than the SLD group, who in turn performed worse than the CON1 group. Importantly, the SLD group outperformed the FASD group on adaptive behavior in spite of comparable cognitive levels. This is the first study to compare adaptive abilities in a group of children and adolescents with FASD relative to both a contrast group of IQ-matched children with SLD and to a group of typically-developing peers. Finally, in Study 5, through magnetic resonance spectroscopic imaging (MRS) evidence of longstanding neurochemical alterations were observed in adolescents and young adults with FASD related to alcohol exposure in utero 14-20 years earlier. Neurochemical alterations were seen in several brain areas: in frontal and parietal cortices, corpus callosum, thalamus and frontal white matter areas as well as in the cerebellar dentate nucleus. The findings are compatible with neuropsychological findings in FASD. Glial cells seemed to be more affected than neurons. In conclusion, more societal efforts and resources should be focused on recognizing and diagnosing FASD, and supporting subgroups with elevated risk of poor outcome. Without adequate intervention children and adolescents with FASD run a great risk of marginalization and social maladjustment, costly not only to society but also to the lives of the many young people with FASD.

Defending behavior in bullying situations

In order to encourage children and adolescents to defend and support their victimized peers, it is important to identify factors that either maximize or minimize the probability that students will engage in such behaviors. This thesis is composed of four studies designed to elucidate how a variety of factors work in conjunction to explain why some children defend their victimized classmates, whereas others remain passive or reinforce the bully. The conceptual framework of this thesis is drawn from several theoretical considerations, including social cognitive learning theory, the expectancy-value framework as well as the literature emphasizing the importance of empathy in motivating behaviors. Also the child-by-environment perspective and the socialecological perspective influenced this research. Accordingly, several intra- and interpersonal characteristics (e.g., social cognitions, empathy, and social status) as well as group-level factors (e.g., norms) that may either enhance or reduce the probability that students defend their victimized peers are investigated. In Studies I and II, the focus is on social cognitions, and special attention is paid to take into account the domain-specificity of cognition-behavior processes. Self-efficacy for defending is still an interest of study III, but the role of affective empathy on defending is also investigated. Also social status variables (preference and perceived popularity) are evaluated as possible moderators of links between intrapersonal factors and defending. In Study IV, the focus is expanded further by concentrating on characteristics of children’s proximal environments (i.e., classroom). Bullying norms and collective perceptions (i.e., connectedness among the students and the teachers’ ability to deal with bullying situations) are examined. Data are drawn from two research projects: the Kaarina Cohort Study (consisting of fourth and eighth graders) and the randomized controlled trial (RCT) evaluating the effects of the KiVa antibullying program (consisting of third to fifth graders). The results of the thesis suggest that defending the victims of bullying is influenced by a variety of individual level motivational characteristics, such as social cognitions and affective empathy. Also, both perceived popularity and social preference play a role in defending, and the findings support the conceptualization that behavior results from the interplay between the characteristics of an individual child and their social-relational environment. Classroom context further influences students’ defending behavior. Thus, antibullying efforts targeting peer bystanders should aim to influence intra- and interpersonal characteristics of children and adolescents as well as their social environment.

Effects of obtained benefits on willingness to pay and word-of-mouth behavior in newspaper industry

The purpose of this master’s thesis was to investigate the effects which benefits obtained from reading a newspaper and using its website have on behavioral outcomes such as word-of-mouth behavior and willingness to pay. Several other antecedents of willingness to pay have been used as the control variables. However, their interrelations haven’t been hypothesized. The empirical part focused on a case company – Finnish regional newspaper. Empirical research has been conducted using a quantitative method and data was collected via online survey placed on newspaper’s website during 2010. 1001 responses have been collected. The results showed that benefits obtained both from traditional printed newspaper and from online one have positive effects on the word-of-mouth about this newspaper and its website. However, it has been revealed that benefits obtained from reading the newspaper don’t have effect on the willingness to pay for this newspaper. Additionally, only interpersonal and convenience benefits obtained from using the newspaper’s website influence on the willingness to pay for it. Finally, willingness to pay for the bundle of printed newspaper and its website access is affected positively only by the information/learning benefits obtained from reading the newspaper and by the interpersonal benefits obtained from using the newspaper’s website.

Cultural distance and negotiation behavior : Perception of entrepreneurs operating in Turku

This thesis investigates the influence of cultural distance on entrepreneurs’ negotiation behaviour. For this purpose, Turku was chosen as the unit of analysis due to the exponential demographic change experienced during the last two decades that has derived in a more diversified local environment. The research aim set for this study was to identify to what extent entrepreneurs face cultural distance, how cultural distance influences the entrepreneur’s negotiation behaviour and how can it be addressed in order to turn dissimilarities into opportunities. This study presented the relation and apparent dichotomy of cultural distance and global culture, including the component of diversity. The impact of cultural distance in the entrepreneurial mindset and its consequent effect in negotiation behaviour was presented too. Addressing questions about the way individuals perceive, behave and interact allowed the use of interviews for this qualitative research study. In the empirical part of this study it was found that negotiation behaviour differed in terms of how congenial entrepreneurs felt when managing cultural distance, encompassing their performance. It was also acknowledged that after time and effort, some of the personal traits were enhanced while others reduced, allowing for more flexibility and adaptation. Furthermore, depending on the level of trust and shared interests, entrepreneurs determined their attitudinal approach, being adaptive or reactive subject to situational aspects. Additionally, it was found that the acquisition of cultural savvy not necessarily conveyed to more creativity. This experiential learning capability led to the proposition of new ways of behaviour. Likewise, it was proposed that growing cultural intelligence bridge distances, reducing mistrusts and misunderstandings. The capability of building more collaborative relationships allows entrepreneurs to see cultural distance as a cultural perspective instead of as a threat. Therefore it was recommended to focus on proximity rather than distance to better identify and exploit untapped opportunities and better perform when negotiating in whichever cultural conditions.

Long-lasting mnemotropic effect of substance P and its N-terminal fragment (SP1-7) on avoidance learning

We investigated the long-lasting effect of peripheral injection of the neuropeptide substance P (SP) and of some N- or C-terminal SP fragments (SPN and SPC, respectively) on retention test performance of avoidance learning. Male Wistar rats (220 to 280 g) were trained in an inhibitory step-down avoidance task and tested 24 h or 21 days later. Immediately after the training trial rats received an intraperitoneal injection of SP (50 µg/kg), SPN 1-7 (167 µg/kg) or SPC 7-11 (134 µg/kg). Control groups were injected with vehicle or SP 5 h after the training trial. The immediate post-training administration of SP and SPN, but not SPC, facilitated avoidance behavior in rats tested 24 h or 21 days later, i.e., the retention test latencies of the SP and SPN groups were significantly longer (P<0.05, Mann-Whitney U-test) during both training-test intervals. These observations suggest that the memory-enhancing effect of SP is long-lasting and that the amino acid sequence responsible for this effect is encoded by its N-terminal part

The brain decade in debate: I. Neurobiology of learning and memory

This article is a transcription of an electronic symposium in which some active researchers were invited by the Brazilian Society for Neuroscience and Behavior (SBNeC) to discuss the last decade's advances in neurobiology of learning and memory. The way different parts of the brain are recruited during the storage of different kinds of memory (e.g., short-term vs long-term memory, declarative vs procedural memory) and even the property of these divisions were discussed. It was pointed out that the brain does not really store memories, but stores traces of information that are later used to create memories, not always expressing a completely veridical picture of the past experienced reality. To perform this process different parts of the brain act as important nodes of the neural network that encode, store and retrieve the information that will be used to create memories. Some of the brain regions are recognizably active during the activation of short-term working memory (e.g., prefrontal cortex), or the storage of information retrieved as long-term explicit memories (e.g., hippocampus and related cortical areas) or the modulation of the storage of memories related to emotional events (e.g., amygdala). This does not mean that there is a separate neural structure completely supporting the storage of each kind of memory but means that these memories critically depend on the functioning of these neural structures. The current view is that there is no sense in talking about hippocampus-based or amygdala-based memory since this implies that there is a one-to-one correspondence. The present question to be solved is how systems interact in memory. The pertinence of attributing a critical role to cellular processes like synaptic tagging and protein kinase A activation to explain the memory storage processes at the cellular level was also discussed.

A multimethod examination of contributors to successful on-the-job learning for vocational students

Rapid changes in working life and competence requirements of different professions have increased interest in workplace learning. It is considered an effective way to learn and update professional skills by performing daily tasks in an authentic environment. Especially, ensuring a supply of skilled future workers is a crucial issue for firms facing tight competition and a shortage of competent employees due to the retirement of current professionals. In order to develop and make the most of workplace learning, it is important to focus on workplace learning environments and the individual characteristics of those participating in workplace learning. The literature has suggested various factors that influence adults' and professionals’ workplace learning of profession-related skills, but lacks empirical studies on contextual and individual-related factors that positively affect students' workplace learning. Workers with vocational education form a large group in modern firms. Therefore, elements of vocational students’ successful workplace learning during their studies, before starting their career paths, need to be examined. To fill this gap in the literature, this dissertation examines contributors to vocational students’ workplace learning in Finland, where students’ workplace learning is included in the vocational education and training system. The study is divided into two parts: the introduction, comprised of the overview of the relevant literature and the conclusion of the entire study, and five separate articles. Three of the articles utilize quantitative methods and two use qualitative methods to examine factors that contribute to vocational students’ workplace learning. The results show that, from the students’ perspective, attitudinal, motivational, and organizationrelated factors enhance the student’s development of professionalism during the on-the-job learning period. Specifically, the organization-related factors such as innovative climate, guidance, and interactions with seniors have a strong positive impact on the students’ perceived development of professional skills because, for example, the seniors’ guidance and provision of new viewpoints for the tasks helps the vocational students to gain autonomy at work performance. A multilevel analysis shows that of those factors enhancing workplace learning from the student perspective, innovative climate, knowledge transfer accuracy, and the students’ performance orientation were significantly related to the workplace instructors’ assessment regarding the students’ professional performance. Furthermore, support from senior colleagues and the students’ self-efficacy were both significantly associated with the formal grades measuring how well the students managed to learn necessary professional skills. In addition, the results suggest that the students’ on-the-job learning can be divided into three main phases, of which two require efforts from both the student and the on-the-job learning organization. The first phase includes the student’s application of basic professional skills, demonstration of potential in performing daily tasks, and orientation provided by the organization at the beginning of the on-the-job learning period. In the second phase, the student actively develops profession-related skills by performing daily tasks, thus learning a fluent working style while observing the seniors’ performance. The organization offers relevant tasks and follows the student’s development. The third level indicates a student who has reached the professional level described as a full occupation. The results suggest that constructing the vocational students’ successful on-the-job learning period requires feedback from seniors, opportunities to learn to manage entire work processes, self-efficacy on the part of the students, proactive behavior, and initiative in learning. The study contributes to research on workplace learning in three ways: firstly, it identifies the key individual- and organization-based factors that influence the vocational students’ successful on-the-job learning from their perspective and examines mutual relationships between these factors. Second, the study provides knowledge of how the factors related to the students’ view of successful workplace learning are associated with the workplace instructors’ perspective and the formal grades. Third, the present study finds elements needed to construct a successful on-the-job learning for the students.

JNK regulates dendrite and spine architecture in the central nervous system influencing spatial learning and motor tasks

JNK1 is a MAP-kinase that has proven a significant player in the central nervous system. It regulates brain development and the maintenance of dendrites and axons. Several novel phosphorylation targets of JNK1 were identified in a screen performed in the Coffey lab. These proteins were mainly involved in the regulation of neuronal cytoskeleton, influencing the dynamics and stability of microtubules and actin. These structural proteins form the dynamic backbone for the elaborate architecture of the dendritic tree of a neuron. The initiation and branching of the dendrites requires a dynamic interplay between the cytoskeletal building blocks. Both microtubules and actin are decorated by associated proteins which regulate their dynamics. The dendrite-specific, high molecular weight microtubule associated protein 2 (MAP2) is an abundant protein in the brain, the binding of which stabilizes microtubules and influences their bundling. Its expression in non-neuronal cells induces the formation of neurite-like processes from the cell body, and its function is highly regulated by phosphorylation. JNK1 was shown to phosphorylate the proline-rich domain of MAP2 in vivo in a previous study performed in the group. Here we verify three threonine residues (T1619, T1622 and T1625) as JNK1 targets, the phosphorylation of which increases the binding of MAP2 to microtubules. This binding stabilizes the microtubules and increases process formation in non-neuronal cells. Phosphorylation-site mutants were engineered in the lab. The non-phosphorylatable mutant of MAP2 (MAP2- T1619A, T1622A, T1625A) in these residues fails to bind microtubules, while the pseudo-phosphorylated form, MAP2- T1619D, T1622D, Thr1625D, efficiently binds and induces process formation even without the presence of active JNK1. Ectopic expression of the MAP2- T1619D, T1622D, Thr1625D in vivo in mouse brain led to a striking increase in the branching of cortical layer 2/3 (L2/3) pyramidal neurons, compared to MAP2-WT. The dendritic complexity defines the receptive field of a neuron and dictates the output to the postsynaptic cells. Previous studies in the group indicated altered dendrite architecture of the pyramidal neurons in the Jnk1-/- mouse motor cortex. Here, we used Lucifer Yellow loading and Sholl analysis of neurons in order to study the dendritic branching in more detail. We report a striking, opposing effect in the absence of Jnk1 in the cortical layers 2/3 and 5 of the primary motor cortex. The basal dendrites of pyramidal neurons close to the pial surface at L2/3 show a reduced complexity. In contrast, the L5 neurons, which receive massive input from the L2/3 neurons, show greatly increased branching. Another novel substrate identified for JNK1 was MARCKSL1, a protein that regulates actin dynamics. It is highly expressed in neurons, but also in various cancer tissues. Three phosphorylation target residues for JNK1 were identified, and it was demonstrated that their phosphorylation reduces actin turnover and retards migration of these cells. Actin is the main cytoskeletal component in dendritic spines, the site of most excitatory synapses in pyramidal neurons. The density and gross morphology of the Lucifer Yellow filled dendrites were characterized and we show reduced density and altered morphology of spines in the motor cortex and in the hippocampal area CA3. The dynamic dendritic spines are widely considered to function as the cellular correlate during learning. We used a Morris water maze to test spatial memory. Here, the wild-type mice outperformed the knock-out mice during the acquisition phase of the experiment indicating impaired special memory. The L5 pyramidal neurons of the motor cortex project to the spinal cord and regulate the movement of distinct muscle groups. Thus the altered dendrite morphology in the motor cortex was expected to have an effect on the input-output balance in the signaling from the cortex to the lower motor circuits. A battery of behavioral tests were conducted for the wild-type and Jnk1-/- mice, and the knock-outs performed poorly compared to wild-type mice in tests assessing balance and fine motor movements. This study expands our knowledge of JNK1 as an important regulator of the dendritic fields of neurons and their manifestations in behavior.

The neurobiology of infant maternal odor learning

Infant rats must learn to identify their mother’s diet-dependent odor. Once learned, maternal odor controls pups’ approach to the mother, their social behavior and nipple attachment. Here we present a review of the research from four different laboratories, which suggests that neural and behavioral responses to the natural maternal odor and neonatal learned odors are similar. Together, these data indicate that pups have a unique learning circuit relying on the olfactory bulb for neural plasticity and on the hyperfunctioning noradrenergic locus coeruleus flooding the olfactory bulb with norepinephrine to support the neural changes. Another important factor making this system unique is the inability of the amygdala to become incorporated into the infant learning circuit. Thus, infant rats appear to be primed in early life to learn odors that will evoke approach responses supporting attachment to the caregiver.

Low-dose thioperamide injected into the cerebellar vermis of mice immediately after exposure to the elevated plus-maze impairs their avoidance behavior on re-exposure to the apparatus

The present study investigated the effect of thioperamide (THIO), an H3 histaminergic receptor antagonist, microinjected into the cerebellar vermis on emotional memory consolidation in male Swiss albino mice re-exposed to the elevated plus-maze (EPM). We implanted a guide cannula into the cerebellar vermis using stereotactic surgery. On the third day after surgery, we performed behavioral tests for two consecutive days. On the first day (exposure), the mice (n=10/group) were exposed to the EPM and received THIO (0.06, 0.3, or 1.5 ng/0.1 µL) immediately after the end of the session. Twenty-four hours later, the mice were re-exposed to the EPM under the same experimental conditions, but without drug injection. A reduction in the exploration of the open arms upon re-exposure to the EPM (percentage of number of entries and time spent in open arms) compared with the initial exposure was used as an indicator of learning and memory. One-way analysis of variance (ANOVA) followed by the Duncan post hoc test was used to analyze the data. Upon re-exposure, exploratory activity in the open arms was reduced in the control group, and with the two highest THIO doses: 0.3 and 1.5 ng/0.1 µL. No reduction was seen with the lowest THIO dose (0.06 ng/0.1 µL), indicating inhibition of the consolidation of emotional memory. None of the doses interfered with the animals' locomotor activity. We conclude that THIO at the lowest dose (0.06 ng/0.1 µL) microinjected into the cerebellum impaired emotional memory consolidation in mice.

A school-based cognitive behavior modification intervention strategy for learned helpless mentally retarded students

This study assessed the usefulness of a cognitive behavior modification (CBM) intervention package with mentally retarded students in overcoming learned helplessness and improving learning strategies. It also examined the feasibility of instructing teachers in the use of such a training program for a classroom setting. A modified single subject design across individuals was employed using two groups of three subjects. Three students from each of two segregated schools for the mentally retarded were selected using a teacher questionnaire and pupil checklist of the most learned helpless students enrolled there. Three additional learned helplessness assessments were conducted on each subject before and after the intervention in order to evaluate the usefulness of the program in alleviating learned helplessness. A classroom environment was created with the three students from each school engaged in three twenty minute work sessions a week with the experimenter and a tutor experimenter (TE) as instructors. Baseline measurements were established on seven targeted behaviors for each subject: task-relevant speech, task-irrelevant speech, speech denoting a positive evaluation of performance, speech denoting a negative evaluation of performance, proportion of time on task, non-verbal positive evaluation of performance and non-verbal negative evaluation of performance. The intervention package combined a variety of CBM techniques such as Meichenbaum's (1977) Stop, Look and Listen approach, role rehearsal and feedback. During the intervention each subject met with his TE twice a week for an individual half-hour session and one joint twenty minute session with all three students, the experimentor and one TE. Five weeks after the end of this experiment one follow up probe was conducted. All baseline, post-intervention and probe sessions were videotaped. The seven targeted behaviors were coded and comparisons of baseline, post intervention, and probe testing were presented in graph form. Results showed a reduction in learned helplessness in all subjects. Improvement was noted in each of the seven targeted behaviors for each of the six subjects. This study indicated that mentally retarded children can be taught to reduce learned helplessness with the aid of a CBM intervention package. It also showed that CBM is a viable approach in helping mentally retarded students acquire more effective learning strategies. Because the TEs (Tutor experimenters) had no trouble learning and implementing this program, it was considered feasible for teachers to use similar methods in the classroom.

Towards Non Invasive Diagnosis of Scoliosis Using Semi-supervised Learning Approach

In this paper, a new methodology for the prediction of scoliosis curve types from non invasive acquisitions of the back surface of the trunk is proposed. One hundred and fifty-nine scoliosis patients had their back surface acquired in 3D using an optical digitizer. Each surface is then characterized by 45 local measurements of the back surface rotation. Using a semi-supervised algorithm, the classifier is trained with only 32 labeled and 58 unlabeled data. Tested on 69 new samples, the classifier succeeded in classifying correctly 87.0% of the data. After reducing the number of labeled training samples to 12, the behavior of the resulting classifier tends to be similar to the reference case where the classifier is trained only with the maximum number of available labeled data. Moreover, the addition of unlabeled data guided the classifier towards more generalizable boundaries between the classes. Those results provide a proof of feasibility for using a semi-supervised learning algorithm to train a classifier for the prediction of a scoliosis curve type, when only a few training data are labeled. This constitutes a promising clinical finding since it will allow the diagnosis and the follow-up of scoliotic deformities without exposing the patient to X-ray radiations.

Reinforcement Learning by Policy Search

One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.

Teaching an Old Robot New Tricks: Learning Novel Tasks via Interaction with People and Things

As AI has begun to reach out beyond its symbolic, objectivist roots into the embodied, experientialist realm, many projects are exploring different aspects of creating machines which interact with and respond to the world as humans do. Techniques for visual processing, object recognition, emotional response, gesture production and recognition, etc., are necessary components of a complete humanoid robot. However, most projects invariably concentrate on developing a few of these individual components, neglecting the issue of how all of these pieces would eventually fit together. The focus of the work in this dissertation is on creating a framework into which such specific competencies can be embedded, in a way that they can interact with each other and build layers of new functionality. To be of any practical value, such a framework must satisfy the real-world constraints of functioning in real-time with noisy sensors and actuators. The humanoid robot Cog provides an unapologetically adequate platform from which to take on such a challenge. This work makes three contributions to embodied AI. First, it offers a general-purpose architecture for developing behavior-based systems distributed over networks of PC's. Second, it provides a motor-control system that simulates several biological features which impact the development of motor behavior. Third, it develops a framework for a system which enables a robot to learn new behaviors via interacting with itself and the outside world. A few basic functional modules are built into this framework, enough to demonstrate the robot learning some very simple behaviors taught by a human trainer. A primary motivation for this project is the notion that it is practically impossible to build an "intelligent" machine unless it is designed partly to build itself. This work is a proof-of-concept of such an approach to integrating multiple perceptual and motor systems into a complete learning agent.