Reinforcement Learning by Policy Search

One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.

Interaction and Intelligent Behavior

We introduce basic behaviors as primitives for control and learning in situated, embodied agents interacting in complex domains. We propose methods for selecting, formally specifying, algorithmically implementing, empirically evaluating, and combining behaviors from a basic set. We also introduce a general methodology for automatically constructing higher--level behaviors by learning to select from this set. Based on a formulation of reinforcement learning using conditions, behaviors, and shaped reinforcement, out approach makes behavior selection learnable in noisy, uncertain environments with stochastic dynamics. All described ideas are validated with groups of up to 20 mobile robots performing safe--wandering, following, aggregation, dispersion, homing, flocking, foraging, and learning to forage.

Can you tell if they're learning?

The proliferation of Web-based learning objects makes finding and evaluating online resources problematic. While established Learning Analytics methods use Web interaction to evaluate learner engagement, there is uncertainty regarding the appropriateness of these measures. In this paper we propose a method for evaluating pedagogical activity in Web-based comments using a pedagogical framework, and present a preliminary study that assigns a Pedagogical Value (PV) to comments. This has value as it categorises discussion in terms of pedagogical activity rather than Web interaction. Results show that PV is distinct from typical interactional measures; there are negative or insignificant correlations with established Learning Analytics methods, but strong correlations with relevant linguistic indicators of learning, suggesting that the use of pedagogical frameworks may produce more accurate indicators than interaction analysis, and that linguistic rather than interaction analysis has the potential to automatically identify learning behaviour.

Occupational therapy and its relevance in social projects. relationships, learning and opportunities: collection of experiences of University of Alberta Mscot students in Bogotá, Colombia

Occupational therapists are equipped to promote wellbeing through occupation and to enable participation and meaningful engagement of people in their social and physical environments (WFOT, 2012). As such, the role of the occupational therapists is profoundly linked to the social, cultural and environmental characteristics of the contexts in which occupations take place. The central role that context plays in occupational performance creates an interesting dichotomy for the occupational therapist: on one hand, a profound understanding of cultural and social factors is required from the Occupational Therapy (OT) in order to develop a meaningful and successful collaboration with the person; on the other hand, the ability of the occupational therapists to recognize and explore the contextual factor of an occupation-person dyad transcends cultural and spatial barriers. As a result, occupational therapists are equipped to engage in international collaboration and practice, and as such face unique and enriching challenges. International fieldwork experiences have become a tool through which occupational therapists in training can develop the critical skills for understanding the impact of cultural and social factors on occupation. An OT student in an international fieldwork experience faces numerous challenges in leading a process that is both relevant and respectful to the characteristics of the local context: language, cultural perceptions of occupation and personhood, religious backgrounds, health care access, etc. These challenges stand out as ethical considerations that must be considered when navigating an international fieldwork experience (AOTA, 2009). For more than five years now, the Faculty of Rehabilitation Medicine (FRM) of the University of Alberta (UoFA) and the School of Medicine and Health Sciences at the Universidad del Rosario (UR), Bogota, Colombia, have sustained a productive and meaningful international collaboration. This collaboration includes a visit by Dr. Albert Cook, professor of the FRM and former dean, to the UR as the main guest speaker in the International Congress of Technologies for Disability Support (IBERDISCAP) in 2008. Furthermore, Dr. Cook was a speaker in the research seminar of the Assistive Technology Research Group of the Universidad del Rosario. Following Dr. Cook’s visit, Professors Liliana Álvarez and Adriana Ríos travelled to Edmonton and initiated collaboration with the FRM, resulting in the signing of an agreement between the FRM and the UR in 2009, agreement that has been maintained to this day. The main goal of this agreement is to increase academic and cultural cooperation between the UR and the UofA. Other activities have included the cooperation between Dr. Kim Adams (who has largely maintained interest and effort in supporting the capacity building of the UR rehabilitation programs in coordinating the provision of research placement opportunities for UR students at the UofA), an Assistive Technology course for clinicians and students led by Dr. Adams, and a research project that researched the use of basic cell phones to provide social interaction and health information access for people with disabilities in a low-income community in Colombia (led by Tim Barlott, OT, MSc, under the supervision of Dr. Adams). Since the beginning, the occupational therapy programs of the Universidad del Rosario and the University of Alberta have promoted this collaboration and have strived to engage in interactions that provide further development opportunities for students and staff. As part of this process, the international placement experience of UofA OT students was born under the leadership of: Claudia Rozo, OT program director at UR, placement and academic leadership of Elvis Castro and Angélica Monsalve, professors of the occupational therapy program at UR; and Dr. Lili Liu, OT department director at UofA, Cori Schmitz, Academic coordinator of clinical education at the UofA; and Tim Barlott and Liliana Álvarez leading the international and cross-cultural aspect of this collaboration.This publication summarizes and illustrates the process of international placement in community settings in Colombia, undertaken by occupational therapy students of the University of Alberta. It is our hope that this document can provide and document the ethical considerations of international fieldwork experience, the special characteristics of communities and the ways in which cultural and social competences are developed and help international students navigate the international setting. We also hope that this document will stimulate discussion among professional and academic communities about the importance and richness of international placement experiences and encourage staff and students to articulate their daily efforts with the global occupational therapy agenda.

An artificial economy based on reinforcement learning and agent based modeling

In this paper, we employ techniques from artificial intelligence such as reinforcement learning and agent based modeling as building blocks of a computational model for an economy based on conventions. First we model the interaction among firms in the private sector. These firms behave in an information environment based on conventions, meaning that a firm is likely to behave as its neighbors if it observes that their actions lead to a good pay off. On the other hand, we propose the use of reinforcement learning as a computational model for the role of the government in the economy, as the agent that determines the fiscal policy, and whose objective is to maximize the growth of the economy. We present the implementation of a simulator of the proposed model based on SWARM, that employs the SARSA(λ) algorithm combined with a multilayer perceptron as the function approximation for the action value function.

Gradient-based reinforcement learning techniques for underwater robotics behavior learning

Darrerament, l'interès pel desenvolupament d'aplicacions amb robots submarins autònoms (AUV) ha crescut de forma considerable. Els AUVs són atractius gràcies al seu tamany i el fet que no necessiten un operador humà per pilotar-los. Tot i això, és impossible comparar, en termes d'eficiència i flexibilitat, l'habilitat d'un pilot humà amb les escasses capacitats operatives que ofereixen els AUVs actuals. L'utilització de AUVs per cobrir grans àrees implica resoldre problemes complexos, especialment si es desitja que el nostre robot reaccioni en temps real a canvis sobtats en les condicions de treball. Per aquestes raons, el desenvolupament de sistemes de control autònom amb l'objectiu de millorar aquestes capacitats ha esdevingut una prioritat. Aquesta tesi tracta sobre el problema de la presa de decisions utilizant AUVs. El treball presentat es centra en l'estudi, disseny i aplicació de comportaments per a AUVs utilitzant tècniques d'aprenentatge per reforç (RL). La contribució principal d'aquesta tesi consisteix en l'aplicació de diverses tècniques de RL per tal de millorar l'autonomia dels robots submarins, amb l'objectiu final de demostrar la viabilitat d'aquests algoritmes per aprendre tasques submarines autònomes en temps real. En RL, el robot intenta maximitzar un reforç escalar obtingut com a conseqüència de la seva interacció amb l'entorn. L'objectiu és trobar una política òptima que relaciona tots els estats possibles amb les accions a executar per a cada estat que maximitzen la suma de reforços totals. Així, aquesta tesi investiga principalment dues tipologies d'algoritmes basats en RL: mètodes basats en funcions de valor (VF) i mètodes basats en el gradient (PG). Els resultats experimentals finals mostren el robot submarí Ictineu en una tasca autònoma real de seguiment de cables submarins. Per portar-la a terme, s'ha dissenyat un algoritme anomenat mètode d'Actor i Crític (AC), fruit de la fusió de mètodes VF amb tècniques de PG.

Implicit and explicit processes in a second-language learning task

Two experiments examined the learning of a set of Greek pronunciation rules through explicit and implicit modes of rule presentation. Experiment 1 compared the effectiveness of implicit and explicit modes of presentation in two modalities, visual and auditory. Subjects in the explicit or rule group were presented with the rule set, and those in the implicit or natural group were shown a set of Greek words, composed of letters from the rule set, linked to their pronunciations. Subjects learned the Greek words to criterion and were then given a series of tests which aimed to tap different types of knowledge. The results showed an advantage of explicit study of the rules. In addition, an interaction was found between mode of presentation and modality. Explicit instruction was more effective in the visual than in the auditory modality, whereas there was no modality effect for implicit instruction. Experiment 2 examined a possible reason for the advantage of the rule groups by comparing different combinations of explicit and implicit presentation in the study and learning phases. The results suggested that explicit presentation of the rules is only beneficial when it is followed by practice at applying them.

The symbiotic roles of empirical experimentation and thought experimentation in the learning of physics

This study was an attempt to identify the epistemological roots of knowledge when students carry out hands-on experiments in physics. We found that, within the context of designing a solution to a stated problem, subjects constructed and ran thought experiments intertwined within the processes of conducting physical experiments. We show that the process of alternating between these two modes- empirically experimenting and experimenting in thought- leads towards a convergence on scientifically acceptable concepts. We call this process mutual projection. In the process of mutual projection, external representations were generated. Objects in the physical environment were represented in an imaginary world and these representations were associated with processes in the physical world. It is through this coupling that constituents of both the imaginary world and the physical world gain meaning. We further show that the external representations are rooted in sensory interaction and constitute a semi-symbolic pictorial communication system, a sort of primitive 'language', which is developed as the practical work continues. The constituents of this pictorial communication system are used in the thought experiments taking place in association with the empirical experimentation. The results of this study provide a model of physics learning during hands-on experimentation.

Learning and natal host influence host preference, handling time and sex allocation behaviour in a pupal parasitoid

The host choice and sex allocation decisions of a foraging female parasitoid will have an enormous influence on the life-history characteristics of her offspring. The pteromalid Pachycrepoideus vindemiae is a generalist idiobiont pupal parasitoid of many species of cyclorrhaphous Diptera. Wasps reared in Musca domestica were larger, had higher attack rates and greater male mating success than those reared in Drosophila melanogaster. In no-choice situations, naive female R vindemiae took significantly less time to accept hosts conspecific with their natal host. Parasitoids that emerged from M. domestica pupae spent similar amounts of time ovipositing in both D. melanogaster and M. domestica. Those parasitoids that had emerged from D. melanogaster spent significantly longer attacking M. domestica pupae. The host choice behaviour of female P. vindemiae was influenced by an interaction between natal host and experience. Female R vindemiae reared in M. domestica only showed a preference among hosts when allowed to gain experience attacking M. domestica, preferentially attacking that species. Similarly, female parasitoids reared on D. melanogaster only showed a preference among hosts when allowed to gain experience attacking D. melanogaster, again preferentially attacking that species. Wasp natal host also influenced sex allocation behaviour. While wasps from both hosts oviposited more females in the larger host, M. domestica, wasps that emerged from M. domestica had significantly more male-biased offspring sex ratios. These results indicate the importance of learning and natal host size in determining R vindemiae attack rates. mating success, host preference and sex allocation behaviour, all critical components of parasitoid fitness.

Using learning networks to understand complex systems: a case study of biological, geophysical and social research in the Amazon

Developing high-quality scientific research will be most effective if research communities with diverse skills and interests are able to share information and knowledge, are aware of the major challenges across disciplines, and can exploit economies of scale to provide robust answers and better inform policy. We evaluate opportunities and challenges facing the development of a more interactive research environment by developing an interdisciplinary synthesis of research on a single geographic region. We focus on the Amazon as it is of enormous regional and global environmental importance and faces a highly uncertain future. To take stock of existing knowledge and provide a framework for analysis we present a set of mini-reviews from fourteen different areas of research, encompassing taxonomy, biodiversity, biogeography, vegetation dynamics, landscape ecology, earth-atmosphere interactions, ecosystem processes, fire, deforestation dynamics, hydrology, hunting, conservation planning, livelihoods, and payments for ecosystem services. Each review highlights the current state of knowledge and identifies research priorities, including major challenges and opportunities. We show that while substantial progress is being made across many areas of scientific research, our understanding of specific issues is often dependent on knowledge from other disciplines. Accelerating the acquisition of reliable and contextualized knowledge about the fate of complex pristine and modified ecosystems is partly dependent on our ability to exploit economies of scale in shared resources and technical expertise, recognise and make explicit interconnections and feedbacks among sub-disciplines, increase the temporal and spatial scale of existing studies, and improve the dissemination of scientific findings to policy makers and society at large. Enhancing interaction among research efforts is vital if we are to make the most of limited funds and overcome the challenges posed by addressing large-scale interdisciplinary questions. Bringing together a diverse scientific community with a single geographic focus can help increase awareness of research questions both within and among disciplines, and reveal the opportunities that may exist for advancing acquisition of reliable knowledge. This approach could be useful for a variety of globally important scientific questions.

Livestock and learning: evaluation of a prototype 3D virtual learning environment for livestock practitioners in India and Bolivia

Livestock are a key asset for the global poor. However, access to relevant information is a critical issue for both the poor and the practitioners who serve them. Therefore, the authors describe a web-based Virtual Learning Environment to disseminate educational materials on priority animal health constraints in Bolivia and India. The aim was to explore demand for 3D among development practitioners in the South. Two wider arguments from the ICT4D literature framed the analysis: the concept of 3D as a ‘lead technology’ and the relevance of Internet skills to the adoption of a 3D format. The results illustrated that neither construct influenced demand. Rather, study participants were ready adopters but desired greater levels of interaction and thereby, a more collaborative learning environment. Therefore, 3D has a number of potential benefits to enhance knowledge sharing among community practitioners in the Global South.

Exploring business and IT alignment mechanisms: toward a learning perspective

Numerous studies have attempted to develop strategic alignment mechanisms. The strategic alignment mechanism is broken down into two categories namely: strategy process and strategy content. Our review shows that alignment research has been carried out in isolation. We see this as having had the effect of limiting the extent to which executives can understand elements of performance. We confer with a number of researchers in postulating that using a mechanism such as multilevel learning to combine strategy content and strategy process under one metaphor can greatly facilitate, through exploration and exploitation, the understanding not only of human interactions within a firm, but also of the interaction existent between a firm and its environment. The findings in this study further support the idea of integrating strategy process and content to have a better understating of alignment maturity and impact on business performance. It also elaborates the affect of misalignment in companies on performance.

Latent learning in end stage renal disease (ESRD)

Cognitive functions such as attention and memory are known to be impaired in End Stage Renal Disease (ESRD), but the sites of the neural changes underlying these impairments are uncertain. Patients and controls took part in a latent learning task, which had previously shown a dissociation between patients with Parkinson’s disease and those with medial temporal damage. ESRD patients (n=24) and age and education-matched controls (n=24) were randomly assigned to either an exposed or unexposed condition. In Phase 1 of the task, participants learned that a cue (word) on the back of a schematic head predicted that the subsequently seen face would be smiling. For the exposed (but not unexposed) condition, an additional (irrelevant) colour cue was shown during presentation. In Phase 2, a different association, between colour and facial expression, was learned. Instructions were the same for each phase: participants had to predict whether the subsequently viewed face was going to be happy or sad. No difference in error rate between the groups was found in Phase 1, suggesting that patients and controls learned at a similar rate. However, in Phase 2, a significant interaction was found between group and condition, with exposed controls performing significantly worse than unexposed (therefore demonstrating learned irrelevance). In contrast, exposed patients made a similar number of errors to unexposed in Phase 2. The pattern of results in ESRD was different from that previously found in Parkinson’s disease, suggesting a different neural origin.

Design patterns for promoting peer interaction in discussion forums in MOOCs

Design patterns are a way of sharing evidence-based solutions to educational design problems. The design patterns presented in this paper were produced through a series of workshops, which aimed to identify Massive Open Online Course (MOOC) design principles from workshop participants’ experiences of designing, teaching and learning on these courses. MOOCs present a challenge for the existing pedagogy of online learning, particularly as it relates to promoting peer interaction and discussion. MOOC cohort sizes, participation patterns and diversity of learners mean that discussions can remain superficial, become difficult to navigate, or never develop beyond isolated posts. In addition, MOOC platforms may not provide sufficient tools to support moderation. This paper draws on four case studies of designing and teaching on a range of MOOCs presenting seven design narratives relating to the experience in these MOOCs. Evidence presented in the narratives is abstracted in the form of three design patterns created through a collaborative process using techniques similar to those used in collective autoethnography. The patterns: “Special Interest Discussions”, “Celebrity Touch” and “Look and Engage”, draw together shared lessons and present possible solutions to the problem of creating, managing and facilitating meaningful discussion in MOOCs through the careful use of staged learning activities and facilitation strategies.