On the Convergence of Stochastic Iterative Dynamic Programming Algorithms

Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments. These algorithms, including the TD(lambda) algorithm of Sutton (1988) and the Q-learning algorithm of Watkins (1989), can be motivated heuristically as approximations to dynamic programming (DP). In this paper we provide a rigorous proof of convergence of these DP-based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem. The theorem establishes a general class of convergent algorithms to which both TD(lambda) and Q-learning belong.

Integrating methods for developing sustainability indicators that can facilitate learning and action

Bossel's (2001) systems-based approach for deriving comprehensive indicator sets provides one of the most holistic frameworks for developing sustainability indicators. It ensures that indicators cover all important aspects of system viability, performance, and sustainability, and recognizes that a system cannot be assessed in isolation from the systems upon which it depends and which in turn depend upon it. In this reply, we show how Bossel's approach is part of a wider convergence toward integrating participatory and reductionist approaches to measure progress toward sustainable development. However, we also show that further integration of these approaches may be able to improve the accuracy and reliability of indicators to better stimulate community learning and action. Only through active community involvement can indicators facilitate progress toward sustainable development goals. To engage communities effectively in the application of indicators, these communities must be actively involved in developing, and even in proposing, indicators. The accuracy, reliability, and sensitivity of the indicators derived from local communities can be ensured through an iterative process of empirical and community evaluation. Communities are unlikely to invest in measuring sustainability indicators unless monitoring provides immediate and clear benefits. However, in the context of goals, targets, and/or baselines, sustainability indicators can more effectively contribute to a process of development that matches local priorities and engages the interests of local people.

Envisioning sustainable water futures in a transdisciplinary learning process: combining normative, explorative, and participatory scenario approaches

Competing water demands for household consumption as well as the production of food, energy, and other uses pose challenges for water supply and sustainable development in many parts of the world. Designing creative strategies and learning processes for sustainable water governance is thus of prime importance. While this need is uncontested, suitable approaches still have to be found. In this article we present and evaluate a conceptual approach to scenario building aimed at transdisciplinary learning for sustainable water governance. The approach combines normative, explorative, and participatory scenario elements. This combination allows for adequate consideration of stakeholders’ and scientists’ systems, target, and transformation knowledge. Application of the approach in the MontanAqua project in the Swiss Alps confirmed its high potential for co-producing new knowledge and establishing a meaningful and deliberative dialogue between all actors involved. The iterative and combined approach ensured that stakeholders’ knowledge was adequately captured, fed into scientific analysis, and brought back to stakeholders in several cycles, thereby facilitating learning and co-production of new knowledge relevant for both stakeholders and scientists. However, the approach also revealed a number of constraints, including the enormous flexibility required of stakeholders and scientists in order for them to truly engage in the co-production of new knowledge. Overall, the study showed that shifts from strategic to communicative action are possible in an environment of mutual trust. This ultimately depends on creating conditions of interaction that place scientists’ and stakeholders’ knowledge on an equal footing.

Testing the 'Learning for Sustainabiliy' Approach in a Training for Trainers: Contribution to new learning theory and approaches

Mainstreaming the LforS approach is a challenge due to dive rging institutional priorities, customs, and expectations of classically traine d staff. A workshop to test LforS theory and practice, and explore how to mainstream it, took place in a concrete context in a rural district of Mozambique, focusing on agricultural, forest and water resources. The evaluation showed that the principles of interaction applied pe rmitted to link rational know ledge with practical experience through mutual learning and iterative self-reflection. The combination of learning techniques was considered usef ul; participants called for further opportunities to apply the LforS methodology, proposing next steps.

BEYOND WORDS: HOW LEARNING TO READ AND MENTAL HEALTH ARE RELATED FOR STRUGGLING READERS

The overarching purpose of this research program was to describe how intervening for academic deficits may be accompanied by changes in mental health. This multi-dimensional, multi-perspective, and iterative research program was developed to report on two distinct but related studies that addressed the same issue: in what ways does the mental health of students change as they transition from being struggling readers to more able readers? To describe the changes, these studies used a number of qualitative research methodologies—focus groups, individual interviews, and ethnographic case studies. Themes that emerged from the focus group and interview data in the first study were used to create a model that guided observations and interview questions in the second study. The first study described what parents, classroom teachers, and two reading instructors of nine previously struggling readers reported as the outcomes of becoming a more proficient reader. Data from this study indicated three broad domains in which change, as perceived by participants, occurred―cognitive/learning, behavioural/social, and psychological/emotional. Within these three domains, six dimensions were identified as having changed as reading improved: (a) academic achievement, (b) attitude, (c) attention, (d) behaviour, (e) mental health, and (f) empowerment. These domains, dimensions, and 15 constituent elements were used to create the model to guide the subsequent study. The purpose of the second study was to validate and refine this model by using an ethnographic case study approach to explore the ways in which the model accounted for the changes in reading and mental health seen in three boys over the months they participated in the intervention. By investigating the relationship between learning to read and mental health, this research aimed to enhance our understanding of how gains in reading may also improve the mental health of struggling readers. The model was found to be robust and a convenient conceptual framework to further our understanding of this relationship. Importantly, gains made in the cognitive/learning domain through an effective reading intervention, offered in a supportive learning environment, were shown to be accompanied by concomitant gains in both the behavioural/social and psychological/emotional domains—all of which enhance student thriving.

Optimization and Machine Learning Frameworks for Complex Network Analysis

Thesis (Ph.D.)--University of Washington, 2016-06

Measuring environmental education program impacts and learning in the field: Using an action research cycle to develop a tool for use with young students

Despite the increasing importance of, and interest in, documenting the impact of environmental education programs on students' learning for sustainability, few tools are currently available to measure young students' environmental learning across all the dimensions of knowledge, skills, attitudes and behaviours. This paper reports on the development of such a tool, using an iterative action research process with 134 students, aged six to eleven, attending programs at an Environmental Education Centre in Queensland. The resulting instrument, the Environmental Learning Outcomes Survey (ELOS) incorporates observations of students' engagement in learning processes as well as measuring learning outcomes, and allows both of these aspects to be linked to particular components of the environmental education program. Test data using the instrument are reported to illustrate its potential usefulness. It is envisaged that the refined instrument will enable researchers to measure student environmental learning in the field, investigate environmental education program impacts and identify aspects of programs that are most effective in facilitating student learning. [Author abstract]

An incremental tri-partite approach to ontology learning

In this paper we present a new approach to ontology learning. Its basis lies in a dynamic and iterative view of knowledge acquisition for ontologies. The Abraxas approach is founded on three resources, a set of texts, a set of learning patterns and a set of ontological triples, each of which must remain in equilibrium. As events occur which disturb this equilibrium various actions are triggered to re-establish a balance between the resources. Such events include acquisition of a further text from external resources such as the Web or the addition of ontological triples to the ontology. We develop the concept of a knowledge gap between the coverage of an ontology and the corpus of texts as a measure triggering actions. We present an overview of the algorithm and its functionalities.

Gaussian processes:iterative sparse approximations

In recent years there has been an increased interest in applying non-parametric methods to real-world problems. Significant research has been devoted to Gaussian processes (GPs) due to their increased flexibility when compared with parametric models. These methods use Bayesian learning, which generally leads to analytically intractable posteriors. This thesis proposes a two-step solution to construct a probabilistic approximation to the posterior. In the first step we adapt the Bayesian online learning to GPs: the final approximation to the posterior is the result of propagating the first and second moments of intermediate posteriors obtained by combining a new example with the previous approximation. The propagation of em functional forms is solved by showing the existence of a parametrisation to posterior moments that uses combinations of the kernel function at the training points, transforming the Bayesian online learning of functions into a parametric formulation. The drawback is the prohibitive quadratic scaling of the number of parameters with the size of the data, making the method inapplicable to large datasets. The second step solves the problem of the exploding parameter size and makes GPs applicable to arbitrarily large datasets. The approximation is based on a measure of distance between two GPs, the KL-divergence between GPs. This second approximation is with a constrained GP in which only a small subset of the whole training dataset is used to represent the GP. This subset is called the em Basis Vector, or BV set and the resulting GP is a sparse approximation to the true posterior. As this sparsity is based on the KL-minimisation, it is probabilistic and independent of the way the posterior approximation from the first step is obtained. We combine the sparse approximation with an extension to the Bayesian online algorithm that allows multiple iterations for each input and thus approximating a batch solution. The resulting sparse learning algorithm is a generic one: for different problems we only change the likelihood. The algorithm is applied to a variety of problems and we examine its performance both on more classical regression and classification tasks and to the data-assimilation and a simple density estimation problems.

Recursive self-organizing map as a contractive iterative function system

Recently, there has been a considerable research activity in extending topographic maps of vectorial data to more general data structures, such as sequences or trees. However, the representational capabilities and internal representations of the models are not well understood. We rigorously analyze a generalization of the Self-Organizing Map (SOM) for processing sequential data, Recursive SOM (RecSOM [1]), as a non-autonomous dynamical system consisting off a set of fixed input maps. We show that contractive fixed input maps are likely to produce Markovian organizations of receptive fields o the RecSOM map. We derive bounds on parameter $\beta$ (weighting the importance of importing past information when processing sequences) under which contractiveness of the fixed input maps is guaranteed.

Learning to manage complexity through simulation: students' challenges and possible strategies

Many have called for medical students to learn how to manage complexity in healthcare. This study examines the nuances of students' challenges in coping with a complex simulation learning activity, using concepts from complexity theory, and suggests strategies to help them better understand and manage complexity.Wearing video glasses, participants took part in a simulation ward-based exercise that incorporated characteristics of complexity. Video footage was used to elicit interviews, which were transcribed. Using complexity theory as a theoretical lens, an iterative approach was taken to identify the challenges that participants faced and possible coping strategies using both interview transcripts and video footage.Students' challenges in coping with clinical complexity included being: a) unprepared for 'diving in', b) caught in an escalating system, c) captured by the patient, and d) unable to assert boundaries of acceptable practice.Many characteristics of complexity can be recreated in a ward-based simulation learning activity, affording learners an embodied and immersive experience of these complexity challenges. Possible strategies for managing complexity themes include: a) taking time to size up the system, b) attuning to what emerges, c) reducing complexity, d) boundary practices, and e) working with uncertainty. This study signals pedagogical opportunities for recognizing and dealing with complexity.