Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

Exploring the effectiveness of development research with a monitoring and learning approach

There is a growing demand for better understanding of the link between research, policy and practice in development. This article provides findings from a study that aimed to gain insights into how researchers engage with their non-academic partners. It draws on experiences from the National Centre of Competence in Research North-South programme, a development research network of Swiss, African, Asian and Latin American institutions. Conceptually, this study is concerned with research effectiveness as a means to identify knowledge useful for society. Research can be improved and adapted when monitoring the effects of interactions between researchers and non-academic partners. Therefore, a monitoring and learning approach was chosen. This study reveals researchers' strategies in engaging with non-academic partners and points to framing conditions considered decisive for soccessful interactions. It concludes that reserachrs need to systematically analyse the socio-political context in which they intervene. By providing insights from the ground and reflecting on them in the light of the latest theoretical concepts, this article contributes to the emerging literature founded on practice-based experience.

Fidelity of Implementation of Research-Based Instructional Strategies (RBIS) in Engineering Science Courses

Background Increasing attention is being paid to improvement in undergraduate science, technology, engineering, and mathematics (STEM) education through increased adoption of research-based instructional strategies (RBIS), but high-quality measures of faculty instructional practice do not exist to monitor progress. Purpose/Hypothesis The measure of how well an implemented intervention follows the original is called fidelity of implementation. This theory was used to address the research questions: What is the fidelity of implementation of selected RBIS in engineering science courses? That is, how closely does engineering science classroom practice reflect the intentions of the original developers? Do the critical components that characterize an RBIS discriminate between engineering science faculty members who claimed use of the RBIS and those who did not? Design/Method A survey of 387 U.S. faculty teaching engineering science courses (e.g., statics, circuits, thermodynamics) included questions about class time spent on 16 critical components and use of 11 corresponding RBIS. Fidelity was quantified as the percentage of RBIS users who also spent time on corresponding critical components. Discrimination between users and nonusers was tested using chi square. Results Overall fidelity of the 11 RBIS ranged from 11% to 80% of users spending time on all required components. Fidelity was highest for RBIS with one required component: case-based teaching, just-in-time teaching, and inquiry learning. Thirteen of 16 critical components discriminated between users and nonusers for all RBIS to which they were mapped. Conclusions Results were consistent with initial mapping of critical components to RBIS. Fidelity of implementation is a potentially useful framework for future work in STEM undergraduate education.

Estimates of Use of Research-Based Instructional Strategies in Core Electrical or Computer Engineering Courses

Many research-based instruction strategies (RBISs) have been developed; their superior efficacy with respect to student learning has been demonstrated in many studies. Collecting and interpreting evidence about: 1) the extent to which electrical and computer engineering (ECE) faculty members are using RBISs in core, required engineering science courses, and 2) concerns that they express about using them, are important aspects of understanding how engineering education is evolving. The authors surveyed ECE faculty members, asking about their awareness and use of selected RBISs. The survey also asked what concerns ECE faculty members had about using RBISs. Respondent data showed that awareness of RBISs was very high, but estimates of use of RBISs, based on survey data, varied from 10% to 70%, depending on characteristics of the strategy. The most significant concern was the amount of class time that using an RBIS might take; efforts to increase use of RBISs must address this.

Spike-based decision learning of nash equilibria in two-player games

Humans and animals face decision tasks in an uncertain multi-agent environment where an agent's strategy may change in time due to the co-adaptation of others strategies. The neuronal substrate and the computational algorithms underlying such adaptive decision making, however, is largely unknown. We propose a population coding model of spiking neurons with a policy gradient procedure that successfully acquires optimal strategies for classical game-theoretical tasks. The suggested population reinforcement learning reproduces data from human behavioral experiments for the blackjack and the inspector game. It performs optimally according to a pure (deterministic) and mixed (stochastic) Nash equilibrium, respectively. In contrast, temporal-difference(TD)-learning, covariance-learning, and basic reinforcement learning fail to perform optimally for the stochastic strategy. Spike-based population reinforcement learning, shown to follow the stochastic reward gradient, is therefore a viable candidate to explain automated decision learning of a Nash equilibrium in two-player games.

Learning from failure--rationale and design for a study about discontinuation of randomized trials (DISCO study)

Background Randomized controlled trials (RCTs) may be discontinued because of apparent harm, benefit, or futility. Other RCTs are discontinued early because of insufficient recruitment. Trial discontinuation has ethical implications, because participants consent on the premise of contributing to new medical knowledge, Research Ethics Committees (RECs) spend considerable effort reviewing study protocols, and limited resources for conducting research are wasted. Currently, little is known regarding the frequency and characteristics of discontinued RCTs. Methods/Design Our aims are, first, to determine the prevalence of RCT discontinuation for specific reasons; second, to determine whether the risk of RCT discontinuation for specific reasons differs between investigator- and industry-initiated RCTs; third, to identify risk factors for RCT discontinuation due to insufficient recruitment; fourth, to determine at what stage RCTs are discontinued; and fifth, to examine the publication history of discontinued RCTs. We are currently assembling a multicenter cohort of RCTs based on protocols approved between 2000 and 2002/3 by 6 RECs in Switzerland, Germany, and Canada. We are extracting data on RCT characteristics and planned recruitment for all included protocols. Completion and publication status is determined using information from correspondence between investigators and RECs, publications identified through literature searches, or by contacting the investigators. We will use multivariable regression models to identify risk factors for trial discontinuation due to insufficient recruitment. We aim to include over 1000 RCTs of which an anticipated 150 will have been discontinued due to insufficient recruitment. Discussion Our study will provide insights into the prevalence and characteristics of RCTs that were discontinued. Effective recruitment strategies and the anticipation of problems are key issues in the planning and evaluation of trials by investigators, Clinical Trial Units, RECs and funding agencies. Identification and modification of barriers to successful study completion at an early stage could help to reduce the risk of trial discontinuation, save limited resources, and enable RCTs to better meet their ethical requirements.

Influence of Engineering Instructors' Teaching and Learning Beliefs on Pedagogies in Engineering Science Courses

This study explored how academics' beliefs about teaching and learning influenced their teaching in engineering science courses typically taught in the second or third year of 4-year engineering undergraduate degrees. Data were collected via a national survey of 166 U. S. statics instructors and interviews at two different institutions with 17 instructors of engineering science courses such as thermodynamics, circuits and statics. The study identified a number of common beliefs about how to best support student learning of these topics; each is discussed in relation to the literature about student development and learning. Specific recommendations are given for educational developers to encourage use of research-based instructional strategies in these courses.

Functional neuroimaging of emotional learning and autonomic reactions

This article provides a selective overview of the functional neuroimaging literature with an emphasis on emotional activation processes. Emotions are fast and flexible response systems that provide basic tendencies for adaptive action. From the range of involved component functions, we first discuss selected automatic mechanisms that control basic adaptational changes. Second, we illustrate how neuroimaging work has contributed to the mapping of the network components associated with basic emotion families (fear, anger, disgust, happiness), and secondary dimensional concepts that organise the meaning space for subjective experience and verbal labels (emotional valence, activity/intensity, approach/withdrawal, etc.). Third, results and methodological difficulties are discussed in view of own neuroimaging experiments that investigated the component functions involved in emotional learning. The amygdala, prefrontal cortex, and striatum form a network of reciprocal connections that show topographically distinct patterns of activity as a correlate of up and down regulation processes during an emotional episode. Emotional modulations of other brain systems have attracted recent research interests. Emotional neuroimaging calls for more representative designs that highlight the modulatory influences of regulation strategies and socio-cultural factors responsible for inhibitory control and extinction. We conclude by emphasising the relevance of the temporal process dynamics of emotional activations that may provide improved prediction of individual differences in emotionality.

LEARNING FROM NEAR MISSES IN MEDICATION ERRORS: A BAYESIAN APPROACH

Medical errors originating in health care facilities are a significant source of preventable morbidity, mortality, and healthcare costs. Voluntary error report systems that collect information on the causes and contributing factors of medi- cal errors regardless of the resulting harm may be useful for developing effective harm prevention strategies. Some patient safety experts question the utility of data from errors that did not lead to harm to the patient, also called near misses. A near miss (a.k.a. close call) is an unplanned event that did not result in injury to the patient. Only a fortunate break in the chain of events prevented injury. We use data from a large voluntary reporting system of 836,174 medication errors from 1999 to 2005 to provide evidence that the causes and contributing factors of errors that result in harm are similar to the causes and contributing factors of near misses. We develop Bayesian hierarchical models for estimating the log odds of selecting a given cause (or contributing factor) of error given harm has occurred and the log odds of selecting the same cause given that harm did not occur. The posterior distribution of the correlation between these two vectors of log-odds is used as a measure of the evidence supporting the use of data from near misses and their causes and contributing factors to prevent medical errors. In addition, we identify the causes and contributing factors that have the highest or lowest log-odds ratio of harm versus no harm. These causes and contributing factors should also be a focus in the design of prevention strategies. This paper provides important evidence on the utility of data from near misses, which constitute the vast majority of errors in our data.

[In search of strategies for preventing brain damage as a sequela of bacterial meningitis]

Multiplication of bacteria within the central nervous system compartment triggers a host response with an overshooting inflammatory reaction which leads to brain parenchyma damage. Some of the inflammatory and neurotoxic mediators involved in the processes leading to neuronal injury during bacterial meningitis have been identified in recent years. As a result, the therapeutic approach to the disease has widened from eradication of the bacterial pathogen with antibiotics to attenuation of the detrimental effects of host defences. Corticosteroids represent an example of the adjuvant therapeutic strategies aimed at downmodulating excessive inflammation in the infected central nervous system. Pathophysiological concepts derived from an experimental rat model of bacterial meningitis revealed possible therapeutic strategies for prevention of brain damage. The insights gained led to the evaluation of new therapeutic modalities such as anticytokine agents, matrix metalloproteinase inhibitors, antioxidants, and antagonists of endothelin and glutamate. Bacterial meningitis is still associated with persistent neurological sequelae in approximately one third of surviving patients. Future research in the model will evaluate whether the neuroprotective agents identified so far have the potential to attenuate learning disabilities as a long-term consequence of bacterial meningitis.

Effects of instructional changes on student learning of electrochemistry in an IB chemistry course

This study investigated the effectiveness of incorporating several new instructional strategies into an International Baccalaureate (IB) chemistry course in terms of how they supported high school seniors’ understanding of electrochemistry. The three new methods used were (a) providing opportunities for visualization of particle movement by student manipulation of physical models and interactive computer simulations, (b) explicitly addressing common misconceptions identified in the literature, and (c) teaching an algorithmic, step-wise approach for determining the products of an aqueous solution electrolysis. Changes in student understanding were assessed through test scores on both internally and externally administered exams over a two-year period. It was found that visualization practice and explicit misconception instruction improved student understanding, but the effect was more apparent in the short-term. The data suggested that instruction time spent on algorithm practice was insufficient to cause significant test score improvement. There was, however, a substantial increase in the percentage of the experimental group students who chose to answer an optional electrochemistry-related external exam question, indicating an increase in student confidence. Implications for future instruction are discussed.

INVESTIGATION OF STRATEGIES TO PROMOTE EFFECTIVE TEACHER PROFESSIONAL DEVELOPMENT EXPERIENCES IN EARTH SCIENCE

This dissertation serves as a call to geoscientists to share responsibility with K-12 educators for increasing Earth science literacy. When partnerships are created among K-12 educators and geoscientists, the synergy created can promote Earth science literacy in students, teachers, and the broader community. The research described here resulted in development of tools that can support effective professional development for teachers. One tool is used during the planning stages to structure a professional development program, another set of tools supports measurement of the effectiveness of a development program, and the third tool supports sustainability of professional development programs. The Michigan Teacher Excellence Program (MiTEP), a Math/Science Partnership project funded by the National Science Foundation, served as the test bed for developing and testing these tools. The first tool, the planning tool, is the Earth Science Literacy Principles (ESLP). The ESLP served as a planning tool for the two-week summer field courses as part of the MiTEP program. The ESLP, published in 2009, clearly describe what an Earth science literate person should know. The ESLP consists of nine big ideas and their supporting fundamental concepts. Using the ESLP for planning a professional development program assisted both instructors and teacher-participants focus on important concepts throughout the professional development activity. The measurement tools were developed to measure change in teachers’ Earth science content-area knowledge and perceptions related to teaching and learning that result from participating in a professional development program. The first measurement tool, the Earth System Concept Inventory (ESCI), directly measures content-area knowledge through a succession of multiple-choice questions that are aligned with the content of the professional development experience. The second measurement, an exit survey, collects qualitative data from teachers regarding their impression of the professional development. Both the ESCI and the exit survey were tested for validity and reliability. Lesson study is discussed here as a strategy for sustaining professional development in a school or a district after the end of a professional development activity. Lesson study, as described here, was offered as a formal course. Teachers engaged in lesson study worked collaboratively to design and test lessons that improve the teachers’ classroom practices. Data regarding the impact of the lesson study activity were acquired through surveys, written documents, and group interviews. The data are interpreted to indicate that the lesson study process improved teacher quality and classroom practices. In the case described here, the lesson study process was adopted by the teachers’ district and currently serves as part of the district’s work in Professional Learning Communities, resulting in ongoing professional development throughout the district.

Learning Curves of Temporary Workers in Manual Order Picking Activities

Person-to-stock order picking is highly flexible and requires minimal investment costs in comparison to automated picking solutions. For these reasons, tradi-tional picking is widespread in distribution and production logistics. Due to its typically large proportion of manual activities, picking causes the highest operative personnel costs of all intralogistics process. The required personnel capacity in picking varies short- and mid-term due to capacity requirement fluctuations. These dynamics are often balanced by employing minimal permanent staff and using seasonal help when needed. The resulting high personnel fluctuation necessitates the frequent training of new pickers, which, in combination with in-creasingly complex work contents, highlights the im-portance of learning processes in picking. In industrial settings, learning is often quantified based on diminishing processing time and cost requirements with increasing experience. The best-known industrial learning curve models include those from Wright, de Jong, Baloff and Crossman, which are typically applied to the learning effects of an entire work crew rather than of individuals. These models have been validated in largely static work environments with homogeneous work contents. Little is known of learning effects in picking systems. Here, work contents are heterogeneous and individual work strategies vary among employees. A mix of temporary and steady employees with varying degrees of experience necessitates the observation of individual learning curves. In this paper, the individual picking performance development of temporary employees is analyzed and compared to that of steady employees in the same working environment.