Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-Temporal Credit Assignment in Neuronal Population Learning

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Computer-aided diagnosis of carotid atherosclerosis based on ultrasound image statistics, laws' texture and neural networks

Quantitative characterisation of carotid atherosclerosis and classification into symptomatic or asymptomatic is crucial in planning optimal treatment of atheromatous plaque. The computer-aided diagnosis (CAD) system described in this paper can analyse ultrasound (US) images of carotid artery and classify them into symptomatic or asymptomatic based on their echogenicity characteristics. The CAD system consists of three modules: a) the feature extraction module, where first-order statistical (FOS) features and Laws' texture energy can be estimated, b) the dimensionality reduction module, where the number of features can be reduced using analysis of variance (ANOVA), and c) the classifier module consisting of a neural network (NN) trained by a novel hybrid method based on genetic algorithms (GAs) along with the back propagation algorithm. The hybrid method is able to select the most robust features, to adjust automatically the NN architecture and to optimise the classification performance. The performance is measured by the accuracy, sensitivity, specificity and the area under the receiver-operating characteristic (ROC) curve. The CAD design and development is based on images from 54 symptomatic and 54 asymptomatic plaques. This study demonstrates the ability of a CAD system based on US image analysis and a hybrid trained NN to identify atheromatous plaques at high risk of stroke.

Learning Software-Maintenance Tasks in the Transition Phase of Offshore Outsourcing Projects: Two Learning-Theoretical Perspectives

The increasing practice of offshore outsourcing software maintenance has posed the challenge of effectively transferring knowledge to individual software engineers of the vendor. In this theoretical paper, we discuss the implications of two learning theories, the model of work-based learning (MWBL) and cognitive load theory (CLT), for knowledge transfer during the transition phase. Taken together, the theories suggest that learning mechanisms need to be aligned with the type of knowledge (tacit versus explicit), task characteristics (complexity and recurrence), and the recipients’ expertise. The MWBL proposes that learning mechanisms need to include conceptual and practical activities based on the relative importance of explicit and tacit knowledge. CLT explains how effective portfolios of learning mechanisms change over time. While jobshadowing, completion tasks, and supportive information may prevail at the outset of transition, they may be replaced by the work on conventional tasks towards the end of transition.

Asthma and allergies: is the farming environment (still) protective in Poland? The GABRIEL Advanced Studies

BACKGROUND Evidence exists that a farming environment in childhood may provide protection against atopic respiratory disease. In the GABRIEL project based in Poland and Alpine regions of Germany, Austria and Switzerland, we aimed to assess whether a farming environment in childhood is protective against allergic diseases in Poland and whether specific exposures explain any protective effect. METHODS In rural Poland, 23 331 families of schoolchildren completed a questionnaire enquiring into farming practices and allergic diseases (Phase I). A subsample (n = 2586) participated in Phase II involving a more detailed questionnaire on specific farm exposures with objective measures of atopy. RESULTS Farming differed between Poland and the Alpine centres; in the latter, cattle farming was prevalent, whereas in Poland 18% of village farms kept ≥1 cow and 34% kept ≥1 pig. Polish children in villages had lower prevalences of asthma and hay fever than children from towns, and in the Phase II population, farm children had a reduced risk of atopy measured by IgE (aOR = 0.72, 95% CI 0.57, 0.91) and skin prick test (aOR = 0.65, 95% CI 0.50, 0.86). Early-life contact with grain was inversely related to the risk of atopy measured by IgE (aOR = 0.66, 95% CI 0.47, 0.92) and appeared to explain part of the farming effect. CONCLUSION While farming in Poland differed from that in the Alpine areas as did the exposure-response associations, we found in communities engaged in small-scale, mixed farming, there was a protective farming effect against objective measures of atopy potentially related to contact with grain or associated farm activities.

International variation in GP treatment strategies for subclinical hypothyroidism in older adults: a case-based survey

Background: There is limited evidence about the impact of treatment for subclinical hypothyroidism, especially among older people. Aim: To investigate the variation in GP treatment strategies for older patients with subclinical hypothyroidism depending on country and patient characteristics. Design and setting: Case-based survey of GPs in the Netherlands, Germany, England, Ireland, Switzerland, and New Zealand. Method: The treatment strategy of GPs (treatment yes/no, starting-dose thyroxine) was assessed for eight cases presenting a woman with subclinical hypothyroidism. The cases differed in the patient characteristics of age (70 versus 85 years), vitality status (vital versus vulnerable), and thyroid-stimulating hormone (TSH) concentration (6 versus 15 mU/L). Results: A total of 526 GPs participated (the Netherlands n = 129, Germany n = 61, England n = 22, Ireland n = 21, Switzerland n = 262, New Zealand n = 31; overall response 19%). Across countries, differences in treatment strategy were observed. GPs from the Netherlands (mean treatment percentage 34%), England (40%), and New Zealand (39%) were less inclined to start treatment than GPs in Germany (73%), Ireland (62%), and Switzerland (52%) (P = 0.05). Overall, GPs were less inclined to start treatment in 85-year-old than in 70-year-old females (pooled odds ratio [OR] 0.74 [95% confidence interval [CI] = 0.63 to 0.87]). Females with a TSH of 15 mU/L were more likely to get treated than those with a TSH of 6 mU/L (pooled OR 9.49 [95% CI = 5.81 to 15.5]). Conclusion: GP treatment strategies of older people with subclinical hypothyroidism vary largely by country and patient characteristics. This variation underlines the need for a new generation of international guidelines based on the outcomes of randomised clinical trials set within primary care

Multimodal sequence learning

While sequence learning research models complex phenomena, previous studies have mostly focused on unimodal sequences. The goal of the current experiment is to put implicit sequence learning into a multimodal context: to test whether it can operate across different modalities. We used the Task Sequence Learning paradigm to test whether sequence learning varies across modalities, and whether participants are able to learn multimodal sequences. Our results show that implicit sequence learning is very similar regardless of the source modality. However, the presence of correlated task and response sequences was required for learning to take place. The experiment provides new evidence for implicit sequence learning of abstract conceptual representations. In general, the results suggest that correlated sequences are necessary for implicit sequence learning to occur. Moreover, they show that elements from different modalities can be automatically integrated into one unitary multimodal sequence.