Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

Spatio-temporal credit assignment in population learning

We present a model for plasticity induction in reinforcement learning which is based on a cascade of synaptic memory traces. In the cascade of these so called eligibility traces presynaptic input is first corre lated with postsynaptic events, next with the behavioral decisions and finally with the external reinforcement. A population of leaky integrate and fire neurons endowed with this plasticity scheme is studied by simulation on different tasks. For operant co nditioning with delayed reinforcement, learning succeeds even when the delay is so large that the delivered reward reflects the appropriateness, not of the immediately preceeding response, but of a decision made earlier on in the stimulus - decision sequence . So the proposed model does not rely on the temporal contiguity between decision and pertinent reward and thus provides a viable means of addressing the temporal credit assignment problem. In the same task, learning speeds up with increasing population si ze, showing that the plasticity cascade simultaneously addresses the spatial problem of assigning credit to the different population neurons. Simulations on other task such as sequential decision making serve to highlight the robustness of the proposed sch eme and, further, contrast its performance to that of temporal difference based approaches to reinforcement learning.

Falcarinol is a covalent cannabinoid CB1 receptor antagonist and induces pro-allergic effects in skin

The skin irritant polyyne falcarinol (panaxynol, carotatoxin) is found in carrots, parsley, celery, and in the medicinal plant Panax ginseng. In our ongoing search for new cannabinoid (CB) receptor ligands we have isolated falcarinol from the endemic Sardinian plant Seseli praecox. We show that falcarinol exhibits binding affinity to both human CB receptors but selectively alkylates the anandamide binding site in the CB(1) receptor (K(i)=594nM), acting as covalent inverse agonist in CB(1) receptor-transfected CHO cells. Given the inherent instability of purified falcarinol we repeatedly isolated this compound for biological characterization and one new polyyne was characterized. In human HaCaT keratinocytes falcarinol increased the expression of the pro-allergic chemokines IL-8 and CCL2/MCP-1 in a CB(1) receptor-dependent manner. Moreover, falcarinol inhibited the effects of anandamide on TNF-alpha stimulated keratinocytes. In vivo, falcarinol strongly aggravated histamine-induced oedema reactions in skin prick tests. Both effects were also obtained with the CB(1) receptor inverse agonist rimonabant, thus indicating the potential role of the CB(1) receptor in skin immunopharmacology. Our data suggest anti-allergic effects of anandamide and that falcarinol-associated dermatitis is due to antagonism of the CB(1) receptor in keratinocytes, leading to increased chemokine expression and aggravation of histamine action.

Feature Versus Spatial Strategies by Orangutans (Pongo abelii) and Human Children (Homo sapiens) in a Cross-Dimensional Task

Despite the fact that photographic stimuli are used across experimental contexts with both human and nonhuman subjects, the nature of individuals' perceptions of these stimuli is still not well understood. In the present experiments, we tested whether three orangutans and 36 human children could use photographic information presented on a computer screen to solve a perceptually corresponding problem in the physical domain. Furthermore, we tested the cues that aided in this process by pitting featural information against spatial position in a series of probe trials. We found that many of the children and one orangutan were successfully able to use the information cross-dimensionally; however, the other two orangutans and almost a quarter of the children failed to acquire the task. Species differences emerged with respect to ease of task acquisition. More striking, however, were the differences in cues that participants used to solve the task: Whereas the orangutan used a spatial strategy, the majority of children used a feature one. Possible reasons for these differences are discussed from both evolutionary and developmental perspectives. The novel results found here underscore the need for further testing in this area to design appropriate experimental paradigms in future comparative research settings.

Can You Hear the Shape of a Drum?

If you had perfect pitch and listened to a recording of the sounds a drum made when struck, could you determine the shape of the drum? This question is an example of an inverse problem; inverse problems arise in medical imaging, oil prospecting, spectroscopy, and many other fields. We’ll first discuss the analogous question in the simpler setting of plucking a string. Then we’ll tackle the problem for drums and see that there are some surprises. Finally, I will give a brief indication of how this problem relates to some of my recent research. The emphasis will be on the ideas rather than on the technical details, so there will be pretty pictures instead of equations.

H1N1 outbreak in a Swiss military boot camp--observations and suggestions

The A(H1N1)pdm09 influenza virus is a highly contagious pathogen which caused the 2009 influenza pandemic. The virus is known to affect mainly younger people and may be a problem in crowded living conditions. The aim of the study was to describe a major A(H1N1)pdm09 outbreak in a Swiss military boot camp and to develop suggestions for similar future situations.

Temporal factors in violence related injuries--An 11 year trend analysis of violence-related injuries from a Swiss Emergency Department

Injury from interpersonal violence is a major social and medical problem in the industrialized world. Little is known about the trends in prevalence and injury pattern or about the demographic characteristics of the patients involved.

First insights into the phylogenetic diversity of Mycobacterium tuberculosis in Nepal

Background Tuberculosis (TB) is a major public health problem in Nepal. Strain variation in Mycobacterium tuberculosis may influence the outcome of TB infection and disease. To date, the phylogenetic diversity of M. tuberculosis in Nepal is unknown. Methods and Findings We analyzed 261 M. tuberculosis isolates recovered from pulmonary TB patients recruited between August 2009 and August 2010 in Nepal. M. tuberculosis lineages were determined by single nucleotide polymorphisms (SNP) typing and spoligotyping. Drug resistance was determined by sequencing the hot spot regions of the relevant target genes. Overall, 164 (62.8%) TB patients were new, and 97 (37.2%) were previously treated. Any drug resistance was detected in 50 (19.2%) isolates, and 16 (6.1%) were multidrug-resistant. The most frequent M. tuberculosis lineage was Lineage 3 (CAS/Delhi) with 106 isolates (40.6%), followed by Lineage 2 (East-Asian lineage, includes Beijing genotype) with 84 isolates (32.2%), Lineage 4 (Euro-American lineage) with 41 (15.7%) isolates, and Lineage 1 (Indo-Oceanic lineage) with 30 isolates (11.5%). Based on spoligotyping, we found 45 different spoligotyping patterns that were previously described. The Beijing (83 isolates, 31.8%) and CAS spoligotype (52, 19.9%) were the dominant spoligotypes. A total of 36 (13.8%) isolates could not be assigned to any known spoligotyping pattern. Lineage 2 was associated with female sex (adjusted odds ratio [aOR] 2.58, 95% confidence interval [95% CI] 1.42–4.67, p = 0.002), and any drug resistance (aOR 2.79; 95% CI 1.43–5.45; p = 0.002). We found no evidence for an association of Lineage 2 with age or BCG vaccination status. Conclusions We found a large genetic diversity of M. tuberculosis in Nepal with representation of all four major lineages. Lineages 3 and 2 were dominating. Lineage 2 was associated with clinical characteristics. This study fills an important gap on the map of the M. tuberculosis genetic diversity in the Asian region.

The impact of body mass index on the development of systemic inflammatory response syndrome and sepsis in patients with polytrauma

PURPOSE: Obesity is a growing problem in industrial nations. Our aim was to examine how overweight patients coped with systemic inflammatory response syndrome (SIRS) after polytrauma. METHODS: A total of 651 patients were included in this retrospective study, with an ISS≥16 and age≥16 years. The sample was subdivided into three groups: body mass index (BMI; all in kg/m(2))<25, BMI 25-30 and BMI>30, or low, intermediate and high BMI. The SIRS score was measured over 31 days after admission together with measurements of C-reactive protein (CRP), interleukin-6 (IL-6) and procalcitonin (PCT). Data are given as the mean±SEM if not otherwise indicated. Kruskal-Wallis and χ(2) tests were used for statistical analysis and the significance level was set at p<.05. RESULTS: The maximum SIRS score was reached in the low BMI-group at 3.4±0.4, vs. 2.3±0.1 and 2.5±0.2 in the intermediate BMI-group and high BMI-group, respectively (p<.0001). However, the maximum SIRS score was reached earlier in the BMI 25-30 group at 1.8±0.2 days, vs. 3.4±0.4 and 2.5±0.2 days in the BMI<25 and BMI>30 groups, respectively (p<.0001). The incidence of sepsis was significantly higher in the low BMI group at 46.1%, vs. 0.2% and 0% in the BMI 25-30 and BMI>30 groups, respectively (p<.0001). No significant differences in the CRP, IL-6 or PCT levels were found between groups. CONCLUSIONS: A higher BMI seemed to be protective for these patients with polytrauma-associated inflammatory problems.

Exercise induces rapid interstitial lung water accumulation in patients with chronic mountain sickness

Chronic mountain sickness (CMS) is a major public health problem in mountainous regions of the world. In its more advanced stages, exercise intolerance is often found, but the underlying mechanism is not known. Recent evidence indicates that exercise-induced pulmonary hypertension is markedly exaggerated in CMS. We speculated that this problem may cause pulmonary fluid accumulation and aggravate hypoxemia during exercise.

Malnutrition in pediatric patients with cancer at diagnosis and throughout therapy: A multicenter cohort study

Malnutrition is a common problem in pediatric patients with cancer. Reported prevalence varies widely and has often been assessed only in a subset of childhood types of cancer. This study aimed to describe the prevalence of malnutrition among pediatric patients newly diagnosed with cancer, to describe the occurrence and course of malnutrition during therapy and to identify factors associated with malnutrition during therapy.

[Hypertension in the elderly]

Many aspects of hypertension diagnosis and treatment are similar in young and old patients. However, some differences exist. Due to the increasing vascular stiffness most elderly patients have isolated systolic hypertension and its prevalence in the population is high. Blood pressure should be measured in the sitting position and also with the patient standing to exclude orthostatic hypotension, a frequent problem in elderly patients. Pseudohypertension, a source of inadequate measurements in elderly patients, should be recognized. In comparison to other health problems there is good scientific evidence for antihypertensive treatment in elderly patients. As treatment does not only improve survival, but also reduces cardiovascular events such as non-fatal stroke or myocardial infarction, antihypertensive therapy is an important measure to prevent functional decline and disability.