Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

Spatio-temporal credit assignment in population learning

We present a model for plasticity induction in reinforcement learning which is based on a cascade of synaptic memory traces. In the cascade of these so called eligibility traces presynaptic input is first corre lated with postsynaptic events, next with the behavioral decisions and finally with the external reinforcement. A population of leaky integrate and fire neurons endowed with this plasticity scheme is studied by simulation on different tasks. For operant co nditioning with delayed reinforcement, learning succeeds even when the delay is so large that the delivered reward reflects the appropriateness, not of the immediately preceeding response, but of a decision made earlier on in the stimulus - decision sequence . So the proposed model does not rely on the temporal contiguity between decision and pertinent reward and thus provides a viable means of addressing the temporal credit assignment problem. In the same task, learning speeds up with increasing population si ze, showing that the plasticity cascade simultaneously addresses the spatial problem of assigning credit to the different population neurons. Simulations on other task such as sequential decision making serve to highlight the robustness of the proposed sch eme and, further, contrast its performance to that of temporal difference based approaches to reinforcement learning.

Spatio-Temporal Credit Assignment in Neuronal Population Learning

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Association between circulating cytokine levels, diabetes and insulin resistance in a population-based sample (CoLaus study)

OBJECTIVE: The associations between inflammation, diabetes and insulin resistance remain controversial. Hence, we assessed the associations between diabetes, insulin resistance (using HOMA-IR) and metabolic syndrome with the inflammatory markers high sensitivity C-reactive protein (hs-CRP), interleukin-1beta (IL-1β), interleukin-6 (IL-6) and tumour necrosis factor-α (TNF-α). DESIGN: CROSS-SECTIONAL STUDY: PARTICIPANTS: 2884 MEN AND 3201 WOMEN AGED 35 TO 75: METHODS: CRP was assessed by immunoassay and cytokines by multiplexed flow cytometric assay. In a subgroup of 532 participants an oral glucose tolerance test was performed to screen for impaired glucose tolerance (IGT). RESULTS: IL-6, TNF-α and hs-CRP were significantly and positively correlated with fasting plasma glucose, insulin and HOMA-IR. Participants with diabetes had higher IL-6, TNF-α and hs-CRP levels than participants without diabetes; this difference persisted for hs-CRP after multivariate adjustment. Participants with metabolic syndrome had increased IL-6, TNF-α and hs-CRP levels; these differences persisted after multivariate adjustment. Participants in the highest quartile of HOMA-IR had increased IL-6, TNF-α and hs-CRP levels; these differences persisted for TNF-α and hs-CRP after multivariate adjustment. No association was found between IL-1β levels and all diabetes and insulin resistance markers studied. Finally, participants with IGT had higher hs-CRP levels than participants with a normal OGTT, but this difference disappeared after controlling for body mass index (BMI). CONCLUSION: subjects with diabetes, metabolic syndrome and increased insulin resistance present with increased levels of IL6, TNF-α and hs-CRP, while no association was found with IL-1β. The increased inflammatory state of subjects with IGT is partially explained by increased BMI. © 2012 Blackwell Publishing Ltd.

Stocking of captive-bred fish can cause long-term population decline and gene pool replacement: predictions from a population dynamics model incorporating density-dependent mortality

Releasing captive-bred fish into natural environments (stocking) is common in fisheries worldwide. Although stocking is believed to have a positive effect on fish abundance over the short term, little is known about the long-term consequences of recurrent stocking and its influence on natural populations. In fact, there are growing concerns that genetically maladapted captive-bred fish can eventually reduce the abundance of natural population. In this study, we develop a simple model to quantitatively investigate the condition under which recurrent stocking has long-term effects on the natural population. Using a population dynamics model that takes into account a density-dependent recruitment, a gene responsible for the fitness difference between wild and captive-bred fish, and hybridization between them, we show that there is little or no contribution of recurrent stocking to the stock enhancement without a replacement of the wild gene pool by the captive-bred gene pool. The model further predicted that stocking of an intermediate level causes a reduction, rather than enhancement, of population size over the long term. The population decline due to stocking was attributed to the fitness disadvantage of captive-bred fish and strong overcompensation at recruitment stage. These results suggest that it would be difficult to simultaneously attain population size recovery and conservation of the local gene pool when captive-bred fish have fitness disadvantage in the wild, although caution is needed when applying the predictions from the simplified model to a specific species or population.

Individual and population level effects of partner notification for Chlamydia trachomatis

Partner notification (PN or contact tracing) is an important aspect of treating bacterial sexually transmitted infections (STIs), such as Chlamydia trachomatis. It facilitates the identification of new infected cases that can be treated through individual case management. PN also acts indirectly by limiting onward transmission in the general population. However, the impact of PN, both at the level of individuals and the population, remains unclear. Since it is difficult to study the effects of PN empirically, mathematical and computational models are useful tools for investigating its potential as a public health intervention. To this end, we developed an individual-based modeling framework called Rstisim. It allows the implementation of different models of STI transmission with various levels of complexity and the reconstruction of the complete dynamic sexual partnership network over any time period. A key feature of this framework is that we can trace an individual's partnership history in detail and investigate the outcome of different PN strategies for C. trachomatis. For individual case management, the results suggest that notifying three or more partners from the preceding 18 months yields substantial numbers of new cases. In contrast, the successful treatment of current partners is most important for preventing re-infection of index cases and reducing further transmission of C. trachomatis at the population level. The findings of this study demonstrate the difference between individual and population level outcomes of public health interventions for STIs.

Seroprevalence of Babesia caballi and Theileria equi in the Swiss horse population

In Switzerland, the prevalence and incidence of equine piroplasma parasite (EPP) infections are unknown. In order to obtain a first insight into the prevalence, a representative sample of 689 sera of horses from Switzerland was serologically tested for the presence of antibodies directed against T. equi and B. caballi using the Indirect Fluorescence Antibody Test (IFAT). A total of 50 (7.3%) horses were seropositive for EPP: overall, the seroprevalence of T. equi was significantly higher than that of B. caballi (p=0.002). The seropositivities in indigenous horses (animals bred and raised in Switzerland) and in imported horses were 4.8% (11/230) and 8.5% (39/459), respectively. Unlike in indigenous horses, where no significant difference in seroprevalences could be observed between the two parasite species, the seroprevalence of T. equi was significantly higher (p<0.001) than that of B. caballi in imported horses. Horses imported from France, Spain and Portugal exhibited a significantly higher seroprevalence, and horses imported from Germany a significantly lower seroprevalence of EPP compared to indigenous horses. There were no associations between sex, age, weight loss, surgery or blood transfusions with T. equi and B. caballi seroprevalences. The overall seroprevalence of 7.3% clearly shows that infection with EPP is a threat to the health of the horses in Switzerland. With the presumed expansion of permissive tick vectors, EPP infections will potentially increase in importance in the future. Therefore, continuous monitoring is mandatory.

Heritability, determinants and reference values of renal length: a family-based population study

OBJECTIVES: In this population-based study, reference values were generated for renal length, and the heritability and factors associated with kidney length were assessed. METHODS: Anthropometric parameters and renal ultrasound measurements were assessed in randomly selected nuclear families of European ancestry (Switzerland). The adjusted narrow sense heritability of kidney size parameters was estimated by maximum likelihood assuming multivariate normality after power transformation. Gender-specific reference centiles were generated for renal length according to body height in the subset of non-diabetic non-obese participants with normal renal function. RESULTS: We included 374 men and 419 women (mean ± SD, age 47 ± 18 and 48 ± 17 years, BMI 26.2 ± 4 and 24.5 ± 5 kg/m(2), respectively) from 205 families. Renal length was 11.4 ± 0.8 cm in men and 10.7 ± 0.8 cm in women; there was no difference between right and left renal length. Body height, weight and estimated glomerular filtration rate (eGFR) were positively associated with renal length, kidney function negatively, age quadratically, whereas gender and hypertension were not. The adjusted heritability estimates of renal length and volume were 47.3 ± 8.5 % and 45.5 ± 8.8 %, respectively (P < 0.001). CONCLUSION: The significant heritability of renal length and volume highlights the familial aggregation of this trait, independently of age and body size. Population-based references for renal length provide a useful guide for clinicians. KEY POINTS: • Renal length and volume are heritable traits, independent of age and size. • Based on a European population, gender-specific reference values/percentiles are provided for renal length. • Renal length correlates positively with body length and weight. • There was no difference between right and left renal lengths in this study. • This negates general teaching that the left kidney is larger and longer.

Does population screening for Chlamydia trachomatis raise anxiety among those tested? Findings from a population based chlamydia screening study

BACKGROUND: The advent of urine testing for Chlamydia trachomatis has raised the possibility of large-scale screening for this sexually transmitted infection, which is now the most common in the United Kingdom. The purpose of this study was to investigate the effect of an invitation to be screened for chlamydia and of receiving a negative result on levels of anxiety, depression and self-esteem. METHODS: 19,773 men and women aged 16 to 39 years, selected at random from 27 general practices in two large city areas (Bristol and Birmingham) were invited by post to send home-collected urine samples or vulvo-vaginal swabs for chlamydia testing. Questionnaires enquiring about anxiety, depression and self-esteem were sent to random samples of those offered screening: one month before the dispatch of invitations; when participants returned samples; and after receiving a negative result. RESULTS: Home screening was associated with an overall reduction in anxiety scores. An invitation to participate did not increase anxiety levels. Anxiety scores in men were lower after receiving the invitation than at baseline. Amongst women anxiety was reduced after receipt of negative test results. Neither depression nor self-esteem scores were affected by screening. CONCLUSION: Postal screening for chlamydia does not appear to have a negative impact on overall psychological well-being and can lead to a decrease in anxiety levels among respondents. There is, however, a clear difference between men and women in when this reduction occurs.

Japanese-US common-arm analysis of paclitaxel plus carboplatin in advanced non-small-cell lung cancer: a model for assessing population-related pharmacogenomics

PURPOSE To explore whether population-related pharmacogenomics contribute to differences in patient outcomes between clinical trials performed in Japan and the United States, given similar study designs, eligibility criteria, staging, and treatment regimens. METHODS We prospectively designed and conducted three phase III trials (Four-Arm Cooperative Study, LC00-03, and S0003) in advanced-stage, non-small-cell lung cancer, each with a common arm of paclitaxel plus carboplatin. Genomic DNA was collected from patients in LC00-03 and S0003 who received paclitaxel (225 mg/m(2)) and carboplatin (area under the concentration-time curve, 6). Genotypic variants of CYP3A4, CYP3A5, CYP2C8, NR1I2-206, ABCB1, ERCC1, and ERCC2 were analyzed by pyrosequencing or by PCR restriction fragment length polymorphism. Results were assessed by Cox model for survival and by logistic regression for response and toxicity. Results Clinical results were similar in the two Japanese trials, and were significantly different from the US trial, for survival, neutropenia, febrile neutropenia, and anemia. There was a significant difference between Japanese and US patients in genotypic distribution for CYP3A4*1B (P = .01), CYP3A5*3C (P = .03), ERCC1 118 (P < .0001), ERCC2 K751Q (P < .001), and CYP2C8 R139K (P = .01). Genotypic associations were observed between CYP3A4*1B for progression-free survival (hazard ratio [HR], 0.36; 95% CI, 0.14 to 0.94; P = .04) and ERCC2 K751Q for response (HR, 0.33; 95% CI, 0.13 to 0.83; P = .02). For grade 4 neutropenia, the HR for ABCB1 3425C-->T was 1.84 (95% CI, 0.77 to 4.48; P = .19). CONCLUSION Differences in allelic distribution for genes involved in paclitaxel disposition or DNA repair were observed between Japanese and US patients. In an exploratory analysis, genotype-related associations with patient outcomes were observed for CYP3A4*1B and ERCC2 K751Q. This common-arm approach facilitates the prospective study of population-related pharmacogenomics in which ethnic differences in antineoplastic drug disposition are anticipated.

Comparison of pain-resilient working individuals to population-based case controls with/without momentary low back pain

Background: Few studies have examined the 20% of individuals who never experience an episode of low back pain (LBP). To date, no investigation has been undertaken that examines a group who claim to have never experienced LBP in their lifetime in comparison to two population-based case–control groups with and without momentary LBP. This study investigates whether LBP-resilient workers between 50 and 65 years had better general health, demonstrated more positive health behaviour and were better able to achieve routine activities compared with both case–control groups. Methods: Forty-two LBP-resilient participants completed the same pain assessment questionnaire as a population-based LBP sample from a nationwide, large-scale cross-sectional survey in Switzerland. The LBP-resilient participants were pairwise compared to the propensity score-matched case controls by exploring differences in demographic and work characteristics, and by calculating odds ratios (ORs) and effect sizes. A discriminant analysis explored group differences, while the multiple logistic regression analysis specified single indicators which accounted for group differences. Results: LBP-resilient participants were healthier than the case controls with momentary LBP and achieved routine activities more easily. Compared to controls without momentary LBP, LBP-resilient participants had a higher vitality, a lower workload, a healthier attitude towards health and behaved more healthily by drinking less alcohol. Conclusions: By demonstrating a difference between LBP-resilient participants and controls without momentary LBP, the question that arises is what additional knowledge can be attained. Three underlying traits seem to be relevant about LBP-resilient participants: personality, favourable work conditions and subjective attitudes/attributions towards health. These rationales have to be considered with respect to LBP prevention.

Twenty-three per cent of the Swiss adult population are using complementary medicine

Aim of the study: This study investigated the use among the Swiss adult population and the regional dissemination in Switzerland of various methods of complementary medicine (CM). It focused on CM methods that required visiting a physician or therapist and excluded e.g. over-the-counter drugs. Data and Methods: Data of the Swiss Health Survey 2007 were obtained from the Swiss Federal Statistical Office. This survey is performed every 5 years in a sample and is representative of the Swiss resident population from the age of 15 on. It consists of a telephone interview followed by a written questionnaire (2007: 18'760 and 14'432 respondents, respectively) and includes questions about people's state of health, general living conditions, lifestyle, health insurance and usage of health services. Users and non-users of CM were compared using logistic regression models. Results: 23.0 % of the Swiss adult population (women: 30.5 %, men: 15.2 %) used CM during the 12 months before the survey. Homeopathy (6.4 %), osteopathy (5.4 %) and acupuncture (4.9 %) were the most popular methods. The average number of treatments within 12 months for these three methods was 3.1 ± 3.6, 3.5 ± 3.3 and 6.6 ± 5.8, respectively. For treatments with homeopathy and acupuncture, medical practitioners were more commonly consulted than non-medical practitioners, for treatments with osteopathy no difference was found. By means of logistic regression, CM users and non-users were compared. There were significant differences in the use of CM between genders, age groups, levels of education and areas of living. Women, people aged 25 to 64 years, and people with higher levels of education used CM more commonly than men, people below 25 or above 64 years of age, or those with poorer education. Lake Geneva region and central Switzerland had a higher proportion of CM users than the other regions. Discussion: Almost one fourth of the Swiss adult population had used CM within 12 months before the survey. User profiles were comparable to those in other countries. Despite a generally lower self-perceived health status, elderly people were less likely to use CM. Reference: Klein SD, Frei-Erb M, Wolf U. Usage of complementary medicine across Switzerland. Results of the Swiss Health Survey 2007. Swiss Med Wkly. 2012;142:w13666.

Twenty-three per cent of the Swiss adult population are using complementary medicine

Aim of the study Various forms of complementary medicine (CM) play an important role in the Swiss health care system, they are appreciated by a majority of the population and mostly used complementarily rather than alternatively to conventional medicine. This study investigates, how many people in Switzerland are actually being treated with CM, and what the most popular methods of CM are. Data Data of the Swiss Health Survey 2007 were obtained from the Swiss Federal Statistical Office. This survey is performed every 5 years amongst a sample of the Swiss resident population above 15 years of age. It consists of a telephone interview followed by a written questionnaire (2007: 18'760 and 14'432 respondents, respectively) and includes questions about people's state of health, general living conditions, lifestyle, health insurance and usage of health services. Results 23.0% of the Swiss adult population (women: 30.5%, men: 15.2%) used CM during the 12 months before the survey. Homeopathy (6.4%), osteopathy (5.4%) and acupuncture (4.9%) were the most popular methods. The average number of treatments within 12 months for these three methods was 3.1, 3.5 and 6.6, respectively. For treatments with homeopathy and acupuncture, medical practitioners were more commonly consulted than non-medical practitioners, for treatments with osteopathy no difference was found. By means of logistic regression, CM users and non-users were compared. There were significant differences in the use of CM between genders, age groups, levels of education and areas of living. Women, people aged 25 to 64 years, and people with higher education used CM more commonly than men, people below 25 or above 64 years of age, or those with poorer education. Lake Geneva region and central Switzerland had a higher proportion of CM users than the other regions. Discussion While 2 years ago, 67.0% of the Swiss population approved a referendum in favour of CM, we find that 23.0% are in fact using it. Current political discussions focus on effectiveness, cost effectiveness and suitability of CM to decide which methods should be permanently covered by the basic health insurance.

Acute post-stroke blood pressure relative to premorbid levels in intracerebral haemorrhage versus major ischaemic stroke: a population-based study.

BACKGROUND It is often assumed that blood pressure increases acutely after major stroke, resulting in so-called post-stroke hypertension. In view of evidence that the risks and benefits of blood pressure-lowering treatment in acute stroke might differ between patients with major ischaemic stroke and those with primary intracerebral haemorrhage, we compared acute-phase and premorbid blood pressure levels in these two disorders. METHODS In a population-based study in Oxfordshire, UK, we recruited all patients presenting with stroke between April 1, 2002, and March 31, 2012. We compared all acute-phase post-event blood pressure readings with premorbid readings from 10-year primary care records in all patients with acute major ischaemic stroke (National Institutes of Health Stroke Scale >3) versus those with acute intracerebral haemorrhage. FINDINGS Of 653 consecutive eligible patients, premorbid and acute-phase blood pressure readings were available for 636 (97%) individuals. Premorbid blood pressure (total readings 13,244) had been measured on a median of 17 separate occasions per patient (IQR 8-31). In patients with ischaemic stroke, the first acute-phase systolic blood pressure was much lower than after intracerebral haemorrhage (158·5 mm Hg [SD 30·1] vs 189·8 mm Hg [38·5], p<0·0001; for patients not on antihypertensive treatment 159·2 mm Hg [27·8] vs 193·4 mm Hg [37·4], p<0·0001), was little higher than premorbid levels (increase of 10·6 mm Hg vs 10-year mean premorbid level), and decreased only slightly during the first 24 h (mean decrease from <90 min to 24 h 13·6 mm Hg). By contrast with findings in ischaemic stroke, the mean first systolic blood pressure after intracerebral haemorrhage was substantially higher than premorbid levels (mean increase of 40·7 mm Hg, p<0·0001) and fell substantially in the first 24 h (mean decrease of 41·1 mm Hg; p=0·0007 for difference from decrease in ischaemic stroke). Mean systolic blood pressure also increased steeply in the days and weeks before intracerebral haemorrhage (regression p<0·0001) but not before ischaemic stroke. Consequently, the first acute-phase blood pressure reading after primary intracerebral haemorrhage was more likely than after ischaemic stroke to be the highest ever recorded (OR 3·4, 95% CI 2·3-5·2, p<0·0001). In patients with intracerebral haemorrhage seen within 90 min, the highest systolic blood pressure within 3 h of onset was 50 mm Hg higher, on average, than the maximum premorbid level whereas that after ischaemic stroke was 5·2 mm Hg lower (p<0·0001). INTERPRETATION Our findings suggest that systolic blood pressure is substantially raised compared with usual premorbid levels after intracerebral haemorrhage, whereas acute-phase systolic blood pressure after major ischaemic stroke is much closer to the accustomed long-term premorbid level, providing a potential explanation for why the risks and benefits of lowering blood pressure acutely after stroke might be expected to differ. FUNDING Wellcome Trust, Wolfson Foundation, UK Medical Research Council, Stroke Association, British Heart Foundation, National Institute for Health Research.