Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Association of alcohol craving and proximal outcomes of a residential treatment program for patients with alcohol use disorders

Background: Alcohol craving is an essential construct in research and treatment of alcohol use disorders (AUD). Craving is mostly investigated in association with concurrent variables or distal treatment outcomes at follow-up. Objectives: The aim of this study is to examine craving at admission and its relevance for essential proximal outcomes at discharge from AUD treatment such as positive alcohol expectancy, abstinent-related self-efficacy, and substance-related coping, as well as patients’ demographic and AUD characteristics. Methods: In total, 36 patients were recruited within an inpatient treatment AUD program. Results: An association between craving and positive alcohol expectancies at discharge was found in the regression model even when the respective expectancies, age, gender, and severity of alcohol dependence at admission were controlled for (F(2,29)1⁄432.71, p50.001). Craving explained 2.3% of the variance of change in positive alcohol expectancy. Conclusion: The results suggest a low predictive value of craving for positive alcohol expectancy. In addition, we found significant associations between the craving and the severity of AUD and alcohol consumption before admission. Future studies should include proximal outcomes related to treatment efficacy as well as distal outcomes.

Workshop: ASSIP - Attempted Suicide Short Intervention Program. Theory, Practice, and 24 Months Follow-up RCT.

Background: ASSIP is a manualized brief therapy based on a model of suicide as goal-directed action, aimed at establishing a therapeutic alliance in a patient-oriented, collaborative approach. The main goals of the three-session program ASSIP are for patients to understand, from an observer’s position, patterns leading to a suicidal crisis, recognize triggers and warning signs, and to establish individual safety strategies for future suicidal crises. An ongoing therapeutic support is provided with regular letters over 24 months. Method: The study was conducted in a naturalistic setting. 120 Patients were randomly assigned to an intervention group (60 participants) treated with ASSIP combined with follow-up contact through letters, and a control group (60 participants) receiving a single session of clinical assessment. Both groups had treatment as usual. Patients completed a set of psychosocial and clinical questionnaires every six months over a period of 24 months. Results: In the ASSIP group 5 patients made a total of 5 reattempts, compared to 15 patients with 41 reattempts in the control group. The survival analysis yielded a significant difference with a Wald Chi2 of .000003. The ASSIP group had significantly lower suicidal ideation and fewer days of inpatient treatment compared to the control group. Higher scores in the Penn Helping Alliance Questionnaire were associated with lower suicidal ideation during follow-up. Conclusions: ASSIP is a highly effective brief therapy for patients with recent suicide attempts. Forming a strong therapeutic alliance is considered to be a major factor for outcome. ASSIP can be used with minimal training by experienced therapists. An English version of the manual will be published in May 2015.