Personalized Tuning of a Reinforcement Learning Control Algorithm for Glucose Regulation

Artificial pancreas is in the forefront of research towards the automatic insulin infusion for patients with type 1 diabetes. Due to the high inter- and intra-variability of the diabetic population, the need for personalized approaches has been raised. This study presents an adaptive, patient-specific control strategy for glucose regulation based on reinforcement learning and more specifically on the Actor-Critic (AC) learning approach. The control algorithm provides daily updates of the basal rate and insulin-to-carbohydrate (IC) ratio in order to optimize glucose regulation. A method for the automatic and personalized initialization of the control algorithm is designed based on the estimation of the transfer entropy (TE) between insulin and glucose signals. The algorithm has been evaluated in silico in adults, adolescents and children for 10 days. Three scenarios of initialization to i) zero values, ii) random values and iii) TE-based values have been comparatively assessed. The results have shown that when the TE-based initialization is used, the algorithm achieves faster learning with 98%, 90% and 73% in the A+B zones of the Control Variability Grid Analysis for adults, adolescents and children respectively after five days compared to 95%, 78%, 41% for random initialization and 93%, 88%, 41% for zero initial values. Furthermore, in the case of children, the daily Low Blood Glucose Index reduces much faster when the TE-based tuning is applied. The results imply that automatic and personalized tuning based on TE reduces the learning period and improves the overall performance of the AC algorithm.

Learned irrelevance and associative learning is attenuated in individuals at risk for psychosis but not in asymptomatic first-degree relatives of schizophrenia patients: translational state markers of psychosis?

Learned irrelevance (LIrr) refers to a form of selective learning that develops as a result of prior noncorrelated exposures of the predicted and predictor stimuli. In learning situations that depend on the associative link between the predicted and predictor stimuli, LIrr is expressed as a retardation of learning. It represents a form of modulation of learning by selective attention. Given the relevance of selective attention impairment to both positive and cognitive schizophrenia symptoms, the question remains whether LIrr impairment represents a state (relating to symptom manifestation) or trait (relating to schizophrenia endophenotypes) marker of human psychosis. We examined this by evaluating the expression of LIrr in an associative learning paradigm in (1) asymptomatic first-degree relatives of schizophrenia patients (SZ-relatives) and in (2) individuals exhibiting prodromal signs of psychosis ("ultrahigh risk" [UHR] patients) in each case relative to demographically matched healthy control subjects. There was no evidence for aberrant LIrr in SZ-relatives, but LIrr as well as associative learning were attenuated in UHR patients. It is concluded that LIrr deficiency in conjunction with a learning impairment might be a useful state marker predictive of psychotic state but a relatively weak link to a potential schizophrenia endophenotype.

Dopamine-related deficit in reward learning after catecholamine depletion in unmedicated, remitted subjects with bulimia nervosa

Disturbances in reward processing have been implicated in bulimia nervosa (BN). Abnormalities in processing reward-related stimuli might be linked to dysfunctions of the catecholaminergic neurotransmitter system, but findings have been inconclusive. A powerful way to investigate the relationship between catecholaminergic function and behavior is to examine behavioral changes in response to experimental catecholamine depletion (CD). The purpose of this study was to uncover putative catecholaminergic dysfunction in remitted subjects with BN who performed a reinforcement-learning task after CD. CD was achieved by oral alpha-methyl-para-tyrosine (AMPT) in 19 unmedicated female subjects with remitted BN (rBN) and 28 demographically matched healthy female controls (HC). Sham depletion administered identical capsules containing diphenhydramine. The study design consisted of a randomized, double-blind, placebo-controlled crossover, single-site experimental trial. The main outcome measures were reward learning in a probabilistic reward task analyzed using signal-detection theory. Secondary outcome measures included self-report assessments, including the Eating Disorder Examination-Questionnaire. Relative to healthy controls, rBN subjects were characterized by blunted reward learning in the AMPT-but not in placebo-condition. Highlighting the specificity of these findings, groups did not differ in their ability to perceptually distinguish between stimuli. Increased CD-induced anhedonic (but not eating disorder) symptoms were associated with a reduced response bias toward a more frequently rewarded stimulus. In conclusion, under CD, rBN subjects showed reduced reward learning compared with healthy control subjects. These deficits uncover disturbance of the central reward processing systems in rBN related to altered brain catecholamine levels, which might reflect a trait-like deficit increasing vulnerability to BN.

First clinical trial of tomographic neurofeedback in attention-deficit/hyperactivity disorder: evaluation of voluntary cortical control

Tomographic neurofeedback (tNF) training was evaluated as a treatment for attention-deficit/hyperactivity disorder (ADHD). To investigate the specificity of the treatment, outcomes were related to learning during tNF.

Functional neuroimaging of emotional learning and autonomic reactions

This article provides a selective overview of the functional neuroimaging literature with an emphasis on emotional activation processes. Emotions are fast and flexible response systems that provide basic tendencies for adaptive action. From the range of involved component functions, we first discuss selected automatic mechanisms that control basic adaptational changes. Second, we illustrate how neuroimaging work has contributed to the mapping of the network components associated with basic emotion families (fear, anger, disgust, happiness), and secondary dimensional concepts that organise the meaning space for subjective experience and verbal labels (emotional valence, activity/intensity, approach/withdrawal, etc.). Third, results and methodological difficulties are discussed in view of own neuroimaging experiments that investigated the component functions involved in emotional learning. The amygdala, prefrontal cortex, and striatum form a network of reciprocal connections that show topographically distinct patterns of activity as a correlate of up and down regulation processes during an emotional episode. Emotional modulations of other brain systems have attracted recent research interests. Emotional neuroimaging calls for more representative designs that highlight the modulatory influences of regulation strategies and socio-cultural factors responsible for inhibitory control and extinction. We conclude by emphasising the relevance of the temporal process dynamics of emotional activations that may provide improved prediction of individual differences in emotionality.

When conditioned responses "fire back": bidirectional cross-activation creates learning opportunities in synesthesia

In grapheme-color synesthesia, the letter "c" printed in black may be experienced as red, but typically the color red does not trigger the experience of the letter "c." Therefore, at the level of subjective experience, cross-activation is usually unidirectional. However, recent evidence from digit-color synesthesia suggests that at an implicit level bidirectional cross-activation can occur. Here we demonstrate that this finding is not restricted to this specific type of synesthesia. We introduce a new method that enables the investigation of bidirectionality in other types of synesthesia. We found that a group of grapheme-color synesthetes, but not a control group, showed a startle in response to a color-inducing grapheme after a startle response was conditioned to the specific corresponding color. These results implicate that when the startle response was associated with the real color an association between shock and the grapheme was also established. By this mechanism (i.e. implicit cross-activation) the conditioned response to the real color generalized to the synesthetic color. We suggest that parietal brain areas are responsible for this neural backfiring.

The free radical scavenger alpha-phenyl-tert-butyl nitrone aggravates hippocampal apoptosis and learning deficits in experimental pneumococcal meningitis

The effect of adjuvant therapy with the radical scavenger alpha-phenyl-tert-butyl nitrone (PBN; 100 mg/kg given intraperitoneally every 8 h for 5 days) on brain injury and learning function was evaluated in an infant rat model of pneumococcal meningitis. Meningitis led to cortical necrotic injury (median, 3.97% [range, 0%-38.9%] of the cortex), which was reduced to a median of 0% (range, 0%-30.9%) of the cortex (P<.001) by PBN. However, neuronal apoptosis in the hippocampal dentate gyrus was increased by PBN, compared with that by saline (median score, 1.15 [range, 0.04-1.73] vs. 0.31 [range, 0-0.92]; P<.001). Learning function 3 weeks after cured infection, as assessed by the Morris water maze, was decreased, compared with that in uninfected control animals (P<.001). Parallel to the increase in hippocampal apoptosis, PBN further impaired learning in infected animals, compared with that in saline-treated animals (P<.02). These results contrast with those of an earlier study, in which PBN reduced cortical and hippocampal neuronal injury in group B streptococcal meningitis. Thus, in pneumococcal meningitis, antioxidant therapy with PBN aggravates hippocampal injury and learning deficits.

Interference during the implicit learning of two different motor sequences

It has been demonstrated that learning a second motor task after having learned a first task may interfere with the long-term consolidation of the first task. However, little is known about immediate changes in the representation of the motor memory in the early acquisition phase within the first minutes of the learning process. Therefore, we investigated such early interference effects with an implicit serial reaction time task in 55 healthy subjects. Each subject performed either a sequence learning task involving two different sequences, or a random control task. The results showed that learning the first sequence led to only a slight, short-lived interference effect in the early acquisition phase of the second sequence. Overall, learning of neither sequence was impaired. Furthermore, the two processes, sequence-unrelated task learning (i.e. general motor training) and the sequence learning itself did not appear to interfere with each other. In conclusion, although the long-term consolidation of a motor memory has been shown to be sensitive to other interfering memories, the present study suggests that the brain is initially able to acquire more than one new motor sequence within a short space of time without significant interference.

An insulin infusion advisory system for type 1 diabetes patients based on non-linear model predictive control methods

In this paper, an Insulin Infusion Advisory System (IIAS) for Type 1 diabetes patients, which use insulin pumps for the Continuous Subcutaneous Insulin Infusion (CSII) is presented. The purpose of the system is to estimate the appropriate insulin infusion rates. The system is based on a Non-Linear Model Predictive Controller (NMPC) which uses a hybrid model. The model comprises a Compartmental Model (CM), which simulates the absorption of the glucose to the blood due to meal intakes, and a Neural Network (NN), which simulates the glucose-insulin kinetics. The NN is a Recurrent NN (RNN) trained with the Real Time Recurrent Learning (RTRL) algorithm. The output of the model consists of short term glucose predictions and provides input to the NMPC, in order for the latter to estimate the optimum insulin infusion rates. For the development and the evaluation of the IIAS, data generated from a Mathematical Model (MM) of a Type 1 diabetes patient have been used. The proposed control strategy is evaluated at multiple meal disturbances, various noise levels and additional time delays. The results indicate that the implemented IIAS is capable of handling multiple meals, which correspond to realistic meal profiles, large noise levels and time delays.

Governing individual learning in the transition phase of software maintenance offshoring: a dynamic perspective

Prior studies suggest that clients need to actively govern knowledge transfer to vendor staff in offshore outsourcing. In this paper, we analyze longitudinal data from four software maintenance offshore out-sourcing projects to explore why governance may be needed for knowledge transfer and how governance and the individual learning of vendor engineers inter-act over time. Our results suggest that self-control is central to learning, but may be hampered by low levels of trust and expertise at the outset of projects. For these foundations to develop, clients initially need to exert high amounts of formal and clan controls to enforce learning activities against barriers to knowledge sharing. Once learning activities occur, trust and expertise increase and control portfolios may show greater emphases on self-control.

Agrarian learning processes in Switzerland 1918–1939

This paper examines the social impacts of weather extremes and the processes of social and communicative learning a society undertakes to find alternative ways to deal with the consequences of a crisis. In the beginning of the 20th Century hunger seemed to be expelled from Europe. Switzerland – like many other European countries – was involved in a global interdependent trade system, which provided necessary goods. But at the end of World War I very cold and wet summers in 1916/17 (causing crop failure) and the difficulties in war-trade led to malnutrition and enormous price risings of general living-standards in Switzerland, which shocked the people and caused revolutionary uprisings in 1918. The experience of malnutrition during the last two years of war made clear that the traditional ways of food supply in Switzerland lacked crisis stability. Therefore various agents in the field of food production, distribution and consumption searched for alternative ways of food supply. In that sense politicians, industrialists, consumer-groups, left-wing communitarians and farmers developed several strategies for new ways in food production. Traditionally there were political conflicts in Switzerland between farmers and consumers regarding price policies, which led mainly to the conflict in 1918. Consumers accused famers of holding back food to control extortionate prices while the farmers pointed to the bad harvest causing the price rising. The collaboration of these groups in search for new forms of food-stability made social integration possible again. In addition to other crisis-factors, weather extremes can have disastrous impacts and destroy a society’s self-confidence to its core. But even such crisis can lead to processes of substantial learning that allows a regeneration of confidence and show positive influence on political stabilization. The paper focuses on the process of learning and the alternative methods of food production that were suggested by various agents working in the field during the Interwar period. To achieve that goal documents of the various associations are analyzed and newspapers have been taken into consideration. Through the method of discourse-analysis of food-production during the Interwar period, possible solutions that crossed the minds of the agents should be brought to light.

Early adolescents' enjoyment experienced in learning situations and its relation to student achievement

While many studies confirm that positive emotions, including enjoyment, lead to better student achievement, less empirical evidence exists about possible mediator variables that link achievement to enjoyment. It is proposed that achievement and enjoyment form a circular dependency; enjoyment in learning leads to higher achievement but a degree of achievement is required to enjoy learning. This study provides insight into the reverse of the much studied enjoyment to achievement link and provides practical recommendations on how to use these findings. Founded in Control-value theory, which suggests that control and value cognitions are important variables that mediate the connection between enjoyment and achievement, this study explores the reciprocal achievement-cognition-enjoyment link. The reciprocal link was investigated by applying a one year longitudinal design to students of grade 6 and 7 (N = 356). This age group was chosen because early adolescence represents a critical period during which a strong decrease in positive learning emotions is observed. Part of the work involved identifying factors that might be responsible for this negative development. Results of cross-lagged path analysis identified reciprocal effects between student achievement and enjoyment with control and value cognitions functioning as partial mediators. High achievement goes with high control and value cognitions, which in turn positively affect enjoyment. However, cross-lagged correlations could only be partly confirmed. The results are discussed in terms of theoretical and practical implications.

Probabilistic contingency learning with limbic or prefrontal damage

AB A fundamental capacity of the human brain is to learn relations (contingencies) between environmental stimuli and the consequences of their occurrence. Some contingencies are probabilistic; that is, they predict an event in some situations but not in all. Animal studies suggest that damage to limbic structures or the prefrontal cortex may disturb probabilistic learning. The authors studied the learning of probabilistic contingencies in amnesic patients with limbic lesions, patients with prefrontal cortex damage, and healthy controls. Across 120 trials, participants learned contingent relations between spatial sequences and a button press. Amnesic patients had learning comparable to that of control subjects but failed to indicate what they had learned. Across the last 60 trials, amnesic patients and control subjects learned to avoid a noncontingent choice better than frontal patients. These results indicate that probabilistic learning does not depend on the brain structures supporting declarative memory.

Early metacognitive abilities: The interplay of monitoring and control processes in 5- to 7-year-old children

The goal of the current investigation was to compare two monitoring processes (judgments of learning [JOLs] and confidence judgments [CJs]) and their corresponding control processes (allocation of study time and selection of answers to maximize accuracy, respectively) in 5- to 7-year-old children (N=101). Children learned the meaning of Japanese characters and provided JOLs after a study phase and CJs after a memory test. They were given the opportunity to control their learning in self-paced study phases, and to control their accuracy by placing correct answers into a treasure chest and incorrect answers into a trash can. All three age groups gave significantly higher CJs for correct compared to incorrect answers, with no age-related differences in the magnitude of this difference, suggesting robust metacognitive monitoring skills in children as young as 5. Furthermore, a link between JOLs and study time was found in the 6- and 7-year-olds, such that children spent more time studying items with low JOLs compared to items with high JOLs. Also, 6- and 7-year-olds but not 5-year-olds spent more time studying difficult items compared to easier items. Moreover, age-related improvements were found in children's use of CJs to guide their selection of answers: although children as young as 5 placed their most confident answers in the treasure chest and least confident answers in the trash can, this pattern was more robust in older children. Overall, results support the view that some metacognitive judgments may be acted upon with greater ease than others among young children.