Personalized Tuning of a Reinforcement Learning Control Algorithm for Glucose Regulation

Artificial pancreas is in the forefront of research towards the automatic insulin infusion for patients with type 1 diabetes. Due to the high inter- and intra-variability of the diabetic population, the need for personalized approaches has been raised. This study presents an adaptive, patient-specific control strategy for glucose regulation based on reinforcement learning and more specifically on the Actor-Critic (AC) learning approach. The control algorithm provides daily updates of the basal rate and insulin-to-carbohydrate (IC) ratio in order to optimize glucose regulation. A method for the automatic and personalized initialization of the control algorithm is designed based on the estimation of the transfer entropy (TE) between insulin and glucose signals. The algorithm has been evaluated in silico in adults, adolescents and children for 10 days. Three scenarios of initialization to i) zero values, ii) random values and iii) TE-based values have been comparatively assessed. The results have shown that when the TE-based initialization is used, the algorithm achieves faster learning with 98%, 90% and 73% in the A+B zones of the Control Variability Grid Analysis for adults, adolescents and children respectively after five days compared to 95%, 78%, 41% for random initialization and 93%, 88%, 41% for zero initial values. Furthermore, in the case of children, the daily Low Blood Glucose Index reduces much faster when the TE-based tuning is applied. The results imply that automatic and personalized tuning based on TE reduces the learning period and improves the overall performance of the AC algorithm.

Pulsatile control of rotary blood pumps: Does the modulation waveform matter?

Mechanical support of a failing heart is typically performed with rotary blood pumps running at constant speed, which results in a limited control on cardiac workload and nonpulsatile hemodynamics. A potential solution to overcome these limitations is to modulate the pump speed to create pulses. This study aims at developing a pulsatile control algorithm for rotary pumps, while investigating its effect on left ventricle unloading and the hemodynamics.

A Physiological Controller for Turbodynamic Ventricular Assist Devices Based on a Measurement of the Left Ventricular Volume

The current article presents a novel physiological control algorithm for ventricular assist devices (VADs), which is inspired by the preload recruitable stroke work. This controller adapts the hydraulic power output of the VAD to the end-diastolic volume of the left ventricle. We tested this controller on a hybrid mock circulation where the left ventricular volume (LVV) is known, i.e., the problem of measuring the LVV is not addressed in the current article. Experiments were conducted to compare the response of the controller with the physiological and with the pathological circulation, with and without VAD support. A sensitivity analysis was performed to analyze the influence of the controller parameters and the influence of the quality of the LVV signal on the performance of the control algorithm. The results show that the controller induces a response similar to the physiological circulation and effectively prevents over- and underpumping, i.e., ventricular suction and backflow from the aorta to the left ventricle, respectively. The same results are obtained in the case of a disturbed LVV signal. The results presented in the current article motivate the development of a robust, long-term stable sensor to measure the LVV.

Adaptive Algorithms for Personalized Diabetes Treatment

Dynamic systems, especially in real-life applications, are often determined by inter-/intra-variability, uncertainties and time-varying components. Physiological systems are probably the most representative example in which population variability, vital signal measurement noise and uncertain dynamics render their explicit representation and optimization a rather difficult task. Systems characterized by such challenges often require the use of adaptive algorithmic solutions able to perform an iterative structural and/or parametrical update process towards optimized behavior. Adaptive optimization presents the advantages of (i) individualization through learning of basic system characteristics, (ii) ability to follow time-varying dynamics and (iii) low computational cost. In this chapter, the use of online adaptive algorithms is investigated in two basic research areas related to diabetes management: (i) real-time glucose regulation and (ii) real-time prediction of hypo-/hyperglycemia. The applicability of these methods is illustrated through the design and development of an adaptive glucose control algorithm based on reinforcement learning and optimal control and an adaptive, personalized early-warning system for the recognition and alarm generation against hypo- and hyperglycemic events.

GoCARB in the Context of an Artificial Pancreas

Background: In an artificial pancreas (AP), the meals are either manually announced or detected and their size estimated from the blood glucose level. Both methods have limitations, which result in suboptimal postprandial glucose control. The GoCARB system is designed to provide the carbohydrate content of meals and is presented within the AP framework. Method: The combined use of GoCARB with a control algorithm is assessed in a series of 12 computer simulations. The simulations are defined according to the type of the control (open or closed loop), the use or not-use of GoCARB and the diabetics’ skills in carbohydrate estimation. Results: For bad estimators without GoCARB, the percentage of the time spent in target range (70-180 mg/dl) during the postprandial period is 22.5% and 66.2% for open and closed loop, respectively. When the GoCARB is used, the corresponding percentages are 99.7% and 99.8%. In case of open loop, the time spent in severe hypoglycemic events (<50 mg/dl) is 33.6% without the GoCARB and is reduced to 0.0% when the GoCARB is used. In case of closed loop, the corresponding percentage is 1.4% without the GoCARB and is reduced to 0.0% with the GoCARB. Conclusion: The use of GoCARB improves the control of postprandial response and glucose profiles especially in the case of open loop. However, the most efficient regulation is achieved by the combined use of the control algorithm and the GoCARB.

An insulin infusion advisory system based on autotuning nonlinear model-predictive control

This paper aims at the development and evaluation of a personalized insulin infusion advisory system (IIAS), able to provide real-time estimations of the appropriate insulin infusion rate for type 1 diabetes mellitus (T1DM) patients using continuous glucose monitors and insulin pumps. The system is based on a nonlinear model-predictive controller (NMPC) that uses a personalized glucose-insulin metabolism model, consisting of two compartmental models and a recurrent neural network. The model takes as input patient's information regarding meal intake, glucose measurements, and insulin infusion rates, and provides glucose predictions. The predictions are fed to the NMPC, in order for the latter to estimate the optimum insulin infusion rates. An algorithm based on fuzzy logic has been developed for the on-line adaptation of the NMPC control parameters. The IIAS has been in silico evaluated using an appropriate simulation environment (UVa T1DM simulator). The IIAS was able to handle various meal profiles, fasting conditions, interpatient variability, intraday variation in physiological parameters, and errors in meal amount estimations.

Conservation of the RNA Transport Machineries and Their Coupling to Translation Control across Eukaryotes

Restriction of proteins to discrete subcellular regions is a common mechanism to establish cellular asymmetries and depends on a coordinated program of mRNA localization and translation control. Many processes from the budding of a yeast to the establishment of metazoan embryonic axes and the migration of human neurons, depend on this type of cell polarization. How factors controlling transport and translation assemble to regulate at the same time the movement and translation of transported mRNAs, and whether these mechanisms are conserved across kingdoms is not yet entirely understood. In this review we will focus on some of the best characterized examples of mRNA transport machineries, the "yeast locasome" as an example of RNA transport and translation control in unicellular eukaryotes, and on the Drosophila Bic-D/Egl/Dyn RNA localization machinery as an example of RNA transport in higher eukaryotes. This focus is motivated by the relatively advanced knowledge about the proteins that connect the localizing mRNAs to the transport motors and the many well studied proteins involved in translational control of specific transcripts that are moved by these machineries. We will also discuss whether the core of these RNA transport machineries and factors regulating mRNA localization and translation are conserved across eukaryotes.

Acetyl-L-carnitine feeding to unloaded rats triggers in soleus muscle the coordinated expression of genes involved in mitochondrial biogenesis

The expressional profile of mitochondrial transcripts and of genes involved in the mitochondrial biogenesis pathway induced by ALCAR daily supplementation in soleus muscle of control and unloaded 3-month-old rats has been analyzed. It has been found that ALCAR treatment is able to upregulate the expression level of mitochondrial transcripts (COX I, ATP6, ND6, 16 S rRNA) in both control and unloaded animals. Interestingly, ALCAR feeding to unloaded rats resulted in the increase of transcript level for master factors involved in mitochondrial biogenesis (PGC-1alpha, NRF-1, TFAM). It also prevented the unloading-induced downregulation of mRNA levels for kinases able to transduce metabolic (AMPK) and neuronal stimuli (CaMKIIbeta) into mitochondrial biogenesis. No significant effect on the expressional level of such genes was found in control ALCAR-treated rats. In addition, ALCAR feeding was able to prevent the loss of mitochondrial protein content due to unloading condition. Correlation analysis revealed a strong coordination in the expression of genes involved in mitochondrial biogenesis only in ALCAR-treated suspended animals, supporting a differentiated effect of ALCAR treatment in relation to the loading state of the soleus muscle. In conclusions, we demonstrated the ability of ALCAR supplementation to promote only in soleus muscle of hindlimb suspended rats an orchestrated expression of genes involved in mitochondrial biogenesis, which might counteract the unloading-induced metabolic changes, preventing the loss of mitochondrial proteins.

Outside-in signaling through integrins and cadherins: a central mechanism to control epidermal growth and differentiation?

The process of epidermal renewal persists throughout the entire life of an organism. It begins when a keratinocyte progenitor leaves the stem cell compartment, undergoes a limited number of mitotic divisions, exits the cell cycle, and commits to terminal differentiation. At the end of this phase, the postmitotic keratinocytes detach from the basement membrane to build up the overlaying stratified epithelium. Although highly coordinated, this sequence of events is endowed with a remarkable versatility, which enables the quiescent keratinocyte to reintegrate into the cell cycle and become migratory when necessary, for example after wounding. It is this versatility that represents the Achilles heel of epithelial cells allowing for the development of severe pathologies. Over the past decade, compelling evidence has been provided that epithelial cancer cells achieve uncontrolled proliferation following hijacking of a "survival program" with PI3K/Akt and a "proliferation program" with growth factor receptor signaling at its core. Recent insights into adhesion receptor signaling now propose that integrins, but also cadherins, can centrally control these programs. It is suggested that the two types of adhesion receptors act as sensors to transmit extracellular stimuli in an outside-in mode, to inversely modulate epidermal growth factor receptor signaling and ensure cell survival. Hence, cell-matrix and cell-cell adhesion receptors likely play a more powerful and wide-ranging role than initially anticipated. This Perspective article discusses the relevance of this emerging field for epidermal growth and differentiation, which can be of importance for severe pathologies such as tumorigenesis and invasive metastasis, as well as psoriasis and Pemphigus vulgaris.

An integral conservative gridding--algorithm using Hermitian curve interpolation

The problem of re-sampling spatially distributed data organized into regular or irregular grids to finer or coarser resolution is a common task in data processing. This procedure is known as 'gridding' or 're-binning'. Depending on the quantity the data represents, the gridding-algorithm has to meet different requirements. For example, histogrammed physical quantities such as mass or energy have to be re-binned in order to conserve the overall integral. Moreover, if the quantity is positive definite, negative sampling values should be avoided. The gridding process requires a re-distribution of the original data set to a user-requested grid according to a distribution function. The distribution function can be determined on the basis of the given data by interpolation methods. In general, accurate interpolation with respect to multiple boundary conditions of heavily fluctuating data requires polynomial interpolation functions of second or even higher order. However, this may result in unrealistic deviations (overshoots or undershoots) of the interpolation function from the data. Accordingly, the re-sampled data may overestimate or underestimate the given data by a significant amount. The gridding-algorithm presented in this work was developed in order to overcome these problems. Instead of a straightforward interpolation of the given data using high-order polynomials, a parametrized Hermitian interpolation curve was used to approximate the integrated data set. A single parameter is determined by which the user can control the behavior of the interpolation function, i.e. the amount of overshoot and undershoot. Furthermore, it is shown how the algorithm can be extended to multidimensional grids. The algorithm was compared to commonly used gridding-algorithms using linear and cubic interpolation functions. It is shown that such interpolation functions may overestimate or underestimate the source data by about 10-20%, while the new algorithm can be tuned to significantly reduce these interpolation errors. The accuracy of the new algorithm was tested on a series of x-ray CT-images (head and neck, lung, pelvis). The new algorithm significantly improves the accuracy of the sampled images in terms of the mean square error and a quality index introduced by Wang and Bovik (2002 IEEE Signal Process. Lett. 9 81-4).

Movement control of manipulative tasks in patients with Gilles de la Tourette syndrome

When a hand-held object is moved, grip and load force are accurately coordinated for establishing grasp stability. In the present work, the question was raised whether patients with Gilles de la Tourette syndrome (TS), who show tic-like movements, are impaired in grip-load force control when executing a manipulative task. To this end, we assessed force regulation during action patterns that required rhythmical unimanual or bimanual (iso-directional/anti-directional) movements. Results showed that the profile of grip-load force ratio was characterized by maxima and minima that were realized at upward and downward hand positions, respectively. TS patients showed increased force ratios during unimanual and bimanual movements, compared with control subjects, indicative of an inaccurate specification of the precision grip. Functional imaging data complemented the behavioural results and revealed that secondary motor areas showed no (or greatly reduced) activation in TS patients when executing the movement tasks as compared with baseline conditions. This indicates that the metabolic level in the secondary motor areas was equal during rest and task performance. At the neuronal level, this observation suggests that these cortical areas were continuously involved in movement preparation. Based on these data, we conclude that the ongoing activation of secondary motor areas may be explained by the TS patients' involuntary urges to move. Accordingly, interference will prevent an accurate planning of voluntary behaviour. Together, these findings reveal modulations in movement organization in patients with TS and exemplify degrading consequences for manual function.

An insulin infusion advisory system for type 1 diabetes patients based on non-linear model predictive control methods

In this paper, an Insulin Infusion Advisory System (IIAS) for Type 1 diabetes patients, which use insulin pumps for the Continuous Subcutaneous Insulin Infusion (CSII) is presented. The purpose of the system is to estimate the appropriate insulin infusion rates. The system is based on a Non-Linear Model Predictive Controller (NMPC) which uses a hybrid model. The model comprises a Compartmental Model (CM), which simulates the absorption of the glucose to the blood due to meal intakes, and a Neural Network (NN), which simulates the glucose-insulin kinetics. The NN is a Recurrent NN (RNN) trained with the Real Time Recurrent Learning (RTRL) algorithm. The output of the model consists of short term glucose predictions and provides input to the NMPC, in order for the latter to estimate the optimum insulin infusion rates. For the development and the evaluation of the IIAS, data generated from a Mathematical Model (MM) of a Type 1 diabetes patient have been used. The proposed control strategy is evaluated at multiple meal disturbances, various noise levels and additional time delays. The results indicate that the implemented IIAS is capable of handling multiple meals, which correspond to realistic meal profiles, large noise levels and time delays.

Numerical solution of optimal control problems with constant control delays

We investigate a class of optimal control problems that exhibit constant exogenously given delays in the control in the equation of motion of the differential states. Therefore, we formulate an exemplary optimal control problem with one stock and one control variable and review some analytic properties of an optimal solution. However, analytical considerations are quite limited in case of delayed optimal control problems. In order to overcome these limits, we reformulate the problem and apply direct numerical methods to calculate approximate solutions that give a better understanding of this class of optimization problems. In particular, we present two possibilities to reformulate the delayed optimal control problem into an instantaneous optimal control problem and show how these can be solved numerically with a stateof- the-art direct method by applying Bock’s direct multiple shooting algorithm. We further demonstrate the strength of our approach by two economic examples.

EPA-1380 – Coordinated body-movement reflects psychotherapy outcome

Social interaction is a core aspect of human life that affects individuals’ physical and mental health. Social interaction usually leads to mutual engagement in diverse areas of mental, emotional, physiological and physical activity involving both interacting persons and subsequently impacting the outcome of interactions. A common approach to the analysis of social interaction is the study of the verbal content transmitted between sender and receiver. However, additional important processes and dynamics are occurring in other domains too, for example in the area of nonverbal behaviour: In a series of studies, we have looked at nonverbal synchrony – the coordination of two persons’ movement patterns – and it‘s association with relationship quality and with the outcome of interactions. Using a computer-based algorithm (Motion Energy Analysis, MEA: Ramseyer & Tschacher, 2011), which automatically quantifies a person‘s body-movement, we were able to objectively calculate nonverbal synchrony in a large number of dyads interacting in various settings. In a first step, we showed that the phenomenon of nonverbal synchrony exists at a level that is significantly higher than expected by chance. In a second step, we ascertained that across different settings – including patient-therapist dyads and healthy dyads – more synchronized movement was associated with better relationship quality and better interactional outcomes. The quality of a relationship is thus embodied by the synchronized movement patterns emerging between partners. Our studies suggest that embodied cognition is a valuable approach to research in social interaction, providing important clues for an improved understanding of interaction dynamics.