Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Reinforcement learning in dendritic structures

The discovery of binary dendritic events such as local NMDA spikes in dendritic subbranches led to the suggestion that dendritic trees could be computationally equivalent to a 2-layer network of point neurons, with a single output unit represented by the soma, and input units represented by the dendritic branches. Although this interpretation endows a neuron with a high computational power, it is functionally not clear why nature would have preferred the dendritic solution with a single but complex neuron, as opposed to the network solution with many but simple units. We show that the dendritic solution has a distinguished advantage over the network solution when considering different learning tasks. Its key property is that the dendritic branches receive an immediate feedback from the somatic output spike, while in the corresponding network architecture the feedback would require additional backpropagating connections to the input units. Assuming a reinforcement learning scenario we formally derive a learning rule for the synaptic contacts on the individual dendritic trees which depends on the presynaptic activity, the local NMDA spikes, the somatic action potential, and a delayed reinforcement signal. We test the model for two scenarios: the learning of binary classifications and of precise spike timings. We show that the immediate feedback represented by the backpropagating action potential supplies the individual dendritic branches with enough information to efficiently adapt their synapses and to speed up the learning process.

Gradient estimation in dendritic reinforcement learning

We study synaptic plasticity in a complex neuronal cell model where NMDA-spikes can arise in certain dendritic zones. In the context of reinforcement learning, two kinds of plasticity rules are derived, zone reinforcement (ZR) and cell reinforcement (CR), which both optimize the expected reward by stochastic gradient ascent. For ZR, the synaptic plasticity response to the external reward signal is modulated exclusively by quantities which are local to the NMDA-spike initiation zone in which the synapse is situated. CR, in addition, uses nonlocal feedback from the soma of the cell, provided by mechanisms such as the backpropagating action potential. Simulation results show that, compared to ZR, the use of nonlocal feedback in CR can drastically enhance learning performance. We suggest that the availability of nonlocal feedback for learning is a key advantage of complex neurons over networks of simple point neurons, which have previously been found to be largely equivalent with regard to computational capability.

Minimum ten year results of total hip arthroplasty with the acetabular reinforcement ring in avascular osteonecrosis

Total hip arthroplasty (THA) still carries a higher failure rate in patients with avascular necrosis of the femoral head (AVN) than in a similar patient population with THA for other reasons. This is particularly true for the acetabular component. One of the major factors accounting for this is the compromised acetabular bone quality with structural defects subsequent to collapsing of the femoral head in high-grade AVN. In this study we implanted an acetabular reinforcement ring with hook (ARRH), which had been used successfully for other indications with acetabular bone stock deficiency, in 32 consecutive THA's in 29 patients with AVN. Five patients died during the observation period of causes unrelated to the surgery, one patient was lost to follow-up and one patient could not be followed up due to chronic illness, leaving 25 hips (23 patients) with a minimum follow-up of ten years (mean: 11.8; range: 10-15). The mean Merle d'Aubigne score increased significantly from 7.7 preoperatively to 16.6 postoperatively (p < 0.001). One revision was performed for aseptic stem loosening. Of the unrevised hips, one acetabular component was classified as definitively loose. The cumulative 12-year survivorship for THA with ARRH in AVN was 95.2% (confidence interval: 86.1-100%) for both components, 100% for the cup and 95.2% for the stem (86.1-100%).

Preliminary results following reinforcement of the pulmonary autograft to prevent dilatation after the Ross procedure

OBJECTIVE: The Ross operation remains a controversially discussed procedure, because concern exists regarding late dilatation of the neoaortic root and progressive regurgitation of the autograft valve. We present our early experience with an external reinforcement of the autograft, which is inserted into a prosthetic Dacron graft with an artificial aortic root configuration. This detail should help to prevent neoaortic root dilatation. PATIENTS AND METHODS: Between 2006 and 2007, 12 patients (mean age 16 +/- 38 years; range 15-38 years) underwent a Ross procedure by this technique. Indications were aortic regurgitation (n = 2), aortic stenosis (n = 5), and combined aortic stenosis and insufficiency (n = 5). A bicuspid aortic valve was present in 9 patients. Balloon valvuloplasty had been performed in 7 patients. Follow-up was performed by clinical and echocardiographic examinations. RESULTS: No early or late deaths occurred in this small series, and freedom from reoperation is 100%. Echocardiographic follow-up confirmed absence of aortic insufficiency in 11 patients after a mean of 11 months (range 2-30 months). In 1 patient, a small asymmetric regurgitation jet was already observed at discharge echocardiography. As expected, no neoaortic root dilatation was observed during follow-up. All patients are in New York Heart Association class I. CONCLUSIONS: The present technique is a simple and reproducible technical step that does not require significant additional time. Inclusion of the autograft within a root prosthesis may be especially indicated in situations known for late autograft dilatation, namely, bicuspid aortic valve, predominant aortic insufficiency, and ascending aortic enlargement.

Acetabular reinforcement ring in primary total hip arthroplasty: a minimum 10-year follow-up

INTRODUCTION: We report the results of a titanium acetabular reinforcement ring with a hook (ARRH) in primary total hip arthroplasty (THA), which was introduced in 1987 and continues to be used routinely in our center. The favorable results of this device in arthroplasty for developmental dysplasia and difficult revisions motivated its use in primary THA. With this implant only minimal acetabular reaming is necessary, anatomic positioning is achieved by placing the hook around the teardrop and a homogenous base for cementing the polyethylene cup is provided. MATERIALS AND METHODS: Between April 1987 and December 1991, 241 THAs with insertion of an ARRH were performed in 178 unselected, consecutive patients (average age 58 years; range 30-84 years) with a secondary osteoarthrosis in 41% of the cases. RESULTS: At the time of the latest follow-up, 33 patients (39 hips) had died and 17 cases had been lost to follow-up. The median follow-up was 122 months with a minimum of 10 years. Eight hips had been revised, leaving 177 hips in 120 living patients without revision. Six cups were revised because of aseptic loosening. Two hips were revised for sepsis. The mean Merle d'Aubigné score for the remaining hips was 16 (range 7-18) at the latest follow-up. For aseptic loosening, the probability of survival of the cup was 0.97 (95% confidence interval, 0.94-0.99). However, analysis of radiographs implied loosening in seven other cups without clinical symptoms. CONCLUSIONS: The results of primary THA using an acetabular reinforcement ring parallel the excellent results of these implants often observed in difficult primary and revision arthroplasty at a minimum of 10 years. Survivorship is comparable to modern cementless implants. Medial migration that occurs with loosening of the acetabular component seems to be prevented with this implant. Radiographic loosening signs can exist without clinical symptoms.

Kindlin-3 regulates integrin activation and adhesion reinforcement of effector T cells

Activated T cells use very late antigen-4/α4β1 integrin for capture, rolling on, and firm adhesion to endothelial cells, and use leukocyte function-associated antigen-1/αLβ2 integrin for subsequent crawling and extravasation. Inhibition of α4β1 is sufficient to prevent extravasation of activated T cells and is successfully used to combat autoimmune diseases, such as multiple sclerosis. Here we show that effector T cells lacking the integrin activator Kindlin-3 extravasate and induce experimental autoimmune encephalomyelitis in mice immunized with autoantigen. In sharp contrast, adoptively transferred autoreactive T cells from Kindlin-3-deficient mice fail to extravasate into the naïve CNS. Mechanistically, autoreactive Kindlin-3-null T cells extravasate when the CNS is inflamed and the brain microvasculature expresses high levels of integrin ligands. Flow chamber assays under physiological shear conditions confirmed that Kindlin-3-null effector T cells adhere to high concentrations of vascular cell adhesion molecule-1 and intercellular adhesion molecule-1, albeit less efficiently than WT T cells. Although these arrested T cells polarize and start crawling, only few remain firmly adherent over time. Our data demonstrate that the requirement of Kindlin-3 for effector T cells to induce α4β1 and αLβ2 integrin ligand binding and stabilization of integrin-ligand bonds is critical when integrin ligand levels are low, but of less importance when integrin ligand levels are high.

Personalized Tuning of a Reinforcement Learning Control Algorithm for Glucose Regulation

Artificial pancreas is in the forefront of research towards the automatic insulin infusion for patients with type 1 diabetes. Due to the high inter- and intra-variability of the diabetic population, the need for personalized approaches has been raised. This study presents an adaptive, patient-specific control strategy for glucose regulation based on reinforcement learning and more specifically on the Actor-Critic (AC) learning approach. The control algorithm provides daily updates of the basal rate and insulin-to-carbohydrate (IC) ratio in order to optimize glucose regulation. A method for the automatic and personalized initialization of the control algorithm is designed based on the estimation of the transfer entropy (TE) between insulin and glucose signals. The algorithm has been evaluated in silico in adults, adolescents and children for 10 days. Three scenarios of initialization to i) zero values, ii) random values and iii) TE-based values have been comparatively assessed. The results have shown that when the TE-based initialization is used, the algorithm achieves faster learning with 98%, 90% and 73% in the A+B zones of the Control Variability Grid Analysis for adults, adolescents and children respectively after five days compared to 95%, 78%, 41% for random initialization and 93%, 88%, 41% for zero initial values. Furthermore, in the case of children, the daily Low Blood Glucose Index reduces much faster when the TE-based tuning is applied. The results imply that automatic and personalized tuning based on TE reduces the learning period and improves the overall performance of the AC algorithm.

Tracing reinforcement through asymmetrical partner preference in the European common vole Microtus arvalis

Background The mechanistic basis of speciation and in particular the contribution of behaviour to the completion of the speciation process is often contentious. Contact zones between related taxa provide a situation where selection against hybridization might reinforce separation by behavioural mechanisms, which could ultimately fully isolate the taxa. One of the most abundant European mammals, the common vole Microtus arvalis, forms multiple natural hybrid zones where rapidly diverging evolutionary lineages meet in secondary contact. Very narrow zones of hybridization spanning only a few kilometres and sex-specific gene flow patterns indicate reduced fitness of natural hybrids and incipient speciation between some of the evolutionary lineages. In this study, we examined the contribution of behavioural mechanisms to the speciation process in these rodents by fine-mapping allopatric and parapatric populations in the hybrid zone between the Western and Central lineages and experimental testing of the partner preferences of wild, pure-bred and hybrid female common voles. Results Genetic analysis based on microsatellite markers revealed the presence of multiple parapatric and largely non-admixed populations at distances of about 10 km at the edge of the area of natural hybridization between the Western and Central lineages. Wild females from Western parapatric populations and lab-born F1 hybrids preferred males from the Western lineage whereas wild females of Central parapatric origin showed no measurable preference. Furthermore, wild and lab-born females from allopatric populations of the Western or Central lineages showed no detectable preference for males from either lineage. Conclusions The detected partner preferences are consistent with asymmetrical reinforcement of pre-mating reproductive isolation mechanisms in the European common vole and with earlier results suggesting that hybridization is more detrimental to the Western lineage. As a consequence, these differences in behaviour might contribute to a further geographical stabilization of this moving hybrid zone. Such behavioural processes could also provide a mechanistic perspective for frequently-detected asymmetrical introgression patterns in the largely allopatrically diversifying Microtus genus and other rapidly speciating rodents.