Altruistic learning.

The origin of altruism remains one of the most enduring puzzles of human behaviour. Indeed, true altruism is often thought either not to exist, or to arise merely as a miscalculation of otherwise selfish behaviour. In this paper, we argue that altruism emerges directly from the way in which distinct human decision-making systems learn about rewards. Using insights provided by neurobiological accounts of human decision-making, we suggest that reinforcement learning in game-theoretic social interactions (habitisation over either individuals or games) and observational learning (either imitative of inference based) lead to altruistic behaviour. This arises not only as a result of computational efficiency in the face of processing complexity, but as a direct consequence of optimal inference in the face of uncertainty. Critically, we argue that the fact that evolutionary pressure acts not over the object of learning ('what' is learned), but over the learning systems themselves ('how' things are learned), enables the evolution of altruism despite the direct threat posed by free-riders.

The misbehavior of value and the discipline of the will.

Most reinforcement learning models of animal conditioning operate under the convenient, though fictive, assumption that Pavlovian conditioning concerns prediction learning whereas instrumental conditioning concerns action learning. However, it is only through Pavlovian responses that Pavlovian prediction learning is evident, and these responses can act against the instrumental interests of the subjects. This can be seen in both experimental and natural circumstances. In this paper we study the consequences of importing this competition into a reinforcement learning context, and demonstrate the resulting effects in an omission schedule and a maze navigation task. The misbehavior created by Pavlovian values can be quite debilitating; we discuss how it may be disciplined.

Cortical substrates for exploratory decisions in humans.

Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.

Opponent appetitive-aversive neural processes underlie predictive learning of pain relief.

Termination of a painful or unpleasant event can be rewarding. However, whether the brain treats relief in a similar way as it treats natural reward is unclear, and the neural processes that underlie its representation as a motivational goal remain poorly understood. We used fMRI (functional magnetic resonance imaging) to investigate how humans learn to generate expectations of pain relief. Using a pavlovian conditioning procedure, we show that subjects experiencing prolonged experimentally induced pain can be conditioned to predict pain relief. This proceeds in a manner consistent with contemporary reward-learning theory (average reward/loss reinforcement learning), reflected by neural activity in the amygdala and midbrain. Furthermore, these reward-like learning signals are mirrored by opposite aversion-like signals in lateral orbitofrontal cortex and anterior cingulate cortex. This dual coding has parallels to 'opponent process' theories in psychology and promotes a formal account of prediction and expectation during pain.

Composite electrospun gelatin fiber-alginate gel scaffolds for mechanically robust tissue engineered cornea

A severe shortage of good quality donor cornea is now an international crisis in public health. Alternatives for donor tissue need to be urgently developed to meet the increasing demand for corneal transplantation. Hydrogels have been widely used as scaffolds for corneal tissue regeneration due to their large water content, similar to that of native tissue. However, these hydrogel scaffolds lack the fibrous structure that functions as a load-bearing component in the native tissue, resulting in poor mechanical performance. This work shows that mechanical properties of compliant hydrogels can be substantially enhanced with electrospun nanofiber reinforcement. Electrospun gelatin nanofibers were infiltrated with alginate hydrogels, yielding transparent fiber-reinforced hydrogels. Without prior crosslinking, electrospun gelatin nanofibers improved the tensile elastic modulus of the hydrogels from 78±19. kPa to 450±100. kPa. Stiffer hydrogels, with elastic modulus of 820±210. kPa, were obtained by crosslinking the gelatin fibers with carbodiimide hydrochloride in ethanol before the infiltration process, but at the expense of transparency. The developed fiber-reinforced hydrogels show great promise as mechanically robust scaffolds for corneal tissue engineering applications. © 2013 Elsevier Ltd.

On the relation of slow feature analysis and Laplacian eigenmaps

The past decade has seen a rise of interest in Laplacian eigenmaps (LEMs) for nonlinear dimensionality reduction. LEMs have been used in spectral clustering, in semisupervised learning, and for providing efficient state representations for reinforcement learning. Here, we show that LEMs are closely related to slow feature analysis (SFA), a biologically inspired, unsupervised learning algorithm originally designed for learning invariant visual representations. We show that SFA can be interpreted as a function approximation of LEMs, where the topological neighborhoods required for LEMs are implicitly defined by the temporal structure of the data. Based on this relation, we propose a generalization of SFA to arbitrary neighborhood relations and demonstrate its applicability for spectral clustering. Finally, we review previous work with the goal of providing a unifying view on SFA and LEMs. © 2011 Massachusetts Institute of Technology.

Code-specific policy gradient rules for spiking neurons

Although it is widely believed that reinforcement learning is a suitable tool for describing behavioral learning, the mechanisms by which it can be implemented in networks of spiking neurons are not fully understood. Here, we show that different learning rules emerge from a policy gradient approach depending on which features of the spike trains are assumed to influence the reward signals, i.e., depending on which neural code is in effect. We use the framework of Williams (1992) to derive learning rules for arbitrary neural codes. For illustration, we present policy-gradient rules for three different example codes - a spike count code, a spike timing code and the most general "full spike train" code - and test them on simple model problems. In addition to classical synaptic learning, we derive learning rules for intrinsic parameters that control the excitability of the neuron. The spike count learning rule has structural similarities with established Bienenstock-Cooper-Munro rules. If the distribution of the relevant spike train features belongs to the natural exponential family, the learning rules have a characteristic shape that raises interesting prediction problems.

Bayesian Learning of Noisy Markov Decision Processes

We consider the inverse reinforcement learning problem, that is, the problem of learning from, and then predicting or mimicking a controller based on state/action data. We propose a statistical model for such data, derived from the structure of a Markov decision process. Adopting a Bayesian approach to inference, we show how latent variables of the model can be estimated, and how predictions about actions can be made, in a unified framework. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from the posterior distribution. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

Fracture energy of the concrete-FRP interface in strengthened beams

To determine the load at which FRPs debond from concrete beams using global-energy-balance-based fracture mechanics concepts, the single most important parameter is the fracture energy of the concrete-FRP interface, which is easy to define but difficult to determine. Debonding propagates in the narrow zone of concrete, between the FRP and the (tension) steel reinforcement bars in the beam, and the presence of nearby steel bars prevents the fracture process zone, which in concrete is normally extensive, from developing fully. The paper presents a detailed discussion of the mechanism of the FRP debonding, and shows that the initiation of debonding can be regarded as a Mode I (tensile) fracture in concrete, despite being loaded primarily in shear. It is shown that the incorporation of this fracture energy in the debonding model developed by the authors, details of which are presented elsewhere, gives predictions that match the test results reported in the literature. © 2013 Elsevier Ltd.

Shear transfer across a diagonal crack in reinforced concrete strengthened with externally bonded FRP fabric

Aging concrete infrastructure in developed economies and more recently constructed concrete infrastructure in the developing world are frequently found to be deficient in structural strength relative to current needs. This can be attributed to a variety of factors including deterioration, construction defects, accidental damage, changes in understanding and failure to design for future loading requirements. Strengthening existing concrete structures can be a cost and carbon effective alternative to replacement. A competitive option for the strengthening of concrete slab-on-beam structures that are deficient in shear capacity is the U-wrapping of the down-stand beam portion of the shear span with externally bonded FRP fabric. While guidance exists for the strengthening of reinforced concrete by U-wrapping, the interaction between internal steel reinforcement, concrete and external FRP in the presence of a dominant diagonal shear crack is not well understood. An approach adopted in previous work has been to explore this interaction through conventional push-off testing. In conventional push-off testing, unlike in a beam, the shear plane is parallel to the direction of loading and perpendicular to the principal fibre orientation. This paper presents a novel push-off test variation in which the shear plane is inclined at 45° to the direction of loading and the principal fibre orientation. A variety of reinforcement ratios, FRP thicknesses and FRP end conditions are modelled. The implications of inclined cracking on debonding of FRP are investigated. The suitability and relevance of inclined push-off tests for further work in this area is also assessed. © 2013, NetComposite Limited.

On-line policy optimisation of Bayesian spoken dialogue systems via human interaction

A partially observable Markov decision process has been proposed as a dialogue model that enables robustness to speech recognition errors and automatic policy optimisation using reinforcement learning (RL). However, conventional RL algorithms require a very large number of dialogues, necessitating a user simulator. Recently, Gaussian processes have been shown to substantially speed up the optimisation, making it possible to learn directly from interaction with human users. However, early studies have been limited to very low dimensional spaces and the learning has exhibited convergence problems. Here we investigate learning from human interaction using the Bayesian Update of Dialogue State system. This dynamic Bayesian network based system has an optimisation space covering more than one hundred features, allowing a wide range of behaviours to be learned. Using an improved policy model and a more robust reward function, we show that stable learning can be achieved that significantly outperforms a simulator trained policy. © 2013 IEEE.

Behavior of RC beams retrofitted with CARDIFRC after thermal cycling

This study investigates the effect of thermal cycling on the performance of concrete beams retrofitted with CARDIFRC, a new class of high performance fiber-reinforced cement-based material that is compatible with concrete. Twenty four beams were subjected to 24 h thermal cycles between 25 and 90°C. One third of the beams were reinforced either in flexure only or in flexure and shear with conventional steel reinforcement and used as control specimens. The remaining sixteen beams were retrofitted with CARDIFRC strips to provide external flexural and/or shear strengthening. All beams were exposed to a varied number of 24 h thermal cycles ranging from 0 to 90 and were tested in four-point bending at room temperature. The tests indicated that the retrofitted members were stronger and stiffer than control beams, and more importantly, that their failure initiated in flexure without any signs of interfacial delamination cracking. The results of these tests are presented and compared to analytical predictions. The predictions show good correlation with the experimental results. © 2010 ASCE.

Analytical model and experimental validation of friction laws for composites under low loads

In order to account for interfacial friction of composite materials, an analytical model based on contact geometry and local friction is proposed. A contact area includes several types of microcontacts depending on reinforcement materials and their shape. A proportion between these areas is defined by in-plane contact geometry. The model applied to a fibre-reinforced composite results in the dependence of friction on surface fibre fraction and local friction coefficients. To validate this analytical model, an experimental study on carbon fibrereinforced epoxy composites under low normal pressure was performed. The effects of fibre volume fraction and fibre orientation were studied, discussed and compared with analytical model results. © Springer Science+Business Media, LLC 2012.

A reversible infinite HMM using normalised random measures

Copyright 2014 by the author(s). We present a nonparametric prior over reversible Markov chains. We use completely random measures, specifically gamma processes, to construct a countably infinite graph with weighted edges. By enforcing symmetry to make the edges undirected we define a prior over random walks on graphs that results in a reversible Markov chain. The resulting prior over infinite transition matrices is closely related to the hierarchical Dirichlet process but enforces reversibility. A reinforcement scheme has recently been proposed with similar properties, but the de Finetti measure is not well characterised. We take the alternative approach of explicitly constructing the mixing measure, which allows more straightforward and efficient inference at the cost of no longer having a closed form predictive distribution. We use our process to construct a reversible infinite HMM which we apply to two real datasets, one from epigenomics and one ion channel recording.

Comparison of moment-curvature models for fiber-reinforced polymer plate-end debonding studies using global energy balance approach

This paper compares a number of different moment-curvature models for cracked concrete sections that contain both steel and external fiber-reinforced polymer (FRP) reinforcement. The question of whether to use a whole-section analysis or one that considers the FRP separately is discussed. Five existing and three new models are compared with test data for moment-curvature or load deflection behavior, and five models are compared with test results for plate-end debonding using a global energy balance approach (GEBA). A proposal is made for the use of one of the simplified models. The availability of a simplified model opens the way to the production of design aids so that the GEBA can be made available to practicing engineers through design guides and parametric studies. Copyright © 2014, American Concrete Institute.