Does temporal discounting explain unhealthy behavior? A systematic review and reinforcement learning perspective.

The tendency to make unhealthy choices is hypothesized to be related to an individual's temporal discount rate, the theoretical rate at which they devalue delayed rewards. Furthermore, a particular form of temporal discounting, hyperbolic discounting, has been proposed to explain why unhealthy behavior can occur despite healthy intentions. We examine these two hypotheses in turn. We first systematically review studies which investigate whether discount rates can predict unhealthy behavior. These studies reveal that high discount rates for money (and in some instances food or drug rewards) are associated with several unhealthy behaviors and markers of health status, establishing discounting as a promising predictive measure. We secondly examine whether intention-incongruent unhealthy actions are consistent with hyperbolic discounting. We conclude that intention-incongruent actions are often triggered by environmental cues or changes in motivational state, whose effects are not parameterized by hyperbolic discounting. We propose a framework for understanding these state-based effects in terms of the interplay of two distinct reinforcement learning mechanisms: a "model-based" (or goal-directed) system and a "model-free" (or habitual) system. Under this framework, while discounting of delayed health may contribute to the initiation of unhealthy behavior, with repetition, many unhealthy behaviors become habitual; if health goals then change, habitual behavior can still arise in response to environmental cues. We propose that the burgeoning development of computational models of these processes will permit further identification of health decision-making phenotypes.

Experimental and numerical investigation of the seismic behavior of rectangular tunnels in soft soils

Underground structures constitute crucial components of the transportation networks. Considering their significance for modern societies, their proper seismic design is of great importance. However, this design may become very tricky, accounting of the lack of knowledge regarding their seismic behavior. Several issues that are significantly affecting this behavior (i.e. earth pressures on the structure, seismic shear stresses around the structure, complex deformation modes for rectangular structures during shaking etc.) are still open. The problem is wider for the non-circular (i.e. rectangular) structures, were the soilstructure interaction effects are expected to be maximized. The paper presents representative experimental results from a test case of a series of dynamic centrifuge tests that were performed on rectangular tunnels embedded in dry sand. The tests were carried out at the centrifuge facility of the University of Cambridge, within the Transnational Task of the SERIES EU research program. The presented test case is also numerically simulated and studied. Preliminary full dynamic time history analyses of the coupled soil-tunnel system are performed, using ABAQUS. Soil non-linearity and soil-structure interaction are modeled, following relevant specifications for underground structures and tunnels. Numerical predictions are compared to experimental results and discussed. Based on this comprehensive experimental and numerical study, the seismic behavior of rectangular embedded structures is better understood and modeled, consisting an important step in the development of appropriate specifications for the seismic design of rectangular shallow tunnels.

Self-organization of reflexive behavior from spontaneous motor activity.

In mammals, the development of reflexes is often regarded as an innate process. However, recent findings show that fetuses are endowed with favorable conditions for ontogenetic development. In this article, we hypothesize that the circuitry of at least some mammalian reflexes can be self-organized from the sensory and motor interactions brought forth in a musculoskeletal system. We focus mainly on three reflexes: the myotatic reflex, the reciprocal inhibition reflex, and the reverse myotatic reflex. To test our hypothesis, we conducted a set of experiments on a simulated musculoskeletal system using pairs of agonist and antagonist muscles. The reflex connectivity is obtained by producing spontaneous motor activity in each muscle and by correlating the resulting sensor and motor signals. Our results show that, under biologically plausible conditions, the reflex circuitry thus obtained is consistent with that identified in relation to the analogous mammalian reflexes. In addition, they show that the reflex connectivity obtained depends on the morphology of the musculoskeletal system as well as on the environment that it is embedded in.

Morphological computation for adaptive behavior and cognition

Traditionally, in robotics, artificial intelligence and neuroscience, there has been a focus on the study of the control or the neural system itself. Recently there has been an increasing interest in the notion of embodiment not only in robotics and artificial intelligence, but also in the neurosciences, psychology and philosophy. In this paper, we introduce the notion of morphological computation, and demonstrate how it can be exploited on the one hand for designing intelligent, adaptive robotic systems, and on the other hand for understanding natural systems. While embodiment has often been used in its trivial meaning, i.e. "intelligence requires a body", the concept has deeper and more important implications, concerned with the relation between physical and information (neural, control) processes. Morphological computation is about connecting body, brain and environment. A number of case studies are presented to illustrate the concept. We conclude with some speculations about potential lessons for neuroscience and robotics. © 2006 Elsevier B.V. All rights reserved.

Behavior learning of a face robot based on the characteristics of human instruction

Computer simulation experiments were performed to examine the effectiveness of OR- and comparative-reinforcement learning algorithms. In the simulation, human rewards were given as +1 and -1. Two models of human instruction that determine which reward is to be given in every step of a human instruction were used. Results show that human instruction may have a possibility of including both model-A and model-B characteristics, and it can be expected that the comparative-reinforcement learning algorithm is more effective for learning by human instructions.