Curiosity driven reinforcement learning for motion planning on humanoids

Most previous work on artificial curiosity (AC) and intrinsic motivation focuses on basic concepts and theory. Experimental results are generally limited to toy scenarios, such as navigation in a simulated maze, or control of a simple mechanical system with one or two degrees of freedom. To study AC in a more realistic setting, we embody a curious agent in the complex iCub humanoid robot. Our novel reinforcement learning (RL) framework consists of a state-of-the-art, low-level, reactive control layer, which controls the iCub while respecting constraints, and a high-level curious agent, which explores the iCub's state-action space through information gain maximization, learning a world model from experience, controlling the actual iCub hardware in real-time. To the best of our knowledge, this is the first ever embodied, curious agent for real-time motion planning on a humanoid. We demonstrate that it can learn compact Markov models to represent large regions of the iCub's configuration space, and that the iCub explores intelligently, showing interest in its physical constraints as well as in objects it finds in its environment.

Interlaminar mechanical properties of carbon fiber reinforced plastic laminates modified with graphene oxide interleaf

By taking the advantage of the excellent mechanical properties and high specific surface area of graphene oxide (GO) sheets, we develop a simple and effective strategy to improve the interlaminar mechanical properties of carbon fiber reinforced plastic (CFRP) laminates. With the incorporation of graphene oxide reinforced epoxy interleaf into the interface of CFRP laminates, the Mode-I fracture toughness and resistance were greatly increased. The experimental results of double cantilever beam (DCB) tests demonstrated that, with 2 g/m2 addition of GO, the Mode-I fracture toughness and resistance of the specimen increase by 170.8% and 108.0%, respectively, compared to those of the plain specimen. The improvement mechanisms were investigated by the observation of fracture surface with scanning electron microscopies. Moreover, finite element analyses were performed based on the cohesive zone model to verify the experimental fracture toughness and to predict the interfacial tensile strength of CFRP laminates.

Extending genetic linkage analysis to diffusion tensor images to map single gene effects on brain fiber architecture

We extended genetic linkage analysis - an analysis widely used in quantitative genetics - to 3D images to analyze single gene effects on brain fiber architecture. We collected 4 Tesla diffusion tensor images (DTI) and genotype data from 258 healthy adult twins and their non-twin siblings. After high-dimensional fluid registration, at each voxel we estimated the genetic linkage between the single nucleotide polymorphism (SNP), Val66Met (dbSNP number rs6265), of the BDNF gene (brain-derived neurotrophic factor) with fractional anisotropy (FA) derived from each subject's DTI scan, by fitting structural equation models (SEM) from quantitative genetics. We also examined how image filtering affects the effect sizes for genetic linkage by examining how the overall significance of voxelwise effects varied with respect to full width at half maximum (FWHM) of the Gaussian smoothing applied to the FA images. Raw FA maps with no smoothing yielded the greatest sensitivity to detect gene effects, when corrected for multiple comparisons using the false discovery rate (FDR) procedure. The BDNF polymorphism significantly contributed to the variation in FA in the posterior cingulate gyrus, where it accounted for around 90-95% of the total variance in FA. Our study generated the first maps to visualize the effect of the BDNF gene on brain fiber integrity, suggesting that common genetic variants may strongly determine white matter integrity.

Brain fiber architecture, genetics, and intelligence: a high angular resolution diffusion imaging (HARDI) study

We developed an analysis pipeline enabling population studies of HARDI data, and applied it to map genetic influences on fiber architecture in 90 twin subjects. We applied tensor-driven 3D fluid registration to HARDI, resampling the spherical fiber orientation distribution functions (ODFs) in appropriate Riemannian manifolds, after ODF regularization and sharpening. Fitting structural equation models (SEM) from quantitative genetics, we evaluated genetic influences on the Jensen-Shannon divergence (JSD), a novel measure of fiber spatial coherence, and on the generalized fiber anisotropy (GFA) a measure of fiber integrity. With random-effects regression, we mapped regions where diffusion profiles were highly correlated with subjects' intelligence quotient (IQ). Fiber complexity was predominantly under genetic control, and higher in more highly anisotropic regions; the proportion of genetic versus environmental control varied spatially. Our methods show promise for discovering genes affecting fiber connectivity in the brain.

Mapping genetic influences on brain fiber architecture with high angular resolution diffusion imaging (HARDI)

We report the first 3D maps of genetic effects on brain fiber complexity. We analyzed HARDI brain imaging data from 90 young adult twins using an information-theoretic measure, the Jensen-Shannon divergence (JSD), to gauge the regional complexity of the white matter fiber orientation distribution functions (ODF). HARDI data were fluidly registered using Karcher means and ODF square-roots for interpol ation; each subject's JSD map was computed from the spatial coherence of the ODFs in each voxel's neighborhood. We evaluated the genetic influences on generalized fiber anisotropy (GFA) and complexity (JSD) using structural equation models (SEM). At each voxel, genetic and environmental components of data variation were estimated, and their goodness of fit tested by permutation. Color-coded maps revealed that the optimal models varied for different brain regions. Fiber complexity was predominantly under genetic control, and was higher in more highly anisotropic regions. These methods show promise for discovering factors affecting fiber connectivity in the brain.

Genetics of brain fiber architecture and intellectual performance

The study is the first to analyze genetic and environmental factors that affect brain fiber architecture and its genetic linkage with cognitive function. We assessed white matter integrity voxelwise using diffusion tensor imaging at high magnetic field (4 Tesla), in 92 identical and fraternal twins. White matter integrity, quantified using fractional anisotropy (FA), was used to fit structural equation models (SEM) at each point in the brain, generating three-dimensional maps of heritability. We visualized the anatomical profile of correlations between white matter integrity and full-scale, verbal, and performance intelligence quotients (FIQ, VIQ, and PIQ). White matter integrity (FA) was under strong genetic control and was highly heritable in bilateral frontal (a 2 = 0.55, p = 0.04, left; a 2 = 0.74, p = 0.006, right), bilateral parietal (a 2 = 0.85, p < 0.001, left; a 2 = 0.84, p < 0.001, right), and left occipital (a 2 = 0.76, p = 0.003) lobes, and was correlated with FIQ and PIQ in the cingulum, optic radiations, superior fronto- occipital fasciculus, internal capsule, callosal isthmus, and the corona radiata (p = 0.04 for FIQ and p = 0.01 for PIQ, corrected for multiple comparisons). In a cross-trait mapping approach, common genetic factors mediated the correlation between IQ and white matter integrity, suggesting a common physiological mechanism for both, and common genetic determination. These genetic brain maps reveal heritable aspects of white matter integrity and should expedite the discovery of single-nucleotide polymorphisms affecting fiber connectivity and cognition.

Information-theoretic analysis of brain white matter fiber orientation distribution functions

We propose a new information-theoretic metric, the symmetric Kullback-Leibler divergence (sKL-divergence), to measure the difference between two water diffusivity profiles in high angular resolution diffusion imaging (HARDI). Water diffusivity profiles are modeled as probability density functions on the unit sphere, and the sKL-divergence is computed from a spherical harmonic series, which greatly reduces computational complexity. Adjustment of the orientation of diffusivity functions is essential when the image is being warped, so we propose a fast algorithm to determine the principal direction of diffusivity functions using principal component analysis (PCA). We compare sKL-divergence with other inner-product based cost functions using synthetic samples and real HARDI data, and show that the sKL-divergence is highly sensitive in detecting small differences between two diffusivity profiles and therefore shows promise for applications in the nonlinear registration and multisubject statistical analysis of HARDI data.

An evaluation of differential reinforcement of other behavior, differential reinforcement of incompatible behavior, and restitution for the management of aggressive behaviors

A 46 year old institutionalized retarded woman was referred for treatment because of frequent verbal abuse and physically aggressive behaviors. A multicomponent behavioral intervention consisting of differential reinforcement of other behaviors, differential reinforcement of incompatible behaviors, and restitution was implemented. A sequential withdrawal design was used in order to evaluate the effects of components of the original intervention, and to provide a measure of response maintenance. Results indicated a marked decrease in verbal abuse and elimination of physical aggression. Verbal abuse recurred when the restitution procedure was withdrawn. The relative efficacy of the different interventions is discussed with respect to the behavioral management of aggression.

Heritability of white matter fiber tract shapes: A HARDI study of 198 twins

Genetic analysis of diffusion tensor images (DTI) shows great promise in revealing specific genetic variants that affect brain integrity and connectivity. Most genetic studies of DTI analyze voxel-based diffusivity indices in the image space (such as 3D maps of fractional anisotropy) and overlook tract geometry. Here we propose an automated workflow to cluster fibers using a white matter probabilistic atlas and perform genetic analysis on the shape characteristics of fiber tracts. We apply our approach to large study of 4-Tesla high angular resolution diffusion imaging (HARDI) data from 198 healthy, young adult twins (age: 20-30). Illustrative results show heritability for the shapes of several major tracts, as color-coded maps.

The multivariate A/C/E model and the genetics of fiber architecture

We present a new algorithm to compute the voxel-wise genetic contribution to brain fiber microstructure using diffusion tensor imaging (DTI) in a dataset of 25 monozygotic (MZ) twins and 25 dizygotic (DZ) twin pairs (100 subjects total). First, the structural and DT scans were linearly co-registered. Structural MR scans were nonlinearly mapped via a 3D fluid transformation to a geometrically centered mean template, and the deformation fields were applied to the DTI volumes. After tensor re-orientation to realign them to the anatomy, we computed several scalar and multivariate DT-derived measures including the geodesic anisotropy (GA), the tensor eigenvalues and the full diffusion tensors. A covariance-weighted distance was measured between twins in the Log-Euclidean framework [2], and used as input to a maximum-likelihood based algorithm to compute the contributions from genetics (A), common environmental factors (C) and unique environmental ones (E) to fiber architecture. Quanititative genetic studies can take advantage of the full information in the diffusion tensor, using covariance weighted distances and statistics on the tensor manifold.

White matter integrity measured by fractional anisotropy correlates poorly with actual individual fiber anisotropy

Fractional anisotropy (FA), a very widely used measure of fiber integrity based on diffusion tensor imaging (DTI), is a problematic concept as it is influenced by several quantities including the number of dominant fiber directions within each voxel, each fiber's anisotropy, and partial volume effects from neighboring gray matter. High-angular resolution diffusion imaging (HARDI) can resolve more complex diffusion geometries than standard DTI, including fibers crossing or mixing. The tensor distribution function (TDF) can be used to reconstruct multiple underlying fibers per voxel, representing the diffusion profile as a probabilistic mixture of tensors. Here we found that DTIderived mean diffusivity (MD) correlates well with actual individual fiber MD, but DTI-derived FA correlates poorly with actual individual fiber anisotropy, and may be suboptimal when used to detect disease processes that affect myelination. Analysis of the TDFs revealed that almost 40% of voxels in the white matter had more than one dominant fiber present. To more accurately assess fiber integrity in these cases, we here propose the differential diffusivity (DD), which measures the average anisotropy based on all dominant directions in each voxel.

Brain network efficiency and topology depend on the fiber tracking method: 11 tractography algorithms compared in 536 subjects

As connectivity analyses become more popular, claims are often made about how the brain's anatomical networks depend on age, sex, or disease. It is unclear how results depend on tractography methods used to compute fiber networks. We applied 11 tractography methods to high angular resolution diffusion images of the brain (4-Tesla 105-gradient HARDI) from 536 healthy young adults. We parcellated 70 cortical regions, yielding 70×70 connectivity matrices, encoding fiber density. We computed popular graph theory metrics, including network efficiency, and characteristic path lengths. Both metrics were robust to the number of spherical harmonics used to model diffusion (4th-8th order). Age effects were detected only for networks computed with the probabilistic Hough transform method, which excludes smaller fibers. Sex and total brain volume affected networks measured with deterministic, tensor-based fiber tracking but not with the Hough method. Each tractography method includes different fibers, which affects inferences made about the reconstructed networks.

Analyzing multi-fiber reconstruction in high angular resolution diffusion imaging using the tensor distribution function

High-angular resolution diffusion imaging (HARDI) can reconstruct fiber pathways in the brain with extraordinary detail, identifying anatomical features and connections not seen with conventional MRI. HARDI overcomes several limitations of standard diffusion tensor imaging, which fails to model diffusion correctly in regions where fibers cross or mix. As HARDI can accurately resolve sharp signal peaks in angular space where fibers cross, we studied how many gradients are required in practice to compute accurate orientation density functions, to better understand the tradeoff between longer scanning times and more angular precision. We computed orientation density functions analytically from tensor distribution functions (TDFs) which model the HARDI signal at each point as a unit-mass probability density on the 6D manifold of symmetric positive definite tensors. In simulated two-fiber systems with varying Rician noise, we assessed how many diffusionsensitized gradients were sufficient to (1) accurately resolve the diffusion profile, and (2) measure the exponential isotropy (EI), a TDF-derived measure of fiber integrity that exploits the full multidirectional HARDI signal. At lower SNR, the reconstruction accuracy, measured using the Kullback-Leibler divergence, rapidly increased with additional gradients, and EI estimation accuracy plateaued at around 70 gradients.

Fiber Bragg grating accelerometer based on a transversely rotating stick

Fiber Bragg Grating (FBG) accelerometers using transverse forces with an inertial object placed at the middle of the FBG have a high sensitivity but low resonant frequency. The resonant frequency 26 Hz and sensitivity at 6 Hz 1.29 nm/g were reported based on a 50mm-long FBG accelerometer. We demonstrate that the first FBG accelerometer based on a transversely rotating stick, which can, at the same or even larger size, keep the high sensitivity and significantly increase the low resonant frequency. In our experiments, a 77.5mm-long FBG accelerometer has achieved a similar sensitivity but 65% higher resonant frequency. This novel structure not only significantly widens the potential applications of FBG accelerometers by increasing their resonant frequencies but also provides a new route to design other accelerometers, e.g. micro accelerometers.

Anomalous thermal dynamics of Bragg gratings inscribed in germanosilicate optical fiber

An interesting, periodic appearance of a new peak has been observed in the reflected spectrum of a Fiber Bragg Grating (FBG) inscribed in a germanosilicate fiber during thermal treatment. The new peak occurs on the longer wavelength side of the spectrum during heating and on the shorter wavelength side during cooling, following an identical reverse dynamics. Comparison with a commercial grating with 99.9% reflectivity shows a similar decay dynamics. It is proposed that the distortion due to simultaneous erasure and thermal expansion of the index modulation profile may be responsible for the observed anomaly. The reported results help us in understanding the thermal behavior of FBGs and provide additional insights into the mechanisms responsible for the photosensitivity in germanosilicate fibers.