Curiosity driven reinforcement learning for motion planning on humanoids

Most previous work on artificial curiosity (AC) and intrinsic motivation focuses on basic concepts and theory. Experimental results are generally limited to toy scenarios, such as navigation in a simulated maze, or control of a simple mechanical system with one or two degrees of freedom. To study AC in a more realistic setting, we embody a curious agent in the complex iCub humanoid robot. Our novel reinforcement learning (RL) framework consists of a state-of-the-art, low-level, reactive control layer, which controls the iCub while respecting constraints, and a high-level curious agent, which explores the iCub's state-action space through information gain maximization, learning a world model from experience, controlling the actual iCub hardware in real-time. To the best of our knowledge, this is the first ever embodied, curious agent for real-time motion planning on a humanoid. We demonstrate that it can learn compact Markov models to represent large regions of the iCub's configuration space, and that the iCub explores intelligently, showing interest in its physical constraints as well as in objects it finds in its environment.

An evaluation of differential reinforcement of other behavior, differential reinforcement of incompatible behavior, and restitution for the management of aggressive behaviors

A 46 year old institutionalized retarded woman was referred for treatment because of frequent verbal abuse and physically aggressive behaviors. A multicomponent behavioral intervention consisting of differential reinforcement of other behaviors, differential reinforcement of incompatible behaviors, and restitution was implemented. A sequential withdrawal design was used in order to evaluate the effects of components of the original intervention, and to provide a measure of response maintenance. Results indicated a marked decrease in verbal abuse and elimination of physical aggression. Verbal abuse recurred when the restitution procedure was withdrawn. The relative efficacy of the different interventions is discussed with respect to the behavioral management of aggression.

Out-of-plane deformation and failure of masonry walls with various forms of reinforcement

Out-of-plane behaviour of mortared and mortarless masonry walls with various forms of reinforcement, including unreinforced masonry as a base case is examined using a layered shell element based explicit finite element modelling method. Wall systems containing internal reinforcement, external surface reinforcement and intermittently laced reinforced concrete members and unreinforced masonry panels are considered. Masonry is modelled as a layer with macroscopic orthotropic properties; external reinforcing render, grout and reinforcing bars are modelled as distinct layers of the shell element. Predictions from the layered shell model have been validated using several out-of-plane experimental datasets reported in the literature. The model is used to examine the effectiveness of two retrofitting schemes for an unreinforced masonry wall.

Towards vision-based deep reinforcement learning for robotic motion control

This paper introduces a machine learning based system for controlling a robotic manipulator with visual perception only. The capability to autonomously learn robot controllers solely from raw-pixel images and without any prior knowledge of configuration is shown for the first time. We build upon the success of recent deep reinforcement learning and develop a system for learning target reaching with a three-joint robot manipulator using external visual observation. A Deep Q Network (DQN) was demonstrated to perform target reaching after training in simulation. Transferring the network to real hardware and real observation in a naive approach failed, but experiments show that the network works when replacing camera images with synthetic images.

Seismic behaviour of rigid-faced reinforced soil retaining wall models:reinforcement effect

This paper presents the results of shaking table tests on models of rigid-faced reinforced soil retaining walls in which reinforcement materials of different tensile strength were used. The construction of the model retaining walls in a laminar box mounted on a shaking table, the instrumentation and the results from the shaking table tests are described in detail and the effects of the reinforcement parameters on the acceleration response at different elevations of the retaining wall, horizontal soil pressures and face deformations are presented. It was observed from these tests that the horizontal face displacement response of the rigid-faced retaining walls was significantly affected by the inclusion of reinforcement and even low-strength polymer reinforcement was found to be efficient in significantly reducing the deformation of the face. The acceleration amplifications were, however, observed to be less influenced by the reinforcement parameters. The results obtained from this study are helpful in understanding the relative performance of reinforced soil retaining walls under the different test conditions used in the experiments.

Applications of coralline hydroxyapatite with bioabsorbable containment and reinforcement as bone graft substitute : An experimental study

The aim of the present experimental study was to find out if the applications of coralline hydroxyapatite (HA) can be improved by using bioabsorbable containment or binding substance with particulate HA in mandibular contour augmentation and by using bioabsorbable fibre-reinforced HA blocks in filling bone defects and in anterior lumbar interbody fusion. The use of a separate curved polyglycolide (PGA) containment alone or together with a fast resorbing polyglycolide/polylactide (PGA/PLA) binding substance were compared to the conventional non-contained method in ridge augmentation in sheep. The contained methods decreased HA migration, but the augmentations did not differ significantly. The use of the containment caused a risk for wound dehiscence and infection. Histologically there was a rapid connective tissue ingrowth into the HA graft and it was more abundant with the PGA containment compared to the non-contained augmentation and even additionally rich when the HA particles were bound with PGA/PLA copolymer. However, the bone ingrowth was best in the non-contained augmentation exceeding 10-12 % of the total graft area at 24 weeks. Negligible or no bone ingrowth was seen in the cases where the polymer composite was added to the HA particles and, related to that, foreign-body type cells were seen at the interface between the HA and host bone. The PGA and poly-dl/l-lactide (PDLLA) fibre-reinforced coralline HA blocks were studied in the metaphyseal and in the diaphyseal defects in rabbits. A rapid bone ingrowth was seen inside the both types of implants. Both PGA and PDLLA fibres induced an inflammatory fibrous reaction around themselves but it did not hinder the bone ingrowth. The bone ingrowth pattern was directed according to the loading conditions so that the load-carrying cortical ends of the implants as well as the implants sited in the diaphyseal defects were the most ossified. The fibre-reinforced coralline HA implants were further studied as stand-alone grafts in the lumbar anterior interbody implantation in pigs. The strength of the HA implants proved not to be adequate, the implants fractured in six weeks and the disc space was gradually lost similarly to that of the discectomized spaces. Histologically, small quantities of bone ingrowth was seen in some of the PGA and PDLLA reinforced coralline implants while no bone formation was identified in any of the PDLLA reinforced synthetic porous HA implants. While fragmented, the inner structure of the implants was lost, the bone ingrowth was minimal, and the disc was replaced by the fibrous connective tissue. When evaluated radiologically the grade of ossification was assessed as better than histologically, and, when related to the histologic findings, CT was more dependable than the plain films to show ossification of the implanted disc space. Local kyphosis was a frequent finding along with anterior bone bridging and ligament ossification as a consequence of instability of the implanted segment.

Effect of reinforcement form on the bearing capacity of square footings on sand

This paper presents the results of laboratory model loading tests and numerical studies carried out on square footings supported on geosynthetic reinforced sand beds. The relative performance of different forms of geosynthetic reinforcement (i.e. geocell, planar layers and randomly distributed mesh elements) in foundation beds is compared; using same quantity of reinforcement in each test. A biaxial geogrid and a geonet are used for reinforcing the sand beds. Geonet is used in two forms of reinforcement, viz. Planar layers and geocell, while the biaxial geogrid was used in three forms of reinforcement, viz. planar layers, geocell and randomly distributed mesh elements. Laboratory load tests on unreinforced and reinforced footings are simulated in a numerical model and the results are analyzed to understand the distribution of displacements and stresses below the footing better. Both the experimental and numerical studies demonstrated that the geocell is the most advantageous form of soil reinforcement technique of those investigated, provided there is no rupture of the material during loading. Geogrid used in the form of randomly distributed mesh elements is found to be inferior to the other two forms. Some significant observations on the difference in reinforcement mechanism for different forms of reinforcement are presented in this paper.

Optimal non-linear reinforcement schemes for stochastic automata

Two optimal non-linear reinforcement schemes—the Reward-Inaction and the Penalty-Inaction—for the two-state automaton functioning in a stationary random environment are considered. Very simple conditions of symmetry of the non-linear function figuring in the reinforcement scheme are shown to be necessary and sufficient for optimality. General expressions for the variance and rate of learning are derived. These schemes are compared with the already existing optimal linear schemes in the light of average variance and average rate of learning.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Effect of densification and TiB2 reinforcement on the wear of molybdenum disilicide

Wear tests were done in a pin-on-disc machine by sliding MoSi2 pins against hard-steel discs in a normal load range of 5-140 N and a speed of 0.5 m/s under nominally dry conditions in the ambient. The specific wear rate of the pin undergoes two transitions: severe to mild at low load and mild to severe at high load. The mild-wear domain is distinguished by the formation of a protective mechanically mixed layer of steel and its oxides, transferred from the counterface in particulate form. Increasing the hardness by densification and TiB2 reinforcement lowers the specific wear rate and expands the mild-wear load domain. However, even when the volume wear rate is normalised with respect to the real contact area (load/hardness) the non-dimensional wear factor is still seen to decrease with densification and reinforcement. This indicates that fracture toughness may also play an important role in determining the wear-resistance of these materials. The surface coverage on the pin by the mechanically mixed layer increases with densification and reinforcement.

Processing and fiber content effects on the machinability of compression moulded random direction short GFRP composites

The random direction short Glass Fiber Reinforced Plastics (GFRP) have been prepared by two compression moulding processes, namely the Preform and Sheet Moulding Compound (SMC) processes. Cutting force analysis and surface characterization are conducted on the random direction short GFRPs with varying fiber contents (25 similar to 40%). Edge trimming experiments are preformed using carbide inserts with varing the depth of cut and cutting speed. Machining characteristics of the Preform and SMC processed random direction short GFRPs are evaluated in terms of cutting forces, surface quality, and tool wear. It is found that composite primary processing and fiber contents are major contributing factors influencing the cutting force magnitudes and surface textures. The SMC composites show better surface finish over the Preform composites due to less delamination and fiber pullouts. Moreover, matrix damage and fiber protrusions at the machined edge are reduced by increasing fiber content in the random direction short GFRP composites.

A tutorial survey of reinforcement learning

This paper gives a compact, self-contained tutorial survey of reinforcement learning, a tool that is increasingly finding application in the development of intelligent dynamic systems. Research on reinforcement learning during the past decade has led to the development of a variety of useful algorithms. This paper surveys the literature and presents the algorithms in a cohesive framework.

Reinforcement Learning With Function Approximation for Traffic Signal Control

We propose, for the first time, a reinforcement learning (RL) algorithm with function approximation for traffic signal control. Our algorithm incorporates state-action features and is easily implementable in high-dimensional settings. Prior work, e. g., the work of Abdulhai et al., on the application of RL to traffic signal control requires full-state representations and cannot be implemented, even in moderate-sized road networks, because the computational complexity exponentially grows in the numbers of lanes and junctions. We tackle this problem of the curse of dimensionality by effectively using feature-based state representations that use a broad characterization of the level of congestion as low, medium, or high. One advantage of our algorithm is that, unlike prior work based on RL, it does not require precise information on queue lengths and elapsed times at each lane but instead works with the aforementioned described features. The number of features that our algorithm requires is linear to the number of signaled lanes, thereby leading to several orders of magnitude reduction in the computational complexity. We perform implementations of our algorithm on various settings and show performance comparisons with other algorithms in the literature, including the works of Abdulhai et al. and Cools et al., as well as the fixed-timing and the longest queue algorithms. For comparison, we also develop an RL algorithm that uses full-state representation and incorporates prioritization of traffic, unlike the work of Abdulhai et al. We observe that our algorithm outperforms all the other algorithms on all the road network settings that we consider.

A finite element assessment of flexural strength of prestressed concrete beams with fiber reinforcement

This paper presents an assessment of the flexural behavior of 15 fully/partially prestressed high strength concrete beams containing steel fibers investigated using three-dimensional nonlinear finite elemental analysis. The experimental results consisted of eight fully and seven partially prestressed beams, which were designed to be flexure dominant in the absence of fibers. The main parameters varied in the tests were: the levels of prestressing force (i.e, in partially prestressed beams 50% of the prestress was reduced with the introduction of two high strength deformed bars instead), fiber volume fractions (0%, 0.5%, 1.0% and 1.5%), fiber location (full depth and partial depth over full length and half the depth over the shear span only). A three-dimensional nonlinear finite element analysis was conducted using ANSYS 5.5 [Theory Reference Manual. In: Kohnke P, editor. Elements Reference Manual. 8th ed. September 1998] general purpose finite element software to study the flexural behavior of both fully and partially prestressed fiber reinforced concrete beams. Influence of fibers on the concrete failure surface and stress-strain response of high strength concrete and the nonlinear stress-strain curves of prestressing wire and deformed bar were considered in the present analysis. In the finite element model. tension stiffening and bond slip between concrete and reinforcement (fibers., prestressing wire, and conventional reinforcing steel bar) have also been considered explicitly. The fraction of the entire volume of the fiber present along the longitudinal axis of the prestressed beams alone has been modeled explicitly as it is expected that these fibers would contribute to the mobilization of forces required to sustain the applied loads across the crack interfaces through their bridging action. A comparison of results from both tests and analysis on all 15 specimens confirm that, inclusion of fibers over a partial depth in the tensile side of the prestressed flexural structural members was economical and led to considerable cost saving without sacrificing on the desired performance. However. beams having fibers over half the depth in only the shear span, did not show any increase in the ultimate load or deformational characteristics when compared to plain concrete beams. (C) 2002 Published by Elsevier Science Ltd.

A reinforcement learning approach to automatic generation control

This paper formulates the automatic generation control (AGC) problem as a stochastic multistage decision problem. A strategy for solving this new AGC problem formulation is presented by using a reinforcement learning (RL) approach This method of obtaining an AGC controller does not depend on any knowledge of the system model and more importantly it admits considerable flexibility in defining the control objective. Two specific RL based AGC algorithms are presented. The first algorithm uses the traditional control objective of limiting area control error (ACE) excursions, where as, in the second algorithm, the controller can restore the load-generation balance by only monitoring deviation in tie line flows and system frequency and it does not need to know or estimate the composite ACE signal as is done by all current approaches. The effectiveness and versatility of the approaches has been demonstrated using a two area AGC model. (C) 2002 Elsevier Science B.V. All rights reserved.