Hybrid coordination of reinforcement learning-based behaviors for AUV control

This paper proposes a hybrid coordination method for behavior-based control architectures. The hybrid method takes advantages of the robustness and modularity in competitive approaches as well as optimized trajectories in cooperative ones. This paper shows the feasibility of applying this hybrid method with a 3D-navigation to an autonomous underwater vehicle (AUV). The behaviors are learnt online by means of reinforcement learning. A continuous Q-learning implemented with a feed-forward neural network is employed. Realistic simulations were carried out. The results obtained show the good performance of the hybrid method on behavior coordination as well as the convergence of the behaviors

A behavior-based scheme using reinforcement learning for autonomous underwater vehicles

This paper presents a hybrid behavior-based scheme using reinforcement learning for high-level control of autonomous underwater vehicles (AUVs). Two main features of the presented approach are hybrid behavior coordination and semi on-line neural-Q_learning (SONQL). Hybrid behavior coordination takes advantages of robustness and modularity in the competitive approach as well as efficient trajectories in the cooperative approach. SONQL, a new continuous approach of the Q_learning algorithm with a multilayer neural network is used to learn behavior state/action mapping online. Experimental results show the feasibility of the presented approach for AUVs

Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

Autonomous underwater vehicle control using reinforcement learning policy search methods

Autonomous underwater vehicles (AUV) represent a challenging control problem with complex, noisy, dynamics. Nowadays, not only the continuous scientific advances in underwater robotics but the increasing number of subsea missions and its complexity ask for an automatization of submarine processes. This paper proposes a high-level control system for solving the action selection problem of an autonomous robot. The system is characterized by the use of reinforcement learning direct policy search methods (RLDPS) for learning the internal state/action mapping of some behaviors. We demonstrate its feasibility with simulated experiments using the model of our underwater robot URIS in a target following task

Towards Direct Policy Search Reinforcement Learning for Robot Control

This paper proposes a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot. Although the dominant approach, when using RL, has been to apply value function based algorithms, the system here detailed is characterized by the use of direct policy search methods. Rather than approximating a value function, these methodologies approximate a policy using an independent function approximator with its own parameters, trying to maximize the future expected reward. The policy based algorithm presented in this paper is used for learning the internal state/action mapping of a behavior. In this preliminary work, we demonstrate its feasibility with simulated experiments using the underwater robot GARBI in a target reaching task

Compression-based Image Registration

Image registration is an important component of image analysis used to align two or more images. In this paper, we present a new framework for image registration based on compression. The basic idea underlying our approach is the conjecture that two images are correctly registered when we can maximally compress one image given the information in the other. The contribution of this paper is twofold. First, we show that the image registration process can be dealt with from the perspective of a compression problem. Second, we demonstrate that the similarity metric, introduced by Li et al., performs well in image registration. Two different versions of the similarity metric have been used: the Kolmogorov version, computed using standard real-world compressors, and the Shannon version, calculated from an estimation of the entropy rate of the images

Statistical similarity between the compression of a porous material and earthquakes

It has been long stated that there are profound analogies between fracture experiments and earthquakes; however, few works attempt a complete characterization of the parallelisms between these so separate phenomena. We study the Acoustic Emission events produced during the compression of Vycor (SiO&sub&2&/sub&). The Gutenberg-Richter law, the modified Omori's law, and the law of aftershock productivity hold for a minimum of 5 decades, are independent of the compression rate, and keep stationary for all the duration of the experiments. The waiting-time distribution fulfills a unified scaling law with a power-law exponent close to 2.45 for long times, which is explained in terms of the temporal variations of the activity rate.

Management of corn stalk waste as reinforcement for polypropylene injection moulded composites

The main objective of this study was the management of corn stalk waste as reinforcement for polypropylene (PP) injection moulded composites as an alternative to wood flour and fibers. In the first step, corn stalk waste was subjected to various treatments, and four different corn stalk derivatives (flour and fibers) able to be used as reinforcement of composite materials were prepared and characterized. These derivatives are corn stalk flour, thermo-mechanical, semi-chemical, and chemical fibers. They were characterized in terms of their yield, lignin content, Kappa number, fiber length/diameter ratio, fines, coarseness, viscosity, and the length at the break of a standard sheet of paper. Results showed that the corn stalk derivatives have different physico-chemical properties. In the second step, the prepared flour and fibers were explored as a reinforcing element for PP composites. Coupled and non-coupled PP composites were prepared and tested for tensile properties. For overall trend, with the addition of a coupling agent, tensile properties of composites significantly improved, as compared with non-coupled samples. In addition, a morphological study revealed the positive effect of the coupling agent on the interfacial bonding. The composites prepared with semichemical fiber gave better results in comparison with the rest of the corn stalk derivatives due to its chemical characteristics

CB1 cannabinoid receptor modulates MDMA acute responses and reinforcement

Background: 3, 4-methylenedioxymethamphetamine (MDMA) is a popular recreational drug widely abused by young people. The endocannabinoid system is involved in the addictive processes induced by different drugs of abuse. However, the role of this system in the pharmacological effects of MDMA has not been yet clarified.Methods: Locomotion, body temperature and anxiogenic-like responses were evaluated after acute MDMA administration in CB1 knockout mice. Additionally, MDMA rewarding properties were investigated in the place conditioning and the intravenous self-administration paradigms. Extracellular levels of DA in the nucleus accumbens were also analyzed after a single administration of MDMA by in vivo microdialysis. Results: Acute MDMA administration increased locomotor activity, body temperature and anxiogenic-like responses in wild type mice, but these responses were lower or abolished in knockout animals. MDMA produced similar conditioned place preference and increased dopamine extracellular levels in the nucleus accumbens in both genotypes. Nevertheless, CB1 knockout mice failed to self-administer MDMA at any of the doses used. Conclusions: These results indicate that CB1 cannabinoid receptors play an important role in the acute prototypical effects of MDMA, and are essential in the acquisition of an operant behavior to self-administer this drug.

The use of the SeDeM diagram expert system for the formulation of Captopril SR matrix tablets by direct compression

The SeDeM Diagram Expert System has been used to study excipients, Captopril and designed formulations for their galenic characterization and to ascertain the critical points of the formula affecting product quality to obtain suitable formulations of Captopril Direct Compression SR Matrix Tablets. The application of the Sedem Diagram Expert System enables selecting excipients with in order to optimize the formula in the preformulation and formulation studies. The methodology is based on the implementation of ICH Q8, establishing the design space of the formula with the use of experiment design, using the parameters of the SeDeM Diagram Expert System as system responses.

Reinforcement learning utilizes proxemics: an avatar learns to manipulate the position of people in immersive virtual reality

A reinforcement learning (RL) method was used to train a virtual character to move participants to a specified location. The virtual environment depicted an alleyway displayed through a wide field-of-view head-tracked stereo head-mounted display. Based on proxemics theory, we predicted that when the character approached within a personal or intimate distance to the participants, they would be inclined to move backwards out of the way. We carried out a between-groups experiment with 30 female participants, with 10 assigned arbitrarily to each of the following three groups: In the Intimate condition the character could approach within 0.38m and in the Social condition no nearer than 1.2m. In the Random condition the actions of the virtual character were chosen randomly from among the same set as in the RL method, and the virtual character could approach within 0.38m. The experiment continued in each case until the participant either reached the target or 7 minutes had elapsed. The distributions of the times taken to reach the target showed significant differences between the three groups, with 9 out of 10 in the Intimate condition reaching the target significantly faster than the 6 out of 10 who reached the target in the Social condition. Only 1 out of 10 in the Random condition reached the target. The experiment is an example of applied presence theory: we rely on the many findings that people tend to respond realistically in immersive virtual environments, and use this to get people to achieve a task of which they had been unaware. This method opens up the door for many such applications where the virtual environment adapts to the responses of the human participants with the aim of achieving particular goals.

A robust audio watermarking scheme based on MPEG 1 Layer 3 Compression

This paper describes an audio watermarking scheme based on lossy compression. The main idea is taken from an image watermarking approach where the JPEG compression algorithm is used to determine where and how the mark should be placed. Similarly, in the audio scheme suggested in this paper, an MPEG 1 Layer 3 algorithm is chosen for compression to determine the position of the mark bits and, thus, the psychoacoustic masking of the MPEG 1 Layer 3compression is implicitly used. This methodology provides with a high robustness degree against compression attacks. The suggested scheme is also shown to succeed against most of the StirMark benchmark attacks for audio.

Reinforcement learning utilizes proxemics: an avatar learns to manipulate the position of people in immersive virtual reality

A reinforcement learning (RL) method was used to train a virtual character to move participants to a specified location. The virtual environment depicted an alleyway displayed through a wide field-of-view head-tracked stereo head-mounted display. Based on proxemics theory, we predicted that when the character approached within a personal or intimate distance to the participants, they would be inclined to move backwards out of the way. We carried out a between-groups experiment with 30 female participants, with 10 assigned arbitrarily to each of the following three groups: In the Intimate condition the character could approach within 0.38m and in the Social condition no nearer than 1.2m. In the Random condition the actions of the virtual character were chosen randomly from among the same set as in the RL method, and the virtual character could approach within 0.38m. The experiment continued in each case until the participant either reached the target or 7 minutes had elapsed. The distributions of the times taken to reach the target showed significant differences between the three groups, with 9 out of 10 in the Intimate condition reaching the target significantly faster than the 6 out of 10 who reached the target in the Social condition. Only 1 out of 10 in the Random condition reached the target. The experiment is an example of applied presence theory: we rely on the many findings that people tend to respond realistically in immersive virtual environments, and use this to get people to achieve a task of which they had been unaware. This method opens up the door for many such applications where the virtual environment adapts to the responses of the human participants with the aim of achieving particular goals.

Integrin-Specific Mechanoresponses to Compression and Extension Probed by Cylindrical Flat-Ended AFM Tips in Lung Cells

Cells from lung and other tissues are subjected to forces of opposing directions that are largely transmitted through integrin-mediated adhesions. How cells respond to force bidirectionality remains ill defined. To address this question, we nanofabricated flat-ended cylindrical Atomic Force Microscopy (AFM) tips with ~1 µm2 cross-section area. Tips were uncoated or coated with either integrin-specific (RGD) or non-specific (RGE/BSA) molecules, brought into contact with lung epithelial cells or fibroblasts for 30 s to form focal adhesion precursors, and used to probe cell resistance to deformation in compression and extension. We found that cell resistance to compression was globally higher than to extension regardless of the tip coating. In contrast, both tip-cell adhesion strength and resistance to compression and extension were the highest when probed at integrin-specific adhesions. These integrin-specific mechanoresponses required an intact actin cytoskeleton, and were dependent on tyrosine phosphatases and Ca2+ signaling. Cell asymmetric mechanoresponse to compression and extension remained after 5 minutes of tip-cell adhesion, revealing that asymmetric resistance to force directionality is an intrinsic property of lung cells, as in most soft tissues. Our findings provide new insights on how lung cells probe the mechanochemical properties of the microenvironment, an important process for migration, repair and tissue homeostasis.