Incremental natural-gradient actor-critic algorithms

We present four new reinforcement learning algorithms based on actor-critic and natural-gradient ideas, and provide their convergence proofs. Actor-critic rein- forcement learning methods are online approximations to policy iteration in which the value-function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent. Methods based on policy gradients in this way are of special interest because of their com- patibility with function approximation methods, which are needed to handle large or infinite state spaces. The use of temporal difference learning in this way is of interest because in many applications it dramatically reduces the variance of the gradient estimates. The use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further re- duce variance in some cases. Our results extend prior two-timescale convergence results for actor-critic methods by Konda and Tsitsiklis by using temporal differ- ence learning in the actor and by incorporating natural gradients, and they extend prior empirical studies of natural actor-critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms.

Simulation of RNA silencing pathway for time-dependent transgene transcription rate

The synthesis of dsRNA is analyzed using a pathway model with amplifications caused by the aberrant RNAs. The transgene influx rate is assumed time-decaying considering the fact that the number of transgenes can not be infinite. The dynamics of the transgene induced RNA silencing is investigated using a system of coupled nonautonomous ordinary nonlinear differential equations which describe the model phenomenologically. The silencing phenomena are detected after a period of transcription. Important contributions of certain parameters are discussed with several numerical examples.

Effect of irradiation on the microstructure and mechanical behavior of nanocrystalline nickel

The effect of 4.0 MeV proton irradiation on the microstructure and mechanical properties of nanocrystalline (nc) nickel was investigated. The irradiation damage induced in the sample was of the order of 0.004 dpa. Transmission electron microscopy of irradiated samples indicated the presence of dislocation loops within the grains. An increase in hardness and strain-rate sensitivity (m) of nc-Ni with irradiation was noted. The rate-controlling deformation mechanism in irradiated nc-Ni was identified to be interaction of dislocations with irradiation-induced defects. (C) 2011 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.

Active Vibration Suppression of One-dimensional Nonlinear Structures Using Optimal Dynamic Inversion

A flexible robot arm can be modeled as an Euler-Bernoulli beam which are infinite degrees of freedom (DOF) system. Proper control is needed to track the desired motion of a robotic arm. The infinite number of DOF of beams are reduced to finite number for controller implementation, which brings in error (due to their distributed nature). Therefore, to represent reality better distributed parameter systems (DPS) should be controlled using the systems partial differential equation (PDE) directly. In this paper, we propose to use a recently developed optimal dynamic inversion technique to design a controller to suppress nonlinear vibration of a beam. The method used in this paper determines control forces directly from the PDE model of the system. The formulation has better practical significance, because it leads to a closed form solution of the controller (hence avoids computational issues).

Pricing a Shared Access Link for Fair and Efficient Operation with Variable User Data Rates

The literature on pricing implicitly assumes an "infinite data" model, in which sources can sustain any data rate indefinitely. We assume a more realistic "finite data" model, in which sources occasionally run out of data; this leads to variable user data rates. Further, we assume that users have contracts with the service provider, specifying the rates at which they can inject traffic into the network. Our objective is to study how prices can be set such that a single link can be shared efficiently and fairly among users in a dynamically changing scenario where a subset of users occasionally has little data to send. User preferences are modelled by concave increasing utility functions. Further, we introduce two additional elements: a convex increasing disutility function and a convex increasing multiplicative congestion-penally function. The disutility function takes the shortfall (contracted rate minus present rate) as its argument, and essentially encourages users to send traffic at their contracted rates, while the congestion-penalty function discourages heavy users from sending excess data when the link is congested. We obtain simple necessary and sufficient conditions on prices for fair and efficient link sharing; moreover, we show that a single price for all users achieves this. We illustrate the ideas using a simple experiment.

Resource Pricing for Fair and Efficient Link Sharing with a Finite Data Model

The literature on pricing implicitly assumes an "infinite data" model, in which sources can sustain any data rate indefinitely. We assume a more realistic "finite data" model, in which sources occasionally run out of data. Further, we assume that users have contracts with the service provider, specifying the rates at which they can inject traffic into the network. Our objective is to study how prices can be set such that a single link can be shared efficiently and fairly among users in a dynamically changing scenario where a subset of users occasionally has little data to send. We obtain simple necessary and sufficient conditions on prices such that efficient and fair link sharing is possible. We illustrate the ideas using a simple example

A Simultaneous Deterministic Perturbation Actor-Critic Algorithm with an Application to Optimal Mortgage Refinancing

We develop a simulation-based, two-timescale actor-critic algorithm for infinite horizon Markov decision processes with finite state and action spaces, with a discounted reward criterion. The algorithm is of the gradient ascent type and performs a search in the space of stationary randomized policies. The algorithm uses certain simultaneous deterministic perturbation stochastic approximation (SDPSA) gradient estimates for enhanced performance. We show an application of our algorithm on a problem of mortgage refinancing. Our algorithm obtains the optimal refinancing strategies in a computationally efficient manner

Thermal Weight Functions and Stress Intensity Factors for Bonded Dissimilar Media Using Body Analogy

In this study, an analytical method is presented for the computation of thermal weight functions in two dimensional bi-material elastic bodies containing a crack at the interface and subjected to thermal loads using body analogy method. The thermal weight functions are derived for two problems of infinite bonded dissimilar media, one with a semi-infinite crack and the other with a finite crack along the interface. The derived thermal weight functions are shown to reduce to the already known expressions of thermal weight functions available in the literature for the respective homogeneous elastic body. Using these thermal weight functions, the stress intensity factors are computed for the above interface crack problems when subjected to an instantaneous heat source.

Solute solute and solvent solute interactions in solid solutions of Cu+Sn, Au+Sn and Cu+Au+Sn alloys

The chemical potentials of tin in its α-solid solutions with Cu, Au and Cu + Au alloys have been measured using a gas-solid equilibration technique. The variation of the excess chemical potential of tin with its composition in the alloy is related to the solute-solute repulsive interaction, while the excess chemical potential at infinite dilution of the solute is a measure of solvent-solute interaction energies. It is shown that solute-solute interaction is primarily determined by the concentration of (s + p) electrons in the conduction band, although the interaction energies are smaller than those predicted by either the rigid band model or calculation based on Friedel oscillations in the potential function. Finally, the variation of the solvent-solute interaction with solvent composition in the ternary system can be accounted for in terms of a quasi-chemical treatment which takes into account the clustering of the solvent atoms around the solute.

Investigation of dielectric, piezoelectric and ferroelectric properties of b-axis grown triglycine sulphate single crystal

Large single crystal of triglycine sulphate (dimension 100 mm along monoclinic b-axis and 15 mm in diameter) was grown using the unidirectional solution growth technique. The X-ray diffraction studies confirmed the growth/long axis to be b-axis (polar axis). The dielectric studies were carried out at various temperatures to establish the phase transition temperature. The frequency response of the dielectric constant, dielectric loss and impedance of the crystal along the growth axis, was monitored. These are typically characterized by strong resonance peaks in the kHz region. The piezoelectric coefficients like stiffness constant (C), elastic coefficient (S), electromechanical coupling coefficient (k) and d (31) were calculated using the resonance-antiresonance method. Polarization (P)-Electric field (E) hysteresis loops were recorded at various temperatures to find the temperature-dependent spontaneous polarization of the grown crystal. The pyroelectric coefficients were determined from the pyroelectric current measurement by the Byer and Roundy method. The ferroelectric domain patterns were recorded on (010) plane using scanning electron microscopy and optical microscopy.

Analysis of a linear one-dimensional active noise control system by means of block diagrams and transfer functions

In arriving at the ideal filter transfer function for an active noise control system in a duct, the effect of the auxiliary sources (generally loudspeakers) on the waves generated by the primary source has invariably been neglected in the existing literature, implying a rigid wall or infinite impedance. The present paper presents a fairly general analysis of a linear one-dimensional noise control system by means of block diagrams and transfer functions. It takes into account the passive as well as active role of a terminal primary source, wall-mounted auxiliary source, open duct radiation impedance, and the effects of mean flow and damping. It is proved that the pressure generated by a source against a load impedance can be looked upon as a sum of two pressure waves, one generated by the source against an anechoic termination and the other by reflecting the rearward wave (incident on the source) off the passive source impedance. Application of this concept is illustrated for both the types of sources. A concise closed-form expression for the ideal filter transfer function is thus derived and discussed. Finally, the dynamics of an adaptive noise control system is discussed briefly, relating its standing-wave variables and transfer functions with those of the progressive-wave model presented here.

Extending the Supramolecular Synthon Based Fragment Approach (SBFA) for Transferability of Multipole Charge Density Parameters to Monofluorobenzoic Acids and their Cocrystals with Isonicotinamide: Importance of C-H center dot center dot center dot O, C-H center dot center dot center dot F, and F center dot center dot center dot F Intermolecular Regions

An extension of the supramolecular synthon-based fragment approach (SBFA) method for transferability of multipole charge density parameters to include weak supramolecular synthons is proposed. In particular, the SBFA method is applied to C-H center dot center dot center dot O, C-H center dot center dot center dot F, and F center dot center dot center dot F containing synthons. A high resolution charge density study has been performed on 4-fluorobenzoic acid to build a synthon library for C-H center dot center dot center dot F infinite chain interactions. Libraries for C-H center dot center dot center dot O and F center dot center dot center dot F synthons were taken from earlier work. The SBFA methodology was applied successfully to 2- and 3-fluorobenzoic acids, data sets for which were collected in a routine manner at 100 K, and the modularity of the synthons was demonstrated. Cocrystals of isonicotinamide with all three fluorobenzoic acids were also studied with the SBFA method. The topological analysis of inter- and intramolecular interaction regions was performed using Bader's AIM approach. This study shows that the SBFA method is generally applicable to generate charge density maps using information from multiple intermolecular regions.

Improving Superscalar Instruction Dispatch And Issue By Exploiting Dynamic Code Sequences

Superscalar processors currently have the potential to fetch multiple basic blocks per cycle by employing one of several recently proposed instruction fetch mechanisms. However, this increased fetch bandwidth cannot be exploited unless pipeline stages further downstream correspondingly improve. In particular,register renaming a large number of instructions per cycle is diDcult. A large instruction window, needed to receive multiple basic blocks per cycle, will slow down dependence resolution and instruction issue. This paper addresses these and related issues by proposing (i) partitioning of the instruction window into multiple blocks, each holding a dynamic code sequence; (ii) logical partitioning of the registerjle into a global file and several local jles, the latter holding registers local to a dynamic code sequence; (iii) the dynamic recording and reuse of register renaming information for registers local to a dynamic code sequence. Performance studies show these mechanisms improve performance over traditional superscalar processors by factors ranging from 1.5 to a little over 3 for the SPEC Integer programs. Next, it is observed that several of the loops in the benchmarks display vector-like behavior during execution, even if the static loop bodies are likely complex for compile-time vectorization. A dynamic loop vectorization mechanism that builds on top of the above mechanisms is briefly outlined. The mechanism vectorizes up to 60% of the dynamic instructions for some programs, albeit the average number of iterations per loop is quite small.

Growth and characterization of excimer laser-ablated BaBi2Nb2O9 thin films

The pulsed-laser ablation technique has been employed to deposit polycrystalline thin films of layered-structure ferroelectric BaBi2Nb2O9 (BBN). Low-substrate-temperature growth (Ts = 400 °C) followed by ex situ annealing at 800 °C for 30 min was performed to obtain a preferred orientation. Ferroelectricity in the films was verified by examining the polarization with the applied electric field and was also confirmed from the capacitance–voltage characteristics. The films exhibited well-defined hysteresis loops, and the values of saturation (Ps) and remanent (Pr) polarization were 4.0 and 1.2 μC/cm2, respectively. The room-temperature dielectric constant and dissipation factor were 214 and 0.04, respectively, at a frequency of 100 kHz. A phase transition from a ferroelectric to paraelectric state of the BBN thin film was observed at 220 °C. The dissipation factor of the film was observed to increase after the phase transition due to a probable influence of dc conduction at high temperatures. The real and imaginary part of the dielectric constant also exhibited strong frequency dispersion at high temperatures.

Additions to the z transform table

The ztransform method is a widely used tool for the analysis and synthesis of discrete systems. In this note a table of ztransform pairs when F(z) is an irrational function of z is given. The table is also useful for obtaining closed-form sums for some infinite series.