An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Symmetry Properties of the Large-Deviation Function of the Velocity of a Self-Propelled Polar Particle

A geometrically polar granular rod confined in 2D geometry, subjected to a sinusoidal vertical oscillation, undergoes noisy self-propulsion in a direction determined by its polarity. When surrounded by a medium of crystalline spherical beads, it displays substantial negative fluctuations in its velocity. We find that the large-deviation function (LDF) for the normalized velocity is strongly non-Gaussian with a kink at zero velocity, and that the antisymmetric part of the LDF is linear, resembling the fluctuation relation known for entropy production, even when the velocity distribution is clearly non-Gaussian. We extract an analogue of the phase-space contraction rate and find that it compares well with an independent estimate based on the persistence of forward and reverse velocities.

Deblurring in a noncoherent optical processing system: pupil function synthesis and experimental implementation

The bipolar point spread function (PSF) corresponding to the Wiener filter tor correcting linear-motion-blurred pictures is implemented in a noncoherent optical processor. The following two approaches are taken for this implementation: (1) the PSF is modulated and biased so that the resulting function is non-negative and (2) the PSF is split into its positive and sign-reversed negative parts, and these two parts are dealt with separately. The phase problem associated with arriving at the pupil function from these modified PSFs is solved using both analytical and combined analytical-iterative techniques available in the literature. The designed pupil functions are experimentally implemented, and deblurring in a noncoherent processor is demonstrated. The postprocessing required (i.e., demodulation in the first approach to modulating the PSF and intensity subtraction in the second approach) are carried out either in a coherent processor or with the help of a PC-based vision system. The deblurred outputs are presented.

In vivo and in vitro studies on the differential role of luteinizing hormone and follicle-stimulating hormone in regulating follicular function in the bonnet monkey (Macaca radiata) using specific gonadotropin antibodies

Although a distinct need for FSH in the regulation of follicular maturation in the primate is well recognized, it is not clear how FSH controls the functionality of different cellular compartments of the follicle. It is also not evident whether there is a requirement for LH in follicular maturation in the primate. In the first part of the present study, female bonnet monkeys were administered a well-characterized ovine (o) LH antiserum to neutralize endogenous monkey LH for different periods during the follicular phase, and the effect on the overall follicular maturation process was assessed by analyzing serum estrogen (E) and progesterone (P) profiles. Neither continuous LH deprivation from Day 8 of the cycle nor deprivation of LH on any one day between Days 6 and 10 had a significant effect on serum E and P profiles and the follicular maturation process. The period for which the antiserum was effective was dependent upon the dose injected; 1 ml of the antiserum given on Day 8 blocked ovulation but not follicular maturation. To assess the effect of deprivation of LH/FSH at the cellular level, animals were deprived in vivo of LH (on Days 8 and 9 of the cycle) or of FSH (on Day 6 of the cycle) by injection of highly characterized hCG and ovine (o) FSH antisera, respectively; the in vitro responsiveness of granulosa and thecal cells isolated on Day 10 from these animals was then determined.(ABSTRACT TRUNCATED AT 250 WORDS)

Single-molecule thermodynamics: the heat distribution function of a charged particle in a static magnetic field

Interest in the applicability of fluctuation theorems to the thermodynamics of single molecules in external potentials has recently led to calculations of the work and total entropy distributions of Brownian oscillators in static and time-dependent electromagnetic fields. These calculations, which are based on solutions to a Smoluchowski equation, are not easily extended to a consideration of the other thermodynamic quantity of interest in such systems-the heat exchanges of the particle alone-because of the nonlinear dependence of the heat on a particle's stochastic trajectory. In this paper, we show that a path integral approach provides an exact expression for the distribution of the heat fluctuations of a charged Brownian oscillator in a static magnetic field. This approach is an extension of a similar path integral approach applied earlier by our group to the calculation of the heat distribution function of a trapped Brownian particle, which was found, in the limit of long times, to be consistent with experimental data on the thermal interactions of single micron-sized colloids in a viscous solvent.

Lossless and lossy image compression using Boolean function minimization

A novel approach for lossless as well as lossy compression of monochrome images using Boolean minimization is proposed. The image is split into bit planes. Each bit plane is divided into windows or blocks of variable size. Each block is transformed into a Boolean switching function in cubical form, treating the pixel values as output of the function. Compression is performed by minimizing these switching functions using ESPRESSO, a cube based two level function minimizer. The minimized cubes are encoded using a code set which satisfies the prefix property. Our technique of lossless compression involves linear prediction as a preprocessing step and has compression ratio comparable to that of JPEG lossless compression technique. Our lossy compression technique involves reducing the number of bit planes as a preprocessing step which incurs minimal loss in the information of the image. The bit planes that remain after preprocessing are compressed using our lossless compression technique based on Boolean minimization. Qualitatively one cannot visually distinguish between the original image and the lossy image and the value of mean square error is kept low. For mean square error value close to that of JPEG lossy compression technique, our method gives better compression ratio. The compression scheme is relatively slower while the decompression time is comparable to that of JPEG.

Critical earthquake input power spectral density function models for engineering structures

The problem of determining optimal power spectral density models for earthquake excitation which satisfy constraints on total average power, zero crossing rate and which produce the highest response variance in a given linear system is considered. The solution to this problem is obtained using linear programming methods. The resulting solutions are shown to display a highly deterministic structure and, therefore, fail to capture the stochastic nature of the input. A modification to the definition of critical excitation is proposed which takes into account the entropy rate as a measure of uncertainty in the earthquake loads. The resulting problem is solved using calculus of variations and also within linear programming framework. Illustrative examples on specifying seismic inputs for a nuclear power plant and a tall earth dam are considered and the resulting solutions are shown to be realistic.

Dielectric function and optical conductivity of TiOx (0.8

Reflection electron energy-loss spectra are reported for the family of compounds TiOx over the entire homogeneity range (0.8 < a: < 1.3). The spectra exhibit a plasmon feature on the low-energy side, while several interband transitions are prominent at higher energies. The real and imaginary parts of dielectric functions and optical conductivity for these compounds are determined using the Kramers-Kronig analysis. The results exhibit systematic behavior with varying oxygen stoichiometry.

Engineered dimer interface mutants of triosephosphate isomerase: the role of inter-subunit interactions in enzyme function and stability

The role of inter-subunit interactions in maintaining optimal catalytic activity in triosephosphate isomerase (TIM) has been probed, using the Plasmodium falciparum enzyme as a model. Examination of subunit interface contacts in the crystal structures suggests that residue 75 (Thr, conserved) and residue 13 (Cys, variable) make the largest number of inter-subunit contacts. The mutants Cys13Asp (C13D) and Cys13Glu (C13E) have been constructed and display significant reduction in catalytic activity when compared with wild-type (WT) enzyme (similar to 7.4-fold decrease in k(cat) for the C13D and similar to 3.3-fold for the C13E mutants). Analytical gel filtration demonstrates that the C13D mutant dissociates at concentrations < 1.25 mu M, whereas the WT and the C13E enzymes retain the dimeric structure. The order of stability of the mutants in the presence of chemical denaturants, like urea and guanidium chloride, is WT > Cys13Glu > Cys13Asp. Irreversible thermal precipitation temperatures follow the same order as well. Modeling studies establish that the Cys13Asp mutation is likely to cause a significantly greater structural perturbation than Cys13Glu. Analysis of sequence and structural data for TIMs from diverse sources suggests that residues 13 and 82 form a pair of proximal sites, in which a limited number of residue pairs may be accommodated.

Evidence for follicle-stimulating hormone mediation in the hemiorchidectomy-induced compensatory increase in the function of the remaining testis of the adult male bonnet monkey (Macaca radiata)

Hemiorchidectomy (HO) in the adult male bonnet monkey results in a selective increase in circulating concentrations of FSH and testosterone, and this is accompanied by compensatory increase in sperm production by the remaining testis. We investigated the possible role of increased FSH concentration that occurs after HO in the compensatory increase in the activity of the remaining testis. Of eight adult male bonnet monkeys that underwent HO, four received i.v. injections every other day for 30 days of a well-characterized ovine FSH antiserum (a/s) that cross-reacts with monkey FSH. The remaining four males received normal monkey serum (NMS) as control treatment in a protocol similar to that employed for ais-treated males. Blood samples were collected between 2100 and 2200 h before and 1/2, 1, 3, 5, 7, 14, 22, and 29 days after HO. Testicular weight, number of 3 beta-hydroxy steroid dehydrogenase-positive (3 beta-HSD+) cells, and DNA flow cytometric analysis of germ cell populations were obtained for testes collected before and at the termination of NMS or ais treatment. In NMS-treated males, circulating serum FSH concentrations progressively increased to reach a maximal level by Day 7 after HO (1.95 +/- 0.3 vs. 5.6 +/- 0.7 ng/ml on Days -1 and 7, respectively). Within 30 min of ais injection, FSH antibodies were detected in circulation, and the antibody level was maintained at a constant level between Day 7 and end of treatment (exhibiting 50-60% binding to I-125-hFSH). Although circulating mean nocturnal serum testosterone concentration showed an initial decrease, it rose gradually to pre-HO concentrations by Day 7 in NMS-treated males. In contrast, nocturnal mat serum testosterone concentrations in a/s-treated males remained lower than in NMS-treated controls (p < 0.05) up to Day 22 and thereafter only marginally increased. Testicular weights increased (p < 0.05) over the pre-HO weight in NMS- but not in ais-treated males. After HO, the number of 3 beta-HSD+ cells (Leydig cells) was markedly increased but was significantly (p < 0.05) higher in NMS-treated males compared to a/s-treated males. A significant (p < 0.05) reduction in the primary spermatocyte population of germ cells was observed in ais-treated compared to NMS-treated males. These results suggest that the increased FSH occurring after HO could be intimately involved in increasing the compensatory functional activity of the remaining testis in the male bonnet monkey.

Direct Evaluation Of Transient Stability Improvement With Static Var Compensators Using A Structure Preserving Energy Function

The ability of Static Var Compensators (SVCs) to rapidly and continuously control reactive power in response to changing system conditions can result in the improvement of system stability and also increase the power transfer in the transmission system. This paper concerns the application of strategically located SVCs to enhance the transient stability limits and the direct evaluation of the effect of these SVCs on transient stability using a Structure Preserving Energy Function (SPEF). The SVC control system can be modelled from the steady- state control characteristic to accurately simulate its effect on transient stability. Treating the SVC as a voltage-dependent reactive power load leads to the derivation of a path-independent SPEF for the SVC. Case studies on a 10-machine test system using multiple SVCs illustrate the effects of SVCs on transient stability and its accurate prediction.

Reinforcement Learning With Function Approximation for Traffic Signal Control

We propose, for the first time, a reinforcement learning (RL) algorithm with function approximation for traffic signal control. Our algorithm incorporates state-action features and is easily implementable in high-dimensional settings. Prior work, e. g., the work of Abdulhai et al., on the application of RL to traffic signal control requires full-state representations and cannot be implemented, even in moderate-sized road networks, because the computational complexity exponentially grows in the numbers of lanes and junctions. We tackle this problem of the curse of dimensionality by effectively using feature-based state representations that use a broad characterization of the level of congestion as low, medium, or high. One advantage of our algorithm is that, unlike prior work based on RL, it does not require precise information on queue lengths and elapsed times at each lane but instead works with the aforementioned described features. The number of features that our algorithm requires is linear to the number of signaled lanes, thereby leading to several orders of magnitude reduction in the computational complexity. We perform implementations of our algorithm on various settings and show performance comparisons with other algorithms in the literature, including the works of Abdulhai et al. and Cools et al., as well as the fixed-timing and the longest queue algorithms. For comparison, we also develop an RL algorithm that uses full-state representation and incorporates prioritization of traffic, unlike the work of Abdulhai et al. We observe that our algorithm outperforms all the other algorithms on all the road network settings that we consider.

The role of lysine-256 in the structure and function of sheep liver recombinant serine hydroxymethyltransferase

The active site lysine residue, K256, involved in Schiffs base linkage with pyridoxal-5'-phosphate (PEP) in sheep liver recombinant serine hydroxymethyltransferase (rSHMT) was changed to glutamine or arginine by site-directed mutagenesis. The purified K256Q and K256R SHMTs had less than 0.1% of catalytic activity with serine and H(4)folate as substrates compared to rSHMT. The mutant enzymes also failed to exhibit the characteristic visible absorbance spectrum (lambda(max) 425 nm) and did not produce the quinonoid intermediate (lambda(max) 495 nm) upon the addition of glycine and H(4)folate. The mutant enzymes were unable to catalyze aldol cleavage of beta-phenylserine and transamination of D-alanine. These results suggested that the mutation of the lysine had resulted in the inability of the enzyme to bind to the cofactor. Therefore, the K256Q SHMT was isolated as a dimer and the K256R SHMT as a mixture of dimers and tetramers which were converted to dimers slowly. On the other hand, rSHMT was stable as a tetramer for several months, further confirming the role of PLP in maintenance of oligomeric structure. The mutant enzymes also failed to exhibit the increased thermal stability upon the addition of serine, normally observed with rSHMT. The enhanced thermal stability has been attributed to a change in conformation of the enzyme from open to closed form leading to reaction specificity. The mutant enzymes were unable to undergo this conformational change probably because of the absence of bound cofactor.

Velocity distribution function for a dilute granular material in shear flow

The velocity distribution function for the steady shear flow of disks (in two dimensions) and spheres (in three dimensions) in a channel is determined in the limit where the frequency of particle-wall collisions is large compared to particle-particle collisions. An asymptotic analysis is used in the small parameter epsilon, which is naL in two dimensions and na(2)L in three dimensions, where; n is the number density of particles (per unit area in two dimensions and per unit volume in three dimensions), L is the separation of the walls of the channel and a is the particle diameter. The particle-wall collisions are inelastic, and are described by simple relations which involve coefficients of restitution e(t) and e(n) in the tangential and normal directions, and both elastic and inelastic binary collisions between particles are considered. In the absence of binary collisions between particles, it is found that the particle velocities converge to two constant values (u(x), u(y)) = (+/-V, O) after repeated collisions with the wall, where u(x) and u(y) are the velocities tangential and normal to the wall, V = (1 - e(t))V-w/(1 + e(t)), and V-w and -V-w, are the tangential velocities of the walls of the channel. The effect of binary collisions is included using a self-consistent calculation, and the distribution function is determined using the condition that the net collisional flux of particles at any point in velocity space is zero at steady state. Certain approximations are made regarding the velocities of particles undergoing binary collisions :in order to obtain analytical results for the distribution function, and these approximations are justified analytically by showing that the error incurred decreases proportional to epsilon(1/2) in the limit epsilon --> 0. A numerical calculation of the mean square of the difference between the exact flux and the approximate flux confirms that the error decreases proportional to epsilon(1/2) in the limit epsilon --> 0. The moments of the velocity distribution function are evaluated, and it is found that [u(x)(2)] --> V-2, [u(y)(2)] similar to V-2 epsilon and -[u(x)u(y)] similar to V-2 epsilon log(epsilon(-1)) in the limit epsilon --> 0. It is found that the distribution function and the scaling laws for the velocity moments are similar for both two- and three-dimensional systems.