An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

SuSeFLAV 1.2: Program for supersymmetric mass spectra with seesaw mechanism and rare lepton flavor violating decays

Accurate supersymmetric spectra are required to confront data from direct and indirect searches of supersymmetry. SuSeFLAV is a numerical tool capable of computing supersymmetric spectra precisely for various supersymmetric breaking scenarios applicable even in the presence of flavor violation. The program solves MSSM RGEs with complete 3 x 3 flavor mixing at 2-loop level and one loop finite threshold corrections to all MSSM parameters by incorporating radiative electroweak symmetry breaking conditions. The program also incorporates the Type-I seesaw mechanism with three massive right handed neutrinos at user defined mass scales and mixing. It also computes branching ratios of flavor violating processes such as l(j) -> l(i)gamma, l(j) -> 3 l(i), b -> s gamma and supersymmetric contributions to flavor conserving quantities such as (g(mu) - 2). A large choice of executables suitable for various operations of the program are provided. Program summary Program title: SuSeFLAV Catalogue identifier: AEOD_v1_0 Program summary URL: http://cpc.cs.qub.ac.uk/summaries/AEOD_v1_0.html Program obtainable from: CPC Program Library, Queen's University, Belfast, N. Ireland Licensing provisions: GNU General Public License No. of lines in distributed program, including test data, etc.: 76552 No. of bytes in distributed program, including test data, etc.: 582787 Distribution format: tar.gz Programming language: Fortran 95. Computer: Personal Computer, Work-Station. Operating system: Linux, Unix. Classification: 11.6. Nature of problem: Determination of masses and mixing of supersymmetric particles within the context of MSSM with conserved R-parity with and without the presence of Type-I seesaw. Inter-generational mixing is considered while calculating the mass spectrum. Supersymmetry breaking parameters are taken as inputs at a high scale specified by the mechanism of supersymmetry breaking. RG equations including full inter-generational mixing are then used to evolve these parameters up to the electroweak breaking scale. The low energy supersymmetric spectrum is calculated at the scale where successful radiative electroweak symmetry breaking occurs. At weak scale standard model fermion masses, gauge couplings are determined including the supersymmetric radiative corrections. Once the spectrum is computed, the program proceeds to various lepton flavor violating observables (e.g., BR(mu -> e gamma), BR(tau -> mu gamma) etc.) at the weak scale. Solution method: Two loop RGEs with full 3 x 3 flavor mixing for all supersymmetry breaking parameters are used to compute the low energy supersymmetric mass spectrum. An adaptive step size Runge-Kutta method is used to solve the RGEs numerically between the high scale and the electroweak breaking scale. Iterative procedure is employed to get the consistent radiative electroweak symmetry breaking condition. The masses of the supersymmetric particles are computed at 1-loop order. The third generation SM particles and the gauge couplings are evaluated at the 1-loop order including supersymmetric corrections. A further iteration of the full program is employed such that the SM masses and couplings are consistent with the supersymmetric particle spectrum. Additional comments: Several executables are presented for the user. Running time: 0.2 s on a Intel(R) Core(TM) i5 CPU 650 with 3.20 GHz. (c) 2012 Elsevier B.V. All rights reserved.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Partial-Dislocation-Mediated Processes in Nanocrystalline Ni with Nonequilibrium Grain Boundaries

The partial-dislocation-mediated processes have so far eluded high-resolution transmission electron microscopy studies in nanocrystalline nc Ni with nonequilibrium grain boundaries. It is revealed that the nc Ni deformed largely by twinning instead of extended partials. The underlying mechanisms including dissociated dislocations, high residual stresses, and stress concentrations near stacking faults are demonstrated and discussed.

Nonparametric Bayesian Density Modeling with Gaussian Processes

We present the Gaussian process density sampler (GPDS), an exchangeable generative model for use in nonparametric Bayesian density estimation. Samples drawn from the GPDS are consistent with exact, independent samples from a distribution defined by a density that is a transformation of a function drawn from a Gaussian process prior. Our formulation allows us to infer an unknown density from data using Markov chain Monte Carlo, which gives samples from the posterior distribution over density functions and from the predictive distribution on data space. We describe two such MCMC methods. Both methods also allow inference of the hyperparameters of the Gaussian process.

Dynamics of rotationally resolved multiphoton ionization processes in molecules

This dissertation presents the results of studies of several rotationally- resolved resonance enhanced multiphoton ionization (REMPI) processes in some simple molecular systems. The objective of these studies is to quantitatively identify the underlying dynamics of this highly state-specific process which utilizes the narrow bandwidth radiation of a laser to ionize a molecule by first preparing an excited state via multiphoton absorption and subsequently ionizing that state before it can decay. Coupled with high-resolution photoelectron spectroscopy, REMPI is clearly an important probe of molecular excited states and their photoioniza tion dynamics.

A key feature of our studies is that they are carried out using accurate Hartree-Fock orbitals to describe the photoelectron orbitals of the molecular ions. The use of such photoelectron orbitals is important in rotationally-resolved studies where the angular momentum coupling in the photoelectron orbital plays a significant role in the photoionization dynamics. In these studies the Hartree-Fock molecular molecular photoelectron orbitals are obtained by numerical solution of a Lippmann-Schwinger integral equation.

Studies reported here include investigations of (i) ionic rotational branching ratios and their energy dependence for REMPI via the A^2Σ^+(3sσ) and D^2Σ^+(3pσ)states of NO, (ii) the influence of angular momentum constraints on branching ratios at low photoelectron energies for REMPI via low-J levels of the resonant intermediate state, (iii) the strong dependence of photoelectron angular distributions on final ionic rotational state and on the alignment in REMPI of the A^2Σ^+ state of NO, (iv) vibrational state dependence of ionic rotational branching ratios arising from rapid orbital evolution in resonant states (E'^2Σ^+(3pσ) of CH), (v) the influence of rovibronic interactions on the rotational branching ratios seen in REMPI via the D^2Σ^+(3pσ) state of NO, and (vi) effects of laser intensity on the photoionization dynamics of REMPI.

Combustion of CS₂/O₂ in a laminar mixing layer and processes in the CO chemical laser

The combustion of CS₂ and O₂ in a free burning laminar mixing layer at low pressure was investigated using emission spectroscopy. The temperature fields, CO vibrational distributions, and CO concentrations were measured. The data indicate that vibration ally excited CO was produced in the mixing layer flames, but that there were no vibrational population inversions. In comparison with the CS₂/O₂ premixed flames, the mixing layer flames favored greater production of COS and CO₂. Computer modeling was used to study the mechanisms responsible for the production of COS and CO₂, and to study how the branching chain mechanism responsible for production of CO affects the behavior of the mixing layer flame. The influences of the gas additives, N₂O, COS, and CNBr, were also investigated.

Properties of convergence of a class of iterative processes generated by sequences of self-mappings with applications to switched dynamic systems

This article investigates the convergence properties of iterative processes involving sequences of self-mappings of metric or Banach spaces. Such sequences are built from a set of primary self-mappings which are either expansive or non-expansive self-mappings and some of the non-expansive ones can be contractive including the case of strict contractions. The sequences are built subject to switching laws which select each active self-mapping on a certain activation interval in such a way that essential properties of boundedness and convergence of distances and iterated sequences are guaranteed. Applications to the important problem of stability of dynamic switched systems are also given.