A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Some implications of lepton flavor violating processes in a supersymmetric Type II seesaw model at TeV scale

We have conceived a supersymmetric Type II seesaw model at TeV scale, which has some additional particles consisting of scalar and fermionic triplet Higgs states, whose masses are around a few hundred GeV. In this particular model, we have studied constraints on the masses of triplet states arising from the lepton flavor violating (LFV) processes, such as mu -> 3e and mu -> e gamma. We have analyzed the implications of these constraints on other observable quantities such as the muon anomalous magnetic moment and the decay patterns of scalar triplet Higgses. Scalar triplet Higgs states can decay into leptons and into supersymmetric fields. We have found that the constraints from LFV can affect these various decay modes.

Nonlinear dielectric response of BaBi4Ti4O15 ceramics

Nonlinear dielectric response of BaBi4Ti4O15 ceramics synthesized via the conventional solid-state reaction route has been monitored over a wide range of electric field strengths (E-0 = 0.5 - 5 kV/cm). Dielectric permittivity was found to increase linearly within the range of applied field. Rayleigh relations were employed to interpret the nonlinear dielectric response and the contribution of irreversible domain wall motion to the macroscopic permittivity was separated. The values of room temperature Rayleigh dielectric coefficient (alpha) and relative initial permittivity (epsilon'(init)) were found to be 2.28 +/- 0.02 cm/kV and 146.10 +/- 0.07, respectively. A reasonable agreement between the simulated and measured polarization-electric field (P-E) hysteresis loops was observed at an applied electric field of 5 kV/cm.

A novel filtering framework through Girsanov correction for the identification of nonlinear dynamical systems

Using a Girsanov change of measures, we propose novel variations within a particle-filtering algorithm, as applied to the inverse problem of state and parameter estimations of nonlinear dynamical systems of engineering interest, toward weakly correcting for the linearization or integration errors that almost invariably occur whilst numerically propagating the process dynamics, typically governed by nonlinear stochastic differential equations (SDEs). Specifically, the correction for linearization, provided by the likelihood or the Radon-Nikodym derivative, is incorporated within the evolving flow in two steps. Once the likelihood, an exponential martingale, is split into a product of two factors, correction owing to the first factor is implemented via rejection sampling in the first step. The second factor, which is directly computable, is accounted for via two different schemes, one employing resampling and the other using a gain-weighted innovation term added to the drift field of the process dynamics thereby overcoming the problem of sample dispersion posed by resampling. The proposed strategies, employed as add-ons to existing particle filters, the bootstrap and auxiliary SIR filters in this work, are found to non-trivially improve the convergence and accuracy of the estimates and also yield reduced mean square errors of such estimates vis-a-vis those obtained through the parent-filtering schemes.

Crystal growth and nonlinear optical properties of sodium D-isoascorbate monohydrate

Optical quality single crystals of sodium D-isoascorbate monohydrate were grown by a slow cooling technique. The crystal possesses a bulky prismatic morphology. Thermal analyses indicate that the crystals are stable up to 125 degrees C. The optical transmission window ranges from 307 nm to 1450 nm. The principal refractive indices have been measured employing Brewster's angle method. The crystallographic and the principal dielectric axes coincide with each other such that a lies along Z, b along X and c along Y. The optic axis is oriented 58 degrees (at 532 nm) to the crystallographic a axis in the XZ plane and the crystal is negative biaxial. Type 1 and type 2 phase matching curves are generated and experimentally verified. No polarization dependence of the light absorption was observed confirming the validity of Kleinman's symmetry conjecture, leading to a single nonvanishing nonlinear tensor component. According to Hobden's classification the crystal belongs to class 3. The crystal also exhibits second order noncollinear conic sections. The single shot and multiple shot surface laser damage thresholds are determined to be 32.7 GW cm(-2) and 6.5 GW cm(-2) respectively for 1064 nm radiation.

Flow cytometry analysis of human fetal osteoblast fate processes on spark plasma sintered hydroxyapatite-titanium biocomposites

Hydroxyapatite (HA)-based biocomposites have been widely investigated for a multitude of applications and these studies have been largely driven to improve mechanical properties (toughness and strength) without compromising cytocompatibility properties. Apart from routine cell viability/proliferation analysis, limited efforts have been made to quantify the fate processes (cell proliferation, cell cycle, and cell apoptosis) of human fetal osteoblast (hFOB) cells on HA-based composites, in vitro. In this work, the osteoblast cell fate process has been studied on a model hydroxyapatite-titanium (HA-Ti) system using the flow cytometry. In order to retain both HA and Ti, the novel processing technique, that is, spark plasma sintering, was suitably adopted. The cell fate processes of hFOBs, as evaluated using a flow cytometry, revealed statistically insignificant differences among HA-10 wt % Ti and HA and control (tissue culture polystyrene surface) in terms of osteoblast apoptosis, proliferation index as well as division index. For the first time, we provide quantified flow cytometry results to demonstrate that 10 wt % Ti additions to HA do not have any significant influence on the fate processes of human osteoblast-like cells, in vitro.

Linear and nonlinear optical properties of expanded porphyrins: a DMRG study

We study absorption spectra and two photon absorption coefficient of expanded porphyrins (EPs) by the density matrix renormalization group (DMRG) technique. We employ the Pariser-Parr-Pople (PPP) Hamiltonian which includes long-range electron-electron interactions. We find that, in the 4n+2 EPs, there are two prominent low-lying one-photon excitations, while in 4n EPs, there is only one such excitation. We also find that 4n+2 EPs have large two-photon absorption cross sections compared to 4n EPs. The charge density rearrangement in the one-photon excited state is mostly at the pyrrole nitrogen site and at the meso carbon sites. In the two-photon states, the charge density rearrangement occurs mostly at the aza-ring sites. In the one-photon state, the C-C bond length in aza rings shows a tendency to become uniform. In the two-photon state, the bond distortions are on C-N bonds of the pyrrole ring and the adjoining C-C bonds which connect the pyrrole ring to the aza or meso carbon sites.

Dynamic state estimation for identifying earthquake support motions in instrumented structures

The problem of identification of multi-component and (or) spatially varying earthquake support motions based on measured responses in instrumented structures is considered. The governing equations of motion are cast in the state space form and a time domain solution to the input identification problem is developed based on the Kalman and particle filtering methods. The method allows for noise in measured responses, imperfections in mathematical model for the structure, and possible nonlinear behavior of the structure. The unknown support motions are treated as hypothetical additional system states and a prior model for these motions are taken to be given in terms of white noise processes. For linear systems, the solution is developed within the Kalman filtering framework while, for nonlinear systems, the Monte Carlo simulation based particle filtering tools are employed. In the latter case, the question of controlling sampling variance based on the idea of Rao-Blackwellization is also explored. Illustrative examples include identification of multi-component and spatially varying support motions in linear/nonlinear structures.

Validation based sparse Gaussian processes for ordinal regression

This paper proposes a sparse modeling approach to solve ordinal regression problems using Gaussian processes (GP). Designing a sparse GP model is important from training time and inference time viewpoints. We first propose a variant of the Gaussian process ordinal regression (GPOR) approach, leave-one-out GPOR (LOO-GPOR). It performs model selection using the leave-one-out cross-validation (LOO-CV) technique. We then provide an approach to design a sparse model for GPOR. The sparse GPOR model reduces computational time and storage requirements. Further, it provides faster inference. We compare the proposed approaches with the state-of-the-art GPOR approach on some benchmark data sets. Experimental results show that the proposed approaches are competitive.

Nonlinear thermoacoustics of ducted premixed flames: The influence of perturbation convection speed

When a premixed flame is placed within a duct, acoustic waves induce velocity perturbations at the flame's base. These travel down the flame, distorting its surface and modulating its heat release. This can induce self-sustained thermoacoustic oscillations. Although the phase speed of these perturbations is often assumed to equal the mean flow speed, experiments conducted in other studies and Direct Numerical Simulation (DNS) conducted in this study show that it varies with the acoustic frequency. In this paper, we examine how these variations affect the nonlinear thermoacoustic behaviour. We model the heat release with a nonlinear kinematic G-equation, in which the velocity perturbation is modelled on DNS results. The acoustics are governed by linearised momentum and energy equations. We calculate the flame describing function (FDF) using harmonic forcing at several frequencies and amplitudes. Then we calculate thermoacoustic limit cycles and explain their existence and stability by examining the amplitude-dependence of the gain and phase of the FDF. We find that, when the phase speed equals the mean flow speed, the system has only one stable state. When the phase speed does not equal the mean flow speed, however, the system supports multiple limit cycles because the phase of the FDF changes significantly with oscillation amplitude. This shows that the phase speed of velocity perturbations has a strong influence on the nonlinear thermoacoustic behaviour of ducted premixed flames. (C) 2013 The Combustion Institute. Published by Elsevier Inc. All rights reserved.

Signal Detection in Generalized Gaussian Noise by Nonlinear Wavelet Denoising

In this paper, a nonlinear suboptimal detector whose performance in heavy-tailed noise is significantly better than that of the matched filter is proposed. The detector consists of a nonlinear wavelet denoising filter to enhance the signal-to-noise ratio, followed by a replica correlator. Performance of the detector is investigated through an asymptotic theoretical analysis as well as Monte Carlo simulations. The proposed detector offers the following advantages over the optimal (in the Neyman-Pearson sense) detector: it is easier to implement, and it is more robust with respect to error in modeling the probability distribution of noise.

A Nearly Exact Reformulation of the Girsanov Linearization for Stochastically Driven Nonlinear Oscillators

The Girsanov linearization method (GLM), proposed earlier in Saha, N., and Roy, D., 2007, ``The Girsanov Linearisation Method for Stochastically Driven Nonlinear Oscillators,'' J. Appl. Mech., 74, pp. 885-897, is reformulated to arrive at a nearly exact, semianalytical, weak and explicit scheme for nonlinear mechanical oscillators under additive stochastic excitations. At the heart of the reformulated linearization is a temporally localized rejection sampling strategy that, combined with a resampling scheme, enables selecting from and appropriately modifying an ensemble of locally linearized trajectories while weakly applying the Girsanov correction (the Radon-Nikodym derivative) for the linearization errors. The semianalyticity is due to an explicit linearization of the nonlinear drift terms and it plays a crucial role in keeping the Radon-Nikodym derivative ``nearly bounded'' above by the inverse of the linearization time step (which means that only a subset of linearized trajectories with low, yet finite, probability exceeds this bound). Drift linearization is conveniently accomplished via the first few (lower order) terms in the associated stochastic (Ito) Taylor expansion to exclude (multiple) stochastic integrals from the numerical treatment. Similarly, the Radon-Nikodym derivative, which is a strictly positive, exponential (super-) martingale, is converted to a canonical form and evaluated over each time step without directly computing the stochastic integrals appearing in its argument. Through their numeric implementations for a few low-dimensional nonlinear oscillators, the proposed variants of the scheme, presently referred to as the Girsanov corrected linearization method (GCLM), are shown to exhibit remarkably higher numerical accuracy over a much larger range of the time step size than is possible with the local drift-linearization schemes on their own.

NESP: Nonlinear enhancement and selection of plane for optimal segmentation and recognition of scene word images

In this paper, we report a breakthrough result on the difficult task of segmentation and recognition of coloured text from the word image dataset of ICDAR robust reading competition challenge 2: reading text in scene images. We split the word image into individual colour, gray and lightness planes and enhance the contrast of each of these planes independently by a power-law transform. The discrimination factor of each plane is computed as the maximum between-class variance used in Otsu thresholding. The plane that has maximum discrimination factor is selected for segmentation. The trial version of Omnipage OCR is then used on the binarized words for recognition. Our recognition results on ICDAR 2011 and ICDAR 2003 word datasets are compared with those reported in the literature. As baseline, the images binarized by simple global and local thresholding techniques were also recognized. The word recognition rate obtained by our non-linear enhancement and selection of plance method is 72.8% and 66.2% for ICDAR 2011 and 2003 word datasets, respectively. We have created ground-truth for each image at the pixel level to benchmark these datasets using a toolkit developed by us. The recognition rate of benchmarked images is 86.7% and 83.9% for ICDAR 2011 and 2003 datasets, respectively.

Capturability of augmented proportional navigation (APN) guidance with nonlinear engagement dynamics

Proportional Navigation (PN) and its variants are widely used guidance philosophies. However, in the presence of target maneuver, PN guidance law is effective only for a restrictive set of initial geometries. To account for target maneuvers, the concept of Augmented Proportional Navigation (APN) guidance law was introduced and analyzed in a linearized interceptor-target engagement framework presented in literature. However, there is no work in the literature, that addresses the capturability performance of the APN guidance law in a nonlinear engagement framework. This paper presents such an analysis and obtains the conditions for capturability. It also shows that a shorter time of interception is obtained when APN is formulated in the nonlinear framework as proposed in this paper. Simulation results are given to support the theoretical findings.

Simultaneous attitude control and trajectory tracking of a micro quadrotor: a SNAC aided nonlinear dynamic inversion approach

This paper presents an advanced single network adaptive critic (SNAC) aided nonlinear dynamic inversion (NDI) approach for simultaneous attitude control and trajectory tracking of a micro-quadrotor. Control of micro-quadrotors is a challenging problem due to its small size, strong coupling in pitch-yaw-roll and aerodynamic effects that often need to be ignored in the control design process to avoid mathematical complexities. In the proposed SNAC aided NDI approach, the gains of the dynamic inversion design are selected in such a way that the resulting controller behaves closely to a pre-synthesized SNAC controller for the output regulation problem. However, since SNAC is based on optimal control theory, it makes the dynamic inversion controller to operate near optimal and enhances its robustness property as well. More important, it retains two major benefits of dynamic inversion: (i) closed form expression of the controller and (ii) easy scalability to command tracking application even without any apriori knowledge of the reference command. Effectiveness of the proposed controller is demonstrated from six degree-of-freedom simulation studies of a micro-quadrotor. It has also been observed that the proposed SNAC aided NDI approach is more robust to modeling inaccuracies, as compared to the NDI controller designed independently from time domain specifications.