An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

Stabilization of stochastic approximation by step size adaptation

A scheme for stabilizing stochastic approximation iterates by adaptively scaling the step sizes is proposed and analyzed. This scheme leads to the same limiting differential equation as the original scheme and therefore has the same limiting behavior, while avoiding the difficulties associated with projection schemes. The proof technique requires only that the limiting o.d.e. descend a certain Lyapunov function outside an arbitrarily large bounded set. (C) 2012 Elsevier B.V. All rights reserved.

Influence of anion on the quadratic nonlinearity and depolarization ratios of scattered second harmonic light from cation-pi complexes

We have investigated quadratic nonlinearity (beta(HRS)) and linear and circular depolarization ratios (D and D', respectively) of a series of 1:1 complexes of tropyliumtetrafluoroborate as a cation and methyl-substituted benzenes as pi-donors by making polarization resolved hyper-Rayleigh scattering measurements in solution. The measured D and D' values are much lower than the values expected from a typical sandwich or a T-shaped geometry of a complex. In the cation-pi complexes studied here, the D value varies from 1.36 to 1.46 and D' from 1.62 to 1.72 depending on the number of methyl substitutions on the benzene ring. In order to probe it further, beta, D and D' were computed using the Zerner intermediate neglect of differential overlap-correction vector self-consistent reaction field technique including single and double configuration interactions in the absence and presence of BF4- anion. In the absence of the anion, the calculated value of D varies from 4.20 to 4.60 and that of D' from 2.45 to 2.72 which disagree with experimental values. However, by arranging three cation-pi BF4- complexes in a trigonal symmetry, the computed values are brought to agreement with experiments. When such an arrangement was not considered, the calculated beta values were lower than the experimental values by more than a factor of two. This unprecedented influence of the otherwise ``unimportant'' anion in solution on the beta value and depolarization ratios of these cation-pi complexes is highlighted and emphasized in this paper. (C) 2012 American Institute of Physics. http://dx.doi.org/10.1063/1.4716020]

Path-integral formulation for Levy flights: Evaluation of the propagator for free, linear, and harmonic potentials in the over- and underdamped limits

Levy flights can be described using a Fokker-Planck equation, which involves a fractional derivative operator in the position coordinate. Such an operator has its natural expression in the Fourier domain. Starting with this, we show that the solution of the equation can be written as a Hamiltonian path integral. Though this has been realized in the literature, the method has not found applications as the path integral appears difficult to evaluate. We show that a method in which one integrates over the position coordinates first, after which integration is performed over the momentum coordinates, can be used to evaluate several path integrals that are of interest. Using this, we evaluate the propagators for (a) free particle, (b) particle subjected to a linear potential, and (c) harmonic potential. In all the three cases, we have obtained results for both overdamped and underdamped cases. DOI: 10.1103/PhysRevE.86.061105

A constant factor approximation algorithm for boxicity of circular arc graphs

Boxicity of a graph G(V, E) is the minimum integer k such that G can be represented as the intersection graph of k-dimensional axis parallel boxes in Rk. Equivalently, it is the minimum number of interval graphs on the vertex set V such that the intersection of their edge sets is E. It is known that boxicity cannot be approximated even for graph classes like bipartite, co-bipartite and split graphs below O(n0.5-ε)-factor, for any ε > 0 in polynomial time unless NP = ZPP. Till date, there is no well known graph class of unbounded boxicity for which even an nε-factor approximation algorithm for computing boxicity is known, for any ε < 1. In this paper, we study the boxicity problem on Circular Arc graphs - intersection graphs of arcs of a circle. We give a (2+ 1/k)-factor polynomial time approximation algorithm for computing the boxicity of any circular arc graph along with a corresponding box representation, where k ≥ 1 is its boxicity. For Normal Circular Arc(NCA) graphs, with an NCA model given, this can be improved to an additive 2-factor approximation algorithm. The time complexity of the algorithms to approximately compute the boxicity is O(mn+n2) in both these cases and in O(mn+kn2) which is at most O(n3) time we also get their corresponding box representations, where n is the number of vertices of the graph and m is its number of edges. The additive 2-factor algorithm directly works for any Proper Circular Arc graph, since computing an NCA model for it can be done in polynomial time.

Fast acoustic likelihood computation using low-rank matrix approximation

Acoustic modeling using mixtures of multivariate Gaussians is the prevalent approach for many speech processing problems. Computing likelihoods against a large set of Gaussians is required as a part of many speech processing systems and it is the computationally dominant phase for LVCSR systems. We express the likelihood computation as a multiplication of matrices representing augmented feature vectors and Gaussian parameters. The computational gain of this approach over traditional methods is by exploiting the structure of these matrices and efficient implementation of their multiplication.In particular, we explore direct low-rank approximation of the Gaussian parameter matrix and indirect derivation of low-rank factors of the Gaussian parameter matrix by optimum approximation of the likelihood matrix. We show that both the methods lead to similar speedups but the latter leads to far lesser impact on the recognition accuracy. Experiments on a 1138 word vocabulary RM1 task using Sphinx 3.7 system show that, for a typical case the matrix multiplication approach leads to overall speedup of 46%. Both the low-rank approximation methods increase the speedup to around 60%, with the former method increasing the word error rate (WER) from 3.2% to 6.6%, while the latter increases the WER from 3.2% to 3.5%.

Dual band(WLAN & WiMAX) suppressed harmonic microstrip filter with perturbed ground

Design of a dual band pass filter employing microstrip line with defected ground is presented in this paper. A dual band filter at 2.45GHz and 3.5GHz (covering WLAN and WiMAX) with 6% bandwidth has been designed at each frequency. Apertures in ground plane were used to improve the stop band rejection characteristics and coupling levels in the filter. Measured results of the experimental filter were compared against the simulation results for the purpose of validation.

Quadrature approximation properties of the spiral-phase quadrature transform

The notion of the 1-D analytic signal is well understood and has found many applications. At the heart of the analytic signal concept is the Hilbert transform. The problem in extending the concept of analytic signal to higher dimensions is that there is no unique multidimensional definition of the Hilbert transform. Also, the notion of analyticity is not so well under stood in higher dimensions. Of the several 2-D extensions of the Hilbert transform, the spiral-phase quadrature transform or the Riesz transform seems to be the natural extension and has attracted a lot of attention mainly due to its isotropic properties. From the Riesz transform, Larkin et al. constructed a vortex operator, which approximates the quadratures based on asymptotic stationary-phase analysis. In this paper, we show an alternative proof for the quadrature approximation property by invoking the quasi-eigenfunction property of linear, shift-invariant systems. We show that the vortex operator comes up as a natural consequence of applying this property. We also characterize the quadrature approximation error in terms of its energy as well as the peak spatial-domain error. Such results are available for 1-D signals, but their counter part for 2-D signals have not been provided. We also provide simulation results to supplement the analytical calculations.

A post processing technique for improved harmonic/percussion separation in polyphonic speech

In this paper we propose a postprocessing technique for a spectrogram diffusion based harmonic/percussion decom- position algorithm. The proposed technique removes har- monic instrument leakages in the percussion enhanced out- puts of the baseline algorithm. The technique uses median filtering and an adaptive detection of percussive segments in subbands followed by piecewise signal reconstruction using envelope properties to ensure that percussion is enhanced while harmonic leakages are suppressed. A new binary mask is created for the percussion signal which upon applying on the original signal improves harmonic versus percussion separation. We compare our algorithm with two recent techniques and show that on a database of polyphonic Indian music, the postprocessing algorithm improves the harmonic versus percussion decomposition significantly.

A fully discrete C-0 interior penalty Galerkin approximation of the extended Fisher-Kolmogorov equation

A fully discrete C-0 interior penalty finite element method is proposed and analyzed for the Extended Fisher-Kolmogorov (EFK) equation u(t) + gamma Delta(2)u - Delta u + u(3) - u = 0 with appropriate initial and boundary conditions, where gamma is a positive constant. We derive a regularity estimate for the solution u of the EFK equation that is explicit in gamma and as a consequence we derive a priori error estimates that are robust in gamma. (C) 2013 Elsevier B.V. All rights reserved.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Polynomial time and parameterized approximation algorithms for boxicity

The boxicity (cubicity) of a graph G, denoted by box(G) (respectively cub(G)), is the minimum integer k such that G can be represented as the intersection graph of axis parallel boxes (cubes) in ℝ k . The problem of computing boxicity (cubicity) is known to be inapproximable in polynomial time even for graph classes like bipartite, co-bipartite and split graphs, within an O(n 0.5 − ε ) factor for any ε > 0, unless NP = ZPP. We prove that if a graph G on n vertices has a clique on n − k vertices, then box(G) can be computed in time n22O(k2logk) . Using this fact, various FPT approximation algorithms for boxicity are derived. The parameter used is the vertex (or edge) edit distance of the input graph from certain graph families of bounded boxicity - like interval graphs and planar graphs. Using the same fact, we also derive an O(nloglogn√logn√) factor approximation algorithm for computing boxicity, which, to our knowledge, is the first o(n) factor approximation algorithm for the problem. We also present an FPT approximation algorithm for computing the cubicity of graphs, with vertex cover number as the parameter.

Harmonic Analysis of DC-Link Capacitor Current in Sinusoidally Modulated Neutral-Point-Clamped Inverter

The voltage ripple and power loss in the DC-capacitor of a voltage source inverter depend on the harmonic currents flowing through the capacitor. This paper presents double Fourier series based harmonic analysis of DC capacitor current in a three-level neutral point clamped inverter, modulated with sine-triangle PWM. The analytical results are validated experimentally on a 5-kVA three-level inverter prototype. The results of the analysis are used for predicting the power loss in the DC capacitor.

Nonlinear optical second harmonic generation in ZnS quantum dots and observation on optical properties of ZnS/PMMA nanocomposites

ZnS quantum dots (QDs) of different sizes are synthesized by a simple chemical co-precipitation method at room temperature, by varying pH value of the reaction mixture. Samples are characterized by an X-ray diffractometer, transmission electron microscope, energy-dispersive X-ray analysis, etc. Linear optical properties, including UV-visible absorption and photoluminescence emission characteristics, of as-prepared QDs are measured. Size dependent nonlinear optical property, such as second harmonic generation (SHG) of 1064 nm Nd:YAG laser fundamental radiation in the synthesized ZnS QDs, is reported for the first time, to the best of our knowledge, by using the standard Kurtz-Perry powder method. In not to study the possibility of the synthesized ZnS QDs in different device applications ZnS/PMMA (polymethylmethacrylate) nanocomposites are also synthesized. The presence of weak chemical interaction between the polymer matrix and ZnS QDs is confirmed by Fourier transform infrared spectroscopy. Thermal properties of the nanocomposites are studied by differential scanning calorimetry and thermo-gravimetric analysis techniques, which show that the composites are stable up to similar to 300 degrees C temperature. (C) 2013 Elsevier B.V. All rights reserved.