A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Violation of entropic Leggett-Garg inequality in nuclear spins

We report an experimental study of recently formulated entropic Leggett-Garg inequality (ELGI) by Usha Devi et al. Phys. Rev. A 87, 052103 (2013)]. This inequality places a bound on the statistical measurement outcomes of dynamical observables describing a macrorealistic system. Such a bound is not necessarily obeyed by quantum systems, and therefore provides an important way to distinguish quantumness from classical behavior. Here we study ELGI using a two-qubit nuclear magnetic resonance system. To perform the noninvasive measurements required for the ELGI study, we prepare the system qubit in a maximally mixed state as well as use the ``ideal negative result measurement'' procedure with the help of an ancilla qubit. The experimental results show a clear violation of ELGI by over four standard deviations. These results agree with the predictions of quantum theory. The violation of ELGI is attributed to the fact that certain joint probabilities are not legitimate in the quantum scenario, in the sense they do not reproduce all the marginal probabilities. Using a three-qubit system, we also demonstrate that three-time joint probabilities do not reproduce certain two-time marginal probabilities.

Exact internal controllability for a hyperbolic problem in a domain with highly oscillating boundary

In this paper, by using the Hilbert Uniqueness Method (HUM), we study the exact controllability problem described by the wave equation in a three-dimensional horizontal domain bounded at the bottom by a smooth wall and at the top by a rough wall. The latter is assumed to consist in a plane wall covered with periodically distributed asperities whose size depends on a small parameter epsilon > 0, and with a fixed height. Our aim is to obtain the exact controllability for the homogenized equation. In the process, we study the asymptotic analysis of wave equation in two setups, namely solution by standard weak formulation and solution by transposition method.

Application of the random eigenvalue problem in forced response analysis of a linear stochastic structure

The random eigenvalue problem arises in frequency and mode shape determination for a linear system with uncertainties in structural properties. Among several methods of characterizing this random eigenvalue problem, one computationally fast method that gives good accuracy is a weak formulation using polynomial chaos expansion (PCE). In this method, the eigenvalues and eigenvectors are expanded in PCE, and the residual is minimized by a Galerkin projection. The goals of the current work are (i) to implement this PCE-characterized random eigenvalue problem in the dynamic response calculation under random loading and (ii) to explore the computational advantages and challenges. In the proposed method, the response quantities are also expressed in PCE followed by a Galerkin projection. A numerical comparison with a perturbation method and the Monte Carlo simulation shows that when the loading has a random amplitude but deterministic frequency content, the proposed method gives more accurate results than a first-order perturbation method and a comparable accuracy as the Monte Carlo simulation in a lower computational time. However, as the frequency content of the loading becomes random, or for general random process loadings, the method loses its accuracy and computational efficiency. Issues in implementation, limitations, and further challenges are also addressed.

On the Erdos-Szekeres n-interior-point problem

The n-interior-point variant of the Erdos Szekeres problem is the following: for every n, n >= 1, does there exist a g(n) such that every point set in the plane with at least g(n) interior points has a convex polygon containing exactly n interior points. The existence of g(n) has been proved only for n <= 3. In this paper, we show that for any fixed r >= 2, and for every n >= 5, every point set having sufficiently large number of interior points and at most r convex layers contains a subset with exactly n interior points. We also consider a relaxation of the notion of convex polygons and show that for every n, n >= 1, any point set with at least n interior points has an almost convex polygon (a simple polygon with at most one concave vertex) that contains exactly n interior points. (C) 2013 Elsevier Ltd. All rights reserved.

Higher-Dimensional Analogues of the Map Coloring Problem

After a brief discussion of the history of the problem, we propose a generalization of the map coloring problem to higher dimensions.

The problem of clear air turbulence: Changing perspectives in the understanding of the phenomenon

Due to rapid improvements in on-board instrumentation and atmospheric observation systems, in most cases, aircraft are able to steer clear of regions of adverse weather. However, they still encounter unexpected bumpy flight conditions in regions away from storms and clouds. This is the phenomenon of clear air turbulence (CAT), which has been a challenge to our understanding as well as efforts at prediction. While most of such cases result in mild discomfort, a few cases can be violent leading to serious injuries to passengers and damage to the aircraft. The underlying physical mechanisms have been sought to be explained in terms of fluid dynamic instabilities and waves in the atmosphere. The main mechanisms which have been proposed are: (i) Kelvin-Helmholtz instability of shear layers, (ii) waves generated from flow over mountains, (iii) inertia-gravity waves from clouds and other sources, (iv) spontaneous imbalance theory and (v) horizontal vortex tubes. This has also undergone a change over the years. We present an overview of the mechanisms proposed and their implications for prediction.

Cluster labeling for multilingual scatter/gather using comparable corpora

Scatter/Gather systems are increasingly becoming useful in browsing document corpora. Usability of the present-day systems are restricted to monolingual corpora, and their methods for clustering and labeling do not easily extend to the multilingual setting, especially in the absence of dictionaries/machine translation. In this paper, we study the cluster labeling problem for multilingual corpora in the absence of machine translation, but using comparable corpora. Using a variational approach, we show that multilingual topic models can effectively handle the cluster labeling problem, which in turn allows us to design a novel Scatter/Gather system ShoBha. Experimental results on three datasets, namely the Canadian Hansards corpus, the entire overlapping Wikipedia of English, Hindi and Bengali articles, and a trilingual news corpus containing 41,000 articles, confirm the utility of the proposed system.

STOKES' SYSTEM IN A DOMAIN WITH OSCILLATING BOUNDARY: HOMOGENIZATION AND ERROR ANALYSIS OF AN INTERIOR OPTIMAL CONTROL PROBLEM

Homogenization and error analysis of an optimal interior control problem in the framework of Stokes' system, on a domain with rapidly oscillating boundary, are the subject matters of this article. We consider a three dimensional domain constituted of a parallelepiped with a large number of rectangular cylinders at the top of it. An interior control is applied in a proper subdomain of the parallelepiped, away from the oscillating volume. We consider two types of functionals, namely a functional involving the L-2-norm of the state variable and another one involving its H-1-norm. The asymptotic analysis of optimality systems for both cases, when the cross sectional area of the rectangular cylinders tends to zero, is done here. Our major contribution is to derive error estimates for the state, the co-state and the associated pressures, in appropriate functional spaces.

A Phase Transition for the Uniform Distribution in the Pattern Maximum Likelihood Problem

In this paper, we consider the setting of the pattern maximum likelihood (PML) problem studied by Orlitsky et al. We present a well-motivated heuristic algorithm for deciding the question of when the PML distribution of a given pattern is uniform. The algorithm is based on the concept of a ``uniform threshold''. This is a threshold at which the uniform distribution exhibits an interesting phase transition in the PML problem, going from being a local maximum to being a local minimum.

Free turbulent shear layer in a point vortex gas as a problem in nonequilibrium statistical mechanics

This paper attempts to unravel any relations that may exist between turbulent shear flows and statistical mechanics through a detailed numerical investigation in the simplest case where both can be well defined. The flow considered for the purpose is the two-dimensional (2D) temporal free shear layer with a velocity difference Delta U across it, statistically homogeneous in the streamwise direction (x) and evolving from a plane vortex sheet in the direction normal to it (y) in a periodic-in-x domain L x +/-infinity. Extensive computer simulations of the flow are carried out through appropriate initial-value problems for a ``vortex gas'' comprising N point vortices of the same strength (gamma = L Delta U/N) and sign. Such a vortex gas is known to provide weak solutions of the Euler equation. More than ten different initial-condition classes are investigated using simulations involving up to 32 000 vortices, with ensemble averages evaluated over up to 10(3) realizations and integration over 10(4)L/Delta U. The temporal evolution of such a system is found to exhibit three distinct regimes. In Regime I the evolution is strongly influenced by the initial condition, sometimes lasting a significant fraction of L/Delta U. Regime III is a long-time domain-dependent evolution towards a statistically stationary state, via ``violent'' and ``slow'' relaxations P.-H. Chavanis, Physica A 391, 3657 (2012)], over flow time scales of order 10(2) and 10(4)L/Delta U, respectively (for N = 400). The final state involves a single structure that stochastically samples the domain, possibly constituting a ``relative equilibrium.'' The vortex distribution within the structure follows a nonisotropic truncated form of the Lundgren-Pointin (L-P) equilibrium distribution (with negatively high temperatures; L-P parameter lambda close to -1). The central finding is that, in the intermediate Regime II, the spreading rate of the layer is universal over the wide range of cases considered here. The value (in terms of momentum thickness) is 0.0166 +/- 0.0002 times Delta U. Regime II, extensively studied in the turbulent shear flow literature as a self-similar ``equilibrium'' state, is, however, a part of the rapid nonequilibrium evolution of the vortex-gas system, which we term ``explosive'' as it lasts less than one L/Delta U. Regime II also exhibits significant values of N-independent two-vortex correlations, indicating that current kinetic theories that neglect correlations or consider them as O(1/N) cannot describe this regime. The evolution of the layer thickness in present simulations in Regimes I and II agree with the experimental observations of spatially evolving (3D Navier-Stokes) shear layers. Further, the vorticity-stream-function relations in Regime III are close to those computed in 2D Navier-Stokes temporal shear layers J. Sommeria, C. Staquet, and R. Robert, J. Fluid Mech. 233, 661 (1991)]. These findings suggest the dominance of what may be called the Kelvin-Biot-Savart mechanism in determining the growth of the free shear layer through large-scale momentum and vorticity dispersal.

Fermionic superfluid from a bilayer band insulator in an optical lattice

We propose a model to realize a fermionic superfluid state in an optical lattice circumventing the cooling problem. Our proposal exploits the idea of tuning the interaction in a characteristically low-entropy state, a band insulator in an optical bilayer system, to obtain a superfluid. By performing a detailed analysis of the model including fluctuations and augmented by a variational quantum Monte Carlo calculation of the ground state, we show that the superfluid state obtained has a high transition temperature of the order of the hopping energy. Our system is designed to suppress other competing orders such as a charge density wave. We suggest a laboratory realization of this model via an orthogonally shaken optical lattice bilayer.

Error analysis of discontinuous Galerkin methods for the Stokes problem under minimal regularity

In this article, we analyse several discontinuous Galerkin (DG) methods for the Stokes problem under minimal regularity on the solution. We assume that the velocity u belongs to H-0(1)(Omega)](d) and the pressure p is an element of L-0(2)(Omega). First, we analyse standard DG methods assuming that the right-hand side f belongs to H-1(Omega) boolean AND L-1(Omega)](d). A DG method that is well defined for f belonging to H-1(Omega)](d) is then investigated. The methods under study include stabilized DG methods using equal-order spaces and inf-sup stable ones where the pressure space is one polynomial degree less than the velocity space.