An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Electrochemical phase formation involving multicomponents: hemispherical overlap models for varied nucleation and growth conditions

The phenomenological theory of hemispherical growth in the context of phase formation with more than one component is presented. The model discusses in a unified manner both instantaneous and progressive nucleation (at the substrate) as well as arbitrary growth rates (e.g. constant and diffusion controlled growth rates). A generalized version of Avrami ansatz (a mean field description) is used to tackle the ''overlap'' aspects arising from the growing multicentres of the many components involved, observing that the nucleation is confined to the substrate plane only. The time evolution of the total extent of macrogrowth as well as those of the individual components are discussed explicitly for the case of two phases. The asymptotic expressions for macrogrowth are derived. Such analysis depicts a saturation limit (i.e. the maximum extent of growth possible) for the slower growing component and its dependence on the kinetic parameters which, in the electrochemical context, can be controlled through potential. The significance of this model in the context of multicomponent alloy deposition and possible future directions for further development are pointed out.

Exact N-Wave Solutions for the Non-Planar Burgers Equation

An exact representation of N-wave solutions for the non-planar Burgers equation u(t) + uu(x) + 1/2ju/t = 1/2deltau(xx), j = m/n, m < 2n, where m and n are positive integers with no common factors, is given. This solution is asymptotic to the inviscid solution for Absolute value of x < square-root (2Q0 t), where Q0 is a function of the initial lobe area, as lobe Reynolds number tends to infinity, and is also asymptotic to the old age linear solution, as t tends to infinity; the formulae for the lobe Reynolds numbers are shown to have the correct behaviour in these limits. The general results apply to all j = m/n, m < 2n, and are rather involved; explicit results are written out for j = 0, 1, 1/2, 1/3 and 1/4. The case of spherical symmetry j = 2 is found to be 'singular' and the general approach set forth here does not work; an alternative approach for this case gives the large time behaviour in two different time regimes. The results of this study are compared with those of Crighton & Scott (1979).

Weighted subspace methods and spatial smoothing: analysis and comparison

The effect of using a spatially smoothed forward-backward covariance matrix on the performance of weighted eigen-based state space methods/ESPRIT, and weighted MUSIC for direction-of-arrival (DOA) estimation is analyzed. Expressions for the mean-squared error in the estimates of the signal zeros and the DOA estimates, along with some general properties of the estimates and optimal weighting matrices, are derived. A key result is that optimally weighted MUSIC and weighted state-space methods/ESPRIT have identical asymptotic performance. Moreover, by properly choosing the number of subarrays, the performance of unweighted state space methods can be significantly improved. It is also shown that the mean-squared error in the DOA estimates is independent of the exact distribution of the source amplitudes. This results in a unified framework for dealing with DOA estimation using a uniformly spaced linear sensor array and the time series frequency estimation problems.

On the solution of the equation ut+unux+H(x, t, u)=0

We consider the equation u(t) + u(n)u(x) + H(x, t, u) = 0 and derive a transformation relating it to u(t) + u(n)u(x) = 0. Special cases of the equation appearing in applications are discussed. Initial value problems and asymptotic behaviour of the solution are studied.

3-D Warping in Four-Bar Laminated Linkages

This paper deals with the evaluation of the component-laminate load-carrying capacity, i.e., to calculate the loads that cause the failure of the individual layers and the component-laminate as a whole in four-bar mechanism. The component-laminate load-carrying capacity is evaluated using the Tsai-Wu-Hahn failure criterion for various layups. The reserve factor of each ply in the component-laminate is calculated by using the maximum resultant force and the maximum resultant moment occurring at different time steps at the joints of the mechanism. Here, all component bars of the mechanism are made of fiber reinforced laminates and have thin rectangular cross-sections. They could, in general, be pre-twisted and/or possess initial curvature, either by design or by defect. They are linked to each other by means of revolute joints. We restrict ourselves to linear materials with small strains within each elastic body (beam). Each component of the mechanism is modeled as a beam based on geometrically nonlinear 3-D elasticity theory. The component problems are thus split into 2-D analyses of reference beam cross-sections and nonlinear 1-D analyses along the three beam reference curves. For the thin rectangular cross-sections considered here, the 2-D cross-sectional nonlinearity is also overwhelming. This can be perceived from the fact that such sections constitute a limiting case between thin-walled open and closed sections, thus inviting the nonlinear phenomena observed in both. The strong elastic couplings of anisotropic composite laminates complicate the model further. However, a powerful mathematical tool called the Variational Asymptotic Method (VAM) not only enables such a dimensional reduction, but also provides asymptotically correct analytical solutions to the nonlinear cross-sectional analysis. Such closed-form solutions are used here in conjunction with numerical techniques for the rest of the problem to predict more quickly and accurately than would otherwise be possible. Local 3-D stress, strain and displacement fields for representative sections in the component-bars are recovered, based on the stress resultants from the 1-D global beam analysis. A numerical example is presented which illustrates the failure of each component-laminate and the mechanism as a whole.

Low-Complexity Detection and Performance in Multi-Gigabit High Spectral Efficiency Wireless Systems

Recently, we reported a low-complexity likelihood ascent search (LAS) detection algorithm for large MIMO systems with several tens of antennas that can achieve high spectral efficiencies of the order of tens to hundreds of bps/Hz. Through simulations, we showed that this algorithm achieves increasingly near SISO AWGN performance for increasing number of antennas in Lid. Rayleigh fading. However, no bit error performance analysis of the algorithm was reported. In this paper, we extend our work on this low-complexity large MIMO detector in two directions: i) We report an asymptotic bit error probability analysis of the LAS algorithm in the large system limit, where N-t, N-r -> infinity keeping N-t = N-r, where N-t and N-r are the number of transmit and receive antennas, respectively. Specifically, we prove that the error performance of the LAS detector for V-BLAST with 4-QAM in i.i.d. Rayleigh fading converges to that of the maximum-likelihood (ML) detector as N-t, N-r -> infinity keeping N-t = N-r ii) We present simulated BER and nearness to capacity results for V-BLAST as well as high-rate non-orthogonal STBC from Division Algebras (DA), in a more realistic spatially correlated MIMO channel model. Our simulation results show that a) at an uncoded BER of 10(-3), the performance of the LAS detector in decoding 16 x 16 STBC from DA with N-t = = 16 and 16-QAM degrades in spatially correlated fading by about 7 dB compared to that in i.i.d. fading, and 19) with a rate-3/4 outer turbo code and 48 bps/Hz spectral efficiency, the performance degrades by about 6 dB at a coded BER of 10(-4). Our results further show that providing asymmetry in number of antennas such that N-r > N-t keeping the total receiver array length same as that for N-r = N-t, the detector is able to pick up the extra receive diversity thereby significantly improving the BER performance.

Stretch effects extracted from inwardly and outwardly propagating spherical premixed flames

Unsteady propagation of spherical flames, both inward and outward, are studied numerically extensively for single-step reaction and for different Lewis numbers of fuel/oxidizer. The dependence of flame speed ratio (s) and flame temperature ratio are obtained for a range of Lewis numbers and stretch (kappa) values. These results of s versus kappa show that the asymptotic theory by Frankel and Sivashinsky is reasonable for outward propagation. Other theories are unsatisfactory both quantitatively and qualitatively. The stretch effects are much higher for negative stretch than for positive stretch, as also seen in the theory of Frankel and Sivashinsky. The linearity of the flame speed ratio vs stretch relationship is restricted to nondimensional stretch of +/-0.1. It is shown further that the results from cylindrical flames are identical to the spherical flame on flame speed ratio versus nondimensional stretch plot thus confirming the generality of the concept of stretch. The comparison of the variation of (ds/dkappa)kappa=0 with beta(Lc - 1) show an offset between the computed and the asymptotic results of Matalon and Matkowsky. The departure of negative stretch results from this variation is significant. Several earlier experimental results are analysed and set out in the form of s versus kappa plot. Comparison of the results with experiments seem reasonable for negative stretch. The results for positive stretch are satisfactory qualitatively for a few cases. For rich propane-air, there are qualitative differences pointing to the need for full chemistry calculations in the extraction of stretch effects.

Conjectures about phase turbulence in the complex Ginzburg-Landau equation

In the complex Ginzburg-Landau equation, we consider possible ''phase turbulent'' regimes, where asymptotic correlations are controlled by phase fluctuations rather than by topological defects. Conjecturing that the decay of such correlations is governed by the Kardar-Parisi-Zhang (KPZ) model of growing interfaces, we derive the following results: (1) A scaling ansatz implies that equal-time spatial correlations in 1d, 2d, and 3d decay like e(-Ax2 zeta), where A is a nonuniversal constant, and zeta=1/2 in 1d. (2) Temporal correlations decay as exp(-t(2 beta)h(t/L(z))), with the scaling law <(beta)over bar> = <(zeta)over bar>/z, where z = 3/2, 1.58..., and 1.66..., for d = 1,2, and 3 respectively. The scaling function h(y) approaches a constant as y --> 0, and behaves like y(2(beta-<(beta)over bar>)), for large y. If in 3d the associated KPZ model turns out to be in its weak-coupling (''smooth'') phase, then, instead of the above behavior, the CGLE exhibits rotating long-range order whose connected correlations decay like 1/x in space or 1/t(1/2) in time. (3) For system sizes, L, and times t respectively less than a crossover length, L(c), and time, t(c), correlations are governed by the free-field or Edwards-Wilkinson (EW) equation, rather than the KPZ model. In 1d, we find that L(c) is large: L(c) similar to 35,000; for L < L(c) we show numerical evidence for stretched exponential decay of temporal correlations with an exponent consistent with the EW value beta(EW)= 1/4.

Automated Multi-Agent Search Using Centroidal Voronoi Configuration

This paper addresses the problem of automated multiagent search in an unknown environment. Autonomous agents equipped with sensors carry out a search operation in a search space, where the uncertainty, or lack of information about the environment, is known a priori as an uncertainty density distribution function. The agents are deployed in the search space to maximize single step search effectiveness. The centroidal Voronoi configuration, which achieves a locally optimal deployment, forms the basis for the proposed sequential deploy and search strategy. It is shown that with the proposed control law the agent trajectories converge in a globally asymptotic manner to the centroidal Voronoi configuration. Simulation experiments are provided to validate the strategy. Note to Practitioners-In this paper, searching an unknown region to gather information about it is modeled as a problem of using search as a means of reducing information uncertainty about the region. Moreover, multiple automated searchers or agents are used to carry out this operation optimally. This problem has many applications in search and surveillance operations using several autonomous UAVs or mobile robots. The concept of agents converging to the centroid of their Voronoi cells, weighted with the uncertainty density, is used to design a search strategy named as sequential deploy and search. Finally, the performance of the strategy is validated using simulations.

Free convection in power-law fluids near a three-dimensional stagnation point

Numerical results are presented for the free-convection boundary-layer equations of the Ostwald de-Waele non-Newtonian power-law type fluids near a three-dimensional (3-D) stagnation point of attachment on an isothermal surface. The existence of dual solutions that are three-dimensional in nature have been verified by means of a numerical procedure. An asymptotic solution for very large Prandtl numbers has also been derived. Solutions are presented for a range of values of the geometric curvature parameter c, the power-law index n, and the Prandtl number Pr.

Saturation throughput analysis of an input queueing ATM switch with multiclass bursty traffic

In this paper we consider an N x N non-blocking, space division ATM switch with input cell queueing. At each input, the cell arrival process comprises geometrically distributed bursts of consecutive cells for the various outputs. Motivated by the fact that some input links may be connected to metropolitan area networks, and others directly to B-ISDN terminals, we study the situation where there are two classes of inputs with different values of mean burst length. We show that when inputs contend for an output, giving priority to an input with smaller expected burst length yields a saturation throughput larger than if the reverse priority is given. Further, giving priority to less bursty traffic can give better throughput than if all the inputs were occupied by this less bursty traffic. We derive the asymptotic (as N --> infinity) saturation throughputs for each priority class.

Shear-induced enhancement of self-diffusion in interacting colloidal suspensions

We set up the generalized Langevin equations describing coupled single-particle and collective motion in a suspension of interacting colloidal particles in a shear how and use these to show that the measured self-diffusion coefficients in these systems should be strongly dependent on shear rate epsilon. Three regimes are found: (i) an initial const+epsilon(.2), followed by (ii) a large regime of epsilon(.1/2) behavior, crossing over to an asymptotic power-law approach (iii) D-o - const x epsilon(.-1/2) to the Stokes-Einstein value D-o. The shear dependence is isotropic up to very large shear rates and increases with the interparticle interaction strength. Our results provide a straightforward explanation of recent experiments and simulations on sheared colloids.

Stability of the viscous flow of a fluid through a flexible tube

The stability of the Hagen-Poiseuille flow of a Newtonian fluid in a tube of radius R surrounded by an incompressible viscoelastic medium of radius R < r < HR is analysed in the high Reynolds number regime. The dimensionless numbers that affect the fluid flow are the Reynolds number Re = (rho VR/eta), the ratio of the viscosities of the wall and fluid eta(r) = (eta(s)/eta), the ratio of radii H and the dimensionless velocity Gamma = (rho V-2/G)(1/2). Here rho is the density of the fluid, G is the coefficient of elasticity of the wall and V is the maximum fluid velocity at the centre of the tube. In the high Reynolds number regime, an asymptotic expansion in the small parameter epsilon = (1/Re) is employed. In the leading approximation, the viscous effects are neglected and there is a balance between the inertial stresses in the fluid and the elastic stresses in the medium. There are multiple solutions for the leading-order growth rate s((0)), all of which are imaginary, indicating that the fluctuations are neutrally stable, since there is no viscous dissipation of energy or transfer of energy from the mean flow to the fluctuations due to the Reynolds stress. There is an O(epsilon(1/2)) correction to the growth rate, s((1)), due to the presence of a wall layer of thickness epsilon(1/2)R where the viscous stresses are O(epsilon(1/2)) smaller than the inertial stresses. An energy balance analysis indicates that the transfer of energy from the mean flow to the fluctuations due to the Reynolds stress in the wall layer is exactly cancelled by an opposite transfer of equal magnitude due to the deformation work done at the interface, and there is no net transfer from the mean flow to the fluctuations. Consequently, the fluctuations are stabilized by the viscous dissipation in the wall layer, and the real part of s(1) is negative. However, there are certain values of Gamma and wavenumber k where s((1)) = 0. At these points, the wall layer amplitude becomes zero because the tangential velocity boundary condition is identically satisfied by the inviscid flow solution. The real part of the O(epsilon) correction to the growth rate s((2)) turns out to be negative at these points, indicating a small stabilizing effect due to the dissipation in the bulk of the fluid and the wall material. It is found that the minimum value of s((2)) increases proportional to (H-1)(-2) for (H-1) much less than 1 (thickness of wall much less than the tube radius), and decreases proportional to H-4 for H much greater than 1. The damping rate for the inviscid modes is smaller than that for the viscous wall and centre modes in a rigid tube, which have been determined previously using a singular perturbation analysis. Therefore, these are the most unstable modes in the flow through a flexible tube

Gas-phase controlled mass transfer in a foam-bed reactor

Gas-phase controlled absorption of ammonia in foams made of solutions of sulphuric acid has been studied experimentally. Effects of gas-phase concentration of ammonia and type of surfactant on the performance of the foam-bed reactor are investigated. Gas-phase controlled absorption from a spherical bubble is anaylzed using the asymptotic value of Sherwood number (Sh = 6.58), for both negligible as well as significant changes in the volume of the bubble. The experimental data are shown to be in good agreement with the single-stage model of the foam-bed reactor using these asymptotic sub-models, as well as the diffusion-in-sphere analysis available in literature. Influence of effective diffusivity on the time dependence of fractional gas absorption has been found to be unimportant for foam columns with large times of contact. The asymptotic sub-models have been compared and use of the rigid-sphere asymptotic sub-model is recommended for foam columns of practical relevence.