An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Gradient solid electrolytes for thermodynamic measurements: System Na2CO3-Na2SO4

The thermodynamic properties of Na2CO3-Na2SO4 solid solution with hexagonal structure have been measured in the temperature range of 873 to 1073 K, using a composite-gradient solid electrolyte. The cell used can be represented as The composite-gradient solid electrolyte consisted of pure Na2CO3 at one extremity and the solid solution under study at the other, with variation in composition across the electrolyte. A CO2 + O2 + Ar gas mixture was used to fix the chemical potential of sodium at each electrode. The Nernstian response of the cell to changes in partial pressures of CO2 and O2 at the electrodes has been demonstrated. The activity of Na2CO3 in the solid solution was measured by two techniques. In the first method, the electromotive force (emf) of the cell was measured with the same CO2 + O2 + Ar mixture at both electrodes. The resultant emf is directly related to the activity of Na2CO3 at the solid solution electrode. By the second approach, the activity was calculated from the difference in compositions Of CO2 + O2 + Ar mixtures at the two electrodes required to produce a null emf. Both methods gave identical results. The second method is more suitable for gradient solid electrolytes that exhibit significant electronic conduction. The activity of Na2CO3 exhibits positive deviation from Raoult's law. The excess Gibbs' energy of mixing of the solid solution can be represented using a subregular solution model such as the following: DELTAG(E) = X(1 - X)[6500(+/-200)X + 3320(+/-80)(1 - X)J mol-1 where X is the mole fraction of Na2CO3. By combining this information with the phase diagram, mixing properties of the liquid phase are obtained.

An Efficient Simulated Annealing With a Valid Solution Mechanism For TDMA Brodcast Scheduling problem

This paper presents an efficient Simulated Annealing with valid solution mechanism for finding an optimum conflict-free transmission schedule for a broadcast radio network. This is known as a Broadcast Scheduling Problem (BSP) and shown as an NP-complete problem, in earlier studies. Because of this NP-complete nature, earlier studies used genetic algorithms, mean field annealing, neural networks, factor graph and sum product algorithm, and sequential vertex coloring algorithm to obtain the solution. In our study, a valid solution mechanism is included in simulated annealing. Because of this inclusion, we are able to achieve better results even for networks with 100 nodes and 300 links. The results obtained using our methodology is compared with all the other earlier solution methods.

Supersymmetry In The Two-Anyon Problem

We show that the problem of two anyons interacting through a simple harmonic potential or a Coulomb potential is supersymmetric. The supersymmetry operators map a theory described by statistics parameter θ to one described by π+θ. Thus fermions and bosons go into each other, while semions are supersymmetric by themselves. The simple harmonic problem has a Sp(4) symmetry for any value of θ which explains the energy degeneracies.

On the bifurcation in a “modulated” logistic map

We comment on a paper by Luang [On the bifurcation in a ''modulated'' logistic map, Physics Letters A 194(1994) 57]. The numerical evidence given in that paper, for a peculiar type of bifurcation, is shown to be incorrect. The causes of such anomalous results are explained. An accurate bifurcation diagram for the map is also given.

Transport coefficients for the shear dynamo problem at small Reynolds numbers

We build on the formulation developed in S. Sridhar and N. K. Singh J. Fluid Mech. 664, 265 (2010)] and present a theory of the shear dynamo problem for small magnetic and fluid Reynolds numbers, but for arbitrary values of the shear parameter. Specializing to the case of a mean magnetic field that is slowly varying in time, explicit expressions for the transport coefficients alpha(il) and eta(iml) are derived. We prove that when the velocity field is nonhelical, the transport coefficient alpha(il) vanishes. We then consider forced, stochastic dynamics for the incompressible velocity field at low Reynolds number. An exact, explicit solution for the velocity field is derived, and the velocity spectrum tensor is calculated in terms of the Galilean-invariant forcing statistics. We consider forcing statistics that are nonhelical, isotropic, and delta correlated in time, and specialize to the case when the mean field is a function only of the spatial coordinate X-3 and time tau; this reduction is necessary for comparison with the numerical experiments of A. Brandenburg, K. H. Radler, M. Rheinhardt, and P. J. Kapyla Astrophys. J. 676, 740 (2008)]. Explicit expressions are derived for all four components of the magnetic diffusivity tensor eta(ij) (tau). These are used to prove that the shear-current effect cannot be responsible for dynamo action at small Re and Rm, but for all values of the shear parameter.

Theoretical Analysis of the Electromotive Force of a Cell Incorporating a Composition Gradient Solid Electrolyte

New composition gradient solid electrolytes have been designed for application in high temperature solid-state galvanic sensors and in thermodynamic measurements. The functionally gradient electrolyte consists of a solid solution between two or more ionic conductors with a common ion and gradual variation in composition of the other ionic species. Unequal rates of migration of the ions, caused by the presence of the concentration gradient, may result in the development of space charge, manifesting as diffusion potential. Presented is a theoretical analysis of the EMF of cells incorporating gradient solid electrolytes. An analytical expression is derived for diffusion potential, using the thermodynamics of irreversible processes, for different types of concentration gradients and boundary conditions at the electrode/electrolyte interfaces. The diffusion potential of an isothermal cell incorporating these gradient electrolytes becomes negligible if there is only one mobile ion and the transport numbers of the relatively immobile polyionic species and electrons approach zero. The analysis of the EMF of a nonisothermal cell incorporating a composition gradient solid electrolyte indicates that the cell EMF can be expressed in terms of the thermodynamic parameters at the electrodes and the Seebeck coefficient of the gradient electrolyte under standard conditions when the transport number of one of the ions approaches unity.

Novel correlations in two dimensions: Two-body problem

We discuss a many-body Hamiltonian with two- and three-body interactions in two dimensions introduced recently by Murthy, Bhaduri and Sen. Apart from an analysis of some exact solutions in the many-body system, we analyse in detail the two-body problem which is completely solvable. We show that the solution of the two-body problem reduces to solving a known differential equation due to Heun. We show that the two-body spectrum becomes remarkably simple for large interaction strengths and the level structure resembles that of the Landau levels. We also clarify the 'ultraviolet' regularization which is needed to define an inverse-square potential properly and discuss its implications for our model.

On the problem of spurious patterns in neural associative memory models

The problem of spurious patterns in neural associative memory models is discussed, Some suggestions to solve this problem from the literature are reviewed and their inadequacies are pointed out, A solution based on the notion of neural self-interaction with a suitably chosen magnitude is presented for the Hebb learning rule. For an optimal learning rule based on linear programming, asymmetric dilution of synaptic connections is presented as another solution to the problem of spurious patterns, With varying percentages of asymmetric dilution it is demonstrated numerically that this optimal learning rule leads to near total suppression of spurious patterns. For practical usage of neural associative memory networks a combination of the two solutions with the optimal learning rule is recommended to be the best proposition.

Mean first passage time approach to the problem of optimal barrier subdivision for Kramer's escape rate

We consider the effect of subdividing the potential barrier along the reaction coordinate on Kramer's escape rate for a model potential, Using the known supersymmetric potential approach, we show the existence of an optimal number of subdivisions that maximizes the rate, We cast the problem as a mean first passage time problem of a biased random walker and obtain equivalent results, We briefly summarize the results of our investigation on the increase in the escape rate by placing a blow-torch in the unstable part of one of the potential wells. (C) 1999 Elsevier Science B.V. All rights reserved.

The effect of boundaries on the plane Couette flow of granular materials: a bifurcation analysis

The tendency of granular materials in rapid shear flow to form non-uniform structures is well documented in the literature. Through a linear stability analysis of the solution of continuum equations for rapid shear flow of a uniform granular material, performed by Savage (1992) and others subsequently, it has been shown that an infinite plane shearing motion may be unstable in the Lyapunov sense, provided the mean volume fraction of particles is above a critical value. This instability leads to the formation of alternating layers of high and low particle concentrations oriented parallel to the plane of shear. Computer simulations, on the other hand, reveal that non-uniform structures are possible even when the mean volume fraction of particles is small. In the present study, we have examined the structure of fully developed layered solutions, by making use of numerical continuation techniques and bifurcation theory. It is shown that the continuum equations do predict the existence of layered solutions of high amplitude even when the uniform state is linearly stable. An analysis of the effect of bounding walls on the bifurcation structure reveals that the nature of the wall boundary conditions plays a pivotal role in selecting that branch of non-uniform solutions which emerges as the primary branch. This demonstrates unequivocally that the results on the stability of bounded shear how of granular materials presented previously by Wang et al. (1996) are, in general, based on erroneous base states.

The influence of temperature gradient and lowering speed on the melt-solid interface shape of GaxIn1-xSb alloy crystals grown by vertical Bridgman technique

Radially homogeneous bulk alloys of GaxIn1-xSb in the range 0.7 < x < 0.8, have been grown by vertical Bridgman technique. The factors affecting the interface shape during the growth were optimised to achieve zero convexity. From a series of experiments, a critical ratio of the temperature gradient (G) of the furnace at the melting point of the melt composition to the ampoule lowering speed (v) was deduced for attaining the planarity of the melt-solid interface. The studies carried out on directional solidification of Ga0.77In0.23Sb mixed crystals employing planar melt-solid interface exhibited superior quality than those with nonplanar interfaces. The solutions to certain problems encountered during the synthesis and growth of the compound were discussed. (C) 1999 Elsevier Science B.V. All rights reserved.

An inverse problem for the Helmholtz equation involving two semi-infinite fluids

An analytical method is developed for solving an inverse problem for Helmholtz's equation associated with two semi-infinite incompressible fluids of different variable refractive indices, separated by a plane interface. The unknowns of the inverse problem are: (i) the refractive indices of the two fluids, (ii) the ratio of the densities of the two fluids, and (iii) the strength of an acoustic source assumed to be situated at the interface of the two fluids. These are determined from the pressure on the interface produced by the acoustic source. The effect of the surface tension force at the interface is taken into account in this paper. The application of the proposed analytical method to solve the inverse problem is also illustrated with several examples. In particular, exact solutions of two direct problems are first derived using standard classical methods which are then used in our proposed inverse method to recover the unknowns of the corresponding inverse problems. The results are found to be in excellent agreement.