Parametrized actor-critic algorithms for finite-horizon MDPs

Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FHMDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.

The partition of unity finite element method for elastic wave propagation in Reissner-Mindlin plates

This paper reports a numerical method for modelling the elastic wave propagation in plates. The method is based on the partition of unity approach, in which the approximate spectral properties of the infinite dimensional system are embedded within the space of a conventional finite element method through a consistent technique of waveform enrichment. The technique is general, such that it can be applied to the Lagrangian family of finite elements with specific waveform enrichment schemes, depending on the dominant modes of wave propagation in the physical system. A four-noded element for the Reissner-indlin plate is derived in this paper, which is free of shear locking. Such a locking-free property is achieved by removing the transverse displacement degrees of freedom from the element nodal variables and by recovering the same through a line integral and a weak constraint in the frequency domain. As a result, the frequency-dependent stiffness matrix and the mass matrix are obtained, which capture the higher frequency response with even coarse meshes, accurately. The steps involved in the numerical implementation of such element are discussed in details. Numerical studies on the performance of the proposed element are reported by considering a number of cases, which show very good accuracy and low computational cost. Copyright (C)006 John Wiley & Sons, Ltd.

A Fair Contract for Managing Water Scarcity

In public utilities, under supply constraints, fairness considerations lead to a market failure. This paper characterizes a two-period principal-agent contract for demand management, that mitigates this market failure in urban water systems. The contract is designed as an extensive form mechanism using subgame perfect Nash equilibrium (SPNE) as the solution concept. The contract is fair; and is shown to be economically efficient if, in case of deviation by the agent, the gain to the agent and the loss to the principal are small. It is shown that the assumption can be avoided in an infinite horizon contract.

Boundary value problems for third-order nonlinear ordinary differential equations

In this paper, we describe how to analyze boundary value problems for third-order nonlinear ordinary differential equations over an infinite interval. Several physical problems of interest are governed by such systems. The seminumerical schemes described here offer some advantages over solutions obtained by using traditional methods such as finite differences, shooting method, etc. These techniques also reveal the analytic structure of the solution function. For illustrative purposes, several physical problems, mainly drawn from fluid mechanics, are considered; they clearly demonstrate the efficiency of the techniques presented here.

Non local scale effects on ultrasonic wave characteristics nanorods

In this paper, the nonlocal elasticity theory has been incorporated into classical Euler-Bernoulli rod model to capture unique features of the nanorods under the umbrella of continuum mechanics theory. The strong effect of the nonlocal scale has been obtained which leads to substantially different wave behaviors of nanorods from those of macroscopic rods. Nonlocal Euler-Bernoulli bar model is developed for nanorods. Explicit expressions are derived for wavenumbers and wave speeds of nanorods. The analysis shows that the wave characteristics are highly over estimated by the classical rod model, which ignores the effect of small-length scale. The studies also shows that the nonlocal scale parameter introduces certain band gap region in axial wave mode where no wave propagation occurs. This is manifested in the spectrum cures as the region where the wavenumber tends to infinite (or wave speed tends to zero). The results can provide useful guidance for the study and design of the next generation of nanodevices that make use of the wave propagation properties of single-walled carbon nanotubes. (C) 2010 Elsevier B.V. All rights reserved.

Terahertz wave characteristics of a single-walled carbon nanotube containing a fluid flow using the nonlocal Timoshenko beam model

This paper presents the effect of nonlocal scaling parameter on the terahertz wave propagation in fluid filled single walled carbon nanotubes (SWCNTs). The SWCNT is modeled as a Timoshenko beam,including rotary inertia and transverse shear deformation by considering the nonlocal scale effects. A uniform fluid velocity of 1000 m/s is assumed. The analysis shows that, for a fluid filled SWCNT, the wavenumbers of flexural and shear waves will increase and the corresponding wave speeds will decrease as compared to an empty SWCNT. The nonlocal scale parameter introduces certain band gap region in both flexural and shear wave mode where no wave propagation occurs. This is manifested in the wavenumber plots as the region where the wavenumber tends to infinite (or wave speed tends to zero). The frequency at which this phenomenon occurs is called the ``escape frequency''. The effect of fluid density on the terahertz wave propagation in SWCNT is also studied and the analysis shows that as the fluid becomes denser, the wave speeds will decrease. The escape frequency decreases with increase in nonlocal scaling parameter, for both wave modes. We also show that the effect of fluid density and velocity are negligible on the escape frequencies of flexural and shear wave modes. (C) 2010 Elsevier B.V. All rights reserved.

Solving MDPs using two-timescale simulated annealing with multiplicative weights

We develop extensions of the Simulated Annealing with Multiplicative Weights (SAMW) algorithm that proposed a method of solution of Finite-Horizon Markov Decision Processes (FH-MDPs). The extensions developed are in three directions: a) Use of the dynamic programming principle in the policy update step of SAMW b) A two-timescale actor-critic algorithm that uses simulated transitions alone, and c) Extending the algorithm to the infinite-horizon discounted-reward scenario. In particular, a) reduces the storage required from exponential to linear in the number of actions per stage-state pair. On the faster timescale, a 'critic' recursion performs policy evaluation while on the slower timescale an 'actor' recursion performs policy improvement using SAMW. We give a proof outlining convergence w.p. 1 and show experimental results on two settings: semiconductor fabrication and flow control in communication networks.

Solution of MDPS using simulation-based value iteration

This article proposes a three-timescale simulation based algorithm for solution of infinite horizon Markov Decision Processes (MDPs). We assume a finite state space and discounted cost criterion and adopt the value iteration approach. An approximation of the Dynamic Programming operator T is applied to the value function iterates. This 'approximate' operator is implemented using three timescales, the slowest of which updates the value function iterates. On the middle timescale we perform a gradient search over the feasible action set of each state using Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates, thus finding the minimizing action in T. On the fastest timescale, the 'critic' estimates, over which the gradient search is performed, are obtained. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is also presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are performed using the proposed algorithm. The results obtained are verified against classical value iteration where the feasible set is suitably discretized. Over such a discretized setting, a variant of the algorithm of [12] is compared and the proposed algorithm is found to converge faster.

Boundary value problems for third-order nonlinear ordinary differential equations

In this paper, we describe how to analyze boundary value problems for third-order nonlinear ordinary differential equations over an infinite interval. Several physical problems of interest are governed by such systems. The seminumerical schemes described here offer some advantages over solutions obtained by using traditional methods such as finite differences, shooting method, etc. These techniques also reveal the analytic structure of the solution function. For illustrative purposes, several physical problems, mainly drawn from fluid mechanics, are considered; they clearly demonstrate the efficiency of the techniques presented here.

Instability of Coupled Longitudinal and Transverse Electromagnetic Modes of Wave‐Propagation in Bounded Streaming Plasmas

The instability of coupled longitudinal and transverse electromagnetic modes associated with long wavelengths is studied in bounded streaming plasmas. The main conclusions are as follows: (i) For long waves for which O (k 2)=0, in the absence of relative streaming motion of electrons and ions and aωp/c<0.66, the whole spectrum of harmonic waves is excited due to finite temperature and boundary effects consisting of two subseries. One of these subseries can be identified with Tonks-Dattner resonance oscillations for the electrons, and arises primarily due to the electrons with frequencies greater than the electrostatic plasma frequency corresponding to the electron density in the midplane in the undisturbed state. The other series arises primarily due to ion motion. When aωp/c>0.66, in addition to the above spectrum of harmonic waves, the system admits an infinite number of growing and decaying waves. The instability associated with these modes is found to arise due to the interaction of the waves inside the plasma with the external electromagnetic field. (ii) For modes with comparatively shorter wavelengths for which O (k3)=0, the coupling due to finite temperature sets in, and it is found that the two series of harmonic waves obtained in (i) deriving energy from the transverse modes also become unstable. Thus, for these wavelengths the system admits three sets of growing and decaying modes, first two for all values of aωp/c and the third for (aωp/c) > 0.66. (iii) The presence of streaming velocities introduces various other coupling mechanisms, and we find that even for the wavelengths for which O (k2)=0, we get three sets of growing and decaying waves. The numerical values for the growth rates show that the streaming velocities enhance the growth rates of instability significantly.

Non-Linear vibration of an elastic string

Equations proposed in previous work on the non-linear motion of a string show a basic disagreement, which is here traced to an assumption about the longitudinal displacement u. It is shown that it is neither necessary nor justifiable to assume that u is zero; and also that the velocity of propagation of u disturbances in a string is different from that in an infinite medium, although this difference is usually negligible. After formulating the exact equations of motion for the string, a systematic procedure is described for obtaining approximations to these equations to any order, making only the assumption that the strain in the material of the string is small. The lowest order equations in this scheme are non-linear, and are used to describe the response of a string near resonance. Finally, it is shown that in the absence of damping, planar motion of a string is always unstable at sufficiently high amplitudes, the critical amplitude falling to zero at the natural frequency and its subharmonics. The effect of slight damping on this instability is also discussed.

Analysis of continuous beams—a three dimensional elasticity solution

A three dimensional elasticity solution for the analysis of beams continuous over an infinite number of equally spaced supports has been given. The beam has been subjected to normal tractions on its two opposite faces and these loads are identical over each span. The other two faces are traction free. Numerical results have been given for different cases when the beam is loaded on its bottom face. The results obtained have been compared with the results of two dimensional elasticity solution.

Cubic and ultimate groups of complete symmetry

It is shown that the systems of definite actions described by polar and axial tensors of the second rank and their combinations during the superposition of their elements of complete symmetry with the elements of complete symmetry of the "grey" cube, result in 11 cubic crystallographical groups of complete symmetry. There are 35 ultimate groups (i.e., the groups having the axes of symmetry of infinite order) in complete symmetry of finite figures. 14 out of these groups are ultimate groups of symmetry of polar and axial tensors of the second rank and 24 are new groups. All these 24 ultimate groups are conventional groups since they cannot be presented by certain finite figures possessing the axes of symmetry {Mathematical expression}. Geometrical interpretation for some of the groups of complete symmetry is given. The connection between complete symmetry and physical properties of the crystals (electrical, magnetic and optical) is shown.

Bending, vibration and buckling of simply supported thick orthotropic rectangular plates and laminates

This paper presents a unified exact analysis for the statics and dynamics of a class of thick laminates. A three-dimensional, linear, small deformation theory of elasticity solution is developed for the bending, vibration and buckling of simply supported thick orthotropic rectangular plates and laminates. All the nine elastic constants of orthotropy are taken into account. The solution is formally exact and leads to simple infinite series for stresses and displacements in flexure, forced vibration and "beam-column" type problems and to closed form characteristic equations for free vibration and buckling problems. For free vibration of plates, the present analysis yields a triply infinite spectrum of frequencies instead of only one doubly infinite spectrum by thin plate theory or three doubly infinite spectra by Reissner-Mindlin type analyses. Some numerical results are presented for plates and laminates. Comparison of results from thin plate, Reissner and Mindlin analyses with these yield some important conclusions regarding the validity and effects of the assumptions made in the approximate theories.

An exact analysis for vibration of simply-supported homogeneous and laminated thick rectangular plates

A three-dimensional linear, small deformation theory of elasticity solution by the direct method is developed for the free vibration of simply-supported, homogeneous, isotropic, thick rectangular plates. The solution is exact and involves determining a triply infinite sequence of eigenvalues from a doubly infinite set of closed form transcendental equations. As no restrictions are placed on the thickness variation of stresses or displacements, this formulation yields a triply infinite spectrum of frequencies, instead of only one doubly infinite spectrum by thin plate theory and three doubly infinite spectra by Mindlin's thick plate theory. Further, the present analysis yields symmetric thickness modes which neither of the approximate theories can identify. Some numerical results from the two approximate theories are compared with those from the present solution and some important conclusions regarding the effect of the assumptions made in the approximate theories are drawn. The thickness variations of stresses and displacements are also discussed. The analysis is readily extended for laminated plates of isotropic materials. Numerical results are also given for three-ply laminates, and are used to assess the accuracy of thin plate theory predictions for laminates. Extension to general lateral surface conditions and forced vibrations is indicated.