An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Toeplitz Operators with Special Symbols on Segal-Bargmann Spaces

We study the boundedness of Toeplitz operators on Segal-Bargmann spaces in various contexts. Using Gutzmer's formula as the main tool we identify symbols for which the Toeplitz operators correspond to Fourier multipliers on the underlying groups. The spaces considered include Fock spaces, Hermite and twisted Bergman spaces and Segal-Bargmann spaces associated to Riemannian symmetric spaces of compact type.

Nonlinear analysis of adhesively bonded lap joints considering viscoplasticity in adhesives

This paper presents nonlinear finite element analysis of adhesively bonded joints considering the elastoviscoplastic constitutive model of the adhesive material and the finite rotation of the joint. Though the adherends have been assumed to be linearly elastic, the yielding of the adhesive is represented by a pressure sensitive modified von Mises yield function. The stress-strain relation of the adhesive is represented by the Ramberg-Osgood relation. Geometric nonlinearity due to finite rotation in the joint is accounted for using the Green-Lagrange strain tensor and the second Piola-Kirchhoff stress tensor in a total Lagrangian formulation. Critical time steps have been calculated based on the eigenvalues of the transition matrices of the viscoplastic model of the adhesive. Stability of the viscoplastic solution and time dependent behaviour of the joints are examined. A parametric study has been carried out with particular reference to peel and shear stress along the interface. Critical zones for failure of joints have been identified. The study is of significance in the design of lap joints as well as on the characterization of adhesive strength. (C) 1999 Elsevier Science Ltd. All rights reserved.

Tatonnement Mechanisms for Combinatorial Exchanges

Combinatorial exchanges are double sided marketplaces with multiple sellers and multiple buyers trading with the help of combinatorial bids. The allocation and other associated problems in such exchanges are known to be among the hardest to solve among all economic mechanisms. It has been shown that the problems of surplus maximization or volume maximization in combinatorial exchanges are inapproximable even with free disposal. In this paper, the surplus maximization problem is formulated as an integer linear programming problem and we propose a Lagrangian relaxation based heuristic to find a near optimal solution. We develop computationally efficient tâtonnement mechanisms for clearing combinatorial exchanges where the Lagrangian multipliers can be interpreted as the prices of the items set by the exchange in each iteration. Our mechanisms satisfy Individual-rationality and Budget-nonnegativity properties. The computational experiments performed on representative data sets show that the proposed heuristic produces a feasible solution with negligible optimality gap.

An interior penalty method for a sixth-order elliptic equation

We derive and study a C(0) interior penalty method for a sixth-order elliptic equation on polygonal domains. The method uses the cubic Lagrange finite-element space, which is simple to implement and is readily available in commercial software. After introducing some notation and preliminary results, we provide a detailed derivation of the method. We then prove the well-posedness of the method as well as derive quasi-optimal error estimates in the energy norm. The proof is based on replacing Galerkin orthogonality with a posteriori analysis techniques. Using this approach, we are able to obtain a Cea-like lemma with minimal regularity assumptions on the solution. Numerical experiments are presented that support the theoretical findings.

Microwave cavity resonators. Some perturbation effects and their applications

Lagrange's equation is utilized to show the analogy of a lossless microwave cavity resonator with the conventional LC network. A brief discussion on the resonant frequencies of a microwave cavity resonator and the two degenerate companion modes H01 and E11 appearing in a cavity is given. The first order perturbation theory of a small deformation of the wall of a cavity is discussed. The effects of perturbation, such as the change in the resonant frequency and the Q of a cavity, the change in the electromagnetic field configurations and hence mixing of modes are also discussed. An expression for the coupling coefficient between the two degenerate modes H01 and E11 is derived with the help of the field equations. Results indicate that in the absence of perturbation the above two degenerate modes can co-exist without losing their individual identities. Several applications of the perturbation theory, such as the measurement of the dielectric properties of matter, study of ferromagnetic resonance, etc., are described.

Energy absorption behaviours of CSM-based GFRC plates with hemispherical features

In the present study, the mechanical behaviour of CSM (chopped strand mat)-based GFRC (glass fibre-reinforced composite) plates with single and multiple hemispheres under compressive loads has been investigated both experimentally and numerically. The basic stress-strain behaviours arc identified with quasi-static tests on two-ply coupon laminates and short cylinders, and these are followed up with compressive tests in a UTM (universal testing machine) on single- and multiple-hemisphere plates. The ability of an explicit LS-DYNA solver in predicting the complex material behaviour of composite hemispheres, including failure, is demonstrated. The relevance and scalability of the present class of structural components as `force-multipliers' and `energy-multipliers' have been justified by virtue of findings that as the number of hemispheres in a panel increased from one to four, peak load and average absorbed energy rose by factors of approximately four and six, respectively. The performance of a composite hemisphere has been compared to similar-sized steel and aluminium hemispheres, and the former is found to be of distinctly higher specific energy than the steel specimen. A simulation-based study has also been carried out on a composite 2 x 2-hemisphere panel under impact loads and its behaviour approaching that of an ideal energy absorber has been predicted. In summary, the present investigation has established the efficacy of composite plates with hemispherical force multipliers as potential energy-absorbing countermeasures and the suitability of CAE (computer-aided engineering) for their design.

An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

Revisiting Riesz transforms on Heisenberg groups

We characterise higher order Riesz transforms on the Heisenberg group and also show that they satisfy dimension-free bounds under some assumptions on the multipliers. Using transference theorems, we deduce boundedness theorems for Riesz transforms on the reduced Heisenberg group and hence also for the Riesz transforms associated to multiple Hermite and Laguerre expansions.

The defect sequence for contractive tuples

We introduce the defect sequence for a contractive tuple of Hilbert space operators and investigate its properties. The defect sequence is a sequence of numbers, called defect dimensions associated with a contractive tuple. We show that there are upper bounds for the defect dimensions. The tuples for which these upper bounds are obtained, are called maximal contractive tuples. The upper bounds are different in the non-commutative and in the commutative case. We show that the creation operators on the full Fock space and the coordinate multipliers on the Drury-Arveson space are maximal. We also study pure tuples and see how the defect dimensions play a role in their irreducibility. (C) 2012 Elsevier Inc. All rights reserved.

Queueing delay - error probability tradeoff for point-to-point channels with fixed length block codes

We study the tradeoff between the average error probability and the average queueing delay of messages which randomly arrive to the transmitter of a point-to-point discrete memoryless channel that uses variable rate fixed codeword length random coding. Bounds to the exponential decay rate of the average error probability with average queueing delay in the regime of large average delay are obtained. Upper and lower bounds to the optimal average delay for a given average error probability constraint are presented. We then formulate a constrained Markov decision problem for characterizing the rate of transmission as a function of queue size given an average error probability constraint. Using a Lagrange multiplier the constrained Markov decision problem is then converted to a problem of minimizing the average cost for a Markov decision problem. A simple heuristic policy is proposed which approximately achieves the optimal average cost.

An Outward-Wave-Favouring Finite Element-Based Strategy for Exterior Acoustical Problems

This work presents a finite element-based strategy for exterior acoustical problems based on an assumed pressure form that favours outgoing waves. The resulting governing equation, weak formulation, and finite element formulation are developed both for coupled and uncoupled problems. The developed elements are very similar to conventional elements in that they are based on the standard Galerkin variational formulation and use standard Lagrange interpolation functions and standard Gaussian quadrature. In addition and in contrast to wave envelope formulations and their extensions, the developed elements can be used in the immediate vicinity of the radiator/scatterer. The method is similar to the perfectly matched layer (PML) method in the sense that each layer of elements added around the radiator absorbs acoustical waves so that no boundary condition needs to be applied at the outermost boundary where the domain is truncated. By comparing against strategies such as the PML and wave-envelope methods, we show that the relative accuracy, both in the near and far-field results, is considerably higher.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

A POSTERIORI ERROR CONTROL OF DISCONTINUOUS GALERKIN METHODS FOR ELLIPTIC OBSTACLE PROBLEMS

In this article, we derive an a posteriori error estimator for various discontinuous Galerkin (DG) methods that are proposed in (Wang, Han and Cheng, SIAM J. Numer. Anal., 48: 708-733, 2010) for an elliptic obstacle problem. Using a key property of DG methods, we perform the analysis in a general framework. The error estimator we have obtained for DG methods is comparable with the estimator for the conforming Galerkin (CG) finite element method. In the analysis, we construct a non-linear smoothing function mapping DG finite element space to CG finite element space and use it as a key tool. The error estimator consists of a discrete Lagrange multiplier associated with the obstacle constraint. It is shown for non-over-penalized DG methods that the discrete Lagrange multiplier is uniformly stable on non-uniform meshes. Finally, numerical results demonstrating the performance of the error estimator are presented.

Generalized Model Predictive Static Programming and Angle-Constrained Guidance of Air-to-Ground Missiles

A new generalized model predictive static programming technique is presented for rapidly solving a class of finite-horizon nonlinear optimal control problems with hard terminal constraints. Two key features for its high computational efficiency include one-time backward integration of a small-dimensional weighting matrix dynamics, followed bya static optimization formulation that requires only a static Lagrange multiplier to update the control history. It turns out that under Euler integration and rectangular approximation of finite integrals it is equivalent to the existing model predictive static programming technique. In addition to the benchmark double integrator problem, usefulness of the proposed technique is demonstrated by solving a three-dimensional angle-constrained guidance problem for an air-to-ground missile, which demands that the missile must meet constraints on both azimuth and elevation angles at the impact point in addition to achieving near-zero miss distance, while minimizing the lateral acceleration demand throughout its flight path. Simulation studies include maneuvering ground targets along with a first-order autopilot lag. Comparison studies with classical augmented proportional navigation guidance and modern general explicit guidance lead to the conclusion that the proposed guidance is superior to both and has a larger capture region as well.