Parametrized actor-critic algorithms for finite-horizon MDPs

Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FHMDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.

Solving MDPs using two-timescale simulated annealing with multiplicative weights

We develop extensions of the Simulated Annealing with Multiplicative Weights (SAMW) algorithm that proposed a method of solution of Finite-Horizon Markov Decision Processes (FH-MDPs). The extensions developed are in three directions: a) Use of the dynamic programming principle in the policy update step of SAMW b) A two-timescale actor-critic algorithm that uses simulated transitions alone, and c) Extending the algorithm to the infinite-horizon discounted-reward scenario. In particular, a) reduces the storage required from exponential to linear in the number of actions per stage-state pair. On the faster timescale, a 'critic' recursion performs policy evaluation while on the slower timescale an 'actor' recursion performs policy improvement using SAMW. We give a proof outlining convergence w.p. 1 and show experimental results on two settings: semiconductor fabrication and flow control in communication networks.

The Ecologic Foundations of Stylistics in Music and in Language

This contribution suggests that it is possible to describe the transformations of musical style in an analogous way to the transformations of style in language, and also that it can be explained how the ‘musics in contact’ behave in an analogous way to the ‘languages in contact’. According to this idea, the ‘evolution’ of styles in music and in language can be identified and studied as dynamic exchanges in ecological niches. It is suggested, also, that the idiolectic-ecolectic, and acrolectic-basilectic relationships in music and language are functions of cycles in several ‘layers’ and ‘rhythms’. The presence of stylistic varieties and influences in music and in language may imply that they are part of major sign systems within a more complex ecological relationship.

MMSE Receiver for Multiuser Interference Cancellation in Uplink OFDMA

In uplink orthogonal frequency division multiple access (OFDMA) systems, multiuser interference (MUI) occurs due to different carrier frequency offsets (CFO) of different users at the receiver. In this paper, we present a minimum mean square error (MMSE) based approach to MUI cancellation in uplink OFDMA. We derive a recursion to approach the MMSE solution. We present a structure-wise and performance-wise comparison of this recursive MMSE solution with a linear PIC receiver as well as other detectors recently proposed in the literature. We show that the proposed recursive MMSE solution encompasses several known detectors in the literature as special cases.

New computational approaches for wrinkled and slack membranes

The static response of thin, wrinkled membranes is studied using both a tension field approximation based on plane stress conditions and a 3D nonlinear elasticityformulation, discretized through 8-noded Cosserat point elements. While the tension field approach only obtains the wrinkled/slack regions and at best a measure of the extent of wrinkliness, the 3D elasticity solution provides, in principle, the deformed shape of a wrinkled/slack membrane. However, since membranes barely resist compression, the discretized and linearized system equations via both the approaches are ill-conditioned and solutions could thus be sensitive to discretizations errors as well as other sources of noises/imperfections. We propose a regularized, pseudo-dynamical recursion scheme that provides a sequence of updates, which are almost insensitive to theregularizing term as well as the time step size used for integrating the pseudo-dynamical form. This is borne out through several numerical examples wherein the relative performance of the proposed recursion scheme vis-a-vis a regularized Newton strategy is compared. The pseudo-time marching strategy, when implemented using 3D Cosserat point elements, also provides a computationally cheaper, numerically accurate and simpler alternative to that using geometrically exact shell theories for computing large deformations of membranes in the presence of wrinkles. (C) 2010 Elsevier Ltd. All rights reserved.

Critical phenomena in polymer solutions: Scaling of the free energy

The thermodynamics of monodisperse solutions of polymers in the neighborhood of the phase separation temperature is studied by means of Wilson’s recursion relation approach, starting from an effective ϕ4 Hamiltonian derived from a continuum model of a many‐chain system in poor solvents. Details of the chain statistics are contained in the coefficients of the field variables ϕ, so that the parameter space of the Hamiltonian includes the temperature, coupling constant, molecular weight, and excluded volume interaction. The recursion relations are solved under a series of simplifying assumptions, providing the scaling forms of the relevant parameters, which are then used to determine the scaling form of the free energy. The free energy, in turn, is used to calculate the other singular thermodynamic properties of the solution. These are characteristically power laws in the reduced temperature and molecular weight, with the temperature exponents being the same as those of the 3d Ising model. The molecular weight exponents are unique to polymer solutions, and the calculated values compare well with the available experimental data.

A simplified kinetic model for oxidative dehydrogenation of ethylbenzene over Pd-NaBr/Al2O3 catalyst

The oxidative dehydrogenation of ethylbenzene is gaining considerable importance in recent years as a promising alternative for styrene production. This vapour phase reaction has been studied over Pd-NaBr/Al2O3 catalyst in the temperature range 623-793 K in a fixed bed reactor. Kinetic analysis of this reaction has been done using a recursion procedure developed in this work from first principles. The advantage of this method is the absence of any restriction on the conversion level as it uses an integrated rate equation. The rate of styrene formation was found to follow a linear relationship with concentration of ethylbenzene and shows a Langmuir type dependence on the concentration of oxygen.

Thermalization of Green functions and quasinormal modes

We develop a new method to study the thermalization of time dependent retarded Green function in conformal field theories holographically dual to thin shell AdS Vaidya space times. The method relies on using the information of all time derivatives of the Green function at the shell and then evolving it for later times. The time derivatives of the Green function at the shell is given in terms of a recursion formula. Using this method we obtain analytic results for short time thermalization of the Green function. We show that the late time behaviour of the Green function is determined by the first quasinormal mode. We then implement the method numerically. As applications of this method we study the thermalization of the retarded time dependent Green function corresponding to a minimally coupled scalar in the AdS 3 and AdS 5 thin Vaidya shells. We see that as expected the late time behaviour is determined by the first quasinormal mode. We apply the method to study the late time behaviour of the shear vector mode in AdS 5 Vaidya shell. At small momentum the corresponding time dependent Green function is expected to relax to equilibrium by the shear hydrodynamic mode. Using this we obtain the universal ratio of the shear viscosity to entropy density from a time dependent process.

FAST AND ACCURATE BILATERAL FILTERING USING GAUSS-POLYNOMIAL DECOMPOSITION

The bilateral filter is a versatile non-linear filter that has found diverse applications in image processing, computer vision, computer graphics, and computational photography. A common form of the filter is the Gaussian bilateral filter in which both the spatial and range kernels are Gaussian. A direct implementation of this filter requires O(sigma(2)) operations per pixel, where sigma is the standard deviation of the spatial Gaussian. In this paper, we propose an accurate approximation algorithm that can cut down the computational complexity to O(1) per pixel for any arbitrary sigma (constant-time implementation). This is based on the observation that the range kernel operates via the translations of a fixed Gaussian over the range space, and that these translated Gaussians can be accurately approximated using the so-called Gauss-polynomials. The overall algorithm emerging from this approximation involves a series of spatial Gaussian filtering, which can be efficiently implemented (in parallel) using separability and recursion. We present some preliminary results to demonstrate that the proposed algorithm compares favorably with some of the existing fast algorithms in terms of speed and accuracy.

A backward particle interpretation of Feynman-Kac formulae

We design a particle interpretation of Feynman-Kac measures on path spaces based on a backward Markovian representation combined with a traditional mean field particle interpretation of the flow of their final time marginals. In contrast to traditional genealogical tree based models, these new particle algorithms can be used to compute normalized additive functionals "on-the-fly" as well as their limiting occupation measures with a given precision degree that does not depend on the final time horizon. We provide uniform convergence results with respect to the time horizon parameter as well as functional central limit theorems and exponential concentration estimates. Our results have important consequences for online parameter estimation for non-linear non-Gaussian state-space models. We show how the forward filtering backward smoothing estimates of additive functionals can be computed using a forward only recursion.

On innate and specific aspects of human language

Mixel Aurnague, Kepa Korta and Jesus M. Larrazabal (eds.)

Some properties of the coefficients of cyclotomic polynomials

An explicit formula is obtained for the coefficients of the cyclotomic polynomial F_n(x), where n is the product of two distinct odd primes. A recursion formula and a lower bound and an improvement of Bang’s upper bound for the coefficients of F_n(x) are also obtained, where n is the product of three distinct primes. The cyclotomic coefficients are also studied when n is the product of four distinct odd primes. A recursion formula and upper bounds for its coefficients are obtained. The last chapter includes a different approach to the cyclotomic coefficients. A connection is obtained between a certain partition function and the cyclotomic coefficients when n is the product of an arbitrary number of distinct odd primes. Finally, an upper bound for the coefficients is derived when n is the product of an arbitrary number of distinct and odd primes.

一种步进扫描投影光刻机承片台不平度检测新技术

提出一种步进扫描投影光刻机承片台不平度检测新技术。在晶圆与承片台存在不同偏移量时,利用线性差分传感器在线测量晶圆上不同点的局部高度;通过建立临时边界条件,以递推法消除晶圆面形影响,并逐行计算出承片台的相对不平度;通过逐行计算的结果递推相邻行之间的高度差,并将该高度差叠加到每一行,以消除临时边界条件的限制,得到处于同一高度上的承片台不平度;将计算的结果作为初始值,根据最小二乘原理,以邻近的四个测量点作为参考,逐步逼近得到承片台的真实不平度。计算机仿真结果验证了该检测方法的正确性,计算结果逐步收敛并逼近真实值

Influence of local geology on earthquake ground motion

As a simplified approach for estimating theoretically the influence of local subsoils upon the ground motion during an earthquake, the problem of an idealized layered system subjected to vertically incident plane body waves was studied. Both the technique of steady-state analysis and the technique of transient analysis have been used to analyze the problem.

In the steady-state analysis, a recursion formula has been derived for obtaining the response of a layered system to sinusoidally steady-state input. Several conclusions are drawn concerning the nature of the amplification spectrum of a nonviscous layered system having its layer stiffnesses increasing with depth. Numerical examples are given to demonstrate the effect of layer parameters on the amplification spectrum of a layered system.

In the transient analysis, two modified shear beam models have been established for obtaining approximately the response of a layered system to earthquake-like excitation. The method of continuous modal analysis was adopted for approximate analysis of the models, with energy dissipation in the layers, if any, taken into account. Numerical examples are given to demonstrate the accuracy of the models and the effect of a layered system in modifying the input motion.

Conditions are established, under which the theory is applicable to predict the influence of local subsoils on the ground motion during an earthquake. To demonstrate the applicability of the models to actual cases, three examples of actually recorded earthquake events are examined. It is concluded that significant modification of the incoming seismic waves, as predicted by the theory, is likely to occur in well defined soft subsoils during an earthquake, provided that certain conditions concerning the nature of the incoming seismic waves are satisfied.

Predictive control of a Boeing 747 aircraft using an FPGA

New embedded predictive control applications call for more eficient ways of solving quadratic programs (QPs) in order to meet demanding real-time, power and cost requirements. A single precision QP-on-a-chip controller is proposed, implemented in afield-programmable gate array (FPGA) with an iterative linear solver at its core. A novel offline scaling procedure is introduced to aid the convergence of the reduced precision solver. The feasibility of the proposed approach is demonstrated with a real-time hardware-in-the-loop (HIL) experimental setup where an ML605 FPGA board controls a nonlinear model of a Boeing 747 aircraft running on a desktop PC through an Ethernet link. Simulations show that the quality of the closed-loop control and accuracy of individual solutions is competitive with a conventional double precision controller solving linear systems using a Riccati recursion. © 2012 IFAC.