On the Convergence of Stochastic Iterative Dynamic Programming Algorithms

Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments. These algorithms, including the TD(lambda) algorithm of Sutton (1988) and the Q-learning algorithm of Watkins (1989), can be motivated heuristically as approximations to dynamic programming (DP). In this paper we provide a rigorous proof of convergence of these DP-based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem. The theorem establishes a general class of convergent algorithms to which both TD(lambda) and Q-learning belong.

Convergence Rates of Approximation by Translates

In this paper we consider the problem of approximating a function belonging to some funtion space Φ by a linear comination of n translates of a given function G. Ussing a lemma by Jones (1990) and Barron (1991) we show that it is possible to define function spaces and functions G for which the rate of convergence to zero of the erro is 0(1/n) in any number of dimensions. The apparent avoidance of the "curse of dimensionality" is due to the fact that these function spaces are more and more constrained as the dimension increases. Examples include spaces of the Sobolev tpe, in which the number of weak derivatives is required to be larger than the number of dimensions. We give results both for approximation in the L2 norm and in the Lc norm. The interesting feature of these results is that, thanks to the constructive nature of Jones" and Barron"s lemma, an iterative procedure is defined that can achieve this rate.

Bagging Regularizes

Intuitively, we expect that averaging --- or bagging --- different regressors with low correlation should smooth their behavior and be somewhat similar to regularization. In this note we make this intuition precise. Using an almost classical definition of stability, we prove that a certain form of averaging provides generalization bounds with a rate of convergence of the same order as Tikhonov regularization --- similar to fashionable RKHS-based learning algorithms.

Transport of Sub-micron Aerosols in Bifurcations

The convective-diffusive transport of sub-micron aerosols in an oscillatory laminar flow within a 2-D single bifurcation is studied, using order-of-magnitude analysis and numerical simulation using a commercial software (FEMLAB®). Based on the similarity between momentum and mass transfer equations, various transient mass transport regimes are classified and scaled according to Strouhal and beta numbers. Results show that the mass transfer rate is highest at the carinal ridge and there is a phase-shift in diffusive transport time if the beta number is greater than one. It is also shown that diffusive mass transfer becomes independent of the oscillating outer flow if the Strouhal number is greater than one.

The Inertio-Elastic Planar Entry Flow of Low-Viscosity Elastic Fluids in Micro-fabricated Geometries

The non-Newtonian flow of dilute aqueous polyethylene oxide (PEO) solutions through microfabricated planar abrupt contraction-expansions is investigated. The contraction geometries are fabricated from a high-resolution chrome mask and cross-linked PDMS gels using the tools of soft-lithography. The small length scales and high deformation rates in the contraction throat lead to significant extensional flow effects even with dilute polymer solutions having time constants on the order of milliseconds. The dimensionless extra pressure drop across the contraction increases by more than 200% and is accompanied by significant upstream vortex growth. Streak photography and videomicroscopy using epifluorescent particles shows that the flow ultimately becomes unstable and three-dimensional. The moderate Reynolds numbers (0.03 ≤ Re ≤ 44) associated with these high Deborah number (0 ≤ De ≤ 600) microfluidic flows results in the exploration of new regions of the Re-De parameter space in which the effects of both elasticity and inertia can be observed. Understanding such interactions will be increasingly important in microfluidic applications involving complex fluids and can best be interpreted in terms of the elasticity number, El = De/Re, which is independent of the flow kinematics and depends only on the fluid rheology and the characteristic size of the device.