Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Learning the kernel matrix with semidefinite programming

Kernel-based learning algorithms work by embedding the data into a Euclidean space, and then searching for linear relations among the embedded data points. The embedding is performed implicitly, by specifying the inner products between each pair of points in the embedding space. This information is contained in the so-called kernel matrix, a symmetric and positive semidefinite matrix that encodes the relative positions of all points. Specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space - classical model selection problems in machine learning. In this paper we show how the kernel matrix can be learned from data via semidefinite programming (SDP) techniques. When applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm -using the labeled part of the data one can learn an embedding also for the unlabeled part. The similarity between test points is inferred from training points and their labels. Importantly, these learning problems are convex, so we obtain a method for learning both the model class and the function without local minima. Furthermore, this approach leads directly to a convex method for learning the 2-norm soft margin parameter in support vector machines, solving an important open problem.

Active transonic aerofoil design optimization using robust multiobjective evolutionary algorithms

The use of adaptive wing/aerofoil designs is being considered, as they are promising techniques in aeronautic/ aerospace since they can reduce aircraft emissions and improve aerodynamic performance of manned or unmanned aircraft. This paper investigates the robust design and optimization for one type of adaptive techniques: active flow control bump at transonic flow conditions on a natural laminar flow aerofoil. The concept of using shock control bump is to control supersonic flow on the suction/pressure side of natural laminar flow aerofoil that leads to delaying shock occurrence (weakening its strength) or boundary layer separation. Such an active flow control technique reduces total drag at transonic speeds due to reduction of wave drag. The location of boundary-layer transition can influence the position and structure of the supersonic shock on the suction/pressure side of aerofoil. The boundarylayer transition position is considered as an uncertainty design parameter in aerodynamic design due to the many factors, such as surface contamination or surface erosion. This paper studies the shock-control-bump shape design optimization using robust evolutionary algorithms with uncertainty in boundary-layer transition locations. The optimization method is based on a canonical evolution strategy and incorporates the concepts of hierarchical topology, parallel computing, and asynchronous evaluation. The use of adaptive wing/aerofoil designs is being considered, as they are promising techniques in aeronautic/ aerospace since they can reduce aircraft emissions and improve aerodynamic performance of manned or unmanned aircraft. This paper investigates the robust design and optimization for one type of adaptive techniques: active flow control bump at transonic flow conditions on a natural laminar flow aerofoil. The concept of using shock control bump is to control supersonic flow on the suction/pressure side of natural laminar flow aerofoil that leads to delaying shock occurrence (weakening its strength) or boundary-layer separation. Such an active flow control technique reduces total drag at transonic speeds due to reduction of wave drag. The location of boundary-layer transition can influence the position and structure of the supersonic shock on the suction/pressure side of aerofoil. The boundarylayer transition position is considered as an uncertainty design parameter in aerodynamic design due to the many factors, such as surface contamination or surface erosion. This paper studies the shock-control-bump shape design optimization using robust evolutionary algorithms with uncertainty in boundary-layer transition locations. The optimization method is based on a canonical evolution strategy and incorporates the concepts of hierarchical topology, parallel computing, and asynchronous evaluation. Two test cases are conducted: the first test assumes the boundary-layer transition position is at 45% of chord from the leading edge, and the second test considers robust design optimization for the shock control bump at the variability of boundary-layer transition positions. The numerical result shows that the optimization method coupled to uncertainty design techniques produces Pareto optimal shock-control-bump shapes, which have low sensitivity and high aerodynamic performance while having significant total drag reduction.

Learning the Kernel Matrix with Semi-Definite Programming

Kernel-based learning algorithms work by embedding the data into a Euclidean space, and then searching for linear relations among the embedded data points. The embedding is performed implicitly, by specifying the inner products between each pair of points in the embedding space. This information is contained in the so-called kernel matrix, a symmetric and positive definite matrix that encodes the relative positions of all points. Specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space -- classical model selection problems in machine learning. In this paper we show how the kernel matrix can be learned from data via semi-definite programming (SDP) techniques. When applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm -- using the labelled part of the data one can learn an embedding also for the unlabelled part. The similarity between test points is inferred from training points and their labels. Importantly, these learning problems are convex, so we obtain a method for learning both the model class and the function without local minima. Furthermore, this approach leads directly to a convex method to learn the 2-norm soft margin parameter in support vector machines, solving another important open problem. Finally, the novel approach presented in the paper is supported by positive empirical results.

Robust designs for Poisson regression models

We consider the problem of how to construct robust designs for Poisson regression models. An analytical expression is derived for robust designs for first-order Poisson regression models where uncertainty exists in the prior parameter estimates. Given certain constraints in the methodology, it may be necessary to extend the robust designs for implementation in practical experiments. With these extensions, our methodology constructs designs which perform similarly, in terms of estimation, to current techniques, and offers the solution in a more timely manner. We further apply this analytic result to cases where uncertainty exists in the linear predictor. The application of this methodology to practical design problems such as screening experiments is explored. Given the minimal prior knowledge that is usually available when conducting such experiments, it is recommended to derive designs robust across a variety of systems. However, incorporating such uncertainty into the design process can be a computationally intense exercise. Hence, our analytic approach is explored as an alternative.

Radiation effects on natural convection laminar flow from a horizontal circular cylinder

The effect of radiation on natural convection flow from an isothermal circular cylinder has been investigated numerically in this study. The governing boundary layer equations of motion are transformed into a non-dimensional form and the resulting nonlinear systems of partial differential equations are reduced to convenient boundary layer equations, which are then solved numerically by two distinct efficient methods namely: (i) implicit finite differencemethod or the Keller-Box Method (KBM) and (ii) Straight Forward Finite Difference Method (SFFD). Numerical results are presented by velocity and temperature distribution of the fluid as well as heat transfer characteristics, namely the shearing stress and the local heat transfer rate in terms of the local skin-friction coefficient and the local Nusselt number for a wide range of surface heating parameter and radiation-conduction parameter. Due to the effects of the radiation the skin-friction coefficients as well as the rate of heat transfer increased and consequently the momentum and thermal boundary layer thickness enhanced.

Radiation effect on free convection laminar flow along a vertical flat plate with streamwise sinusoidal surface temperature

The effect of thermal radiation on a steady two-dimensional natural convection laminar flow of viscous incompressible optically thick fluid along a vertical flat plate with streamwise sinusoidal surface temperature has been investigated in this study. Using the appropriate variables; the basic governing equations are transformed to convenient form and then solved numerically employing two efficient methods, namely, Implicit finite difference method (IFD) together with Keller box scheme and Straight forward finite difference (SFFD) method. Effects of the variation of the physical parameters, for example, conduction-radiation parameter (Planck number), surface temperature parameter, and the amplitude of the surface temperature, are shown on the skin friction and heat transfer rate quantitatively are shown numerically. Velocity and temperature profiles as well as streamlines and isotherms are also presented and discussed for the variation of conduction-radiation parameter. It is found that both skin-friction and rate of heat transfer are enhanced considerably by increasing the values of conduction radiation parameter, Rd.

The effect of temperature dependent viscosity on MHD natural convection flow from an isothermal sphere

Laminar magnetohydrodynamic (MHD) natural convection flow from an isothermal sphere immersed in a fluid with viscosity proportional to linear function of temperature has been studied. The governing boundary layer equations are transformed into a non-dimensional form and the resulting nonlinear system of partial differential equations are reduced to convenient form which are solved numerically by two very efficient methods, namely, (i) Implicit finite difference method together with Keller box scheme and (ii) Direct numerical scheme. Numerical results are presented by velocity and temperature distribution, streamlines and isotherms of the fluid as well as heat transfer characteristics, namely the local skin-friction coefficients and the local heat transfer rate for a wide range of magnetohydrodynamic paramagnet and viscosity-variation parameter.

Natural convection from a plane vertical porous surface in non-isothermal surroundings

In this paper, laminar natural convection flow from a permeable and isothermal vertical surface placed in non-isothermal surroundings is considered. Introducing appropriate transformations into the boundary layer equations governing the flow derives non-similar boundary layer equations. Results of both the analytical and numerical solutions are then presented in the form of skin-friction and Nusselt number. Numerical solutions of the transformed non-similar boundary layer equations are obtained by three distinct solution methods, (i) the perturbation solutions for small � (ii) the asymptotic solution for large � (iii) the implicit finite difference method for all � where � is the transpiration parameter. Perturbation solutions for small and large values of � are compared with the finite difference solutions for different values of pertinent parameters, namely, the Prandtl number Pr, and the ambient temperature gradient n.

Scaling of thermo-magnetic convection

The scaling to characterize unsteady boundary layer development for thermo-magnetic convection of paramagnetic fluids with the Prandtl number greater than one is developed. Under the consideration is a square cavity with initially quiescent isothermal fluid placed in microgravity condition (g = 0) and subject to a uniform, vertical gradient magnetic field. A distinct magnetic thermal-boundary layer is produced by sudden imposing of a higher temperature on the vertical sidewall and as an effect of magnetic body force generated on paramagnetic fluid. The transient flow behavior of the resulting boundary layer is shown to be described by three stages: the start-up stage, the transitional stage and the steady state. The scaling is verified by numerical simulations with the magnetic momentum parameter m variation and the parameter γRa variation.

Natural convection adjacent to an inclined flat plate and in an attic space under various thermal forcing conditions

The effect of viscous dissipation on natural convection from a vertical plate placed in a thermally stratified environment has been investigated numerically. The reduced equations are integrated by employing the implicit finite difference scheme or Ke1ler-box method and obtained the effect of heat due to viscous dissipation on the local skin-friction and loca1 Nusselt number at various stratification levels, for fluids having Prandtl number equals 10, 50, and 100. Solutions are also obtained using the perturbation technique for small values of viscous dissipation parameters and compared with the Finite Difference solutions. Effect of the heat transfer due to viscous dissipation and the temperature stratification are also shown on the velocity and temperature distributions in the boundary layer region. A numerical study of laminar doubly diffusive free convection flows adjacent to a vertical surface in a stable thermally stratified medium is also considered for this study. Solutions are obtained using the implicit Finite Difference method and compared with the local non-similarity method. The velocity and temperature distributions for different values of stratification parameter are shown graphically. The results show many interesting aspects of complex interaction of the two buoyant mechanisms.

Nanowires of metal oxides for gas sensing applications

Nanowires of different metal oxides (SnO2, ZnO) have been grown by evaporation-condensation process. Their chemical composition has been investigated by using XPS. The standard XPS quantification through main photoelectron peaks, modified Auger parameter and valence band spectra were examined for the accurate determination of oxidation state of metals in the nanowires. Morphological investigation has been conducted by acquiring and analyzing the SEM images. For the simulation of working conditions of sensor, the samples were annealed in ultra high vacuum (UHV) up to 500°C and XPS analysis repeated after this treatment. Finally, the nanowires of SnO 2 have were used to produce a novel gas sensor based on Pt/oxide/SiC structure and operating as Schottky diode. Copyright © 2008 John Wiley & Sons, Ltd.

Stochastic simulation of chemical reactions in spatially complex media

Discrete stochastic simulations, via techniques such as the Stochastic Simulation Algorithm (SSA) are a powerful tool for understanding the dynamics of chemical kinetics when there are low numbers of certain molecular species. However, an important constraint is the assumption of well-mixedness and homogeneity. In this paper, we show how to use Monte Carlo simulations to estimate an anomalous diffusion parameter that encapsulates the crowdedness of the spatial environment. We then use this parameter to replace the rate constants of bimolecular reactions by a time-dependent power law to produce an SSA valid in cases where anomalous diffusion occurs or the system is not well-mixed (ASSA). Simulations then show that ASSA can successfully predict the temporal dynamics of chemical kinetics in a spatially constrained environment.

Propagation of low power low divergence Gaussian fields in unbiased self-defocusing photorefractive media and their interactions

Many optical networks are limited in speed and processing capability due to the necessity for the optical signal to be converted to an electrical signal and back again. In addition, electronically manipulated interconnects in an otherwise optical network lead to overly complicated systems. Optical spatial solitons are optical beams that propagate without spatial divergence. They are capable of phase dependent interactions, and have therefore been extensively researched as suitable all optical interconnects for over 20 years. However, they require additional external components, initially high voltage power sources were required, several years later, high power background illumination had replaced the high voltage. However, these additional components have always remained as the greatest hurdle in realising the applications of the interactions of spatial optical solitons as all optical interconnects. Recently however, self-focusing was observed in an otherwise self-defocusing photorefractive crystal. This observation raises the possibility of the formation of soliton-like fields in unbiased self-defocusing media, without the need for an applied electrical field or background illumination. This thesis will present an examination of the possibility of the formation of soliton-like low divergence fields in unbiased self-defocusing photorefractive media. The optimal incident beam and photorefractive media parameters for the formation of these fields will be presented, together with an analytical and numerical study of the effect of these parameters. In addition, preliminary examination of the interactions of two of these fields will be presented. In order to complete an analytical examination of the field propagating through the photorefractive medium, the spatial profile of the beam after propagation through the medium was determined. For a low power solution, it was found that an incident Gaussian field maintains its Gaussian profile as it propagates. This allowed the beam at all times to be described by an individual complex beam parameter, while also allowing simple analytical solutions to the appropriate wave equation. An analytical model was developed to describe the effect of the photorefractive medium on the Gaussian beam. Using this model, expressions for the required intensity dependent change in both the real and imaginary components of the refractive index were found. Numerical investigation showed that under certain conditions, a low powered Gaussian field could propagate in self-defocusing photorefractive media with divergence of approximately 0.1 % per metre. An investigation into the parameters of a Ce:BaTiO3 crystal showed that the intensity dependent absorption is wavelength dependent, and can in fact transition to intensity dependent transparency. Thus, with careful wavelength selection, the required intensity dependent change in both the real and imaginary components of the refractive index for the formation of a low divergence Gaussian field are physically realisable. A theoretical model incorporating the dependence of the change in real and imaginary components of the refractive index on propagation distance was developed. Analytical and numerical results from this model are congruent with the results from the previous model, showing low divergence fields with divergence less than 0.003 % over the propagation length of the photorefractive medium. In addition, this approach also confirmed the previously mentioned self-focusing effect of the self-defocusing media, and provided an analogy to a negative index GRIN lens with an intensity dependent focal length. Experimental results supported the findings of the numerical analysis. Two low divergence fields were found to possess the ability to interact in a Ce:BaTiO3 crystal in a soliton-like fashion. The strength of these interactions was found to be dependent on the degree of divergence of the individual beams. This research found that low-divergence fields are possible in unbiased self-defocusing photorefractive media, and that soliton-like interactions between two of these fields are possible. However, in order for these types of fields to be used in future all optical interconnects, the manipulation of these interactions, together with the ability for these fields to guide a second beam at a different wavelength, must be investigated.