Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

An infinite-horizon robust filter for uncertain hidden Markov Models with conditional relative entropy constraints

We consider a robust filtering problem for uncertain discrete-time, homogeneous, first-order, finite-state hidden Markov models (HMMs). The class of uncertain HMMs considered is described by a conditional relative entropy constraint on measures perturbed from a nominal regular conditional probability distribution given the previous posterior state distribution and the latest measurement. Under this class of perturbations, a robust infinite horizon filtering problem is first formulated as a constrained optimization problem before being transformed via variational results into an unconstrained optimization problem; the latter can be elegantly solved using a risk-sensitive information-state based filtering.

Intuitionistic fuzzy Choquet integral operator-based approach for black-start decision-making

Determining the optimal of black-start strategies is very important for speeding the restoration speed of a power system after a global blackout. Most existing black-start decision-making methods are based on the assumption that all indexes are independent of each other, and little attention has been paid to the group decision-making method which is more reliable. Given this background, the intuitionistic fuzzy set and further intuitionistic fuzzy Choquet integral operator are presented, and a black-start decision-making method based on this integral operator is presented. Compared to existing methods, the proposed algorithm cannot only deal with the relevance among the indexes, but also overcome some shortcomings of the existing methods. Finally, an example is used to demonstrate the proposed method. © 2012 The Institution of Engineering and Technology.

Prandtl number scaling of the unsteady natural convection boundary layer adjacent to a vertical flat plate for Pr > 1 subject to ramp surface heat flux

It is found in the literature that the existing scaling results for the boundary layer thickness, velocity and steady state time for the natural convection flow over an evenly heated plate provide a very poor prediction of the Prandtl number dependency of the flow. However, those scalings provide a good prediction of two other governing parameters’ dependency, the Rayleigh number and the aspect ratio. Therefore, an improved scaling analysis using a triple-layer integral approach and direct numerical simulations have been performed for the natural convection boundary layer along a semi-infinite flat plate with uniform surface heat flux. This heat flux is a ramp function of time, where the temperature gradient on the surface increases with time up to some specific time and then remains constant. The growth of the boundary layer strongly depends on the ramp time. If the ramp time is sufficiently long, the boundary layer reaches a quasi steady mode before the growth of the temperature gradient is completed. In this mode, the thermal boundary layer at first grows in thickness and then contracts with increasing time. However, if the ramp time is sufficiently short, the boundary layer develops differently, but after the wall temperature gradient growth is completed, the boundary layer develops as though the startup had been instantaneous.

Measuring reef complexity and rugosity from monocular video bathymetric reconstruction

Topographic structural complexity of a reef is highly correlated to coral growth rates, coral cover and overall levels of biodiversity, and is therefore integral in determining ecological processes. Modeling these processes commonly includes measures of rugosity obtained from a wide range of different survey techniques that often fail to capture rugosity at different spatial scales. Here we show that accurate estimates of rugosity can be obtained from video footage captured using underwater video cameras (i.e., monocular video). To demonstrate the accuracy of our method, we compared the results to in situ measurements of a 2m x 20m area of forereef from Glovers Reef atoll in Belize. Sequential pairs of images were used to compute fine scale bathymetric reconstructions of the reef substrate from which precise measurements of rugosity and reef topographic structural complexity can be derived across multiple spatial scales. To achieve accurate bathymetric reconstructions from uncalibrated monocular video, the position of the camera for each image in the video sequence and the intrinsic parameters (e.g., focal length) must be computed simultaneously. We show that these parameters can be often determined when the data exhibits parallax-type motion, and that rugosity and reef complexity can be accurately computed from existing video sequences taken from any type of underwater camera from any reef habitat or location. This technique provides an infinite array of possibilities for future coral reef research by providing a cost-effective and automated method of determining structural complexity and rugosity in both new and historical video surveys of coral reefs.

Non-Newtonian natural convection flow along an isothermal horizontal circular cylinder using modified power-law model

Laminar two-dimensional natural convection boundary-layer flow of non-Newtonian fluids along an isothermal horizontal circular cylinder has been studied using a modified power-law viscosity model. In this model, there are no unrealistic limits of zero or infinite viscosity. Therefore, the boundary-layer equations can be solved numerically by using marching order implicit finite difference method with double sweep technique. Numerical results are presented for the case of shear-thinning as well as shear thickening fluids in terms of the fluid velocity and temperature distributions, shear stresses and rate of heat transfer in terms of the local skin-friction and local Nusselt number respectively.

Spectral approximations to the fractional integral and derivative

In this paper, the spectral approximations are used to compute the fractional integral and the Caputo derivative. The effective recursive formulae based on the Legendre, Chebyshev and Jacobi polynomials are developed to approximate the fractional integral. And the succinct scheme for approximating the Caputo derivative is also derived. The collocation method is proposed to solve the fractional initial value problems and boundary value problems. Numerical examples are also provided to illustrate the effectiveness of the derived methods.

The creativity imperative : implications for education research

Arguing for the importance of understanding the conditions under which certain forms of the social subject become visible and viable, this chapter conceptualises the current educational focus on ‘creativity’ as a technology of governmentality that has arisen from the perceived need for governing authorities to manage and responsibilise populations for the pervasive uncertainties of the global economy. With reference to the document, Tough Choices or Tough Times, a publication of the National Center on Education and the Economy in the United States, we show how creativity has been reframed as a programmable capacity of the modern student, citizen and worker primarily because it is considered an indispensible source of enterprise and innovation. Education and family life are an integral part of this bio-politics and the ongoing ‘economisation’ of social life. Our concern is that this reductionist understanding of creativity precludes other transgressive and culturally enriching creativities that represent the infinite range of subjectivities associated with imaginative human capacity and activity. It is vital therefore that educational research renders this historical process transparent and opens spaces for more socially inclusive, sustainable and productive ways of being such as those indicated by the three respondees.

Eigenstrain boundary integral equations with local Eshelby matrix for stress analysis of ellipsoidal particles

Aiming at the large scale numerical simulation of particle reinforced materials, the concept of local Eshelby matrix has been introduced into the computational model of the eigenstrain boundary integral equation (BIE) to solve the problem of interactions among particles. The local Eshelby matrix can be considered as an extension of the concepts of Eshelby tensor and the equivalent inclusion in numerical form. Taking the subdomain boundary element method as the control, three-dimensional stress analyses are carried out for some ellipsoidal particles in full space with the proposed computational model. Through the numerical examples, it is verified not only the correctness and feasibility but also the high efficiency of the present model with the corresponding solution procedure, showing the potential of solving the problem of large scale numerical simulation of particle reinforced materials.

Joint identification of infinite-frequency added mass and fluid-memory models of marine structures

This paper addresses the problem of joint identification of infinite-frequency added mass and fluid memory models of marine structures from finite frequency data. This problem is relevant for cases where the code used to compute the hydrodynamic coefficients of the marine structure does not give the infinite-frequency added mass. This case is typical of codes based on 2D-potential theory since most 3D-potential-theory codes solve the boundary value associated with the infinite frequency. The method proposed in this paper presents a simpler alternative approach to other methods previously presented in the literature. The advantage of the proposed method is that the same identification procedure can be used to identify the fluid-memory models with or without having access to the infinite-frequency added mass coefficient. Therefore, it provides an extension that puts the two identification problems into the same framework. The method also exploits the constraints related to relative degree and low-frequency asymptotic values of the hydrodynamic coefficients derived from the physics of the problem, which are used as prior information to refine the obtained models.

Open access to infinite content (or "in praise of law reviews")

This Article is about legal scholarly publication in a time of plenitude. It is an attempt to explain why the most pressing questions in legal scholarly publishing are about how we ensure access to an infinity of content. It explains why standard assumptions about resource scarcity in publication are wrong in general, and how the changes in the modality of publication affect legal scholarship. It talks about the economics of open access to legal material, and how this connects to a future where there is infinite content. And because student-edited law reviews fit this future better than their commercially-produced, peer-refereed cousins, this Article is, in part, a defense of the crazy-beautiful institution that is the American law review.

Regulation and integral control of an underactuated robotic system using IDA-PBC with dynamic extension

This paper proposes a method for design of a set-point regulation controller with integral action for an underactuated robotic system. The robot is described as a port-Hamiltonian system, and the control design is based on a coordinate transformation and a dynamic extension. Both the change of coordinates and the dynamic extension add extra degrees of freedom that facilitate the solution of the matching equation associated with interconnection and damping assignment passivity-based control designs (IDA-PBC). The stability of the controlled system is proved using the closed loop Hamiltonian as a Lyapunov candidate function. The performance of the proposed controller is shown in simulation.

Similarity solutions for flow and heat transfer of non-Newtonian fluid over a stretching surface

Similarity solutions are carried out for flow of power law non-Newtonian fluid film on unsteady stretching surface subjected to constant heat flux. Free convection heat transfer induces thermal boundary layer within a semi-infinite layer of Boussinesq fluid. The nonlinear coupled partial differential equations (PDE) governing the flow and the boundary conditions are converted to a system of ordinary differential equations (ODE) using two-parameter groups. This technique reduces the number of independent variables by two, and finally the obtained ordinary differential equations are solved numerically for the temperature and velocity using the shooting method. The thermal and velocity boundary layers are studied by the means of Prandtl number and non-Newtonian power index plotted in curves.