Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

An infinite-horizon robust filter for uncertain hidden Markov Models with conditional relative entropy constraints

We consider a robust filtering problem for uncertain discrete-time, homogeneous, first-order, finite-state hidden Markov models (HMMs). The class of uncertain HMMs considered is described by a conditional relative entropy constraint on measures perturbed from a nominal regular conditional probability distribution given the previous posterior state distribution and the latest measurement. Under this class of perturbations, a robust infinite horizon filtering problem is first formulated as a constrained optimization problem before being transformed via variational results into an unconstrained optimization problem; the latter can be elegantly solved using a risk-sensitive information-state based filtering.

Joint identification of infinite-frequency added mass and fluid-memory models of marine structures

This paper addresses the problem of joint identification of infinite-frequency added mass and fluid memory models of marine structures from finite frequency data. This problem is relevant for cases where the code used to compute the hydrodynamic coefficients of the marine structure does not give the infinite-frequency added mass. This case is typical of codes based on 2D-potential theory since most 3D-potential-theory codes solve the boundary value associated with the infinite frequency. The method proposed in this paper presents a simpler alternative approach to other methods previously presented in the literature. The advantage of the proposed method is that the same identification procedure can be used to identify the fluid-memory models with or without having access to the infinite-frequency added mass coefficient. Therefore, it provides an extension that puts the two identification problems into the same framework. The method also exploits the constraints related to relative degree and low-frequency asymptotic values of the hydrodynamic coefficients derived from the physics of the problem, which are used as prior information to refine the obtained models.

Open access to infinite content (or "in praise of law reviews")

This Article is about legal scholarly publication in a time of plenitude. It is an attempt to explain why the most pressing questions in legal scholarly publishing are about how we ensure access to an infinity of content. It explains why standard assumptions about resource scarcity in publication are wrong in general, and how the changes in the modality of publication affect legal scholarship. It talks about the economics of open access to legal material, and how this connects to a future where there is infinite content. And because student-edited law reviews fit this future better than their commercially-produced, peer-refereed cousins, this Article is, in part, a defense of the crazy-beautiful institution that is the American law review.

Numerical simulation of railway track supporting system using finite-infinite and thin layer elements under impulsive loads

The present study deals with two dimensional, numerical simulation of railway track supporting system subjected to dynamic excitation force. Under plane strain condition, the coupled finite-infinite elements to represent the near and far field stress distribution and thin layer interface element was employed to model the interfacial behavior between sleepers and ballast. To account for the relative debonding, slipping and crushing that could take place in the contact area between the sleepers and ballast, modified Mohr-Coulomb criterion was adopted. Furthermore an attempt has been made to consider the elasto-plastic material non-linearity of the railway track supporting media by employing different constitutive models to represent steel, concrete and supporting materials. Based on the proposed physical and constitutive modeling a code has been developed for dynamic loads. The applicability of the developed F.E code has been demonstrated by analyzing a real railway supporting structure.

Napster: Infinite digital jukebox or pirate bazaar?

This paper considers the copyright litigation over the file-sharing program, Napster. The first section examines the culture of collecting at work in Napster. The next part examines the litigation by the major record companies and Metallica against Napster. The final section considers the future of file-sharing, looking at alternatives to Napster, such as Filetopia, Freenet, Gnutella, MP3board.com and streaming media.

Static analysis of infinite laminated cylindrical shell panel

This paper presents an approximate three-dimensional elasticity solution for an infinitely long, cross-ply laminated circular cylindrical shell panel with simply supported boundary conditions, subjected to an arbitrary discontinuous transverse loading. The solution is based on the principal assumption that the ratio of the thickness of the lamina to its middle surface radius is negligible compared to unity. The validity of this assumption and the range of application of this approximate solution have been established through a comparison with an exact solution. Results of classical and first-order shear deformation shell theories have been compared with the results of the present solution to bring out the accuracy of these theories. It is also shown that for very shallow shell panels the definition of a thin shell should be based on the ratio of thickness to chord width rather than the ratio of thickness to mean radius.

Unsteady Compressible Boundary-Layer Flow On The Stagnation Line Of An Infinite Swept Cylinder

Numerical solutions of flow and heat transfer process on the unsteady flow of a compressible viscous fluid with variable gas properties in the vicinity of the stagnation line of an infinite swept cylinder are presented. Results are given for the case where the unsteady temperature field is produced by (i) a sudden change in the wall temperature (enthalpy) as the impulsive motion is started and (ii) a sudden change in the free-stream velocity. Solutions for the simultaneous development of the thermal and momentum boundary layers are obtained by using quasilinearization technique with an implicit finite difference scheme. Attention is given to the transient phenomenon from the initial flow to the final steady-state distribution. Results are presented for the skin friction and heat transfer coefficients as well as for the velocity and enthalpy profiles. The effects of wail enthalpy parameter, sweep parameter, fluid properties and transpiration cooling on the heat transfer and skin friction are considered.

Diffraction Of Elastic Waves By Two Parallel Rigid Strips Embedded In An Infinite Orthotropic Medium

The elastodynamic response of a pair of parallel rigid strips embedded in an infinite orthotropic medium due to elastic waves incident normally on the strips has been investigated. The mixed boundary value problem has been solved by the Integral Equation method. The normal stress and the vertical displacement have been derived in closed form. Numerical values of stress intensity factors at inner and outer edges of the strips and vertical displacement at points in the plane of the strips for several orthotropic materials have been calculated and plotted graphically to show the effect of material orthotropy.

Infinite Dimensional Slow Modulations In A Delayed Model For Orthogonal Cutting Vibrations

We apply the method of multiple scales (MMS) to a well known model of regenerative cutting vibrations in the large delay regime. By ``large'' we mean the delay is much larger than the time scale of typical cutting tool oscillations. The MMS upto second order for such systems has been developed recently, and is applied here to study tool dynamics in the large delay regime. The second order analysis is found to be much more accurate than first order analysis. Numerical integration of the MMS slow flow is much faster than for the original equation, yet shows excellent accuracy. The main advantage of the present analysis is that infinite dimensional dynamics is retained in the slow flow, while the more usual center manifold reduction gives a planar phase space. Lower-dimensional dynamical features, such as Hopf bifurcations and families of periodic solutions, are also captured by the MMS. Finally, the strong sensitivity of the dynamics to small changes in parameter values is seen clearly.

Josh lovegrove: Infinite

Research background: Infinite by Josh Lovegrove is an extended play album co-produced in collaboration with ARIA-nominated artist Mark Sholtez. The album consists of original songs written by Lovegrove, and songs co-written by Lovegrove, Carfoot and Sholtez. The scholarly context of the project is informed by studies of songwriting and ambiguity by Negus and Astor, new approaches to the study of record production associated with Zagorski-Thomas, and studies of creative labour by Hesmondhalgh and Baker. The project focused on the dynamics of musical performance and production in the recording studio, investigating the interface between the creative tasks of songwriting, production and performance in the recording of popular music. The project asked, in what ways do collaborative songwriting and production processes overlap, how has the nature of creative labour changed as a result of new forms of digital recording technology, and how can these aspects inform developments in the learning and teaching of popular music? Research contribution: The project has demonstrated the nuanced ways that the practices of record production have changed in the face of technological developments, and how this has impacted upon the specific forms and divisions of creative labour. Research significance: The project resulted in a well-reviewed album release that has further established Lovegrove’s reputation as a performer and songwriter. The creative work underpins ongoing research into the nature of popular music production, in particular how the nature of collaborative songwriting can inform innovation in the learning and teaching of popular music.