Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

An infinite-horizon robust filter for uncertain hidden Markov Models with conditional relative entropy constraints

We consider a robust filtering problem for uncertain discrete-time, homogeneous, first-order, finite-state hidden Markov models (HMMs). The class of uncertain HMMs considered is described by a conditional relative entropy constraint on measures perturbed from a nominal regular conditional probability distribution given the previous posterior state distribution and the latest measurement. Under this class of perturbations, a robust infinite horizon filtering problem is first formulated as a constrained optimization problem before being transformed via variational results into an unconstrained optimization problem; the latter can be elegantly solved using a risk-sensitive information-state based filtering.

Joint identification of infinite-frequency added mass and fluid-memory models of marine structures

This paper addresses the problem of joint identification of infinite-frequency added mass and fluid memory models of marine structures from finite frequency data. This problem is relevant for cases where the code used to compute the hydrodynamic coefficients of the marine structure does not give the infinite-frequency added mass. This case is typical of codes based on 2D-potential theory since most 3D-potential-theory codes solve the boundary value associated with the infinite frequency. The method proposed in this paper presents a simpler alternative approach to other methods previously presented in the literature. The advantage of the proposed method is that the same identification procedure can be used to identify the fluid-memory models with or without having access to the infinite-frequency added mass coefficient. Therefore, it provides an extension that puts the two identification problems into the same framework. The method also exploits the constraints related to relative degree and low-frequency asymptotic values of the hydrodynamic coefficients derived from the physics of the problem, which are used as prior information to refine the obtained models.

Open access to infinite content (or "in praise of law reviews")

This Article is about legal scholarly publication in a time of plenitude. It is an attempt to explain why the most pressing questions in legal scholarly publishing are about how we ensure access to an infinity of content. It explains why standard assumptions about resource scarcity in publication are wrong in general, and how the changes in the modality of publication affect legal scholarship. It talks about the economics of open access to legal material, and how this connects to a future where there is infinite content. And because student-edited law reviews fit this future better than their commercially-produced, peer-refereed cousins, this Article is, in part, a defense of the crazy-beautiful institution that is the American law review.

Numerical simulation of railway track supporting system using finite-infinite and thin layer elements under impulsive loads

The present study deals with two dimensional, numerical simulation of railway track supporting system subjected to dynamic excitation force. Under plane strain condition, the coupled finite-infinite elements to represent the near and far field stress distribution and thin layer interface element was employed to model the interfacial behavior between sleepers and ballast. To account for the relative debonding, slipping and crushing that could take place in the contact area between the sleepers and ballast, modified Mohr-Coulomb criterion was adopted. Furthermore an attempt has been made to consider the elasto-plastic material non-linearity of the railway track supporting media by employing different constitutive models to represent steel, concrete and supporting materials. Based on the proposed physical and constitutive modeling a code has been developed for dynamic loads. The applicability of the developed F.E code has been demonstrated by analyzing a real railway supporting structure.

Napster: Infinite digital jukebox or pirate bazaar?

This paper considers the copyright litigation over the file-sharing program, Napster. The first section examines the culture of collecting at work in Napster. The next part examines the litigation by the major record companies and Metallica against Napster. The final section considers the future of file-sharing, looking at alternatives to Napster, such as Filetopia, Freenet, Gnutella, MP3board.com and streaming media.

Josh lovegrove: Infinite

Research background: Infinite by Josh Lovegrove is an extended play album co-produced in collaboration with ARIA-nominated artist Mark Sholtez. The album consists of original songs written by Lovegrove, and songs co-written by Lovegrove, Carfoot and Sholtez. The scholarly context of the project is informed by studies of songwriting and ambiguity by Negus and Astor, new approaches to the study of record production associated with Zagorski-Thomas, and studies of creative labour by Hesmondhalgh and Baker. The project focused on the dynamics of musical performance and production in the recording studio, investigating the interface between the creative tasks of songwriting, production and performance in the recording of popular music. The project asked, in what ways do collaborative songwriting and production processes overlap, how has the nature of creative labour changed as a result of new forms of digital recording technology, and how can these aspects inform developments in the learning and teaching of popular music? Research contribution: The project has demonstrated the nuanced ways that the practices of record production have changed in the face of technological developments, and how this has impacted upon the specific forms and divisions of creative labour. Research significance: The project resulted in a well-reviewed album release that has further established Lovegrove’s reputation as a performer and songwriter. The creative work underpins ongoing research into the nature of popular music production, in particular how the nature of collaborative songwriting can inform innovation in the learning and teaching of popular music.

Chasing Infinite Junctures

A collaborative solo exhibition at Boxcopy, Brisbane. For Chasing Infinite Junctures, Caitlin Franzmann presented an installation as an open site for affect, response and collaboration. The exhibition began with an architectural intervention, which was available for exchange and transformation involving other artists, events and the sensing body of the audience. For five weeks, Caitlin provided a space for new junctures to unfold – artists including Andrew McLellan, Ross Manning, Henry Mills, Louise Bennett, Leena Riethmuller and Sandra Selig will gather, experiment and create in the space.

Developing a vision for an nD modelling tool

The design of a building is a complicated process, having to formulate diverse components through unique tasks involving different personalities and organisations in order to satisfy multi-faceted client requirements. To do this successfully, the project team must encapsulate an integrated design that accommodates various social, economic and legislative factors. Therefore, in this era of increasing global competition integrated design has been increasingly recognised as a solution to deliver value to clients.----- The ‘From 3D to nD modelling’ project at the University of Salford aims to support integrated design; to enable and equip the design and construction industry with a tool that allows users to create, share, contemplate and apply knowledge from multiple perspectives of user requirements (accessibility, maintainability, sustainability, acoustics, crime, energy simulation, scheduling, costing etc.). Thus taking the concept of 3-dimensional computer modelling of the built environment to an almost infinite number of dimensions, to cope with whole-life construction and asset management issues in the design of modern buildings. This paper reports on the development of a vision for how integrated environments that will allow nD-enabled construction and asset management to be undertaken. The project is funded by a four-year platform grant from the Engineering and Physical Sciences Research Council (EPSRC) in the UK; thus awarded to a multi-disciplinary research team, to enable flexibility in the research strategy and to produce leading innovation. This paper reports on the development of a business process and IT vision for how integrated environments will allow nD-enabled construction and asset management to be undertaken. It further develops many of the key issues of a future vision arising from previous CIB W78 conferences.

A study of vibration and vibration control of ship structures

This paper examines the vibration characteristics and vibration control of complex ship structures. It is shown that input mobilities of a ship structure at engine supports, due to out-of-plane force or bending moment excitations, are governed by the flexural stiffness of the engine supports. The frequency averaged input mobilities of the ship structure, due to such excitations, can be represented by those of the corresponding infinite beam. The torsional moment input mobility at the engine support can be estimated from the torsional response of the engine bed section under direct excitation. It is found that the inclusion of ship hull and deck plates in the ship structure model has little effect on the frequency-averaged response of the ship structure. This study also shows that vibration propagation in complex ship structures at low frequencies can be attenuated by imposing irregularities to the ring frame locations in ships. Vibration responses of ship structures due to machinery excitations at higher frequencies can be controlled by structural modifications of the local supporting structures such as engine beds in ships.

Community engagement for sustainable urban futures : editorial preface

This special issue of Futures is concerned with community engagement strategies that help to inform medium and long-term futures studies in order to foster sustainable urban environments. Recent special issues of Futures, such as Human Extinction (41:10) and Utopian Thought (41:4), reflect the increasing significance of sustainability issues, which is why we present another crucial component of sustainability, community engagement. Responding to futurists’ long term concerns about climate change outlined in Futures 41(9) [1], Stevenson concluded that we can no longer support infinite growth, and that our goal should be to reshape the economy to let us live within our means. In the face of the continued and accelerated crisis in environmental, economic and social sustainability, a number of trends informed our call for papers on the possible role of community engagement in contributing to enhanced urban sustainability: • Changes in the public sphere in terms of participation, online deliberation systems, polity of urban futures; • The possible use of user-generated content for urban planning (paralleling the rise of user generated content elsewhere); • The related role of social networking, collective and civic intelligence, and crowd- sourcing in urban futures; • The rise of technologies such as wireless Internet and mobile applications, and the impact of neogeography, simulations and 3D virtual environments that reproduce and analyse complex social phenomena and city systems in urban futures, design and planning.

A Novel Approach to Haptic Tele-operation of Aerial Robot Vehicles

We present a novel, simple and effective approach for tele-operation of aerial robotic vehicles with haptic feedback. Such feedback provides the remote pilot with an intuitive feel of the robot’s state and perceived local environment that will ensure simple and safe operation in cluttered 3D environments common in inspection and surveillance tasks. Our approach is based on energetic considerations and uses the concepts of network theory and port-Hamiltonian systems. We provide a general framework for addressing problems such as mapping the limited stroke of a ‘master’ joystick to the infinite stroke of a ‘slave’ vehicle, while preserving passivity of the closed-loop system in the face of potential time delays in communications links and limited sensor data

Performance analysis of adaptive lattice filters for FM signals and alpha-stable processes

The performance of an adaptive filter may be studied through the behaviour of the optimal and adaptive coefficients in a given environment. This thesis investigates the performance of finite impulse response adaptive lattice filters for two classes of input signals: (a) frequency modulated signals with polynomial phases of order p in complex Gaussian white noise (as nonstationary signals), and (b) the impulsive autoregressive processes with alpha-stable distributions (as non-Gaussian signals). Initially, an overview is given for linear prediction and adaptive filtering. The convergence and tracking properties of the stochastic gradient algorithms are discussed for stationary and nonstationary input signals. It is explained that the stochastic gradient lattice algorithm has many advantages over the least-mean square algorithm. Some of these advantages are having a modular structure, easy-guaranteed stability, less sensitivity to the eigenvalue spread of the input autocorrelation matrix, and easy quantization of filter coefficients (normally called reflection coefficients). We then characterize the performance of the stochastic gradient lattice algorithm for the frequency modulated signals through the optimal and adaptive lattice reflection coefficients. This is a difficult task due to the nonlinear dependence of the adaptive reflection coefficients on the preceding stages and the input signal. To ease the derivations, we assume that reflection coefficients of each stage are independent of the inputs to that stage. Then the optimal lattice filter is derived for the frequency modulated signals. This is performed by computing the optimal values of residual errors, reflection coefficients, and recovery errors. Next, we show the tracking behaviour of adaptive reflection coefficients for frequency modulated signals. This is carried out by computing the tracking model of these coefficients for the stochastic gradient lattice algorithm in average. The second-order convergence of the adaptive coefficients is investigated by modeling the theoretical asymptotic variance of the gradient noise at each stage. The accuracy of the analytical results is verified by computer simulations. Using the previous analytical results, we show a new property, the polynomial order reducing property of adaptive lattice filters. This property may be used to reduce the order of the polynomial phase of input frequency modulated signals. Considering two examples, we show how this property may be used in processing frequency modulated signals. In the first example, a detection procedure in carried out on a frequency modulated signal with a second-order polynomial phase in complex Gaussian white noise. We showed that using this technique a better probability of detection is obtained for the reduced-order phase signals compared to that of the traditional energy detector. Also, it is empirically shown that the distribution of the gradient noise in the first adaptive reflection coefficients approximates the Gaussian law. In the second example, the instantaneous frequency of the same observed signal is estimated. We show that by using this technique a lower mean square error is achieved for the estimated frequencies at high signal-to-noise ratios in comparison to that of the adaptive line enhancer. The performance of adaptive lattice filters is then investigated for the second type of input signals, i.e., impulsive autoregressive processes with alpha-stable distributions . The concept of alpha-stable distributions is first introduced. We discuss that the stochastic gradient algorithm which performs desirable results for finite variance input signals (like frequency modulated signals in noise) does not perform a fast convergence for infinite variance stable processes (due to using the minimum mean-square error criterion). To deal with such problems, the concept of minimum dispersion criterion, fractional lower order moments, and recently-developed algorithms for stable processes are introduced. We then study the possibility of using the lattice structure for impulsive stable processes. Accordingly, two new algorithms including the least-mean P-norm lattice algorithm and its normalized version are proposed for lattice filters based on the fractional lower order moments. Simulation results show that using the proposed algorithms, faster convergence speeds are achieved for parameters estimation of autoregressive stable processes with low to moderate degrees of impulsiveness in comparison to many other algorithms. Also, we discuss the effect of impulsiveness of stable processes on generating some misalignment between the estimated parameters and the true values. Due to the infinite variance of stable processes, the performance of the proposed algorithms is only investigated using extensive computer simulations.