Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

An infinite-horizon robust filter for uncertain hidden Markov Models with conditional relative entropy constraints

We consider a robust filtering problem for uncertain discrete-time, homogeneous, first-order, finite-state hidden Markov models (HMMs). The class of uncertain HMMs considered is described by a conditional relative entropy constraint on measures perturbed from a nominal regular conditional probability distribution given the previous posterior state distribution and the latest measurement. Under this class of perturbations, a robust infinite horizon filtering problem is first formulated as a constrained optimization problem before being transformed via variational results into an unconstrained optimization problem; the latter can be elegantly solved using a risk-sensitive information-state based filtering.

Molecular dynamics studies of the effects of branching characteristics on the crystalline structure of polyethylene

Molecular dynamics simulations were carried out on single chain models of linear low-density polyethylene in vacuum to study the effects of branch length, branch content, and branch distribution on the polymer’s crystalline structure at 300 K. The trans/gauche (t/g) ratios of the backbones of the modeled molecules were calculated and utilized to characterize their degree of crystallinity. The results show that the t/g ratio decreases with increasing branch content regardless of branch length and branch distribution, indicating that branch content is the key molecular parameter that controls the degree of crystallinity. Although t/g ratios of the models with the same branch content vary, they are of secondary importance. However, our data suggests that branch distribution (regular or random) has a significant effect on the degree of crystallinity for models containing 10 hexyl branches/1,000 backbone carbons. The fractions of branches that resided in the equilibrium crystalline structures of the models were also calculated. On average, 9.8% and 2.5% of the branches were found in the crystallites of the molecules with ethyl and hexyl branches while C13 NMR experiments showed that the respective probabilities of branch inclusion for ethyl and hexyl branches are 10% and 6% [Hosoda et al., Polymer 1990, 31, 1999–2005]. However, the degree of branch inclusion seems to be insensitive to the branch content and branch distribution.

Joint identification of infinite-frequency added mass and fluid-memory models of marine structures

This paper addresses the problem of joint identification of infinite-frequency added mass and fluid memory models of marine structures from finite frequency data. This problem is relevant for cases where the code used to compute the hydrodynamic coefficients of the marine structure does not give the infinite-frequency added mass. This case is typical of codes based on 2D-potential theory since most 3D-potential-theory codes solve the boundary value associated with the infinite frequency. The method proposed in this paper presents a simpler alternative approach to other methods previously presented in the literature. The advantage of the proposed method is that the same identification procedure can be used to identify the fluid-memory models with or without having access to the infinite-frequency added mass coefficient. Therefore, it provides an extension that puts the two identification problems into the same framework. The method also exploits the constraints related to relative degree and low-frequency asymptotic values of the hydrodynamic coefficients derived from the physics of the problem, which are used as prior information to refine the obtained models.

Open access to infinite content (or "in praise of law reviews")

This Article is about legal scholarly publication in a time of plenitude. It is an attempt to explain why the most pressing questions in legal scholarly publishing are about how we ensure access to an infinity of content. It explains why standard assumptions about resource scarcity in publication are wrong in general, and how the changes in the modality of publication affect legal scholarship. It talks about the economics of open access to legal material, and how this connects to a future where there is infinite content. And because student-edited law reviews fit this future better than their commercially-produced, peer-refereed cousins, this Article is, in part, a defense of the crazy-beautiful institution that is the American law review.

Toward a deeper understanding of branching dialogue systems

This practice-based exegesis examines the field of writing interactive branching dialogues within video games from the perspective of a professional game writer. Leanne analyses both existing critically-acclaimed titles and her own personal works to create a taxonomy for critique that expands upon the current understanding of games as a literary medium. The final project of the exegesis purposely applies the elements of her new taxonomy in an ineffective manner, making explicit the outcomes and pitfalls of writing multi-layered, tiered dialogues and how tacit assumptions made during the writing process can negatively impact player agency.

Numerical simulation of railway track supporting system using finite-infinite and thin layer elements under impulsive loads

The present study deals with two dimensional, numerical simulation of railway track supporting system subjected to dynamic excitation force. Under plane strain condition, the coupled finite-infinite elements to represent the near and far field stress distribution and thin layer interface element was employed to model the interfacial behavior between sleepers and ballast. To account for the relative debonding, slipping and crushing that could take place in the contact area between the sleepers and ballast, modified Mohr-Coulomb criterion was adopted. Furthermore an attempt has been made to consider the elasto-plastic material non-linearity of the railway track supporting media by employing different constitutive models to represent steel, concrete and supporting materials. Based on the proposed physical and constitutive modeling a code has been developed for dynamic loads. The applicability of the developed F.E code has been demonstrated by analyzing a real railway supporting structure.

Napster: Infinite digital jukebox or pirate bazaar?

This paper considers the copyright litigation over the file-sharing program, Napster. The first section examines the culture of collecting at work in Napster. The next part examines the litigation by the major record companies and Metallica against Napster. The final section considers the future of file-sharing, looking at alternatives to Napster, such as Filetopia, Freenet, Gnutella, MP3board.com and streaming media.

Josh lovegrove: Infinite

Research background: Infinite by Josh Lovegrove is an extended play album co-produced in collaboration with ARIA-nominated artist Mark Sholtez. The album consists of original songs written by Lovegrove, and songs co-written by Lovegrove, Carfoot and Sholtez. The scholarly context of the project is informed by studies of songwriting and ambiguity by Negus and Astor, new approaches to the study of record production associated with Zagorski-Thomas, and studies of creative labour by Hesmondhalgh and Baker. The project focused on the dynamics of musical performance and production in the recording studio, investigating the interface between the creative tasks of songwriting, production and performance in the recording of popular music. The project asked, in what ways do collaborative songwriting and production processes overlap, how has the nature of creative labour changed as a result of new forms of digital recording technology, and how can these aspects inform developments in the learning and teaching of popular music? Research contribution: The project has demonstrated the nuanced ways that the practices of record production have changed in the face of technological developments, and how this has impacted upon the specific forms and divisions of creative labour. Research significance: The project resulted in a well-reviewed album release that has further established Lovegrove’s reputation as a performer and songwriter. The creative work underpins ongoing research into the nature of popular music production, in particular how the nature of collaborative songwriting can inform innovation in the learning and teaching of popular music.

Chasing Infinite Junctures

A collaborative solo exhibition at Boxcopy, Brisbane. For Chasing Infinite Junctures, Caitlin Franzmann presented an installation as an open site for affect, response and collaboration. The exhibition began with an architectural intervention, which was available for exchange and transformation involving other artists, events and the sensing body of the audience. For five weeks, Caitlin provided a space for new junctures to unfold – artists including Andrew McLellan, Ross Manning, Henry Mills, Louise Bennett, Leena Riethmuller and Sandra Selig will gather, experiment and create in the space.

Dimethyl sulfide and other biogenic volatile organic compound emissions from branching coral and reef seawater: Potential sources of secondary aerosol over the Great Barrier Reef

Volatile organic compounds (VOCs) in the headspace of bubble chambers containing branches of live coral in filtered reef seawater were analysed using gas chromatography with mass spectrometry (GC-MS). When the coral released mucus it was a source of dimethyl sulfide (DMS) and isoprene; however, these VOCs were not emitted to the chamber headspace from mucus-free coral. This finding, which suggests that coral is an intermittent source of DMS and isoprene, was supported by the observation of occasional large pulses of atmospheric DMS (DMSa) over Heron Island reef on the southern Great Barrier Reef (GBR), Australia, in the austral winter. The highest DMSa pulse (320 ppt) was three orders of magnitude less than the DMS mixing ratio (460 ppb) measured in the headspace of a dynamically purged bubble chamber containing a mucus-coated branch of Acropora aspera indicating that coral reefs can be strong point sources of DMSa. Static headspace GC-MS analysis of coral fragments identified mainly DMS and seven other minor reduced sulfur compounds including dimethyl disulfide, methyl mercaptan, and carbon disulfide, while coral reef seawater was an indicated source of methylene chloride, acetone, and methyl ethyl ketone. The VOCs emitted by coral and reef seawater are capable of producing new atmospheric particles < 15 nm diameter as observed at Heron Island reef. DMS and isoprene are known to play a role in low-level cloud formation, so aerosol precursors such as these could influence regional climate through a sea surface temperature regulation mechanism hypothesized to operate over the GBR.

Developing a vision for an nD modelling tool

The design of a building is a complicated process, having to formulate diverse components through unique tasks involving different personalities and organisations in order to satisfy multi-faceted client requirements. To do this successfully, the project team must encapsulate an integrated design that accommodates various social, economic and legislative factors. Therefore, in this era of increasing global competition integrated design has been increasingly recognised as a solution to deliver value to clients.----- The ‘From 3D to nD modelling’ project at the University of Salford aims to support integrated design; to enable and equip the design and construction industry with a tool that allows users to create, share, contemplate and apply knowledge from multiple perspectives of user requirements (accessibility, maintainability, sustainability, acoustics, crime, energy simulation, scheduling, costing etc.). Thus taking the concept of 3-dimensional computer modelling of the built environment to an almost infinite number of dimensions, to cope with whole-life construction and asset management issues in the design of modern buildings. This paper reports on the development of a vision for how integrated environments that will allow nD-enabled construction and asset management to be undertaken. The project is funded by a four-year platform grant from the Engineering and Physical Sciences Research Council (EPSRC) in the UK; thus awarded to a multi-disciplinary research team, to enable flexibility in the research strategy and to produce leading innovation. This paper reports on the development of a business process and IT vision for how integrated environments will allow nD-enabled construction and asset management to be undertaken. It further develops many of the key issues of a future vision arising from previous CIB W78 conferences.