Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Learning the Kernel Matrix with Semi-Definite Programming

Kernel-based learning algorithms work by embedding the data into a Euclidean space, and then searching for linear relations among the embedded data points. The embedding is performed implicitly, by specifying the inner products between each pair of points in the embedding space. This information is contained in the so-called kernel matrix, a symmetric and positive definite matrix that encodes the relative positions of all points. Specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space -- classical model selection problems in machine learning. In this paper we show how the kernel matrix can be learned from data via semi-definite programming (SDP) techniques. When applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm -- using the labelled part of the data one can learn an embedding also for the unlabelled part. The similarity between test points is inferred from training points and their labels. Importantly, these learning problems are convex, so we obtain a method for learning both the model class and the function without local minima. Furthermore, this approach leads directly to a convex method to learn the 2-norm soft margin parameter in support vector machines, solving another important open problem. Finally, the novel approach presented in the paper is supported by positive empirical results.

An infinite-horizon robust filter for uncertain hidden Markov Models with conditional relative entropy constraints

We consider a robust filtering problem for uncertain discrete-time, homogeneous, first-order, finite-state hidden Markov models (HMMs). The class of uncertain HMMs considered is described by a conditional relative entropy constraint on measures perturbed from a nominal regular conditional probability distribution given the previous posterior state distribution and the latest measurement. Under this class of perturbations, a robust infinite horizon filtering problem is first formulated as a constrained optimization problem before being transformed via variational results into an unconstrained optimization problem; the latter can be elegantly solved using a risk-sensitive information-state based filtering.

Open access to infinite content (or "in praise of law reviews")

This Article is about legal scholarly publication in a time of plenitude. It is an attempt to explain why the most pressing questions in legal scholarly publishing are about how we ensure access to an infinity of content. It explains why standard assumptions about resource scarcity in publication are wrong in general, and how the changes in the modality of publication affect legal scholarship. It talks about the economics of open access to legal material, and how this connects to a future where there is infinite content. And because student-edited law reviews fit this future better than their commercially-produced, peer-refereed cousins, this Article is, in part, a defense of the crazy-beautiful institution that is the American law review.

Napster: Infinite digital jukebox or pirate bazaar?

This paper considers the copyright litigation over the file-sharing program, Napster. The first section examines the culture of collecting at work in Napster. The next part examines the litigation by the major record companies and Metallica against Napster. The final section considers the future of file-sharing, looking at alternatives to Napster, such as Filetopia, Freenet, Gnutella, MP3board.com and streaming media.

Semi-analytical finite element analysis of a strip footing on an elastic reinforced soil

A plane strain elastic interaction analysis of a strip footing resting on a reinforced soil bed has been made by using a combined analytical and finite element method (FEM). In this approach the stiffness matrix for the footing has been obtained using the FEM, For the reinforced soil bed (halfplane) the stiffness matrix has been obtained using an analytical solution. For the latter, the reinforced zone has been idealised as (i) an equivalent orthotropic infinite strip (composite approach) and (ii) a multilayered system (discrete approach). In the analysis, the interface between the strip footing and reinforced halfplane has been assumed as (i) frictionless and (ii) fully bonded. The contact pressure distribution and the settlement reduction have been given for different depths of footing and scheme of reinforcement in soil. The load-deformation behaviour of the reinforced soil obtained using the above modelling has been compared with some available analytical and model test results. The equivalent orthotropic approach proposed in this paper is easy to program and is shown to predict the reinforcing effects reasonably well.

Static analysis of infinite laminated cylindrical shell panel

This paper presents an approximate three-dimensional elasticity solution for an infinitely long, cross-ply laminated circular cylindrical shell panel with simply supported boundary conditions, subjected to an arbitrary discontinuous transverse loading. The solution is based on the principal assumption that the ratio of the thickness of the lamina to its middle surface radius is negligible compared to unity. The validity of this assumption and the range of application of this approximate solution have been established through a comparison with an exact solution. Results of classical and first-order shear deformation shell theories have been compared with the results of the present solution to bring out the accuracy of these theories. It is also shown that for very shallow shell panels the definition of a thin shell should be based on the ratio of thickness to chord width rather than the ratio of thickness to mean radius.

Unsteady Compressible Boundary-Layer Flow On The Stagnation Line Of An Infinite Swept Cylinder

Numerical solutions of flow and heat transfer process on the unsteady flow of a compressible viscous fluid with variable gas properties in the vicinity of the stagnation line of an infinite swept cylinder are presented. Results are given for the case where the unsteady temperature field is produced by (i) a sudden change in the wall temperature (enthalpy) as the impulsive motion is started and (ii) a sudden change in the free-stream velocity. Solutions for the simultaneous development of the thermal and momentum boundary layers are obtained by using quasilinearization technique with an implicit finite difference scheme. Attention is given to the transient phenomenon from the initial flow to the final steady-state distribution. Results are presented for the skin friction and heat transfer coefficients as well as for the velocity and enthalpy profiles. The effects of wail enthalpy parameter, sweep parameter, fluid properties and transpiration cooling on the heat transfer and skin friction are considered.

Diffraction Of Elastic Waves By Two Parallel Rigid Strips Embedded In An Infinite Orthotropic Medium

The elastodynamic response of a pair of parallel rigid strips embedded in an infinite orthotropic medium due to elastic waves incident normally on the strips has been investigated. The mixed boundary value problem has been solved by the Integral Equation method. The normal stress and the vertical displacement have been derived in closed form. Numerical values of stress intensity factors at inner and outer edges of the strips and vertical displacement at points in the plane of the strips for several orthotropic materials have been calculated and plotted graphically to show the effect of material orthotropy.

Infinite Dimensional Slow Modulations In A Delayed Model For Orthogonal Cutting Vibrations

We apply the method of multiple scales (MMS) to a well known model of regenerative cutting vibrations in the large delay regime. By ``large'' we mean the delay is much larger than the time scale of typical cutting tool oscillations. The MMS upto second order for such systems has been developed recently, and is applied here to study tool dynamics in the large delay regime. The second order analysis is found to be much more accurate than first order analysis. Numerical integration of the MMS slow flow is much faster than for the original equation, yet shows excellent accuracy. The main advantage of the present analysis is that infinite dimensional dynamics is retained in the slow flow, while the more usual center manifold reduction gives a planar phase space. Lower-dimensional dynamical features, such as Hopf bifurcations and families of periodic solutions, are also captured by the MMS. Finally, the strong sensitivity of the dynamics to small changes in parameter values is seen clearly.

Josh lovegrove: Infinite

Research background: Infinite by Josh Lovegrove is an extended play album co-produced in collaboration with ARIA-nominated artist Mark Sholtez. The album consists of original songs written by Lovegrove, and songs co-written by Lovegrove, Carfoot and Sholtez. The scholarly context of the project is informed by studies of songwriting and ambiguity by Negus and Astor, new approaches to the study of record production associated with Zagorski-Thomas, and studies of creative labour by Hesmondhalgh and Baker. The project focused on the dynamics of musical performance and production in the recording studio, investigating the interface between the creative tasks of songwriting, production and performance in the recording of popular music. The project asked, in what ways do collaborative songwriting and production processes overlap, how has the nature of creative labour changed as a result of new forms of digital recording technology, and how can these aspects inform developments in the learning and teaching of popular music? Research contribution: The project has demonstrated the nuanced ways that the practices of record production have changed in the face of technological developments, and how this has impacted upon the specific forms and divisions of creative labour. Research significance: The project resulted in a well-reviewed album release that has further established Lovegrove’s reputation as a performer and songwriter. The creative work underpins ongoing research into the nature of popular music production, in particular how the nature of collaborative songwriting can inform innovation in the learning and teaching of popular music.

Transversely isotropic infinite cylindrical shell subjected to a radial axisymmetric line load

The problem of an infinite transversely isotropic circular cylindrical shell subjected to an axisymmetric radial external line load is investigated using elasticity theory, classical shell theory and shear deformation theory. The results obtained by these methods are compared for two ratios of inner to outer shell radius and for varying degrees of anisotropy. Some typical results are given here to show the effect of anisotropy and the thickness of the shell on the distribution of stresses and displacements.

Portfolio Optimization in a Semi-Markov Modulated Market

We address a portfolio optimization problem in a semi-Markov modulated market. We study both the terminal expected utility optimization on finite time horizon and the risk-sensitive portfolio optimization on finite and infinite time horizon. We obtain optimal portfolios in relevant cases. A numerical procedure is also developed to compute the optimal expected terminal utility for finite horizon problem.

Axisymmetric melting of a long cylinder due to an infinite flux

By employing a new embedding technique, a short-time analytical solution for the axisymmetric melting of a long cylinder due to an infinite flux is presented in this paper. The sufficient condition for starting the instantaneous melting of the cylinder has been derived. The melt is removed as soon as it is formed. The method of solution is simple and straightforward and consists of assuming fictitious initial temperature for some fictitious extension of the actual region.