Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

An infinite-horizon robust filter for uncertain hidden Markov Models with conditional relative entropy constraints

We consider a robust filtering problem for uncertain discrete-time, homogeneous, first-order, finite-state hidden Markov models (HMMs). The class of uncertain HMMs considered is described by a conditional relative entropy constraint on measures perturbed from a nominal regular conditional probability distribution given the previous posterior state distribution and the latest measurement. Under this class of perturbations, a robust infinite horizon filtering problem is first formulated as a constrained optimization problem before being transformed via variational results into an unconstrained optimization problem; the latter can be elegantly solved using a risk-sensitive information-state based filtering.

Joint identification of infinite-frequency added mass and fluid-memory models of marine structures

This paper addresses the problem of joint identification of infinite-frequency added mass and fluid memory models of marine structures from finite frequency data. This problem is relevant for cases where the code used to compute the hydrodynamic coefficients of the marine structure does not give the infinite-frequency added mass. This case is typical of codes based on 2D-potential theory since most 3D-potential-theory codes solve the boundary value associated with the infinite frequency. The method proposed in this paper presents a simpler alternative approach to other methods previously presented in the literature. The advantage of the proposed method is that the same identification procedure can be used to identify the fluid-memory models with or without having access to the infinite-frequency added mass coefficient. Therefore, it provides an extension that puts the two identification problems into the same framework. The method also exploits the constraints related to relative degree and low-frequency asymptotic values of the hydrodynamic coefficients derived from the physics of the problem, which are used as prior information to refine the obtained models.

Open access to infinite content (or "in praise of law reviews")

This Article is about legal scholarly publication in a time of plenitude. It is an attempt to explain why the most pressing questions in legal scholarly publishing are about how we ensure access to an infinity of content. It explains why standard assumptions about resource scarcity in publication are wrong in general, and how the changes in the modality of publication affect legal scholarship. It talks about the economics of open access to legal material, and how this connects to a future where there is infinite content. And because student-edited law reviews fit this future better than their commercially-produced, peer-refereed cousins, this Article is, in part, a defense of the crazy-beautiful institution that is the American law review.

Rapid, low-temperature synthesis of nc-Si in high-density, non-equilibrium plasmas : enabling nanocrystallinity at very low hydrogen dilution

Nanocrystalline silicon thin films were deposited on single-crystal silicon and glass substrates simultaneously by inductively coupled plasma-assisted chemical vapor deposition from the reactive silane reactant gas diluted with hydrogen at a substrate temperature of 200 °C. The effect of hydrogen dilution ratio X (X is defined as the flow rate ratio of hydrogen to silane gas), ranging from 1 to 20, on the structural and optical properties of the deposited films, is extensively investigated by Raman spectroscopy, X-ray diffraction, Fourier transform infrared absorption spectroscopy, UV/VIS spectroscopy, and scanning electron microscopy. Our experimental results reveal that, with the increase of the hydrogen dilution ratio X, the deposition rate Rd and hydrogen content CH are reduced while the crystalline fraction Fc, mean grain size δ and optical bandgap ETauc are increased. In comparison with other plasma enhanced chemical vapor deposition methods of nanocrystalline silicon films where a very high hydrogen dilution ratio X is routinely required (e.g. X > 16), we have achieved nanocrystalline silicon films at a very low hydrogen dilution ratio of 1, featuring a high deposition rate of 1.57 nm/s, a high crystalline fraction of 67.1%, a very low hydrogen content of 4.4 at.%, an optical bandgap of 1.89 eV, and an almost vertically aligned columnar structure with a mean grain size of approximately 19 nm. We have also shown that a sufficient amount of atomic hydrogen on the growth surface essential for the formation of nanocrystalline silicon is obtained through highly-effective dissociation of silane and hydrogen molecules in the high-density inductively coupled plasmas. © 2009 The Royal Society of Chemistry.

Influence of hydrogen dilution on the growth of nanocrystalline silicon carbide films by low-frequency inductively coupled plasma chemical vapor deposition

Nanocrystalline silicon carbide (nc-SiC) films are prepared by low-frequency inductively coupled plasma chemical vapor deposition from feedstock gases silane and methane diluted with hydrogen at a substrate temperature of 500 °C. The effect of different hydrogen dilution ratios X [hydrogen flow (sccm) / silane + methane flow (sccm)] on the growth of nc-SiC films is investigated by X-ray diffraction, scanning electron microscopy, Fourier transform infrared (FTIR) spectroscopy, and X-ray photoelectron spectroscopy (XPS). At a low hydrogen dilution ratio X, cubic silicon carbide is the main crystal phase; whereas at a high hydrogen dilution ratio X, hexagonal silicon carbide is the main crystal phase. The SiC crystal phase transformation may be explained by the different surface mobility of reactive Si-based and C-based radicals deposited at different hydrogen dilution ratios X. The FTIR and XPS analyses show that the Si-C bonds are the main bonds in the films and elemental composition of SiC is nearly stoichiometric with almost equal share of silicon and carbon atoms.

Assessment of body composition in Indian adults: comparison between dual-energy X-ray absorptiometry and isotope dilution technique

Dual-energy X-ray absorptiometry (DXA) and isotope dilution technique have been used as reference methods to validate the estimates of body composition by simple field techniques; however, very few studies have compared these two methods. We compared the estimates of body composition by DXA and isotope dilution (18O) technique in apparently healthy Indian men and women (aged 19–70 years, n 152, 48 % men) with a wide range of BMI (14–40 kg/m2). Isotopic enrichment was assessed by isotope ratio mass spectroscopy. The agreement between the estimates of body composition measured by the two techniques was assessed by the Bland–Altman method. The mean age and BMI were 37 (SD 15) years and 23·3 (SD 5·1) kg/m2, respectively, for men and 37 (SD 14) years and 24·1 (SD 5·8) kg/m2, respectively, for women. The estimates of fat-free mass were higher by about 7 (95 % CI 6, 9) %, those of fat mass were lower by about 21 (95 % CI 218,223) %, and those of body fat percentage (BF%) were lower by about 7·4 (95 % CI 28·2, 26·6) % as obtained by DXA compared with the isotope dilution technique. The Bland–Altman analysis showed wide limits of agreement that indicated poor agreement between the methods. The bias in the estimates of BF% was higher at the lower values of BF%. Thus, the two commonly used reference methods showed substantial differences in the estimates of body composition with wide limits of agreement. As the estimates of body composition are method-dependent, the two methods cannot be used interchangeably

An improved method for undertaking limiting dilution assays for in vitro cloning of Plasmodium falciparum parasites

Background Obtaining single parasite clones is required for many techniques in malaria research. Cloning by limiting dilution using microscopy-based assessment for parasite growth is an arduous and labor-intensive process. An alternative method for the detection of parasite growth in limiting dilution assays is using a commercial ELISA histidine-rich protein II (HRP2) detection kit. Methods Detection of parasite growth was undertaken using HRP2 ELISA and compared to thick film microscopy. An HRP2 protein standard was used to determine the detection threshold of the HRP2 ELISA assay, and a HRP2 release model was used to extrapolate the amount of parasite growth required for a positive result. Results The HRP2 ELISA was more sensitive than microscopy for detecting parasite growth. The minimum level of HRP2 protein detection of the ELISA was 0.11ng/ml. Modeling of HRP2 release determined that 2,116 parasites are required to complete a full erythrocytic cycle to produce sufficient HRP2 to be detected by the ELISA. Under standard culture conditions this number of parasites is likely to be reached between 8 to 14 days of culture. Conclusions This method provides an accurate and simple way for the detection of parasite growth in limiting dilution assays, reducing time and resources required in traditional methods. Furthermore the method uses spent culture media instead of the parasite-infected red blood cells, enabling culture to continue.

Numerical simulation of railway track supporting system using finite-infinite and thin layer elements under impulsive loads

The present study deals with two dimensional, numerical simulation of railway track supporting system subjected to dynamic excitation force. Under plane strain condition, the coupled finite-infinite elements to represent the near and far field stress distribution and thin layer interface element was employed to model the interfacial behavior between sleepers and ballast. To account for the relative debonding, slipping and crushing that could take place in the contact area between the sleepers and ballast, modified Mohr-Coulomb criterion was adopted. Furthermore an attempt has been made to consider the elasto-plastic material non-linearity of the railway track supporting media by employing different constitutive models to represent steel, concrete and supporting materials. Based on the proposed physical and constitutive modeling a code has been developed for dynamic loads. The applicability of the developed F.E code has been demonstrated by analyzing a real railway supporting structure.

The Black Label: trade mark dilution, culture jamming, and the no logo movement

This article considers the ongoing debate over the appropriation of well-known and famous trade marks by the No Logo Movement for the purposes of political and social critique. It focuses upon one sensational piece of litigation in South Africa, Laugh It Off Promotions v. South African Breweries International (Finance) B.V. t/a Sabmark International. In this case, a group called Laugh It Off Promotions subjected the trade marks of the manufacturers of Carling Beer were subjected to parody, social satire, and culture jamming. The beer slogan “Black Label” was turned into a T-Shirt entitled “Black Labour/ White Guilt”. In the ensuing litigation, the High Court of South Africa and the Supreme Court of Appeal were of the opinion that the appropriation of the mark was a case of hate speech. However, the Constitutional Court of South Africa disagreed, finding that the parodies of a well-known, famous trade mark did not constitute trade mark dilution. Moseneke J observed that there was a lack of evidence of economic or material harm; and Sachs J held that there is a need to provide latitude for parody, laughter, and freedom of expression. The decision of the Constitutional Court of South Africa provides some important insights into the nature of trade mark dilution, the role of parody and satire, and the relevance of constitutional protections of freedom of speech and freedom of expression. Arguably, the ruling will be of help in the reformation of trade mark dilution law in other jurisdictions – such as the United States. The decision in Laugh It Off Promotions v. South African Breweries International demonstrates that trade mark law should not be immune from careful constitutional scrutiny.