Context-free and context-sensitive dynamics in recurrent neural networks

Continuous-valued recurrent neural networks can learn mechanisms for processing context-free languages. The dynamics of such networks is usually based on damped oscillation around fixed points in state space and requires that the dynamical components are arranged in certain ways. It is shown that qualitatively similar dynamics with similar constraints hold for a(n)b(n)c(n), a context-sensitive language. The additional difficulty with a(n)b(n)c(n), compared with the context-free language a(n)b(n), consists of 'counting up' and 'counting down' letters simultaneously. The network solution is to oscillate in two principal dimensions, one for counting up and one for counting down. This study focuses on the dynamics employed by the sequential cascaded network, in contrast to the simple recurrent network, and the use of backpropagation through time. Found solutions generalize well beyond training data, however, learning is not reliable. The contribution of this study lies in demonstrating how the dynamics in recurrent neural networks that process context-free languages can also be employed in processing some context-sensitive languages (traditionally thought of as requiring additional computation resources). This continuity of mechanism between language classes contributes to our understanding of neural networks in modelling language learning and processing.

On learning context-free and context-sensitive languages

The long short-term memory (LSTM) is not the only neural network which learns a context sensitive language. Second-order sequential cascaded networks (SCNs) are able to induce means from a finite fragment of a context-sensitive language for processing strings outside the training set. The dynamical behavior of the SCN is qualitatively distinct from that observed in LSTM networks. Differences in performance and dynamics are discussed.

Learning the dynamics of embedded clauses

Recent work by Siegelmann has shown that the computational power of recurrent neural networks matches that of Turing Machines. One important implication is that complex language classes (infinite languages with embedded clauses) can be represented in neural networks. Proofs are based on a fractal encoding of states to simulate the memory and operations of stacks. In the present work, it is shown that similar stack-like dynamics can be learned in recurrent neural networks from simple sequence prediction tasks. Two main types of network solutions are found and described qualitatively as dynamical systems: damped oscillation and entangled spiraling around fixed points. The potential and limitations of each solution type are established in terms of generalization on two different context-free languages. Both solution types constitute novel stack implementations - generally in line with Siegelmann's theoretical work - which supply insights into how embedded structures of languages can be handled in analog hardware.

Generalization by symbolic abstraction in cascaded recurrent networks

Generalization performance in recurrent neural networks is enhanced by cascading several networks. By discretizing abstractions induced in one network, other networks can operate on a coarse symbolic level with increased performance on sparse and structural prediction tasks. The level of systematicity exhibited by the cascade of recurrent networks is assessed on the basis of three language domains. (C) 2004 Elsevier B.V. All rights reserved.

Accelerated leap methods for simulating discrete stochastic chemical kinetics

Biologists are increasingly conscious of the critical role that noise plays in cellular functions such as genetic regulation, often in connection with fluctuations in small numbers of key regulatory molecules. This has inspired the development of models that capture this fundamentally discrete and stochastic nature of cellular biology - most notably the Gillespie stochastic simulation algorithm (SSA). The SSA simulates a temporally homogeneous, discrete-state, continuous-time Markov process, and of course the corresponding probabilities and numbers of each molecular species must all remain positive. While accurately serving this purpose, the SSA can be computationally inefficient due to very small time stepping so faster approximations such as the Poisson and Binomial τ-leap methods have been suggested. This work places these leap methods in the context of numerical methods for the solution of stochastic differential equations (SDEs) driven by Poisson noise. This allows analogues of Euler-Maruyuma, Milstein and even higher order methods to be developed through the Itô-Taylor expansions as well as similar derivative-free Runge-Kutta approaches. Numerical results demonstrate that these novel methods compare favourably with existing techniques for simulating biochemical reactions by more accurately capturing crucial properties such as the mean and variance than existing methods.

Implications of seasonal climate forecasts on world wheat trade: a stochastic, dynamic analysis

Improvements in seasonal climate forecasts have potential economic implications for international agriculture. A stochastic, dynamic simulation model of the international wheat economy is developed to estimate the potential effects of seasonal climate forecasts for various countries' wheat production, exports and world trade. Previous studies have generally ignored the stochastic and dynamic aspects of the effects associated with the use of climate forecasts. This study shows the importance of these aspects. In particular with free trade, the use of seasonal forecasts results in increased producer surplus across all exporting countries. In fact, producers appear to capture a large share of the economic surplus created by using the forecasts. Further, the stochastic dimensions suggest that while the expected long-run benefits of seasonal forecasts are positive, considerable year-to-year variation in the distribution of benefits between producers and consumers should be expected. The possibility exists for an economic measure to increase or decrease over a 20-year horizon, depending on the particular sequence of years.

The role of context in producing item interactions and false memories

Cued recall with an extralist cue poses a challenge for contemporary memory theory in that there is a need to explain how episodic and semantic information are combined. A parallel activation and intersection approach proposes one such means by assuming that an experimental cue will elicit its preexisting semantic network and a context cue will elicit a list memory. These 2 sources of information are then combined by focusing on information that is common to the 2 sources. Two key predictions of that approach are examined: (a) Combining semantic and episodic information can lead to item interactions and false memories, and (b) these effects are limited to memory tasks that involve an episodic context cue. Five experiments demonstrate such item interactions and false memories in cued recall but not in free association. Links are drawn between the use of context in this setting and in other settings.

Engineering of needle-free physical methods to target epidermal cells for DNA vaccination

A challenge in epidermal DNA vaccination is the efficient and targeted delivery of polynucleotides to immunologically sensitive Langerhans cells. This paper investigates this particular challenge for physical delivery approaches. The skin immunology and material properties are examined in the context of the physical cell targeting requirements of the viable epidermis. Selected current physical cell targeting technologies engineered to meet these needs are examined: needle and syringe; diffusion patches; liquid jet injectors; microneedle arrays/patches; and biolistic particle injection. The operating methods and relative performance of these approaches are discussed, with a comment on potential future developments and technologies. (c) 2005 Elsevier Ltd. All rights reserved.

Making Fedora easier to implement with Fez: A free open source content model and workflow management front-end to Fedora

The University of Queensland, Australia has developed Fez, a world-leading user-interface and management system for Fedora-based institutional repositories, which bridges the gap between a repository and users. Christiaan Kortekaas, Andrew Bennett and Keith Webster will review this open source software that gives institutions the power to create a comprehensive repository solution without the hassle..

Free-Surface Aeration and Momentum Exchange at a Bottom Outlet

This study aims to provide some new understanding of the air-water flow properties in high-velocity water jets discharging past an abrupt drop. Such a setup has been little studied to date despite the relevance to bottom outlets. Downstream of the step brink, the free-jet entrains air at both upper and lower air-water interfaces, as well as along the sides. An air-water shear layer develops at the lower nappe interface. At the lower nappe, the velocity redistribution was successfully modelled and the velocity field was found to be similar to that in two-dimensional wake flow. The results highlighted further two distinct flow regions. Close to the brink (Wex < 5000), the flow was dominated by momentum transfer. Further downstream (Wex > 5000), a strong competition between air bubble diffusion and momentum exchanges took place.

Pedagogical work and the 'cult of the body': Considering the role of HPE in the context of the 'new public health'

Physical education, now often explicitly identified with health in contemporary school curricula, continues to be implicated in the (re)production of the 'cult of the body'. We argue that HPE is a form of health promotion that attempts to 'make' healthy citizens of young people in the context of the 'risk society'. In our view there is still work to be done in understanding how and why physical education (as HPE) continues to be implicated in the reproduction of values associated with the cult of body. We are keen to understand why HPE continues to be ineffective in helping young people gain some measure of analytic and embodied 'distance' from the problematic aspects of the cult of the body. This paper offers an analysis of this enduring issue by using some contemporary analytic discourses including 'governmentality', 'risk society' and the 'new public health'.

Quantum noise in optical fibers. I. Stochastic equations

We analyze the quantum dynamics of radiation propagating in a single-mode optical fiber with dispersion, nonlinearity, and Raman coupling to thermal phonons. We start from a fundamental Hamiltonian that includes the principal known nonlinear effects and quantum-noise sources, including linear gain and loss. Both Markovian and frequency-dependent, non-Markovian reservoirs are treated. This treatment allows quantum Langevin equations, which have a classical form except for additional quantum-noise terms, to be calculated. In practical calculations, it is more useful to transform to Wigner or 1P quasi-probability operator representations. These transformations result in stochastic equations that can be analyzed by use of perturbation theory or exact numerical techniques. The results have applications to fiber-optics communications, networking, and sensor technology.

Time-reversal test for stochastic quantum dynamics

The calculation of quantum dynamics is currently a central issue in theoretical physics, with diverse applications ranging from ultracold atomic Bose-Einstein condensates to condensed matter, biology, and even astrophysics. Here we demonstrate a conceptually simple method of determining the regime of validity of stochastic simulations of unitary quantum dynamics by employing a time-reversal test. We apply this test to a simulation of the evolution of a quantum anharmonic oscillator with up to 6.022×1023 (Avogadro's number) of particles. This system is realizable as a Bose-Einstein condensate in an optical lattice, for which the time-reversal procedure could be implemented experimentally.

Canonical Bose gas simulations with stochastic gauges

A technique to simulate the grand canonical ensembles of interacting Bose gases is presented. Results are generated for many temperatures by averaging over energy-weighted stochastic paths, each corresponding to a solution of coupled Gross-Pitaevskii equations with phase noise. The stochastic gauge method used relies on an off-diagonal coherent-state expansion, thus taking into account all quantum correlations. As an example, the second-order spatial correlation function and momentum distribution for an interacting 1D Bose gas are calculated.

Experimental Verification of Decoherence-Free Subspaces

Using spontaneous parametric down-conversion, we produce polarization-entangled states of two photons and characterize them using two-photon tomography to measure the density matrix. A controllable decoherence is imposed on the states by passing the photons through thick, adjustable birefringent elements. When the system is subject to collective decoherence, one particular entangled state is seen to be decoherence-free, as predicted by theory. Such decoherence-free systems may have an important role for the future of quantum computation and information processing.