Bounded parameter Markov Decision Processes with average reward criterion

Bounded parameter Markov Decision Processes (BMDPs) address the issue of dealing with uncertainty in the parameters of a Markov Decision Process (MDP). Unlike the case of an MDP, the notion of an optimal policy for a BMDP is not entirely straightforward. We consider two notions of optimality based on optimistic and pessimistic criteria. These have been analyzed for discounted BMDPs. Here we provide results for average reward BMDPs. We establish a fundamental relationship between the discounted and the average reward problems, prove the existence of Blackwell optimal policies and, for both notions of optimality, derive algorithms that converge to the optimal value function.

Exponentiated gradient algorithms for conditional random fields and max-margin Markov Networks

Log-linear and maximum-margin models are two commonly-used methods in supervised machine learning, and are frequently used in structured prediction problems. Efficient learning of parameters in these models is therefore an important problem, and becomes a key factor when learning from very large data sets. This paper describes exponentiated gradient (EG) algorithms for training such models, where EG updates are applied to the convex dual of either the log-linear or max-margin objective function; the dual in both the log-linear and max-margin cases corresponds to minimizing a convex function with simplex constraints. We study both batch and online variants of the algorithm, and provide rates of convergence for both cases. In the max-margin case, O(1/ε) EG updates are required to reach a given accuracy ε in the dual; in contrast, for log-linear models only O(log(1/ε)) updates are required. For both the max-margin and log-linear cases, our bounds suggest that the online EG algorithm requires a factor of n less computation to reach a desired accuracy than the batch EG algorithm, where n is the number of training examples. Our experiments confirm that the online algorithms are much faster than the batch algorithms in practice. We describe how the EG updates factor in a convenient way for structured prediction problems, allowing the algorithms to be efficiently applied to problems such as sequence learning or natural language parsing. We perform extensive evaluation of the algorithms, comparing them to L-BFGS and stochastic gradient descent for log-linear models, and to SVM-Struct for max-margin models. The algorithms are applied to a multi-class problem as well as to a more complex large-scale parsing task. In all these settings, the EG algorithms presented here outperform the other methods.

Radiation effect on free convection laminar flow along a vertical flat plate with streamwise sinusoidal surface temperature

The effect of thermal radiation on a steady two-dimensional natural convection laminar flow of viscous incompressible optically thick fluid along a vertical flat plate with streamwise sinusoidal surface temperature has been investigated in this study. Using the appropriate variables; the basic governing equations are transformed to convenient form and then solved numerically employing two efficient methods, namely, Implicit finite difference method (IFD) together with Keller box scheme and Straight forward finite difference (SFFD) method. Effects of the variation of the physical parameters, for example, conduction-radiation parameter (Planck number), surface temperature parameter, and the amplitude of the surface temperature, are shown on the skin friction and heat transfer rate quantitatively are shown numerically. Velocity and temperature profiles as well as streamlines and isotherms are also presented and discussed for the variation of conduction-radiation parameter. It is found that both skin-friction and rate of heat transfer are enhanced considerably by increasing the values of conduction radiation parameter, Rd.

A compact interferometric sensor design using three waveguide coupling

The use of metal stripes for the guiding of plasmons is a well established technique for the infrared regime and has resulted in the development of a myriad of passive optical components and sensing devices. However, the plasmons suffer from large losses around sharp bends, making the compact design of nanoscale sensors and circuits problematic. A compact alternative would be to use evanescent coupling between two sufficiently close stripes, and thus we propose a compact interferometer design using evanescent coupling. The sensitivity of the design is compared with that achieved using a hand-held sensor based on the Kretschmann style surface plasmon resonance technique. Modeling of the new interferometric sensor is performed for various structural parameters using finite-difference time-domain and COMSOL Multiphysics. The physical mechanisms behind the coupling and propagation of plasmons in this structure are explained in terms of the allowed modes in each section of the device.

Dynamics of attacker–defender dyads in Association Football : parameters influencing decision-making

Previous work on pattern-forming dynamics of team sports has investigated sub-phases of basketball and rugby union by focussing on one-versus-one (1v1) attacker-defender dyads. This body of work has identified the role of candidate control parameters, interpersonal distance and relative velocity, in predicting the outcomes of team player interactions. These two control parameters have been described as functioning in a nested relationship where relative velocity between players comes to the fore within a critical range of interpersonal distance. The critical influence of constraints on the intentionality of player behaviour has also been identified through the study of 1v1 attacker-defender dyads. This thesis draws from previous work adopting an ecological dynamics approach, which encompasses both Dynamical Systems Theory and Ecological Psychology concepts, to describe attacker-defender interactions in 1v1 dyads in association football. Twelve male youth association football players (average age 15.3 ± 0.5 yrs) performed as both attackers and defenders in 1v1 dyads in three field positions in an experimental manipulation of the proximity to goal and the role of players. Player and ball motion was tracked using TACTO 8.0 software (Fernandes & Caixinha, 2003) to produce two-dimensional (2D) trajectories of players and the ball on the ground. Significant differences were found for player-to-ball interactions depending on proximity to goal manipulations, indicating how key reference points in the environment such as the location of the goal may act as a constraint that shapes decision-making behaviour. Results also revealed that interpersonal distance and relative velocity alone were insufficient for accurately predicting the outcome of a dyad in association football. Instead, combined values of interpersonal distance, ball-to-defender distance, attacker-to-ball distance, attacker-to-ball relative velocity and relative angles were found to indicate the state of dyad outcomes.

The Bayesian conditional independence model for measurement error: applications in ecology

The measurement error model is a well established statistical method for regression problems in medical sciences, although rarely used in ecological studies. While the situations in which it is appropriate may be less common in ecology, there are instances in which there may be benefits in its use for prediction and estimation of parameters of interest. We have chosen to explore this topic using a conditional independence model in a Bayesian framework using a Gibbs sampler, as this gives a great deal of flexibility, allowing us to analyse a number of different models without losing generality. Using simulations and two examples, we show how the conditional independence model can be used in ecology, and when it is appropriate.

Some novel techniques of parameter estimation for the dynamical models in biological systems

Inverse problems based on using experimental data to estimate unknown parameters of a system often arise in biological and chaotic systems. In this paper, we consider parameter estimation in systems biology involving linear and non-linear complex dynamical models, including the Michaelis–Menten enzyme kinetic system, a dynamical model of competence induction in Bacillus subtilis bacteria and a model of feedback bypass in B. subtilis bacteria. We propose some novel techniques for inverse problems. Firstly, we establish an approximation of a non-linear differential algebraic equation that corresponds to the given biological systems. Secondly, we use the Picard contraction mapping, collage methods and numerical integration techniques to convert the parameter estimation into a minimization problem of the parameters. We propose two optimization techniques: a grid approximation method and a modified hybrid Nelder–Mead simplex search and particle swarm optimization (MH-NMSS-PSO) for non-linear parameter estimation. The two techniques are used for parameter estimation in a model of competence induction in B. subtilis bacteria with noisy data. The MH-NMSS-PSO scheme is applied to a dynamical model of competence induction in B. subtilis bacteria based on experimental data and the model for feedback bypass. Numerical results demonstrate the effectiveness of our approach.

Exploring the parameter space of a rabbit ventricular action potential model to investigate the effect of variation on action potential and calcium transients

Computational models for cardiomyocyte action potentials (AP) often make use of a large parameter set. This parameter set can contain some elements that are fitted to experimental data independently of any other element, some elements that are derived concurrently with other elements to match experimental data, and some elements that are derived purely from phenomenological fitting to produce the desired AP output. Furthermore, models can make use of several different data sets, not always derived for the same conditions or even the same species. It is consequently uncertain whether the parameter set for a given model is physiologically accurate. Furthermore, it is only recently that the possibility of degeneracy in parameter values in producing a given simulation output has started to be addressed. In this study, we examine the effects of varying two parameters (the L-type calcium current (I(CaL)) and the delayed rectifier potassium current (I(Ks))) in a computational model of a rabbit ventricular cardiomyocyte AP on both the membrane potential (V(m)) and calcium (Ca(2+)) transient. It will subsequently be determined if there is degeneracy in this model to these parameter values, which will have important implications on the stability of these models to cell-to-cell parameter variation, and also whether the current methodology for generating parameter values is flawed. The accuracy of AP duration (APD) as an indicator of AP shape will also be assessed.

Parameter estimation for a phenomenological model of the cardiac action potential

The action potential (ap) of a cardiac cell is made up of a complex balance of ionic currents which flow across the cell membrane in response to electrical excitation of the cell. Biophysically detailed mathematical models of the ap have grown larger in terms of the variables and parameters required to model new findings in subcellular ionic mechanisms. The fitting of parameters to such models has seen a large degree of parameter and module re-use from earlier models. An alternative method for modelling electrically exciteable cardiac tissue is a phenomenological model, which reconstructs tissue level ap wave behaviour without subcellular details. A new parameter estimation technique to fit the morphology of the ap in a four variable phenomenological model is presented. An approximation of a nonlinear ordinary differential equation model is established that corresponds to the given phenomenological model of the cardiac ap. The parameter estimation problem is converted into a minimisation problem for the unknown parameters. A modified hybrid Nelder–Mead simplex search and particle swarm optimization is then used to solve the minimisation problem for the unknown parameters. The successful fitting of data generated from a well known biophysically detailed model is demonstrated. A successful fit to an experimental ap recording that contains both noise and experimental artefacts is also produced. The parameter estimation method’s ability to fit a complex morphology to a model with substantially more parameters than previously used is established.

The African Institute for Mathematical Sciences

Its mission is to promote Mathematics and Science in Africa and to provide a focal point for Mathematics university training in Africa. It offers scholarships for up to 50 students to come and study for a period of nine months. Of the 50 students, about 15 positions are reserved for females. In the 2006/2007 intake there were over 250 applicants. The students are housed and fed and their return travel from their home town is fully funded. Lecturers also stay at AIMS and share their meals with the students, so that a rapport quickly develops. The students are away from their families and friends for nine months and are absolutely committed to the discipline of Mathematics. When they first arrive, some of them have little ability in English but since all tuition is in English they quickly learn. Some find the transitions difficult but they all support one another and at the end of their time their English skills are very good. The students do a series of subjects that last for about three weeks each, consisting of 30 contact hours, as well as a thesis/project. Each course has a number of assignments associated with it and these get evaluated. AIMS has seven or eight teaching assistants who help with the tutorials, marking, advice, and who are a vital component of AIMS.

A continuous time model for election timing

We consider a continuous time model for election timing in a Majoritarian Parliamentary System where the government maintains a constitutional right to call an early election. Our model is based on the two-party-preferred data that measure the popularity of the government and the opposition over time. We describe the poll process by a Stochastic Differential Equation (SDE) and use a martingale approach to derive a Partial Differential Equation (PDE) for the government’s expected remaining life in office. A comparison is made between a three-year and a four-year maximum term and we also provide the exercise boundary for calling an election. Impacts on changes in parameters in the SDE, the probability of winning the election and maximum terms on the call exercise boundaries are discussed and analysed. An application of our model to the Australian Federal Election for House of Representatives is also given.