Solution of MDPS using simulation-based value iteration

This article proposes a three-timescale simulation based algorithm for solution of infinite horizon Markov Decision Processes (MDPs). We assume a finite state space and discounted cost criterion and adopt the value iteration approach. An approximation of the Dynamic Programming operator T is applied to the value function iterates. This 'approximate' operator is implemented using three timescales, the slowest of which updates the value function iterates. On the middle timescale we perform a gradient search over the feasible action set of each state using Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates, thus finding the minimizing action in T. On the fastest timescale, the 'critic' estimates, over which the gradient search is performed, are obtained. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is also presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are performed using the proposed algorithm. The results obtained are verified against classical value iteration where the feasible set is suitably discretized. Over such a discretized setting, a variant of the algorithm of [12] is compared and the proposed algorithm is found to converge faster.

Optimal threshold policies for admission control in communication networks via discrete parameter stochastic approximation

The problem of admission control of packets in communication networks is studied in the continuous time queueing framework under different classes of service and delayed information feedback. We develop and use a variant of a simulation based two timescale simultaneous perturbation stochastic approximation (SPSA) algorithm for finding an optimal feedback policy within the class of threshold type policies. Even though SPSA has originally been designed for continuous parameter optimization, its variant for the discrete parameter case is seen to work well. We give a proof of the hypothesis needed to show convergence of the algorithm on our setting along with a sketch of the convergence analysis. Extensive numerical experiments with the algorithm are illustrated for different parameter specifications. In particular, we study the effect of feedback delays on the system performance.

SPSA algorithms with measurement reuse

Four algorithms, all variants of Simultaneous Perturbation Stochastic Approximation (SPSA), are proposed. The original one-measurement SPSA uses an estimate of the gradient of objective function L containing an additional bias term not seen in two-measurement SPSA. As a result, the asymptotic covariance matrix of the iterate convergence process has a bias term. We propose a one-measurement algorithm that eliminates this bias, and has asymptotic convergence properties making for easier comparison with the two-measurement SPSA. The algorithm, under certain conditions, outperforms both forms of SPSA with the only overhead being the storage of a single measurement. We also propose a similar algorithm that uses perturbations obtained from normalized Hadamard matrices. The convergence w.p. 1 of both algorithms is established. We extend measurement reuse to design two second-order SPSA algorithms and sketch the convergence analysis. Finally, we present simulation results on an illustrative minimization problem.

Social selection and the evolution of cooperative groups: The example of the cellular slime moulds

In social selection the phenotype of an individual depends on its own genotype as well as on the phenotypes, and so genotypes, of other individuals. This makes it impossible to associate an invariant phenotype with a genotype: the social context is crucial. Descriptions of metazoan development, which often is viewed as the acme of cooperative social behaviour, ignore or downplay this fact. The implicit justification for doing so is based on a group-selectionist point of view. Namely, embryos are clones, therefore all cells have the same evolutionary interest, and the visible differences between cells result from a common strategy. The reasoning is flawed, because phenotypic heterogeneity within groups can result from contingent choices made by cells from a flexible repertoire as in multicellular development. What makes that possible is phenotypic plasticity, namely the ability of a genotype to exhibit different phenotypes. However, co-operative social behaviour with division of labour requires that different phenotypes interact appropriately, not that they belong to the same genotype, or have overlapping genetic interests. We sketch a possible route to the evolution of social groups that involves many steps: (a) individuals that happen to be in spatial proximity benefit simply by virtue of their number; (b) traits that are already present act as preadaptations and improve the efficiency of the group; and (c) new adaptations evolve under selection in the social context-that is, via interactions between individuals-and further strengthen group behaviour. The Dictyostelid or cellular slime mould amoebae (CSMs) become multicellular in an unusual way, by the aggregation of free-living cells. In nature the resulting group can be genetically homogeneous (clonal) or heterogeneous (polyclonal); in either case its development, which displays strong cooperation between cells (to the extent of so-called altruism) is not affected. This makes the CSMs exemplars for the study of social behaviour.

Two-timescale Q-learning Algorithms with an Application to Routing in Networks

We propose two variants of the Q-learning algorithm that (both) use two timescales. One of these updates Q-values of all feasible state-action pairs at each instant while the other updates Q-values of states with actions chosen according to the ‘current ’ randomized policy updates. A sketch of convergence of the algorithms is shown. Finally, numerical experiments using the proposed algorithms for routing on different network topologies are presented and performance comparisons with the regular Q-learning algorithm are shown.

Ontheproof of the Poincare´ conjecture

In 2002, Perelman proved the Poincare conjecture, building on the work of Richard Hamilton on the Ricci flow. In this article, we sketch some of the arguments and attempt to place them in a broader dynamical context.

On the proof of the Poincare´ conjecture

In 2002, Perelman proved the Poincare conjecture, building on the work of Richard Hamilton on the Ricci flow. In this article, we sketch some of the arguments and attempt to place them in a broader dynamical context.

A POSSIBLE EVOLUTIONARY SCENARIO OF HIGHLY MAGNETIZED SUPER-CHANDRASEKHAR WHITE DWARFS: PROGENITORS OF PECULIAR TYPE Ia SUPERNOVAE

Several recently discovered peculiar Type Ia supernovae seem to demand an altogether new formation theory that might help explain the puzzling dissimilarities between them and the standard Type Ia supernovae. The most striking aspect of the observational analysis is the necessity of invoking super-Chandrasekhar white dwarfs having masses similar to 2.1-2.8 M-circle dot, M-circle dot being the mass of Sun, as their most probable progenitors. Strongly magnetized white dwarfs having super-Chandrasekhar masses have already been established as potential candidates for the progenitors of peculiar Type Ia supernovae. Owing to the Landau quantization of the underlying electron degenerate gas, theoretical results yielded the observationally inferred mass range. Here, we sketch a possible evolutionary scenario by which super-Chandrasekhar white dwarfs could be formed by accretion on to a commonly observed magnetized white dwarf, invoking the phenomenon of flux freezing. This opens multiple possible evolution scenarios ending in supernova explosions of super-Chandrasekhar white dwarfs having masses within the range stated above. We point out that our proposal has observational support, such as the recent discovery of a large number of magnetized white dwarfs by the Sloan Digital Sky Survey.

Behaviour simulation in computer aided product concept sketching

In the product conceptualization phase of design, sketches are often used for exploration of diverse behaviour patterns of the components to achieve the required functionality. This paper presents a method to animate the sketch produced using a tablet interface to aid verification of the desired behaviour. A sketch is a spatial organization of strokes whose perceptual organization helps one to visually interpret its components and their interconnections. A Gestalt based segmentation followed by interactive grouping and articulation, presented in this paper, enables one to use a mechanism simulation framework to animate the sketch in a “pick and drag” mode to visualize different configurations of the product and gain insight into the product’s behaviour.

Constrained 3D sketching with haptics and active motion annotation

The aim of this work is to enable seamless transformation of product concepts to CAD models. This necessitates availability of 3D product sketches. The present work concerns intuitive generation of 3D strokes and intrinsic support for space sharing and articulation for the components of the product being sketched. Direct creation of 3D strokes in air lacks in precision, stability and control. The inadequacy of proprioceptive feedback for the task is complimented in this work with stereo vision and haptics. Three novel methods based on pencil-paper interaction analogy for haptic rendering of strokes have been investigated. The pen-tilt based rendering is simpler and found to be more effective. For the spatial conformity, two modes of constraints for the stylus movements, corresponding to the motions on a control surface and in a control volume have been studied using novel reactive and field based haptic rendering schemes. The field based haptics, which in effect creates an attractive force field near a surface, though non-realistic, provided highly effective support for the control-surface constraints. The efficacy of the reactive haptic rendering scheme for the constrained environments has been demonstrated using scribble strokes. This can enable distributed collaborative 3D concept development. The notion of motion constraints, defined through sketch strokes enables intuitive generation of articulated 3D sketches and direct exploration of motion annotations found in most product concepts. The work, thus, establishes that modeling of the constraints is a central issue in 3D sketching.

Checking Liveness Properties of Presburger Counter Systems Using Reachability Analysis

Counter systems are a well-known and powerful modeling notation for specifying infinite-state systems. In this paper we target the problem of checking liveness properties in counter systems. We propose two semi decision techniques towards this, both of which return a formula that encodes the set of reachable states of the system that satisfy a given liveness property. A novel aspect of our techniques is that they use reachability analysis techniques, which are well studied in the literature, as black boxes, and are hence able to compute precise answers on a much wider class of systems than previous approaches for the same problem. Secondly, they compute their results by iterative expansion or contraction, and hence permit an approximate solution to be obtained at any point. We state the formal properties of our techniques, and also provide experimental results using standard benchmarks to show the usefulness of our approaches. Finally, we sketch an extension of our liveness checking approach to check general CTL properties.

Boundary Tracking using a Hybrid Cyclic Pursuit Scheme

There has been a lot of work in the literature, related to the mapping of boundaries of regions, using multiple agents. Most of these are based on optimization techniques or rely on potential fields to drive the agents towards the boundary and then retain them there while they space out evenly along the perimeter or surface (in two-dimensional and three-dimensional cases, respectively). In this paper an algorithm to track the boundary of a region in space is provided based on the cyclic pursuit scheme. This enables the agents to constantly move along the perimeter in a cluster, thereby tracking a dynamically changing boundary. The trajectories of the agents provide a sketch of the boundary. The use of multiple agents may facilitate minimization of tracking error by providing accurate estimates of points on the boundary, besides providing redundancy. Simulation results are provided to highlight the performance of the proposed scheme.

The electronic, chemical and electrocatalytic processes and intermediates on iron oxide surfaces during photoelectrochemical water splitting

We re-assess experimental soft X-ray absorption spectra of the oxygen K-shell which we recorded operando from iron oxide during photoelectrochemical water splitting in KOH electrolyte. In particular, we refer to recently reported transitional electron hole states which originate within the charge carrier depletion layer of the iron oxide and on the iron oxide surface. For the latter we find that an intermediate oxy-peroxo species is formed on the iron oxide with increasing bias potential, which disappears upon further polarization of the electrode, concomitantly with the evolution and disappearance of the aforementioned surface state. The oxygen spectra contain also the spectroscopic signatures of the electrolyte water, the position of which changes with increasing bias potential towards lower X-ray energies, revealing the breaking and formation of hydrogen bonds in the water during the experiment. Combined with potential dependent impedance spectroscopy data we are able to sketch the molecular structure of chemical intermediates and their charge carrier dynamics. (C) 2015 Elsevier B.V. All rights reserved.

Nanoparticle induced miscibility in LCST polymer blends: critically assessing the enthalpic and entropic effects

The use of copolymer and polymer blends widened the possibility of creating materials with multilayered architectures. Hierarchical polymer systems with a wide array of micro and nanostructures are generated by thermally induced phase separation (TIPS) in partially miscible polymer blends. Various parameters like the interaction between the polymers, concentration, solvent/non-solvent ratio, and quenching temperature have to be optimized to obtain these micro/nanophase structures. Alternatively, the addition of nanoparticles is another strategy to design materials with desired hetero-phase structures. The dynamics of the polymer nanocomposite depends on the statistical ordering of polymers around the nanoparticle, which is dependent on the shape of the nanoparticle. The entropic loss due to deformation of polymer chains, like the repulsive interactions due to coiling and the attractive interactions in the case of swelling has been highlighted in this perspective article. The dissipative particle dynamics has been discussed and is correlated with the molecular dynamics simulation in the case of polymer blends. The Cahn Hillard Cook model on variedly shaped immobile fillers has shown difference in the propagation of the composition wave. The nanoparticle shape has a contributing effect on the polymer particle interaction, which can change the miscibility window in the case of these phase separating polymer blends. Quantitative information on the effect of spherical particles on the demixing temperature is well established and further modified to explain the percolation of rod shaped particles in the polymer blends. These models correlate well with the experimental observations in context to the dynamics induced by the nanoparticle in the demixing behavior of the polymer blend. The miscibility of the LCST polymer blend depends on the enthalpic factors like the specific interaction between the components, and the solubility product and the entropic losses occurring due to the formation of any favorable interactions. Hence, it is essential to assess the entropic and enthalpic interactions induced by the nanoparticles independently. The addition of nanoparticles creates heterogeneity in the polymer phase it is localized. This can be observed as an alteration in the relaxation behavior of the polymer. This changes the demixing behavior and the interaction parameter between the polymers. The compositional changes induced due to the incorporation of nanoparticles are also attributed as a reason for the altered demixing temperature. The particle shape anisotropy causes a direction dependent depletion, which changes the phase behavior of the blend. The polymer-grafted nanoparticles with varying grafting density show tremendous variation in the miscibility of the blend. The stretching of the polymer chains grafted on the nanoparticles causes an entropy penalty in the polymer blend. A comparative study on the different shaped particles is not available up to date for understanding these aspects. Hence, we have juxtaposed the various computational studies on nanoparticle dynamics, the shape effect of NPs on homopolymers and also the cases of various polymer blends without nanoparticles to sketch a complete picture on the effect of various particles on the miscibility of LCST blends.