Solving MDPs using two-timescale simulated annealing with multiplicative weights

We develop extensions of the Simulated Annealing with Multiplicative Weights (SAMW) algorithm that proposed a method of solution of Finite-Horizon Markov Decision Processes (FH-MDPs). The extensions developed are in three directions: a) Use of the dynamic programming principle in the policy update step of SAMW b) A two-timescale actor-critic algorithm that uses simulated transitions alone, and c) Extending the algorithm to the infinite-horizon discounted-reward scenario. In particular, a) reduces the storage required from exponential to linear in the number of actions per stage-state pair. On the faster timescale, a 'critic' recursion performs policy evaluation while on the slower timescale an 'actor' recursion performs policy improvement using SAMW. We give a proof outlining convergence w.p. 1 and show experimental results on two settings: semiconductor fabrication and flow control in communication networks.

Solution of MDPS using simulation-based value iteration

This article proposes a three-timescale simulation based algorithm for solution of infinite horizon Markov Decision Processes (MDPs). We assume a finite state space and discounted cost criterion and adopt the value iteration approach. An approximation of the Dynamic Programming operator T is applied to the value function iterates. This 'approximate' operator is implemented using three timescales, the slowest of which updates the value function iterates. On the middle timescale we perform a gradient search over the feasible action set of each state using Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates, thus finding the minimizing action in T. On the fastest timescale, the 'critic' estimates, over which the gradient search is performed, are obtained. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is also presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are performed using the proposed algorithm. The results obtained are verified against classical value iteration where the feasible set is suitably discretized. Over such a discretized setting, a variant of the algorithm of [12] is compared and the proposed algorithm is found to converge faster.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

An Actor-Critic Algorithm for Finite Horizon Markov Decision Processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Distributed Simulation Of Flexible Manufacturing Systems

Flexible Manufacturing Systems (FMS), widely considered as the manufacturing technology of the future, are gaining increasing importance due to the immense advantages they provide in terms of cost, quality and productivity over the conventional manufacturing. An FMS is a complex interconnection of capital intensive resources and high levels of system performance is very crucial for survival in a competing environment.Discrete event simulation is one of the most popular methods for performance evaluation of FMS during planning, design and operation phases. Indeed fast simulators are suggested for selection of optimal strategies for flow control (which part type to enter and at what instant), AGV scheduling (which vehicle to carry which part), routing (which machine to process the part) and part selection (which part for processing next). In this paper we develop a C-net based model for an FMS and use the same for distributed discrete event simulation. We illustrate using examples the efficacy of destributed discrete event simulation for the performance evaluation of FMSs.

Investigation of Vaporized Kerosene Injection and Combustion in a Supersonic Model Combustor

Injection and combustion of vaporized kerosene was experimentally investigated in a Mach 2.5 model combustor at various fuel temperatures and injection pressures. A unique kerosene heating and delivery system, which can prepare heated kerosene up to 820 K at a pressure of 5.5 MPa with negligible fuel coking, was developed. A three-species surrogate was employed to simulate the thermophysical properties of kerosene. The calculated thermophysical properties of surrogate provided insight into the fuel flow control in experiments. Kerosene jet structures at various preheat temperatures injecting into both quiescent environment and a Mach 2.5 crossflow were characterized. It was shown that the use ofvaporized kerosene injection holds the potential of enhancing fuel-air mixing and promoting overall burning. Supersonic combustion tests further confirmed the preceding conjecture by comparing the combustor performances of supercritical kerosene with those of liquid kerosene and effervescent atomization with hydrogen barbotage. Under the similar flow conditions and overall kerosene equivalence ratios, experimental results illustrated that the combustion efficiency of supercritical kerosene increased approximately 10-15% over that of liquid kerosene, which was comparable to that of effervescent atomization.

Rotor - Stator broadband noise prediction

In this paper a semi analytic model for rotor - stator broadband noise is presented. The work can be split into two sections. The first examines the distortion of the rotor wake in mean swirling flow, downstream of the fan. Previous work by Cooper and Peake4 is extended to include dissipative effects. In the second section we consider the interaction of this gust with the downstream stator row. We examine the way in which an unsteady pressure field is generated by the interaction of this wake flow with the stator blades and obtain estimates for the radiated noise. A new method is presented to extend the well known LINSUB code to the third dimension to capture the effect of the spanwise wavenumber and stator lean and sweep. Copyright © 2008 by Adrian Lloyd and Nigel Peake.

The wave model of mesothermal plasma near wakes and Korteweg-de Vries equation

The stationary two-dimensional (x, z) near wakes behind a flat-based projectile which moves at a constant mesothermal speed (V∞) along a z-axis in a rarefied, fully ionized, plasma is studied using the wave model previously proposed by one of the authors (VCL). One-fluid theory is used to depict the free expansion of ambient plasma into the vacuum produced behind a fast-moving projectile. This nonstationary, one-dimensional (x, t) flow which is approximated by the K-dV equation can be transformed, through substitution, t=z/V∞, into a stationary two-dimensional (x, z) near wake flow seen by an observer moving with the body velocity (V∞). The initial value problem of the K-dV equation in (x, t) variables is solved by a specially devised numerical method. Comparisons of the present numerical solution for the asymptotically small and large times with available analytical solutions are made and found in satisfactory agreements.

Investigation of Vaporized Kerosene Injection and Combustion in a Supersonic Model Combustor

Injection and combustion of vaporized kerosene was experimentally investigated in a Mach 2.5 model combustor at various fuel temperatures and injection pressures. A unique kerosene heating and delivery system, which can prepare heated kerosene up to 820 K at a pressure of 5.5 MPa with negligible fuel coking, was developed. A three-species surrogate was employed to simulate the thermophysical properties of kerosene. The calculated thermophysical properties of surrogate provided insight into the fuel flow control in experiments. Kerosene jet structures at various preheat temperatures injecting into both quiescent environment and a Mach 2.5 crossflow were characterized. It was shown that the use ofvaporized kerosene injection holds the potential of enhancing fuel-air mixing and promoting overall burning. Supersonic combustion tests further confirmed the preceding conjecture by comparing the combustor performances of supercritical kerosene with those of liquid kerosene and effervescent atomization with hydrogen barbotage. Under the similar flow conditions and overall kerosene equivalence ratios, experimental results illustrated that the combustion efficiency of supercritical kerosene increased approximately 10-15% over that of liquid kerosene, which was comparable to that of effervescent atomization.

Experimental Characterization Of Electrical Current Leakage In Poly(Dimethylsiloxane) Microfluidic Devices

Poly(dimethylsiloxane) (PDMS) is usually considered as a dielectric material and the PDMS microchannel wall can be treated as an electrically insulated boundary in an applied electric field. However, in certain layouts of microfluidic networks, electrical leakage through the PDMS microfluidic channel walls may not be negligible, which must be carefully considered in the microfluidic circuit design. In this paper, we report on the experimental characterization of the electrical leakage current through PDMS microfluidic channel walls of different configurations. Our numerical and experimental studies indicate that for tens of microns thick PDMS channel walls, electrical leakage through the PDMS wall could significantly alter the electrical field in the main channel. We further show that we can use the electrical leakage through the PDMS microfluidic channel wall to control the electrolyte flow inside the microfluidic channel and manipulate the particle motion inside the microfluidic channel. More specifically, we can trap individual particles at different locations inside the microfluidic channel by balancing the electroosmotic flow and the electrophoretic migration of the particle.

Physical Modeling And Parametric Study On Two-Degree-Of-Freedom Viv Of A Cylinder Near Rigid Wall

Unlike most previous studies on the transverse vortex-induced vibration(VIV) of a cylinder mainly under the wallfree condition (Williamson & Govardhan,2004),this paper experimentally investigates the vortex-induced vibration of a cylinder with two degrees of freedom near a rigid wall exposed to steady flow.The amplitude and frequency responses of the cylinder are discussed.The lee wake flow patterns of the cylinder undergoing VIV were visualized by employing the hydrogen bubble technique.The effects of the gap-to-diameter ratio (e0/D) and the mass ratio on the vibration amplitude and frequency are analyzed.Comparisons of VIV response of the cylinder are made between one degree (only transverse) and two degrees of freedom (streamwise and transverse) and those between the present study and previous ones.The experimental observation indicates that there are two types of streamwise vibration,i.e.the first streamwise vibration (FSV) with small amplitude and the second streamwise vibration (SSV) which coexists with transverse vibration.The vortex shedding pattem for the FSV is approximately symmetric and that for the SSV is alternate.The first streamwise vibration tends to disappear with the decrease of e0/D.For the case of large gap-to-diameter ratios (e.g.e0/D = 0.54～1.58),the maximum amplitudes of the second streamwise vibration and transverse one increase with the increasing gapto-diameter ratio.But for the case of small gap-to-diameter ratios (e.g.e0/D = 0.16,0.23),the vibration amplitude of the cylinder increases slowly at the initial stage (i.e.at small reduced velocity V,),and across the maximum amplitude it decreases quickly at the last stage (i.e.at large Vr).Within the range ofthe examined small mass ratio (m＜4),both streamwise and transverse vibration amplitude of the cylinder decrease with the increase of mass ratio for the fixed value of V,.The vibration range (in terms of Vr ) tends to widen with the decrease of the mass ratio.In the second streamwise vibration region,the vibration frequency of the cylinder with a small mass ratio (e.g.mx = 1.44) undergoes a jump at a certain Vr,.The maximum amplitudes of the transverse vibration for two-degree-of-freedom case is larger than that for one-degree-of-freedom case,but the transverse vibration frequency of the cylinder with two degrees of freedom is lower than that with one degree of freedom (transverse).

Telhados verdes para habitações de interesse social: retenção das águas pluviais e conforto térmico.

O crescimento populacional aliado à migração tem aumentado a pressão sobre o uso do solo urbano perpetuando sucessivos problemas de assentamentos informais e saneamento ambiental nos grandes centros. Esta situação se agrava ainda mais em épocas de chuvas intensas devido à ocorrência de enchentes. Este projeto faz parte de um conjunto de ações integradas de cidadania e inclusão social na região hidrográfica da baixada de Jacarepaguá, especificamente envolvendo a Comunidade da Vila Cascatinha, em Vargem Grande, a fim de gerar subsídios para políticas públicas em áreas de assentamentos informais, integrado ao projeto HIDROCIDADES (CNPq/CTHIDRO/CTAGRO), que visa a conservação da água em meios urbanos e periurbanos associado à cidadania, inclusão social e melhoria da qualidade de vida nas grandes cidades. Este projeto utilizou uma tecnologia adaptada dos telhados verdes para edificação popular (telhado de fibrocimento), com o objetivo de verificar aspectos construtivos, possíveis espécies com potencial de geração de renda, custos, efeitos no retardo do escoamento superficial das águas pluviais e outros benefícios associados a questões climáticas locais e de conforto do ambiente interno. Os resultados gerados demonstraram, entre outros, o estabelecimento de metodologia para implantação dos telhados verdes em habitações populares, o valor dos custos e resultados preliminares de espécies com potencial para geração de renda. Ainda, a implantação dos telhados verdes demonstrou ser promissora no controle do escoamento superficial, na aplicação do sistema de irrigação. Na simulação das chuvas, observou-se uma retenção de até 56% do volume precipitado. Observou-se o retardo da ocorrência do pico de até 8 minutos no telhado vegetado em relação ao telhado testemunho (convencional telhas fibrocimento). Foi observada a eficiência tanto no comportamento térmico interno como também no externo, uma redução da amplitude térmica interna em dia característico de verão (35,9 C), sendo capaz de r eduzir a temperatura interna em cerca de 2,0 C nos períodos mais quentes do dia e cerca de 4,0 C no ambiente externo em comparação com o telhado-testemunha (sem plantio), com potencial de modificação do microclima local.

Behaviour of unsteady transonic shock / boundary layer interactions with three-dimensional effects

An experimental investigation into the response of transonic SBLIs to periodic down-stream pressure perturbations in a parallel walled duct has been conducted. Tests have been carried out with a shock strength of M ∞ = 1.5 for pressure perturbation frequencies in the range 16-90 Hz. Analysis of the steady interaction at M∞ = 1.5 has also been made. The principle measurement techniques were high speed schlieren photography and laser Doppler anemometry. The structure of the steady SBLI was found to be highly three-dimensional, with large corner flows and sidewall SBLIs. These aspects are thought to influence the upstream transmission of pressure information through the interaction by affecting the post-shock flow field, including the extent of regions of secondary supersonic flow. At low frequency, the dynamics of shock motion can be predicted using an inviscid analytical model. At increased frequencies, viscous effects become significant and the shock exhibits unexpected dynamic behaviour, due to a phase lag between the upstream transmission of pressure information in the core flow and in the viscous boundary layers. Flow control in the form of micro-vane vortex generators was found to have a small impact on shock dynamics, due to the effect it had on the post-shock flow field outside the viscous boundary layer region. The relationship between inviscid and viscous effects is developed and potential destabilising mechanisms for SBLIs in practical applications are suggested. Copyright © 2009 by Paul Bruce and Holger Babinsky.