Incremental natural-gradient actor-critic algorithms

We present four new reinforcement learning algorithms based on actor-critic and natural-gradient ideas, and provide their convergence proofs. Actor-critic rein- forcement learning methods are online approximations to policy iteration in which the value-function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent. Methods based on policy gradients in this way are of special interest because of their com- patibility with function approximation methods, which are needed to handle large or infinite state spaces. The use of temporal difference learning in this way is of interest because in many applications it dramatically reduces the variance of the gradient estimates. The use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further re- duce variance in some cases. Our results extend prior two-timescale convergence results for actor-critic methods by Konda and Tsitsiklis by using temporal differ- ence learning in the actor and by incorporating natural gradients, and they extend prior empirical studies of natural actor-critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms.

Natural actor-critic algorithms

We present four new reinforcement learning algorithms based on actor-critic, natural-gradient and functi approximation ideas,and we provide their convergence proofs. Actor-critic reinforcement learning methods are online approximations to policy iteration in which the value-function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent. Methods based on policy gradients in this way are of special interest because of their compatibility with function-approximation methods, which are needed to handle large or infinite state spaces. The use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates. The use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases. Our results extend prior two-timescale convergence results for actor-critic methods by Konda and Tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients. Our results extend prior empirical studies of natural actor-critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms.

A fuzzy dynamic flood routing model for natural channels

A fuzzy dynamic flood routing model (FDFRM) for natural channels is presented, wherein the flood wave can be approximated to a monoclinal wave. This study is based on modification of an earlier published work by the same authors, where the nature of the wave was of gravity type. Momentum equation of the dynamic wave model is replaced by a fuzzy rule based model, while retaining the continuity equation in its complete form. Hence, the FDFRM gets rid of the assumptions associated with the momentum equation. Also, it overcomes the necessity of calculating friction slope (S-f) in flood routing and hence the associated uncertainties are eliminated. The fuzzy rule based model is developed on an equation for wave velocity, which is obtained in terms of discontinuities in the gradient of flow parameters. The channel reach is divided into a number of approximately uniform sub-reaches. Training set required for development of the fuzzy rule based model for each sub-reach is obtained from discharge-area relationship at its mean section. For highly heterogeneous sub-reaches, optimized fuzzy rule based models are obtained by means of a neuro-fuzzy algorithm. For demonstration, the FDFRM is applied to flood routing problems in a fictitious channel with single uniform reach, in a fictitious channel with two uniform sub-reaches and also in a natural channel with a number of approximately uniform sub-reaches. It is observed that in cases of the fictitious channels, the FDFRM outputs match well with those of an implicit numerical model (INM), which solves the dynamic wave equations using an implicit numerical scheme. For the natural channel, the FDFRM Outputs are comparable to those of the HEC-RAS model.

Characterization of groundwater chemistry under the influence of lithologic and anthropogenic factors along a climatic gradient in Upper Cauvery basin, South India

Hydrogeological and climatic effect on chemical behavior of groundwater along a climatic gradient is studied along a river basin. `Semi-arid' (500-800 mm of mean annual rainfall), `sub-humid' (800-1,200 mm/year) and `humid' (1,200-1,500 mm/year) are the climatic zones chosen along the granito-gneissic plains of Kabini basin in South India for the present analysis. Data on groundwater chemistry is initially checked for its quality using NICB ratio (<+/- 5 %), EC versus TZ+ (similar to 0.85 correlation), EC versus TDS and EC versus TH analysis. Groundwater in the three climatic zones is `hard' to `very hard' in terms of Ca-Mg hardness. Polluted wells are identified (> 40 % of pollution) and eliminated for the characterization. Piper's diagram with mean concentrations indicates the evolution of CaNaHCO3 (semi-arid) from CaHCO3 (humid zone) along the climatic gradient. Carbonates dominate other anions and strong acids exceeded weak acids in the region. Mule Hole SEW, an experimental watershed in sub-humid zone, is characterized initially using hydrogeochemistry and is observed to be a replica of entire sub-humid zone (with 25 wells). Extension of the studies for the entire basin (120 wells) showed a chemical gradient along the climatic gradient with sub-humid zone bridging semi-arid and humid zones. Ca/Na molar ratio varies by more than 100 times from semi-arid to humid zones. Semi-arid zone is more silicaceous than sub-humid while humid zone is more carbonaceous (Ca/Cl similar to 14). Along the climatic gradient, groundwater is undersaturated (humid), saturated (sub-humid) and slightly supersaturated (semi-arid) with calcite and dolomite. Concentration-depth profiles are in support of the geological stratification i.e., not approximate to 18 m of saprolite and similar to 25 m of fracture rock with parent gneiss beneath. All the wells are classified into four groups based on groundwater fluctuations and further into `deep' and `shallow' based on the depth to groundwater. Higher the fluctuations, larger is its impact on groundwater chemistry. Actual seasonal patterns are identified using `recharge-discharge' concept based on rainfall intensity instead of traditional monsoon-non-monsoon concept. Non-pumped wells have low Na/Cl and Ca/Cl ratios in recharge period than in discharge period (Dilution). Few other wells, which are subjected to pumping, still exhibit dilution chemistry though water level fluctuations are high due to annual recharge. Other wells which do not receive sufficient rainfall and are constantly pumped showed high concentrations in recharge period rather than in discharge period (Anti-dilution). In summary, recharge-discharge concept demarcates the pumped wells from natural deep wells thus, characterizing the basin.

Durability of rammed earth walls exposed for 20 years to natural weathering

This paper presents a study on the durability of different types of stabilised and unstabilised rammed earth walls. These rammed earth walls were constructed and exposed for 20 years to natural weathering, in a wet continental climate. None of these walls have shown complete collapse to date. A method to measure the rammed earth walls erosion by stereo-photogrammetry has been developed. The result shows that the mean erosion depth of the studied walls is about 2 mm (0.5% wall thickness) in the case of rammed earth wall stabilised with 5% by dry weight of hydraulic lime and about 6.4 mm (1.6% wall thickness) in the case of unstabilised rammed earth walls. The stabilisation enables to not use any plaster to protect the walls. In the case of the unstabilised rammed earth walls, an extrapolated lifetime longer than 60 years can be assessed. This shows a potential for the use of unstabilised rammed earth in the similar climatic conditions with this study. The method of stereo-photogrammetry used to measure the erosion of rammed earth walls on site may also help to calibrate and develop more pertinent laboratory test to assess the durability of rammed earth wall.

Axially homogeneous, zero mean flow buoyancy-driven turbulence in a vertical pipe

We report an experimental study of a new type of turbulent flow that is driven purely by buoyancy. The flow is due to an unstable density difference, created using brine and water, across the ends of a long (length/diameter = 9) vertical pipe. The Schmidt number Sc is 670, and the Rayleigh number (Ra) based on the density gradient and diameter is about 10(8). Under these conditions the convection is turbulent, and the time-averaged velocity at any point is `zero'. The Reynolds number based on the Taylor microscale, Re-lambda, is about 65. The pipe is long enough for there to be an axially homogeneous region, with a linear density gradient, about 6-7 diameters long in the midlength of the pipe. In the absence of a mean flow and, therefore, mean shear, turbulence is sustained just by buoyancy. The flow can be thus considered to be an axially homogeneous turbulent natural convection driven by a constant (unstable) density gradient. We characterize the flow using flow visualization and particle image velocimetry (PIV). Measurements show that the mean velocities and the Reynolds shear stresses are zero across the cross-section; the root mean squared (r.m.s.) of the vertical velocity is larger than those of the lateral velocities (by about one and half times at the pipe axis). We identify some features of the turbulent flow using velocity correlation maps and the probability density functions of velocities and velocity differences. The flow away from the wall, affected mainly by buoyancy, consists of vertically moving fluid masses continually colliding and interacting, while the flow near the wall appears similar to that in wall-bound shear-free turbulence. The turbulence is anisotropic, with the anisotropy increasing to large values as the wall is approached. A mixing length model with the diameter of the pipe as the length scale predicts well the scalings for velocity fluctuations and the flux. This model implies that the Nusselt number would scale as (RaSc1/2)-Sc-1/2, and the Reynolds number would scale as (RaSc-1/2)-Sc-1/2. The velocity and the flux measurements appear to be consistent with the Ra-1/2 scaling, although it must be pointed out that the Rayleigh number range was less than 10. The Schmidt number was not varied to check the Sc scaling. The fluxes and the Reynolds numbers obtained in the present configuration are Much higher compared to what would be obtained in Rayleigh-Benard (R-B) convection for similar density differences.

Unsteady Natural Convection Flow in a Square Cavity Filled with a Porous Medium Due to Impulsive Change in Wall Temperature

Unsteady natural convection flow in a two- dimensional square cavity filled with a porous material has been studied. The flow is initially steady where the left- hand vertical wall has temperature T-h and the right- hand vertical wall is maintained at temperature T-c ( T-h > T-c) and the horizontal walls are insulated. At time t > 0, the left- hand vertical wall temperature is suddenly raised to (T-h) over bar ((T-h) over bar > T-h) which introduces unsteadiness in the flow field. The partial differential equations governing the unsteady natural convection flow have been solved numerically using a finite control volume method. The computation has been carried out until the final steady state is reached. It is found that the average Nusselt number attains a minimum during the transient period and that the time required to reach the final steady state is longer for low Rayleigh number and shorter for high Rayleigh number.

Transitional intermittency in boundary layers subjected to pressure gradient

Results are reported from an extensive series of experiments on boundary layers in which the location of pressure gradient and transition onset could be varied almost independently, by judicious use of tunnel wall liners and transition-fixing devices. The experiments show that the transition zone is sensitive to the pressure gradient especially near onset, and can be significantly asymmetric; no universal similarity appears valid in general. Observed intermittency distributions cannot be explained on the basis of the hypothesis, often made, that the spot propagates at speeds proportional to the local free-stream velocity but is otherwise unaffected by the pressure gradient.

On the moulding of population viscosity by natural selection

In this paper, we explore the conjoint evolution of dispersal and social behaviour. The model investigated is of a population distributed over a number of sites each with a carrying capacity of two adults and an episode of dispersal in the juvenile stage. The fertilities are governed by whether an individual and its neighbour are selfish or co-operative. It is shown that the best dispersal strategy for the co-operative genotype always involves lower levels of dispersal; and further that ecological conditions favouring low levels of dispersal increase the selective advantage of a co-operative genotype. Given this positive feedback, we suggest that in any taxon viscosity and co-operativity will tend to be correlated and bimodally distributed. Hence we predict the existence of two kinds of animal societies; viscous and co-operative (e.g. quasi-social wasps such as Mischocyttarus), and non-viscous and selfish (e.g. communal sphecid wasps such as Cerceris), and relatively few social groups with intermediate levels of co-operativity and viscosity. We also suggest that when one of the two sexes disperses, it will be the sex with lower potential for co-operative behaviour.

Natural convection from a vertical cone in a porous medium due to the combined effects of heat and mass diffusion with non-uniform wall temperature/concentration or heat/mass flux and suction/injection

An analysis has been carried out to study the non-Darcy natural convention flow of Newtonian fluids on a vertical cone embedded in a saturated porous medium with power-law variation of the wall temperature/concentration or heat/mass flux and suction/injection with the streamwise distance x. Both non-similar and self-similar solutions have been obtained. The effects of non-Darcy parameter, ratio of the buoyancy forces due to mass and heat diffusion, variation of wall temperature/concentration or heat/mass flux and suction/injection on the Nusselt and Sherwood numbers have been studied.

Natural Frequencies And Transient Responses Of 3-Phase Rotating Machine Windings

The analysis of transient electrical stresses in the insulation of high voltage rotating machines is rendered difficult because of the existence of capacitive and inductive couplings between phases. The Published theories ignore many of the couplings between phases to obtain the solution. A new procedure is proposed here to determine the transient voltage distribution on rotating machine windings. All the significicant capacitive and inductive couplings between different sections in a phase and between different sections in different phases have been considered in this analysis. The experimental results show good correlation with those computed.

Novel Self-Supported Natural and Synthetic Polymer Membranes for Air Humidification

Novel self-supported natural and synthetic polymer membranes of chitosan-hydroxy ethyl Cellulose-montmorillonite (CS-HEC-MMT) and polyvinyl alcohol (PVA)-polystyrene sulfonic acid (PSSA) are prepared by solution casting method followed by crosslinking. These membranes are employed for air humidification at varying temperatures between 30 degrees C and 70 degrees C and their performances are compared with commercial Nafion membranes. High hater fluxes with desired humidified-air output have been achieved for CS-HEC-MMT and PVA-PSSA hybrid membranes at air-flow rates of 1-10 slpm. Variation in the air/water mixing ratio, dew point, and relative humidity that ultimately results in desired water flux With respect to air-flow rates are also quantified for all the membranes. Water flux values for CS-HEC-MMT are less than those for Nafion (R) and PVA-PSSA membranes, but the operational Stability of CS-HEC-MMT membrane is higher than PVA-PSSA and comparable with Nafion (R) both of which can operate up to 70 degrees C at repetitive cycles of humidification.

Synthetic studies towards the phloroglucin natural product hyperforin: construction of the fully prenylated bicyclic core

An approach towards the highly functionalized bicyclo[3.3.1]nonan-9-one core of the complex PPAP-based natural product hyperforin, with the full complement of prenyl substituents in required stereo-disposition, is delineated.

CFD Analysis of conjugate natural convection in a vertical annulus

Conjugate natural convection in a vertical annulus with a centrally located vertical heat generating rod is studied numerically. The governing equations are discretized on a staggered mesh and are solved using a pressure-correction algorithm. A parametric study is performed by varying the Grashof number, aspect ratio, and the solid-to-fluid thermal conductivity ratio over wide ranges with the Prandtl number fixed at 0.7. Results are presented for the variation of several quantities of interest such as the local Nusselt numbers on the inner and outer boundaries, the axial variation of the centerline and interface temperatures, maximum solid, average solid and average interface temperature variations with Grashof number, and the average Nusselt number variation for the inner and outer boundaries with Grashof number. The average Nusselt number from the conjugate analysis is found to be between the Nusselt numbers of the isothermal and the isoflux cases. The average Nusselt numbers on the inner and outer boundaries show an increasing trend with the Grashof number. Correlations are presented for the Nusselt number and the dimensionless temperatures of interest in terms of the parameters of the problem.