Optimal non-linear reinforcement schemes for stochastic automata

Two optimal non-linear reinforcement schemes—the Reward-Inaction and the Penalty-Inaction—for the two-state automaton functioning in a stationary random environment are considered. Very simple conditions of symmetry of the non-linear function figuring in the reinforcement scheme are shown to be necessary and sufficient for optimality. General expressions for the variance and rate of learning are derived. These schemes are compared with the already existing optimal linear schemes in the light of average variance and average rate of learning.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Simultaneous measurement of aerodynamic and heat transfer data for large angle blunt cones in hypersonic shock tunnel

Aerodynamic forces and fore-body convective surface heat transfer rates over a 60 degrees apex-angle blunt cone have been simultaneously measured at a nominal Mach number of 5.75 in the hypersonic shock tunnel HST2. An aluminum model incorporating a three-component accelerometer-based balance system for measuring the aerodynamic forces and an array of platinum thin-film gauges deposited on thermally insulating backing material flush mounted on the model surface is used for convective surface heat transfer measurement in the investigations. The measured value of the drag coefficient varies by about +/-6% from the theoretically estimated value based on the modified Newtonian theory, while the axi-symmetric Navier-Stokes computations overpredict the drag coefficient by about 9%. The normalized values of measured heat transfer rates at 0 degrees angle of attack are about 11% higher than the theoretically estimated values. The aerodynamic and the heat transfer data presented here are very valuable for the validation of CFD codes used for the numerical computation of How fields around hypersonic vehicles.

Effect of densification and TiB2 reinforcement on the wear of molybdenum disilicide

Wear tests were done in a pin-on-disc machine by sliding MoSi2 pins against hard-steel discs in a normal load range of 5-140 N and a speed of 0.5 m/s under nominally dry conditions in the ambient. The specific wear rate of the pin undergoes two transitions: severe to mild at low load and mild to severe at high load. The mild-wear domain is distinguished by the formation of a protective mechanically mixed layer of steel and its oxides, transferred from the counterface in particulate form. Increasing the hardness by densification and TiB2 reinforcement lowers the specific wear rate and expands the mild-wear load domain. However, even when the volume wear rate is normalised with respect to the real contact area (load/hardness) the non-dimensional wear factor is still seen to decrease with densification and reinforcement. This indicates that fracture toughness may also play an important role in determining the wear-resistance of these materials. The surface coverage on the pin by the mechanically mixed layer increases with densification and reinforcement.

Tunnel transport in polypyrrole at low temperature

The anomalous behaviour of conductivity below 4 K in polypyrrole can be attributed to the possibility of tunnel transport in disordered polaronic systems. The deviation from T-1/3 and T-1/4, depending on disorder, can be due to the onset of tunnel transport between localised states, apart from the hopping contribution to the conductivity. In intermediately and lightly doped polypyrrole films, the tunnel contribution to conductivity increases with decreasing temperature in a narrow temperature range, which is a feature of the presence of polarons taking part in the conduction mechanisms of disordered systems with strong electron-phonon coupling. The transition from hopping to tunneling dominated process can be observed either by the increase in conductivity in some cases or by the saturation of conductivity, depending crucially on the extent of disorder in the sample. In both cases the transition temperature is seen to increase with the reduction in the number of localised states.

The high speed water tunnel facility at the Indian Institute of Science

The present article about the high speed water tunnel facility at the Indian Institute of Science, Bangalore, provides a general description of the tunnel circuit, and brief reports on the performance of the facility and some typical results from investigations carried out in it. A unique aspect of the facility is that it has a horizontal resorber in the form of a large cylindrical tank located in the lower leg of the circuit. The facility has been used, among other things, for flow visualization studies, and investigations on marine propeller hydrodynamics and “synthetic cavitation”. The last topic has been primarily developed at the Indian Institute of Science and shows considerable promise for basic work in cavitation inception and noise.

Correlation of helicopter rotor aeroelastic response with HART-II wind tunnel test data

Purpose - This paper aims to validate a comprehensive aeroelastic analysis for a helicopter rotor with the higher harmonic control aeroacoustic rotor test (HART-II) wind tunnel test data. Design/methodology/approach - Aeroelastic analysis of helicopter rotor with elastic blades based on finite element method in space and time and capable of considering higher harmonic control inputs is carried out. Moderate deflection and coriolis nonlinearities are included in the analysis. The rotor aerodynamics are represented using free wake and unsteady aerodynamic models. Findings - Good correlation between analysis and HART-II wind tunnel test data is obtained for blade natural frequencies across a range of rotating speeds. The basic physics of the blade mode shapes are also well captured. In particular, the fundamental flap, lag and torsion modes compare very well. The blade response compares well with HART-II result and other high-fidelity aeroelastic code predictions for flap and torsion mode. For the lead-lag response, the present analysis prediction is somewhat better than other aeroelastic analyses. Research limitations/implications - Predicted blade response trend with higher harmonic pitch control agreed well with the wind tunnel test data, but usually contained a constant offset in the mean values of lead-lag and elastic torsion response. Improvements in the modeling of the aerodynamic environment around the rotor can help reduce this gap between the experimental and numerical results. Practical implications - Correlation of predicted aeroelastic response with wind tunnel test data is a vital step towards validating any helicopter aeroelastic analysis. Such efforts lend confidence in using the numerical analysis to understand the actual physical behavior of the helicopter system. Also, validated numerical analyses can take the place of time-consuming and expensive wind tunnel tests during the initial stage of the design process. Originality/value - While the basic physics appears to be well captured by the aeroelastic analysis, there is need for improvement in the aerodynamic modeling which appears to be the source of the gap between numerical predictions and HART-II wind tunnel experiments.

Control of barrier height of MIS tunnel diodes using deep level impurities

The barrier height of MIS tunnel diodes is studied considering the effect of deep impurities. It is shown that the barrier height of a given MIS-system can be controlled by changing the density and the activation energy of the defect level. The study leads to the conclusion that deep impurities of character opposite to shallow impurities enhance the barrier height. On the other hand, the barrier height is lowered when the type of the deep impurities is the same as that of shallow impurities.

A tutorial survey of reinforcement learning

This paper gives a compact, self-contained tutorial survey of reinforcement learning, a tool that is increasingly finding application in the development of intelligent dynamic systems. Research on reinforcement learning during the past decade has led to the development of a variety of useful algorithms. This paper surveys the literature and presents the algorithms in a cohesive framework.

Review of hypersonic research investigations in IISc shock tunnel (HST1)

Real gas effects dominate the hypersonic flow fields encountered by modem day hypersonic space vehicles. Measurement of aerodynamic data for the design applications of such aerospace vehicles calls for special kinds of wind tunnels capable of faithfully simulating real gas effects. A shock tunnel is an established facility commonly used along with special instrumentation for acquiring the data for this purpose within a short time period. The hypersonic shock tunnel (HST1), established at the Indian Institute of Science (IISc) in the early 1970s, has been extensively used to measure the aerodynamic data of various bodies of interest at hypersonic Mach numbers in the range 4 to 13. Details of some important measurements made during the period 1975-1995 along with the performance capabilities of the HST1 are presented in this review. In view of the re-emergence of interest in hypersonics across the globe in recent times, the present review highlights the Suitability of the hypersonic shock tunnel at the IISc for future space application studies in India.

Reinforcement Learning With Function Approximation for Traffic Signal Control

We propose, for the first time, a reinforcement learning (RL) algorithm with function approximation for traffic signal control. Our algorithm incorporates state-action features and is easily implementable in high-dimensional settings. Prior work, e. g., the work of Abdulhai et al., on the application of RL to traffic signal control requires full-state representations and cannot be implemented, even in moderate-sized road networks, because the computational complexity exponentially grows in the numbers of lanes and junctions. We tackle this problem of the curse of dimensionality by effectively using feature-based state representations that use a broad characterization of the level of congestion as low, medium, or high. One advantage of our algorithm is that, unlike prior work based on RL, it does not require precise information on queue lengths and elapsed times at each lane but instead works with the aforementioned described features. The number of features that our algorithm requires is linear to the number of signaled lanes, thereby leading to several orders of magnitude reduction in the computational complexity. We perform implementations of our algorithm on various settings and show performance comparisons with other algorithms in the literature, including the works of Abdulhai et al. and Cools et al., as well as the fixed-timing and the longest queue algorithms. For comparison, we also develop an RL algorithm that uses full-state representation and incorporates prioritization of traffic, unlike the work of Abdulhai et al. We observe that our algorithm outperforms all the other algorithms on all the road network settings that we consider.

A finite element assessment of flexural strength of prestressed concrete beams with fiber reinforcement

This paper presents an assessment of the flexural behavior of 15 fully/partially prestressed high strength concrete beams containing steel fibers investigated using three-dimensional nonlinear finite elemental analysis. The experimental results consisted of eight fully and seven partially prestressed beams, which were designed to be flexure dominant in the absence of fibers. The main parameters varied in the tests were: the levels of prestressing force (i.e, in partially prestressed beams 50% of the prestress was reduced with the introduction of two high strength deformed bars instead), fiber volume fractions (0%, 0.5%, 1.0% and 1.5%), fiber location (full depth and partial depth over full length and half the depth over the shear span only). A three-dimensional nonlinear finite element analysis was conducted using ANSYS 5.5 [Theory Reference Manual. In: Kohnke P, editor. Elements Reference Manual. 8th ed. September 1998] general purpose finite element software to study the flexural behavior of both fully and partially prestressed fiber reinforced concrete beams. Influence of fibers on the concrete failure surface and stress-strain response of high strength concrete and the nonlinear stress-strain curves of prestressing wire and deformed bar were considered in the present analysis. In the finite element model. tension stiffening and bond slip between concrete and reinforcement (fibers., prestressing wire, and conventional reinforcing steel bar) have also been considered explicitly. The fraction of the entire volume of the fiber present along the longitudinal axis of the prestressed beams alone has been modeled explicitly as it is expected that these fibers would contribute to the mobilization of forces required to sustain the applied loads across the crack interfaces through their bridging action. A comparison of results from both tests and analysis on all 15 specimens confirm that, inclusion of fibers over a partial depth in the tensile side of the prestressed flexural structural members was economical and led to considerable cost saving without sacrificing on the desired performance. However. beams having fibers over half the depth in only the shear span, did not show any increase in the ultimate load or deformational characteristics when compared to plain concrete beams. (C) 2002 Published by Elsevier Science Ltd.

A reinforcement learning approach to automatic generation control

This paper formulates the automatic generation control (AGC) problem as a stochastic multistage decision problem. A strategy for solving this new AGC problem formulation is presented by using a reinforcement learning (RL) approach This method of obtaining an AGC controller does not depend on any knowledge of the system model and more importantly it admits considerable flexibility in defining the control objective. Two specific RL based AGC algorithms are presented. The first algorithm uses the traditional control objective of limiting area control error (ACE) excursions, where as, in the second algorithm, the controller can restore the load-generation balance by only monitoring deviation in tie line flows and system frequency and it does not need to know or estimate the composite ACE signal as is done by all current approaches. The effectiveness and versatility of the approaches has been demonstrated using a two area AGC model. (C) 2002 Elsevier Science B.V. All rights reserved.

Shock tunnel study of spiked aerodynamic bodies flying at hypersonic Mach numbers

A three-component accelerometer balance system is used to study the drag reduction effect of an aerodisc on large angle blunt cones flying at hypersonic Mach numbers. Measurements in a hypersonic shock tunnel at a freestream Mach number of 5.75 indicate more than 50% reduction in the drag coefficient for a 120degrees apex angle blunt cone with a forward facing aerospike having a flat faced aerodisc at moderate angles of attack. Enhancement of drag has been observed for higher angles of attack due to the impingement of the flow separation shock on the windward side of the cone. The flowfields around the large angle blunt cone with aerospike assembly flying at hypersonic Mach numbers are also simulated numerically using a commercial CFD code. The pressure and density levels on the model surface, which is under the aerodynamic shadow of the flat disc tipped spike, are found very low and a drag reduction of 64.34% has been deduced numerically.