Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.

Spectroscopic and electrical properties of SiO2 films prepared by simple and cost effective sol-gel process

Amorphous SiO2 thin films were prepared on glass and silicon substrates by cost effective sol-gel method. Tetra ethyl ortho silicate (TEOS) was used as the precursor material, ethanol as solvent and concentrated HCl as a catalyst. The films were characterized at different annealing temperatures. The optical transmittance was slightly increased with increase of annealing temperature. The refractive index was found to be 1.484 at 550 nm. The formation of SiO2 film was analyzed from FT-IR spectra. The MOS capacitors were designed using silicon (1 0 0) substrates. The current-voltage (I-V), capacitance-voltage (C-V) and dissipation-voltage (D-V) measurements were taken for all the annealed films deposited on Si (1 0 0). The variation of current density, resistivity and dielectric constant of SiO2 films with different annealing temperatures was investigated and discussed for its usage in applications like MOS capacitor. The results revealed the decrease of dielectric constant and increase of resistivity of SiO2 films with increasing annealing temperature. (C) 2010 Elsevier B.V. All rights reserved.

Speedup Macroblock Mode Decision in H.264/SVC Encoding Using Cost-Sensitive Learning

In this paper we present a novel macroblock mode decision algorithm to speedup H.264/SVC Intra frame encoding. We replace the complex mode-decision calculations by a classifier which has been trained specifically to minimize the reduction in RD performance. This results in a significant speedup in encoding. The results show that machine learning has a great potential and can reduce the complexity substantially with negligible impact on quality. The results show that the proposed method reduces encoding time to about 70% in base layer and up to 50% in enhancement layer of the reference implementation with a negligible loss in quality.

An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Optimal control of a stochastic hybrid system with discounted cost

We address the optimal control problem of a very general stochastic hybrid system with both autonomous and impulsive jumps. The planning horizon is infinite and we use the discounted-cost criterion for performance evaluation. Under certain assumptions, we show the existence of an optimal control. We then derive the quasivariational inequalities satisfied by the value function and establish well-posedness. Finally, we prove the usual verification theorem of dynamic programming.

Risk-sensitive optimal control for Markov decision processes with monotone cost

The existence of an optimal feedback law is established for the risk-sensitive optimal control problem with denumerable state space. The main assumptions imposed are irreducibility and a near monotonicity condition on the one-step cost function. A solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law.

Groundwater Management in India:Physical, Institutional and Policy Alternatives

Groundwater constitutes a vital natural resource for sustaining India’s agricultural economy and meeting the country’s social, ecological and environmental goals. It is a unique resource, widely available, providing security against droughts and yet it is closely linked to surface-water resources and the hydrological cycle. Its availability depends on geo-hydrological conditions and characteristics of aquifers, from deep to alluvium, sediment crystalline rocks to basalt formations; and agro-climate from humid to subhumid and semi-arid to arid. Its reliable supply, uniform quality and temperature, relative turbidity, pollution-safe, minimal evaporation losses, and low cost of development are attributes making groundwater more attractive compared to other resources. It plays a key role in the provision of safe drinking water to rural populations. For example, already almost 80% of domestic water use in rural areas in India is groundwater-supplied, and much of it is being supplied to farms, villages and small towns. Inadequate control of the use of groundwater, indiscriminate application of agrochemicals and unrestrained pollution of the rural environment by other human activities make groundwater usage unsustainable, necessitating proper management in the face of the twin demand for water of good quality for domestic supply and adequate supply for irrigation, ensuring equity, efficiency and sustainability of the resource. Groundwater irrigation has overtaken surface irrigation in the early 1980s, supported by well energization. It is estimated that there are about 24 million energised wells and tube wells now and it is driven by demand rather than availability, evident through the greater occurrence of wells in districts with high population densities. Apart from aquifer characteristics, land fragmentation and landholding size are the factors that decide the density of wells. The ‘rise and fall’ of local economies dependent on groundwater can be summarized as: the green revolution of 1980s, groundwaterbased agrarian boom, early symptoms of groundwater overdraft, and decline of the groundwater socio-ecology. The social characteristics and policy interventions typical of each stage provide a fascinating insight into the human-resource dynamics. This book is a compilation of nine research papers discussing various aspects of groundwater management. It attempts to integrate knowledge about the physical system, the socio-economic system, the institutional set-up and the policy environment to come out with a more realistic analysis of the situation with regard to the nature, characteristics and intensity of resource use, the size of the economy the use generates, and the negative socioeconomic consequences. Complex variables addressed in this regard focusing on northern Gujarat are the stock of groundwater available in the region, its hydrodynamics, its net outflows against inflows, the economics of its intensive use (particularly irrigation in semi-arid and arid regions), its criticality in the regional hydroecological regime, ethical aspects and social aspects of its use. The first chapter by Dinesh Kumar and Singh, dwells on complex groundwater socio-ecology of India, while emphasizing the need for policy measures to address indiscriminate over-exploitation of dwindling resources. The chapter also explores the nature of groundwater economy and the role of electricity prices on it. The next chapter on groundwater issue in north Gujarat provides a description of groundwater resource characteristics followed by a detailed analysis of the groundwater depletion and quality deterioration problems in the region and their undesirable consequences on the economy, ecosystem health and the society. Considering water-buyers and wellowning farmers individually, a methodology for economic valuation of groundwater in regions where its primary usage is in agriculture, and as assessment of the groundwater economy based on case studies from north Gujarat is presented in the fourth chapter. The next chapter focuses on the extent of dependency of milk production on groundwater, which includes the water embedded in green and dry fodder and animal feed. The study made a realistic estimate of irrigation water productivity in terms of the physics and economics of milk production. The sixth chapter analyses the extent of reduction in water usage, increase in yield and overall increase in physical productivity of alfalfa with the use of the drip irrigation system. The chapter also provides a detailed synthesis of the costs and benefits associated with the use of drip irrigation systems. A linear programmingbased optimization model with the objective to minimize groundwater use taking into account the interaction between two distinct components – farming and dairying under the constraints of food security and income stability for different scenarios, including shift in cropping pattern, introduction of water-efficient crops, water- saving technologies in addition to the ‘business as usual’ scenario is presented in the seventh chapter. The results show that sustaining dairy production in the region with reduced groundwater draft requires crop shifts and adoption of water-saving technologies. The eighth chapter provides evidences to prove that the presence of adequate economic incentive would encourage farmers to adopt water-saving irrigation devices, based on the findings of market research with reference to the level of awareness among farmers of technologies and the factors that decide the adoption of water-saving technologies. However, now the marginal cost of using electricity for agricultural pumping is almost zero. The economic incentives are strong and visible only when the farmers are either water-buyers or have to manage irrigation with limited water from tube-well partnerships. The ninth chapter explores the socio-economic viability of increasing the power tariff and inducing groundwater rationing as a tool for managing energy and groundwater demand, considering the current estimate of the country’s annual economic loss of Rs 320 billion towards electricity subsidy in the farm sector. The tenth chapter suggests private tradable property rights and development of water markets as the institutional tool for achieving equity, efficiency and sustainability of groundwater use. It identifies the externalities for local groundwater management and emphasizes the need for managing groundwater by local user groups, supported by a thorough analysis of groundwater socio-ecology in India. An institutional framework for managing the resource based on participatory approach that is capable of internalizing the externalities, comprising implementation of institutional and technical alternatives for resource management is also presented. Major findings of the analyses and key arguments in each chapter are summarized in the concluding chapter. Case studies of the social and economic benefits of groundwater use, where that use could be described as unsustainable, are interesting. The benefits of groundwater use are outlined and described with examples of social and economic impacts of groundwater and the negative aspects of groundwater development with the compilation of environmental problems based on up-to-date research results. This publication with a well-edited compilation of case studies is informative and constitutes a useful publication for students and professionals.

Embedded Thin Film Resistors on BCB by Low-Cost PWB Compatible Electroless Plating by Direct Foil Lamination Transfer Processes

Benzocyclobutene (BCB) has been proposed as a board level dielectric for advanced system-on-package (SOP) module primarily due to its attractive low-loss (for RF application) and thin film (for high density wiring) properties. Realization of embedded resistors on low loss benzocyclobutene (dielectric loss ~0.0008 at > 40 GHz) has been explored in this study. Two approaches, viz, foil transfer and electroless plating have been attempted for deposition of thin film resistors on benzocyclobutene (BCB). Ni-P alloys were plated using conventional electroless plating, and NiCr and NiCrAlSi foils were used for the foil transfer process. This paper reports NiP and NiWP electroless plated embedded resistors on BCB dielectric for the first time in the literature

Biofuel production and implications for land use, food production and environment in India

There is a large interest in biofuels in India as a substitute to petroleum-based fuels, with a purpose of enhancing energy security and promoting rural development. India has announced an ambitious target of substituting 20% of fossil fuel consumption by biodiesel and bioethanol by 2017. India has announced a national biofuel policy and launched a large program to promote biofuel production, particularly on wastelands: its implications need to be studied intensively considering the fact that India is a large developing country with high population density and large rural population depending upon land for their livelihood. Another factor is that Indian economy is experiencing high growth rate, which may lead to enhanced demand for food, livestock products, timber, paper, etc., with implications for land use. Studies have shown that area under agriculture and forest has nearly stabilized over the past 2-3 decades. This paper presents an assessment of the implications of projected large-scale biofuel production on land available for food production, water, biodiversity, rural development and GHG emissions. The assessment will be largely focused on first generation biofuel crops, since the Indian program is currently dominated by these crops. Technological and policy options required for promoting sustainable biofuel production will be discussed. (C) 2010 Elsevier Ltd. All rights reserved.

Optical, electrical and dielectric properties of TiO(2)-SiO(2) films prepared by a cost effective sol-gel process

Titanium dioxide (TiO(2)) and silicon dioxide (SiO(2)) thin films and their mixed films were synthesized by the sol-gel spin coating method using titanium tetra isopropoxide (TTIP) and tetra ethyl ortho silicate (TEOS) as the precursor materials for TiO(2) and SiO(2) respectively. The pure and composite films of TiO(2) and SiO(2) were deposited on glass and silicon substrates. The optical properties were studied for different compositions of TiO(2) and SiO(2) sols and the refractive index and optical band gap energies were estimated. MOS capacitors were fabricated using TiO(2) films on p-silicon (1 0 0) substrates. The current-voltage (I-V) and capacitance-voltage (C-V) characteristics were studied and the electrical resistivity and dielectric constant were estimated for the films annealed at 200 degrees C for their possible use in optoelectronic applications. (C) 2011 Elsevier B.V. All rights reserved.

A low cost approach for the fabrication of microwave phase shifter on laminates

This paper presents a simple and low cost fabrication approach using extended printed circuit board processing techniques for an electrostatically actuated phase shifter on a common microwave laminate. This approach uses 15 mu m thin copper foils for realizing the bridge structures as well as for a spacer. A polymeric thin film deposited by spin coating and patterned using lithographic process is used as a dielectric layer to improve the reliability of the device. The prototype of the phase shifter for X-band operation is fabricated and tested for electrical and electromechanical performance parameters. The realized devices have a figure of merit of 70 degrees/dB for a maximum applied bias potential of 85 V. Since these phase shifters can be conveniently fabricated directly on microwave substrates used for feed distribution networks of phased arrays, the overall addition in cost, dimensions and processing for including these phase shifters in these arrays is minimal.

Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

Error, Confidence, and Cost in Engineering Computations

The questions that one should answer in engineering computations - deterministic, probabilistic/randomized, as well as heuristic - are (i) how good the computed results/outputs are and (ii) how much the cost in terms of amount of computation and the amount of storage utilized in getting the outputs is. The absolutely errorfree quantities as well as the completely errorless computations done in a natural process can never be captured by any means that we have at our disposal. While the computations including the input real quantities in nature/natural processes are exact, all the computations that we do using a digital computer or are carried out in an embedded form are never exact. The input data for such computations are also never exact because any measuring instrument has inherent error of a fixed order associated with it and this error, as a matter of hypothesis and not as a matter of assumption, is not less than 0.005 per cent. Here by error we imply relative error bounds. The fact that exact error is never known under any circumstances and any context implies that the term error is nothing but error-bounds. Further, in engineering computations, it is the relative error or, equivalently, the relative error-bounds (and not the absolute error) which is supremely important in providing us the information regarding the quality of the results/outputs. Another important fact is that inconsistency and/or near-consistency in nature, i.e., in problems created from nature is completely nonexistent while in our modelling of the natural problems we may introduce inconsistency or near-inconsistency due to human error or due to inherent non-removable error associated with any measuring device or due to assumptions introduced to make the problem solvable or more easily solvable in practice. Thus if we discover any inconsistency or possibly any near-inconsistency in a mathematical model, it is certainly due to any or all of the three foregoing factors. We do, however, go ahead to solve such inconsistent/near-consistent problems and do get results that could be useful in real-world situations. The talk considers several deterministic, probabilistic, and heuristic algorithms in numerical optimisation, other numerical and statistical computations, and in PAC (probably approximately correct) learning models. It highlights the quality of the results/outputs through specifying relative error-bounds along with the associated confidence level, and the cost, viz., amount of computations and that of storage through complexity. It points out the limitation in error-free computations (wherever possible, i.e., where the number of arithmetic operations is finite and is known a priori) as well as in the usage of interval arithmetic. Further, the interdependence among the error, the confidence, and the cost is discussed.