An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Mining Land Cover Information Using Multilayer Perceptron and Decision Tree from MODIS Data

Land cover (LC) changes play a major role in global as well as at regional scale patterns of the climate and biogeochemistry of the Earth system. LC information presents critical insights in understanding of Earth surface phenomena, particularly useful when obtained synoptically from remote sensing data. However, for developing countries and those with large geographical extent, regular LC mapping is prohibitive with data from commercial sensors (high cost factor) of limited spatial coverage (low temporal resolution and band swath). In this context, free MODIS data with good spectro-temporal resolution meet the purpose. LC mapping from these data has continuously evolved with advances in classification algorithms. This paper presents a comparative study of two robust data mining techniques, the multilayer perceptron (MLP) and decision tree (DT) on different products of MODIS data corresponding to Kolar district, Karnataka, India. The MODIS classified images when compared at three different spatial scales (at district level, taluk level and pixel level) shows that MLP based classification on minimum noise fraction components on MODIS 36 bands provide the most accurate LC mapping with 86% accuracy, while DT on MODIS 36 bands principal components leads to less accurate classification (69%).

Multi-Agent Rendezvous Algorithm with Rectilinear Decision Domain

The aim of this paper is to develop a computationally efficient decentralized rendezvous algorithm for a group of autonomous agents. The algorithm generalizes the notion of sensor domain and decision domain of agents to enable implementation of simple computational algorithms. Specifically, the algorithm proposed in this paper uses a rectilinear decision domain (RDD) as against the circular decision domain assumed in earlier work. Because of this, the computational complexity of the algorithm reduces considerably and, when compared to the standard Ando's algorithm available in the literature, the RDD algorithm shows very significant improvement in convergence time performance. Analytical results to prove convergence and supporting simulation results are presented in the paper.

Positional Consensus of Multi-Agent Systems Using Linear Programming Based Decentralized Control with Rectilinear Decision Domain

In this paper we develop a Linear Programming (LP) based decentralized algorithm for a group of multiple autonomous agents to achieve positional consensus. Each agent is capable of exchanging information about its position and orientation with other agents within their sensing region. The method is computationally feasible and easy to implement. Analytical results are presented. The effectiveness of the approach is illustrated with simulation results.

Towards building an intelligent decision support system for project management

Management of large projects, especially the ones in which a major component of R&D is involved and those requiring knowledge from diverse specialised and sophisticated fields, may be classified as semi-structured problems. In these problems, there is some knowledge about the nature of the work involved, but there are also uncertainties associated with emerging technologies. In order to draw up a plan and schedule of activities of such a large and complex project, the project manager is faced with a host of complex decisions that he has to take, such as, when to start an activity, for how long the activity is likely to continue, etc. An Intelligent Decision Support System (IDSS) which aids the manager in decision making and drawing up a feasible schedule of activities while taking into consideration the constraints of resources and time, will have a considerable impact on the efficient management of the project. This report discusses the design of an IDSS that helps in project planning phase through the scheduling phase. The IDSS uses a new project scheduling tool, the Project Influence Graph (PIG).

Video coding mode decision as a classification problem

In this paper, we show that it is possible to reduce the complexity of Intra MB coding in H.264/AVC based on a novel chance constrained classifier. Using the pairs of simple mean-variances values, our technique is able to reduce the complexity of Intra MB coding process with a negligible loss in PSNR. We present an alternate approach to address the classification problem which is equivalent to machine learning. Implementation results show that the proposed method reduces encoding time to about 20% of the reference implementation with average loss of 0.05 dB in PSNR.

Risk-sensitive optimal control for Markov decision processes with monotone cost

The existence of an optimal feedback law is established for the risk-sensitive optimal control problem with denumerable state space. The main assumptions imposed are irreducibility and a near monotonicity condition on the one-step cost function. A solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law.

Use of climate information for farm-level decision making: rainfed groundnut in southern India

The production of rainfed crops in semi-arid tropics exhibits large variation in response to the variation in seasonal rainfall. There are several farm-level decisions such as the choice of cropping pattern, whether to invest in fertilizers, pesticides etc., the choice of the period for planting, plant population density etc. for which the appropriate choice (associated with maximum production or minimum risk) depends upon the nature of the rainfall variability or the prediction for a specific year. In this paper, we have addressed the problem of identifying the appropriate strategies for cultivation of rainfed groundnut in the Anantapur region in a semi-arid part of the Indian peninsula. The approach developed involves participatory research with active collaboration with farmers, so that the problems with perceived need are addressed with the modern tools and data sets available. Given the large spatial variation of climate and soil, the appropriate strategies are necessarily location specific. With the approach adopted, it is possible to tap the detailed location specific knowledge of the complex rainfed ecosystem and gain an insight into the variety of options of land use and management practices available to each category of stakeholders. We believe such a participatory approach is essential for identifying strategies that have a favourable cost-benefit ratio over the region considered and hence are associated with a high chance of acceptance by the stakeholders. (C) 2002 Elsevier Science Ltd. All rights reserved.

Role of Management Decision Variables in Determining Technology Progress and Technical Efficiency in Indian Electronic Industry

This paper analyses the influence of management on Technical Efficiency Change (TEC) and Technological Progress (TP) in the communication equipment and consumer electronics sub-sectors of Indian hardware electronics industry. Each sub-sector comprises 13 sample firms for two time periods.The primary objective is to determine the relative contribution of TP and TEC to TFP Growth (TFPG) and to establish the influence of firm specific operational management decision variables on these two components. The study finds that both the sub-sectors have strived and achieved steady TP but not TEC in the period of economic liberalisation to cope with the intensifying competition. The management decisions with respect to asset and profit utilization, vertical integration, among others, improved TP and TE in the sub-sectors. However, R&D investments and technology imports proved costly for TFP indicating inadequate efforts and/or poor resource utilisation by the management. Management was found to be complacent in terms of improving or developing their own technology as indicated by their higher dependence on import of raw materials and no influence of R&D on TP.

Solar energy decision support system

Energy plays a prominent role in human society. As a result of technological and industrial development,the demand for energy is rapidly increasing. Existing power sources that are mainly fossil fuel based are leaving an unacceptable legacy of waste and pollution apart from diminishing stock of fuels.Hence, the focus is now shifted to large-scale propagation of renewable energy. Renewable energy technologies are clean sources of energy that have a much lower environmental impact than conventional energy technologies. Solar energy is one such renewable energy. Most renewable energy comes either directly or indirectly from the sun. Estimation of solar energy potential of a region requires detailed solar radiation climatology, and it is necessary to collect extensive radiation data of high accuracy covering all climatic zones of the region. In this regard, a decision support system (DSS)would help in estimating solar energy potential considering the region’s energy requirement.This article explains the design and implementation of DSS for assessment of solar energy. The DSS with executive information systems and reporting tools helps to tap vast data resources and deliver information. The main hypothesis is that this tool can be used to form a core of practical methodology that will result in more resilient in time and can be used by decision-making bodies to assess various scenarios. It also offers means of entering, accessing, and interpreting the information for the purpose of sound decision making.

Geographic Resources Decision Support System – an Open Source GIS

Fully structured and matured open source spatial and temporal analysis technology seems to be the official carrier of the future for planning of the natural resources especially in the developing nations. This technology has gained enormous momentum because of technical superiority, affordability and ability to join expertise from all sections of the society. Sustainable development of a region depends on the integrated planning approaches adopted in decision making which requires timely and accurate spatial data. With the increased developmental programmes, the need for appropriate decision support system has increased in order to analyse and visualise the decisions associated with spatial and temporal aspects of natural resources. In this regard Geographic Information System (GIS) along with remote sensing data support the applications that involve spatial and temporal analysis on digital thematic maps and the remotely sensed images. Open source GIS would help in wide scale applications involving decisions at various hierarchical levels (for example from village panchayat to planning commission) on economic viability, social acceptance apart from technical feasibility. GRASS (Geographic Resources Analysis Support System, http://wgbis.ces.iisc.ernet.in/grass) is an open source GIS that works on Linux platform (freeware), but most of the applications are in command line argument, necessitating a user friendly and cost effective graphical user interface (GUI). Keeping these aspects in mind, Geographic Resources Decision Support System (GRDSS) has been developed with functionality such as raster, topological vector, image processing, statistical analysis, geographical analysis, graphics production, etc. This operates through a GUI developed in Tcltk (Tool command language / Tool kit) under Linux as well as with a shell in X-Windows. GRDSS include options such as Import /Export of different data formats, Display, Digital Image processing, Map editing, Raster Analysis, Vector Analysis, Point Analysis, Spatial Query, which are required for regional planning such as watershed Analysis, Landscape Analysis etc. This is customised to Indian context with an option to extract individual band from the IRS (Indian Remote Sensing Satellites) data, which is in BIL (Band Interleaved by Lines) format. The integration of PostgreSQL (a freeware) in GRDSS aids as an efficient database management system.

WEPA: Wind energy potential assessment-spatial decision support system

Spatial Decision Support System (SDSS) assist in strategic decision-making activities considering spatial and temporal variables, which help in Regional planning. WEPA is a SDSS designed for assessment of wind potential spatially. A wind energy system transforms the kinetic energy of the wind into mechanical or electrical energy that can be harnessed for practical use. Wind energy can diversify the economies of rural communities, adding to the tax base and providing new types of income. Wind turbines can add a new source of property value in rural areas that have a hard time attracting new industry. Wind speed is extremely important parameter for assessing the amount of energy a wind turbine can convert to electricity: The energy content of the wind varies with the cube (the third power) of the average wind speed. Estimation of the wind power potential for a site is the most important requirement for selecting a site for the installation of a wind electric generator and evaluating projects in economic terms. It is based on data of the wind frequency distribution at the site, which are collected from a meteorological mast consisting of wind anemometer and a wind vane and spatial parameters (like area available for setting up wind farm, landscape, etc.). The wind resource is governed by the climatology of the region concerned and has large variability with reference to space (spatial expanse) and time (season) at any fixed location. Hence the need to conduct wind resource surveys and spatial analysis constitute vital components in programs for exploiting wind energy. SDSS for assessing wind potential of a region / location is designed with user friendly GUI’s (Graphic User Interface) using VB as front end with MS Access database (backend). Validation and pilot testing of WEPA SDSS has been done with the data collected for 45 locations in Karnataka based on primary data at selected locations and data collected from the meteorological observatories of the India Meteorological Department (IMD). Wind energy and its characteristics have been analysed for these locations to generate user-friendly reports and spatial maps. Energy Pattern Factor (EPF) and Power Densities are computed for sites with hourly wind data. With the knowledge of EPF and mean wind speed, mean power density is computed for the locations with only monthly data. Wind energy conversion systems would be most effective in these locations during May to August. The analyses show that coastal and dry arid zones in Karnataka have good wind potential, which if exploited would help local industries, coconut and areca plantations, and agriculture. Pre-monsoon availability of wind energy would help in irrigating these orchards, making wind energy a desirable alternative.

Present status of Gottigere Tank : Indicator of Decision maker’s apathy

Gottigere lake with a water spread area of about 14.98 ha is located in the Bellandur Lake catchment of the South Pennar River basin. In recent years, this lake catchment has been subjected to environmental stress mainly due to the rampant unplanned developmental activities in the catchment. The functional ability of the ecosystem is impaired due to structural changes in the ecosystem. This is evident from poor water quality, breeding of disease vectors, contamination of groundwater in the catchment, frequent flooding in the catchment due to topography alteration, decline in groundwater table, erosion in lake bed, etc. The development plans of the region (current as well as the proposed) ignore the integrated planning approaches considering all components of the ecosystem. Serious threats to the sustainability of the region due to lack of holistic approaches in aquatic resources management are land use changes (removal of vegetation cover, etc.), point and non-point sources of pollution impairing water quality, dumping of solid waste (building waste, etc.). Conservation of lake ecosystem is possible only when the physical and chemical integrity of its catchment is maintained. Alteration in the catchment either due to land use changes (leading to paved surface area from vegetation cover), alteration in topography, construction of roads in the immediate vicinity are detrimental to water yield in the catchment and hence, the sustenance of the lake. Open spaces in the form of lakes and parks aid as kidney and lung in an urban ecosystem, which maintain the health of the people residing in the locality. Identification of core buffer zones and conservation of buffer zones (500 to 1000 m from shore) is to be taken up on priority for conservation and sustainable management of Bangalore lakes. Bangalore is located over a ridge delineating four watersheds, viz. Hebbal, Koramangala, Challaghatta and Vrishabhavathi. Lakes and tanks are an integral part of natural drainage and help in retaining water during rainfall, which otherwise get drained off as flash floods. Each lake harvests rainwater from its catchment and surplus flows downstream spilling into the next lake in the chain. The topography of Bangalore has uniquely supported the creation of a large number of lakes. These lakes form chains, being a series of impoundments across streams. This emphasises the interconnectivity among Bangalore lakes, which has to be retained to prevent Bangalore from flooding or from water scarcity. The main source of replenishment of groundwater is the rainfall. The slope of the terrain allows most of the rainwater to flow as run-off. With the steep gradients available in the major valleys of Bangalore, the rainwater will flow out of the city within four to five hours. Only a small fraction of the rainwater infiltrates into the soil. The infiltration of water into the subsoil has declined with more and more buildings and paved road being constructed in the city. Thus the natural drainage of Bangalore is governed by flows from the central ridge to all lower contours and is connected with various tanks and ponds. There are no major rivers flowing in Bangalore and there is an urgent need to sustain these vital ecosystems through proper conservation and management measures. The proposed peripheral ring road connecting Hosur Road (NH 7) and Mysore Road (SH 17) at Gottigere lake falls within the buffer zone of the lake. This would alter the catchment integrity and hence water yield affecting flora, fauna and local people, and ultimately lead to the disappearance of Gottigere lake. Developmental activities in lake catchments, which has altered lake’s ecological integrity is in violation of the Indian Fisheries Act – 1857, the Indian Forest Act – 1927, Wildlife (Protection) Act – 1972, Water (Prevention and Control of Pollution) Act – 1974, Water (Prevention and Control of Pollution) Act – 1977, Forest (Conservation Act) – 1980, Environmental (Protection) Act – 1986, Wildlife (Protection) Amendment Act – 1991 and National Conservation Strategy and Policy Statement on Environment and Development – 1992. Considering 65% decline of waterbodies in Bangalore (during last three decades), decision makers should immediately take preventive measures to ensure that lake ecosystems are not affected. This report discusses the impacts due to the proposed infrastructure developmental activities in the vicinity of Gottigere tank.

Analysis of an LMS linear equalizer for fading channels in decision directed mode

We consider a time varying wireless fading channel, equalized by an LMS linear equalizer in decision directed mode (DD-LMS-LE). We study how well this equalizer tracks the optimal Wiener equalizer. Initially we study a fixed channel.For a fixed channel, we obtain the existence of DD attractors near the Wiener filter at high SNRs using an ODE (Ordinary Differential Equation) approximating the DD-LMS-LE. We also show, via examples, that the DD attractors may not be close to the Wiener filters at low SNRs. Next we study a time varying fading channel modeled by an Auto-regressive (AR) process of order 2. The DD-LMS equalizer and the AR process are jointly approximated by the solution of a system of ODEs. We show via examples that the LMS equalizer ODE show tracks the ODE corresponding to the instantaneous Wiener filter when the SNR is high. This may not happen at low SNRs.