An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

Rodent seed predation: effects on seed survival, recruitment, abundance, and dispersion of bird-dispersed tropical trees

Tropical tree species vary widely in their pattern of spatial dispersion. We focus on how seed predation may modify seed deposition patterns and affect the abundance and dispersion of adult trees in a tropical forest in India. Using plots across a range of seed densities, we examined whether seed predation levels by terrestrial rodents varied across six large-seeded, bird-dispersed tree species. Since inter-specific variation in density-dependent seed mortality may have downstream effects on recruitment and adult tree stages, we determined recruitment patterns close to and away from parent trees, along with adult tree abundance and dispersion patterns. Four species (Canarium resiniferum, Dysoxylum binectariferum, Horsfieldia kingii, and Prunus ceylanica) showed high predation levels (78.5-98.7%) and increased mortality with increasing seed density, while two species, Chisocheton cumingianus and Polyalthia simiarum, showed significantly lower seed predation levels and weak density-dependent mortality. The latter two species also had the highest recruitment near parent trees, with most abundant and aggregated adults. The four species that had high seed mortality had low recruitment under parent trees, were rare, and had more spaced adult tree dispersion. Biotic dispersal may be vital for species that suffer density-dependent mortality factors under parent trees. In tropical forests where large vertebrate seed dispersers but not seed predators are hunted, differences in seed vulnerability to rodent seed predation and density-dependent mortality can affect forest structure and composition.

Computing Reeb Graphs as a Union of Contour Trees

The Reeb graph of a scalar function tracks the evolution of the topology of its level sets. This paper describes a fast algorithm to compute the Reeb graph of a piecewise-linear (PL) function defined over manifolds and non-manifolds. The key idea in the proposed approach is to maximally leverage the efficient contour tree algorithm to compute the Reeb graph. The algorithm proceeds by dividing the input into a set of subvolumes that have loop-free Reeb graphs using the join tree of the scalar function and computes the Reeb graph by combining the contour trees of all the subvolumes. Since the key ingredient of this method is a series of union-find operations, the algorithm is fast in practice. Experimental results demonstrate that it outperforms current generic algorithms by a factor of up to two orders of magnitude, and has a performance on par with algorithms that are catered to restricted classes of input. The algorithm also extends to handle large data that do not fit in memory.

Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

Optimal MSE solution for a decision feedback equalizer

Due to the inherent feedback in a decision feedback equalizer (DFE) the minimum mean square error (MMSE) or Wiener solution is not known exactly. The main difficulty in such analysis is due to the propagation of the decision errors, which occur because of the feedback. Thus in literature, these errors are neglected while designing and/or analyzing the DFEs. Then a closed form expression is obtained for Wiener solution and we refer this as ideal DFE (IDFE). DFE has also been designed using an iterative and computationally efficient alternative called least mean square (LMS) algorithm. However, again due to the feedback involved, the analysis of an LMS-DFE is not known so far. In this paper we theoretically analyze a DFE taking into account the decision errors. We study its performance at steady state. We then study an LMS-DFE and show the proximity of LMS-DFE attractors to that of the optimal DFE Wiener filter (obtained after considering the decision errors) at high signal to noise ratios (SNR). Further, via simulations we demonstrate that, even at moderate SNRs, an LMS-DFE is close to the MSE optimal DFE. Finally, we compare the LMS DFE attractors with IDFE via simulations. We show that an LMS equalizer outperforms the IDFE. In fact, the performance improvement is very significant even at high SNRs (up to 33%), where an IDFE is believed to be closer to the optimal one. Towards the end, we briefly discuss the tracking properties of the LMS-DFE.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Multispecies coexistence of trees in tropical forests: spatial signals of topographic niche differentiation increase with environmental heterogeneity

Neutral and niche theories give contrasting explanations for the maintenance of tropical tree species diversity. Both have some empirical support, but methods to disentangle their effects have not yet been developed. We applied a statistical measure of spatial structure to data from 14 large tropical forest plots to test a prediction of niche theory that is incompatible with neutral theory: that species in heterogeneous environments should separate out in space according to their niche preferences. We chose plots across a range of topographic heterogeneity, and tested whether pairwise spatial associations among species were more variable in more heterogeneous sites. We found strong support for this prediction, based on a strong positive relationship between variance in the spatial structure of species pairs and topographic heterogeneity across sites. We interpret this pattern as evidence of pervasive niche differentiation, which increases in importance with increasing environmental heterogeneity.

On the SIG-dimension of trees under the L (a)-metric

Let where be a set of points in d-dimensional space with a given metric rho. For a point let r (p) be the distance of p with respect to rho from its nearest neighbor in Let B(p,r (p) ) be the open ball with respect to rho centered at p and having the radius r (p) . We define the sphere-of-influence graph (SIG) of as the intersection graph of the family of sets Given a graph G, a set of points in d-dimensional space with the metric rho is called a d-dimensional SIG-representation of G, if G is isomorphic to the SIG of It is known that the absence of isolated vertices is a necessary and sufficient condition for a graph to have a SIG-representation under the L (a)-metric in some space of finite dimension. The SIG-dimension under the L (a)-metric of a graph G without isolated vertices is defined to be the minimum positive integer d such that G has a d-dimensional SIG-representation under the L (a)-metric. It is denoted by SIG (a)(G). We study the SIG-dimension of trees under the L (a)-metric and almost completely answer an open problem posed by Michael and Quint (Discrete Appl Math 127:447-460, 2003). Let T be a tree with at least two vertices. For each let leaf-degree(v) denote the number of neighbors of v that are leaves. We define the maximum leaf-degree as leaf-degree(x). Let leaf-degree{(v) = alpha}. If |S| = 1, we define beta(T) = alpha(T) - 1. Otherwise define beta(T) = alpha(T). We show that for a tree where beta = beta (T), provided beta is not of the form 2 (k) - 1, for some positive integer k a parts per thousand yen 1. If beta = 2 (k) - 1, then We show that both values are possible.

A Markov Decision Theoretic Approach to Pilot Allocation and Receive Antenna Selection

This paper considers antenna selection (AS) at a receiver equipped with multiple antenna elements but only a single radio frequency chain for packet reception. As information about the channel state is acquired using training symbols (pilots), the receiver makes its AS decisions based on noisy channel estimates. Additional information that can be exploited for AS includes the time-correlation of the wireless channel and the results of the link-layer error checks upon receiving the data packets. In this scenario, the task of the receiver is to sequentially select (a) the pilot symbol allocation, i.e., how to distribute the available pilot symbols among the antenna elements, for channel estimation on each of the receive antennas; and (b) the antenna to be used for data packet reception. The goal is to maximize the expected throughput, based on the past history of allocation and selection decisions, and the corresponding noisy channel estimates and error check results. Since the channel state is only partially observed through the noisy pilots and the error checks, the joint problem of pilot allocation and AS is modeled as a partially observed Markov decision process (POMDP). The solution to the POMDP yields the policy that maximizes the long-term expected throughput. Using the Finite State Markov Chain (FSMC) model for the wireless channel, the performance of the POMDP solution is compared with that of other existing schemes, and it is illustrated through numerical evaluation that the POMDP solution significantly outperforms them.

Transmit power Control with ARQ in energy harvesting sensors: a decision-theoretic approach

This paper addresses the problem of finding optimal power control policies for wireless energy harvesting sensor (EHS) nodes with automatic repeat request (ARQ)-based packet transmissions. The EHS harvests energy from the environment according to a Bernoulli process; and it is required to operate within the constraint of energy neutrality. The EHS obtains partial channel state information (CSI) at the transmitter through the link-layer ARQ protocol, via the ACK/NACK feedback messages, and uses it to adapt the transmission power for the packet (re)transmission attempts. The underlying wireless fading channel is modeled as a finite state Markov chain with known transition probabilities. Thus, the goal of the power management policy is to determine the best power setting for the current packet transmission attempt, so as to maximize a long-run expected reward such as the expected outage probability. The problem is addressed in a decision-theoretic framework by casting it as a partially observable Markov decision process (POMDP). Due to the large size of the state-space, the exact solution to the POMDP is computationally expensive. Hence, two popular approximate solutions are considered, which yield good power management policies for the transmission attempts. Monte Carlo simulation results illustrate the efficacy of the approach and show that the approximate solutions significantly outperform conventional approaches.

Socio-cultural protection of endemic trees in humanised landscape

Culturally protected forest patches or sacred groves have been the integral part of many traditional societies. This age old tradition is a classic instance of community driven nature conservation sheltering native biodiversity and supporting various ecosystem functions particularly hydrology. The current work in Central Western Ghats of Karnataka, India, highlights that even small sacred groves amidst humanised landscapes serve as tiny islands of biodiversity, especially of rare and endemic species. Temporal analysis of landuse dynamics reveals the changing pattern of the studied landscape. There is fast reduction of forest cover (15.14-11.02 %) in last 20 years to meet up the demand of agricultural land and plantation programs. A thorough survey and assessment of woody endemic species distribution in the 25 km(2) study area documented presence of 19 endemic species. The distribution of these species is highly skewed towards the culturally protected patches in comparison to other land use elements. It is found that, among the 19 woody endemic species, those with greater ecological amplitude are widely distributed in the studied landscape in groves as well as other land use forms whereas, natural population of the sensitive endemics are very much restricted in the sacred grove fragments. The recent degradation in the sacred grove system is perhaps, due to weakening of traditional belief systems and associated laxity in grove protection leading to biotic disturbances. Revitalisation of traditional practices related to conservation of sacred groves can go a long way in strengthening natural ecological systems of fragile humid tropical landscape.

Skip Decision and Reference Frame Selection for Low-Complexity H.264/AVC Surveillance Video Coding

H. 264/advanced video coding surveillance video encoders use the Skip mode specified by the standard to reduce bandwidth. They also use multiple frames as reference for motion-compensated prediction. In this paper, we propose two techniques to reduce the bandwidth and computational cost of static camera surveillance video encoders without affecting detection and recognition performance. A spatial sampler is proposed to sample pixels that are segmented using a Gaussian mixture model. Modified weight updates are derived for the parameters of the mixture model to reduce floating point computations. A storage pattern of the parameters in memory is also modified to improve cache performance. Skip selection is performed using the segmentation results of the sampled pixels. The second contribution is a low computational cost algorithm to choose the reference frames. The proposed reference frame selection algorithm reduces the cost of coding uncovered background regions. We also study the number of reference frames required to achieve good coding efficiency. Distortion over foreground pixels is measured to quantify the performance of the proposed techniques. Experimental results show bit rate savings of up to 94.5% over methods proposed in literature on video surveillance data sets. The proposed techniques also provide up to 74.5% reduction in compression complexity without increasing the distortion over the foreground regions in the video sequence.

Directed Nonabelian Sandpile Models on Trees

We define two general classes of nonabelian sandpile models on directed trees (or arborescences), as models of nonequilibrium statistical physics. Unlike usual applications of the well-known abelian sandpile model, these models have the property that sand grains can enter only through specified reservoirs. In the Trickle-down sandpile model, sand grains are allowed to move one at a time. For this model, we show that the stationary distribution is of product form. In the Landslide sandpile model, all the grains at a vertex topple at once, and here we prove formulas for all eigenvalues, their multiplicities, and the rate of convergence to stationarity. The proofs use wreath products and the representation theory of monoids.

In the elephant's seed shadow: the prospects of domestic bovids as replacement dispersers of three tropical Asian trees

As populations of the world's largest animal species decline, it is unclear how ecosystems will react to their local extirpation. Due to the unique ecological characteristics of megaherbivores such as elephants, seed dispersal is one ecosystem process that may be affected as populations of large animals are decimated. In typically disturbed South Asian ecosystems, domestic bovids (cattle, Bos primigenius, and buffalo, Bubalus bubalis) may often be the species most available to replace Asian elephants (Elephas maximus) as endozoochorous dispersers of large-fruited mammal-dispersed species. We use feeding trials, germination trials, and movement data from the tropical moist forests of Buxa Tiger Reserve (India) to examine whether domestic bovids are viable replacements for elephants in the dispersal of three largefruited species: Dillenia indica, Artocarpus chaplasha, and Careya arborea. We find that (1) once consumed, seeds are between 2.5 (C. arborea) and 26.5 (D. indica) times more likely to pass undigested into elephant dung than domestic bovid dung; and (2) seeds from elephant dung germinated as well as or better than seeds taken from bovid dung for all plant species, with D. indica seeds from elephant dung 1.5 times more likely to germinate. Furthermore, since wild elephants have less constrained movements than even free-roaming domestic bovids, we calculate that maximum dispersal by elephants is between 9.5 and 11.2 times farther than that of domestic bovids, with about 20% of elephant-dispersed seeds being moved farther than the maximum distance seeds are moved by bovids. Our findings suggest that, while bovids are able to disperse substantial numbers of seeds over moderate distances for two of the three study species, domestic bovids will be unable to routinely emulate the reliable, long-distance dispersal of seeds executed by elephants in this tropical moist forest. Thus while domestic bovids can attenuate the effects of losing elephants as dispersers, they may not be able to prevent the decline of various mammal-dispersed fruiting species in the face of overhunting, habitat fragmentation, and climate change.