T. E. Harris and branching processes

T. E. Harris was a pioneer par excellence in many fields of probability theory. In this paper, we give a brief survey of the many fundamental contributions of Harris to the theory of branching processes, starting with his doctoral work at Princeton in the late forties and culminating in his fundamental book ``The Theory of Branching Processes,'' published in 1963.

Asymptotic behaviour of point processes with Markov-dependent intervals

This splitting techniques for MARKOV chains developed by NUMMELIN (1978a) and ATHREYA and NEY (1978b) are used to derive an imbedded renewal process in WOLD's point process with MARKOV-correlated intervals. This leads to a simple proof of renewal theorems for such processes. In particular, a key renewal theorem is proved, from which analogues to both BLACKWELL's and BREIMAN's forms of the renewal theorem can be deduced.

Unitary Processes with Independent Increments and Representations of Hilbert Tensor Algebras

The aim of this article is to characterize unitary increment process by a quantum stochastic integral representation on symmetric Fock space. Under certain assumptions we have proved its unitary equivalence to a Hudson-Parthasarathy flow.

Characterization of unitary processes with independent and stationary increments

This is a continuation of the earlier work (Publ. Res. Inst. Math. Sci. 45 (2009) 745-785) to characterize unitary stationary independent increment Gaussian processes. The earlier assumption of uniform continuity is replaced by weak continuity and with technical assumptions on the domain of the generator, unitary equivalence of the process to the solution of an appropriate Hudson-Parthasarathy equation is proved.

Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.

Risk-sensitive optimal control for Markov decision processes with monotone cost

The existence of an optimal feedback law is established for the risk-sensitive optimal control problem with denumerable state space. The main assumptions imposed are irreducibility and a near monotonicity condition on the one-step cost function. A solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law.

Optimal Control of Markov Processes with Age-Dependent Transition Rates

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Stability of degenerate diffusions with state-dependent switching

This paper deals with the ergodic properties of hybrid systems modelled by diffusion processes with state-dependent switching. We investigate the sufficient conditions expressed in terms of the parameters of the underlying process which would ensure the existence of a unique invariant probability and stability in distribution of the flow. It turns out that the conditions would depend on certain averaging mechanisms over the states of the discrete component of the hybrid system. (C) 1999 Academic Press.

An Actor-Critic Algorithm for Finite Horizon Markov Decision Processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

A Simultaneous Deterministic Perturbation Actor-Critic Algorithm with an Application to Optimal Mortgage Refinancing

We develop a simulation-based, two-timescale actor-critic algorithm for infinite horizon Markov decision processes with finite state and action spaces, with a discounted reward criterion. The algorithm is of the gradient ascent type and performs a search in the space of stationary randomized policies. The algorithm uses certain simultaneous deterministic perturbation stochastic approximation (SDPSA) gradient estimates for enhanced performance. We show an application of our algorithm on a problem of mortgage refinancing. Our algorithm obtains the optimal refinancing strategies in a computationally efficient manner

Spectrally Resolved Resonance Energy Transfer from ZnO:MgO Nanocrystals

Resonance energy transfer (RET) from the visible emission of core−shell ZnO:MgO nanocrystals to Nile Red chromophores, following band gap excitation in the UV, has been investigated for four different nanocrystal sizes. With use of steady state and time-resolved fluorescence spectroscopic measurements the wavelength dependent RET efficiencies have been determined. The RET process in ZnO:MgO nanocrystals occurs from emissions involving trap state recombination. There are two such processes with different RET efficiencies for the same particle size. This is shown to be a consequence of the fact that the recombination processes giving rise to the two emissions are located at different distances from the center of the particle so that the donor−acceptor distances for the two are different, even for the same particle size.

Generalization of Dynkin's Identity and Some Applications

Let X(t) be a right continuous temporally homogeneous Markov pro- cess, Tt the corresponding semigroup and A the weak infinitesimal genera- tor. Let g(t) be absolutely continuous and r a stopping time satisfying E.( S f I g(t) I dt) < oo and E.( f " I g'(t) I dt) < oo Then for f e 9iJ(A) with f(X(t)) right continuous the identity Exg(r)f(X(z)) - g(O)f(x) = E( 5 " g'(s)f(X(s)) ds) + E.( 5 " g(s)Af(X(s)) ds) is a simple generalization of Dynkin's identity (g(t) 1). With further restrictions on f and r the following identity is obtained as a corollary: Ex(f(X(z))) = f(x) + k! Ex~rkAkf(X(z))) + n-1E + (n ) )!.E,(so un-1Anf(X(u)) du). These identities are applied to processes with stationary independent increments to obtain a number of new and known results relating the moments of stopping times to the moments of the stopped processes.

Bayesian Analysis of a Superimposed Renewal Process

Stochastic behavior of an aero-engine failure/repair process has been analyzed from a Bayesian perspective. Number of failures/repairs in the component-sockets of this multi-component system are assumed to follow independent renewal processes with Weibull inter-arrival times. Based on the field failure/repair data of a large number of such engines and independent Gamma priors on the scale parameters and log-concave priors on the shape parameters, an exact method of sampling from the resulting posterior distributions of the parameters has been proposed. These generated parameter values are next utilised in obtaining the posteriors of the expected number of system repairs, system failure rate, and the conditional intensity function, which are computed using a recursive formula.