An Actor-Critic Algorithm for Finite Horizon Markov Decision Processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Regularity of invariant densities for 1D systems with random switching

© 2015 IOP Publishing Ltd & London Mathematical Society.This is a detailed analysis of invariant measures for one-dimensional dynamical systems with random switching. In particular, we prove the smoothness of the invariant densities away from critical points and describe the asymptotics of the invariant densities at critical points.

Sensitivity to switching rates in stochastically switched ODEs

We consider a stochastic process driven by a linear ordinary differential equation whose right-hand side switches at exponential times between a collection of different matrices. We construct planar examples that switch between two matrices where the individual matrices and the average of the two matrices are all Hurwitz (all eigenvalues have strictly negative real part), but nonetheless the process goes to infinity at large time for certain values of the switching rate. We further construct examples in higher dimensions where again the two individual matrices and their averages are all Hurwitz, but the process has arbitrarily many transitions between going to zero and going to infinity at large time as the switching rate varies. In order to construct these examples, we first prove in general that if each of the individual matrices is Hurwitz, then the process goes to zero at large time for sufficiently slow switching rate and if the average matrix is Hurwitz, then the process goes to zero at large time for sufficiently fast switching rate. We also give simple conditions that ensure the process goes to zero at large time for all switching rates. © 2014 International Press.

Stochastic switching in infinite dimensions with applications to random parabolic PDE

© 2015 Society for Industrial and Applied Mathematics.We consider parabolic PDEs with randomly switching boundary conditions. In order to analyze these random PDEs, we consider more general stochastic hybrid systems and prove convergence to, and properties of, a stationary distribution. Applying these general results to the heat equation with randomly switching boundary conditions, we find explicit formulae for various statistics of the solution and obtain almost sure results about its regularity and structure. These results are of particular interest for biological applications as well as for their significant departure from behavior seen in PDEs forced by disparate Gaussian noise. Our general results also have applications to other types of stochastic hybrid systems, such as ODEs with randomly switching right-hand sides.

Hidden Markov analysis describes dive patterns in semiaquatic animals

Existing methods of dive analysis, developed for fully aquatic animals, tend to focus on frequency of behaviors rather than transitions between them. They, therefore, do not account for the variability of behavior of semiaquatic animals, and the switching between terrestrial and aquatic environments. This is the first study to use hidden Markov models (HMM) to divide dives of a semiaquatic animal into clusters and thus identify the environmental predictors of transition between behavioral modes. We used 18 existing data sets of the dives of 14 American mink (Neovison vison) fitted with time-depth recorders in lowland England. Using HMM, we identified 3 behavioral states (1, temporal cluster of dives; 2, more loosely aggregated diving within aquatic activity; and 3, terminal dive of a cluster or a single, isolated dive). Based on the higher than expected proportion of dives in State 1, we conclude that mink tend to dive in clusters. We found no relationship between temperature and the proportion of dives in each state or between temperature and the rate of transition between states, meaning that in our study area, mink are apparently not adopting different diving strategies at different temperatures. Transition analysis between states has shown that there is no correlation between ambient temperature and the likelihood of mink switching from one state to another, that is, changing foraging modes. The variables provided good discrimination and grouped into consistent states well, indicating promise for further application of HMM and other state transition analyses in studies of semiaquatic animals.

New markov chain based methods for single and cross-domain sentiment classification

Nowadays communication is switching from a centralized scenario, where communication media like newspapers, radio, TV programs produce information and people are just consumers, to a completely different decentralized scenario, where everyone is potentially an information producer through the use of social networks, blogs, forums that allow a real-time worldwide information exchange. These new instruments, as a result of their widespread diffusion, have started playing an important socio-economic role. They are the most used communication media and, as a consequence, they constitute the main source of information enterprises, political parties and other organizations can rely on. Analyzing data stored in servers all over the world is feasible by means of Text Mining techniques like Sentiment Analysis, which aims to extract opinions from huge amount of unstructured texts. This could lead to determine, for instance, the user satisfaction degree about products, services, politicians and so on. In this context, this dissertation presents new Document Sentiment Classification methods based on the mathematical theory of Markov Chains. All these approaches bank on a Markov Chain based model, which is language independent and whose killing features are simplicity and generality, which make it interesting with respect to previous sophisticated techniques. Every discussed technique has been tested in both Single-Domain and Cross-Domain Sentiment Classification areas, comparing performance with those of other two previous works. The performed analysis shows that some of the examined algorithms produce results comparable with the best methods in literature, with reference to both single-domain and cross-domain tasks, in $2$-classes (i.e. positive and negative) Document Sentiment Classification. However, there is still room for improvement, because this work also shows the way to walk in order to enhance performance, that is, a good novel feature selection process would be enough to outperform the state of the art. Furthermore, since some of the proposed approaches show promising results in $2$-classes Single-Domain Sentiment Classification, another future work will regard validating these results also in tasks with more than $2$ classes.

Empirical Analysis of Credit Risk Regime Switching and Temporal Conditional Default Correlation in Credit Default Swap Valuation: The Market liquidity effect

In this paper, we extend the debate concerning Credit Default Swap valuation to include time varying correlation and co-variances. Traditional multi-variate techniques treat the correlations between covariates as constant over time; however, this view is not supported by the data. Secondly, since financial data does not follow a normal distribution because of its heavy tails, modeling the data using a Generalized Linear model (GLM) incorporating copulas emerge as a more robust technique over traditional approaches. This paper also includes an empirical analysis of the regime switching dynamics of credit risk in the presence of liquidity by following the general practice of assuming that credit and market risk follow a Markov process. The study was based on Credit Default Swap data obtained from Bloomberg that spanned the period January 1st 2004 to August 08th 2006. The empirical examination of the regime switching tendencies provided quantitative support to the anecdotal view that liquidity decreases as credit quality deteriorates. The analysis also examined the joint probability distribution of the credit risk determinants across credit quality through the use of a copula function which disaggregates the behavior embedded in the marginal gamma distributions, so as to isolate the level of dependence which is captured in the copula function. The results suggest that the time varying joint correlation matrix performed far superior as compared to the constant correlation matrix; the centerpiece of linear regression models.

Modelling financial time series with switching state space models

The deficiencies of stationary models applied to financial time series are well documented. A special form of non-stationarity, where the underlying generator switches between (approximately) stationary regimes, seems particularly appropriate for financial markets. We use a dynamic switching (modelled by a hidden Markov model) combined with a linear dynamical system in a hybrid switching state space model (SSSM) and discuss the practical details of training such models with a variational EM algorithm due to [Ghahramani and Hilton,1998]. The performance of the SSSM is evaluated on several financial data sets and it is shown to improve on a number of existing benchmark methods.