Recursive Parameter Estimation for Continuous-time Bivariate Markov Chains

Abstract

A continuous-time bivariate Markov chain comprises a pair of continuous-time random processes which are jointly Markov. One of the two processes is an underlying process while the other is assumed observable. An important special bivariate Markov chain is given by the continuous-time Markov modulated Poisson process (MMPP). The underlying process of an MMPP is a Markov chain, and the observable process is conditionally Poisson. Discretetime bivariate Markov chains may also be de ned, but they shall not be studied in this thesis. Bivariate Markov chains are useful in modeling ion-channel currents in living cells, Internet tra c, and in other problems in queuing theory. In this thesis we focus on recursive estimation of the parameter of a bivariate Markov chain which comprises its in nitesimal generator. We study a stochastic approximation approach using a newly developed recursion for the gradient of the log-likelihood function of the observed signal. The recursive algorithm is compared with a batch expectation-maximization (EM) algorithm developed earlier. The bias and mean squared error obtained in estimating each component of the parameter using each algorithm are evaluated and compared. The recursive algorithm requires far more data to provide an estimate comparable to that obtained by the EM algorithm, but the EM algorithm iterates over the entire data multiple times. The main advantage of the recursive estimator is its ability to adapt to slow changes in the underlying statistics of the model. The EM algorithm is a batch approach which must be re-applied whenever new data becomes available or there is a change in the underlying statistics of the model.