Tutorial on Continuous-Time Markov Processes (2014)

by Christian R. Shelton and Gianfranco Ciardo


Abstract: A continuous-time Markov process (CTMP) is a collection of variables indexed by a continuous quantity, time. It obeys the Markov property that the distribution over a future variable is independent of past variables given the state at the present time. We introduce continuous-time Markov process representations and algorithms for filtering, smoothing, expected sufficient statistics calculations, and model estimation, assuming no prior knowledge of continuous-time processes but some basic knowledge of probability and statistics. We begin by describing “flat” or unstructured Markov processes and then move to structured Markov processes (those arising from state spaces consisting of assignments to variables) including Kronecker, decision-diagram, and continuous-time Bayesian network representations. We provide the first connection between decision-diagrams and continuous-time Bayesian networks.

Download Information

Christian R. Shelton and Gianfranco Ciardo (2014). "Tutorial on Continuous-Time Markov Processes." Journal of Artificial Intelligence Research, 51, 725-778. pdf          

Bibtex citation

@article{SheCia14,
   author = "Christian R. Shelton and Gianfranco Ciardo",
   title = "Tutorial on Continuous-Time {M}arkov Processes",
   journal = "Journal of Artificial Intelligence Research",
   journalabbr = "JAIR",
   volume = 51,
   pages = "725--778",
   month = dec,
   year = 2014,
}