Computer Science and Engineering

Christian R. Shelton, Professor

Tutorial on Continuous-Time Markov Processes (2014)

by Christian R. Shelton and Gianfranco Ciardo

Abstract: A continuous-time Markov process (CTMP) is a collection of variables indexed by a continuous quantity, time. It obeys the Markov property that the distribution over a future variable is independent of past variables given the state at the present time. We introduce continuous-time Markov process representations and algorithms for filtering, smoothing, expected sufficient statistics calculations, and model estimation, assuming no prior knowledge of continuous-time processes but some basic knowledge of probability and statistics. We begin by describing ``flat'' or unstructured Markov processes and then move to structured Markov processes (those arising from state spaces consisting of assignments to variables) including Kronecker, decision-diagram, and continuous-time Bayesian network representations. We provide the first connection between decision-diagrams and continuous-time Bayesian networks.

Download Information

Christian R. Shelton and Gianfranco Ciardo (2014). "Tutorial on Continuous-Time Markov Processes." Journal of Artificial Intelligence Research, 51, 725-778. pdf          

Bibtex citation

   author = "Christian R. Shelton and Gianfranco Ciardo",
   title = "Tutorial on Continuous-Time {M}arkov Processes",
   journal = "Journal of Artificial Intelligence Research",
   volume = 51,
   pages = "725--778",
   month = dec,
   year = 2014,

More Information


University of California, Riverside
Chung Hall, room 327
Riverside, CA 92521
Tel: (951) 827-2554
E-mail: cshelton@cs.ucr.edu


Other Links