Computer Science and Engineering

Christian R. Shelton, Professor

Policy Improvement for POMDPs Using Normalized Importance Sampling (2001)

by Christian R. Shelton

Abstract: We present a new method for estimating the expected return of a POMDP from experience. The estimator does not assume any knowledge of the POMDP and allows the experience to be gathered with an arbitrary set of policies. The return is estimated for any new policy of the POMDP. We motivate the estimator from function-approximation and importance sampling points-of-view and derive its theoretical properties. Although the estimator is biased, it has low variance and the bias is often irrelevant when the estimator is used for pair-wise comparisons. We conclude by extending the estimator to policies with memory and compare its performance in a greedy search algorithm to the REINFORCE algorithm showing an order of magnitude reduction in the number of trials required.

Download Information

Christian R. Shelton (2001). "Policy Improvement for POMDPs Using Normalized Importance Sampling." Technical report. MIT AI Lab, AI Memo 2001-002. pdf   ps ps.gz    

Bibtex citation

   author = "Christian R. Shelton",
   title = "Policy Improvement for {POMDPs} Using Normalized Importance Sampling",
   year = 2001,
   institution = "MIT AI Lab",
   number = "2001-002",
   type = "AI Memo",
   month = Mar,

More Information


University of California, Riverside
Chung Hall, room 327
Riverside, CA 92521
Tel: (951) 827-2554
E-mail: cshelton@cs.ucr.edu


Other Links