Optional Stopping with Bayes Factors: A Categorization and Extension of Folklore Results, with an Application to Invariant Situations

Published in Bayesian Analysis on 01 September 2021.

Publication PDF

Summary

It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stopping. We first give an overview, using elementary probability theory, of three different mathematical meanings that various authors give to this claim: (1) stopping rule independence, (2) posterior calibration and (3) (semi-) frequentist robustness to optional stopping. We then prove theorems to the effect that these claims do indeed hold in a general measure-theoretic setting. For claims of type (2) and (3), such results are new. By allowing for non-integrable measures based on improper priors, we obtain particularly strong results for the practically important case of models with nuisance parameters satisfying a group invariance (such as location or scale). We also discuss the practical relevance of (1)–(3), and conclude that whether Bayes factor methods actually perform well under optional stopping crucially depends on details of models, priors and the goal of the analysis.

Outreach

We were invited to write a post related to our paper on the blog of the Young Statisticians Europe (YSE). It was written by Rianne.

Citation

For more details and additional results, read the full paper.
  @Article{hendriksen-2020-option-stopp,
  author          = {Hendriksen, Allard and de Heide, Rianne and Grünwald,
                  Peter},
  title           = {Optional Stopping With {Bayes} Factors: a Categorization
                  and Extension of Folklore Results, With an Application To
                  Invariant Situations},
  journal         = {Bayesian Analysis},
  volume          = 16,
  number          = 3,
  pages           = {961 -- 989},
  year            = 2021,
  doi             = {10.1214/20-BA1234},
  URL             = {https://doi.org/10.1214/20-BA1234},
  keywords        = {Bayes factors, Bayesian testing, group invariance, optional
                  stopping, right Haar prior},
  publisher       = {International Society for Bayesian Analysis},
}