next up previous contents
Next: Frequentists and Bayesian `sects' Up: Appendix on probability and Previous: Bayesian networks   Contents

Why do frequentistic hypothesis tests `often work'?

The problem of classifying hypotheses according to their credibility is natural in the Bayesian framework. Let us recall briefly the following way of drawing conclusions about two hypotheses in the light of some data:
$\displaystyle \frac{P(H_i\,\vert\,\mbox{Data})}{P(H_j\,\vert\,\mbox{Data})}$ $\displaystyle =$ $\displaystyle \frac{P(\mbox{Data}\,\vert\,H_i)}{P(\mbox{Data}\,\vert\,H_j)}\cdot
\frac{P_\circ(H_i)}{P_\circ(H_j)}\,.$ (8.6)

This form is very convenient, because: At this point we can finally reply to the question: ``why do commonly-used methods of hypothesis testing usually work?'' (see Sections [*] and [*]).
Figure: Testing a hypothesis $ H_\circ$ implies that one is ready to replace it with an alternative hypothesis.

By reference to Fig. [*] (imagine for a moment the figure without the curve $ H_1$), the argument that $ \theta_m$ provides evidence against $ H_\circ$ is intuitively accepted and often works, not (only) because of probabilistic considerations of $ \theta$ in the light of $ H_\circ$, but because it is often reasonable to imagine an alternative hypothesis $ H_1$ that

  1. maximizes the likelihood $ f(\theta_m\,\vert\,H_1)$ or, at least

    $\displaystyle \frac{P(\theta_m\,\vert\,H_1)}{P(\theta_m\,\vert\,H_\circ)} \gg 1\,;$

  2. has a comparable prior [ $ P_\circ(H_1)\approx P_\circ(H_\circ)$], such that
    $\displaystyle \frac{P(H_1\,\vert\,\theta_m)}{P(H_\circ\,\vert\,\theta_m)}
= \fr...
\frac{P_\circ(H_1)}{P_\circ(H_\circ)}$ $\displaystyle \approx$ $\displaystyle \frac{P(\theta_m\,\vert\,H_1)}{P(\theta_m\,\vert\,H_\circ)} \longrightarrow \gg 1\,.$  

So, even though there is no objective or logical reason why the frequentistic scheme should work, the reason why it often does is that in many cases the test is made when one has serious doubts about the null hypothesis. But a peak appearing in the middle of a distribution, or any excess of events, is not, in itself, a hint of new physics (Fig. [*] is an invitation to meditation...).
Figure: Experimental obituary (courtesy of Alvaro de Rujula[71]).
My recommendations are therefore the following. As a conclusion to these remarks, and to invite the reader to take with much care the assumption of equiprobability of hypothesis (a hidden assumption in many frequentistic methods), I would like to add this quotation by Poincaré [6]:
``To make my meaning clearer, I go back to the game of écarté mentioned before.8.17 My adversary deals for the first time and turns up a king. What is the probability that he is a sharper? The formulae ordinarily taught give 8/9, a result which is obviously rather surprising. If we look at it closer, we see that the conclusion is arrived at as if, before sitting down at the table, I had considered that there was one chance in two that my adversary was not honest. An absurd hypothesis, because in that case I should certainly not have played with him; and this explains the absurdity of the conclusion. The function on the à priori probability was unjustified, and that is why the conclusion of the à posteriori probability led me into an inadmissible result. The importance of this preliminary convention is obvious. I shall even add that if none were made, the problem of the à posteriori probability would have no meaning. It must be always made either explicitly or tacitly.''

next up previous contents
Next: Frequentists and Bayesian `sects' Up: Appendix on probability and Previous: Bayesian networks   Contents
Giulio D'Agostini 2003-05-15