P-values Vs Bayes factors

Having discussed at length this topic elsewhere (see in particular sections 1.8, and 10.8 of [8]), I sketch here the main points, with the help of some plots. This is obviously a didactic example and does not enter at all into the (very complicate and CPU time consuming) details of the analysis of the interferometer data (see footnote 18). In particular a direct observation will be considered, while in general hypothesis tests are performed on a statistic chosen with large freedom.²⁷ So we just consider here simple models $H_i$ that could produce the quantity $x$ according to pdf's $f(x\,\vert\,H_i,I)$.

• As reminded above, according to probability theory what matters for the update of relative beliefs is the ratio of the pdf's. For example the observation $x_m=5$ shown in the upper plot of Fig.4 modifies our beliefs in favor of $H_3$, with respect to $H_1$ and $H_2$, no matter the size of the area under the pdf's right of $x_m$.

  Figure: Several models that could have produced the observed value of $x_m$[8].

  
• In particular $H_2$ is ruled out (`falsified') because, being $f(x_m\,\vert\,H_2)=0$, it cannot produce the observation, despite it provides the highest probability of $X > x_m$.<sup>28</sup>
• It follows that, if the values of pdf's $f(x_m\,\vert\,H_i)$ are equal for all $H_i$, as in the lower plot of Fig.4, then the experiment is irrelevant and we hold our beliefs, independently of how far $x_m$ occurs from the expected values $\mbox{E}[X\,\vert\,H_i]$, or of the size of the area left or right $x_m$.
• The reason why p-values `often work' (and can then be useful alarm bells when getting experiments running, or validating freshly collected data), is quite simple.
  • Small p-values are normally associated to small values of the pdf, as shown in the upper plot of Fig.5.

    Figure: Pdf's of $X$ given the null hypothesis $H_0$ and the alternative hypothesis $H_1$.

    
  • It is then conceivable an alternative hypothesis $H_1$ such that $f(x_m\,\vert\,H_1) \gg f(x_m\,\vert\,H_0)$, as shown in the bottom plot of Fig.5.
  • Then, if this is the case, the observed $x_m$ would push our beliefs towards $H_1$, in the sense $\mbox{BF}(H_1:H_0) = \frac{f(x_m\,\vert\,H_1)} {f(x_m\,\vert\,H_0)} \gg 1$.
  • BUT we need to take into account also the priors odds $P(H_1\,\vert\,I)/P(H_0\,\vert\,I)$.
  • In the extreme case such a conceivable $H_1$ could not exist, or it could be not believable,²⁹ or it could be just ad hoc, as it happens in recent years, with a plethora of `theorists' who give credit to any fluctuation. If this is the case, as it is often the case in frontier physics, then

    $\Rightarrow$

    $P(H_1\,\vert\,I)/P(H_0\,\vert\,I) \rightarrow 0$

    $\Rightarrow$

    the smallness of the p-value is irrelevant!

  (Note that if, instead of the smallness of the value of the pdf, the rational were really the smallness of the area below the pdf, than the absurd situation might arise in which one could choose a ``rejection area'' anywhere, as shown in chapter 1 of [8].)
• Finally, in order to understand the apparent paradox of large p-value and indeed very large BF, think at a very predictive model $H_1$, whose pdf of the observable $x$ overlaps with that of $H_0$, like in the upper plot of Fig.6.

  Figure: Pdf's of $X$ given the null hypothesis $H_0$ and the alternative hypothesis $H_1$ (case of overlapping pdf's).

  
  We clearly see that $f(x_m\,\vert\,H_1) \gg f(x_m\,\vert\,H_0)$, thus resulting in a Bayes factor highly in favor of $H_1$, although the p-value calculated from the null hypothesis $H_0$ would be absolutely insignificant. Something like that occurs in the analysis of the gravitational wave analysis, the case of Cinderella being the most striking one.³⁰
• And `paradoxically' - this is just a colloquial term, since there is no paradox at all - large deviations from the expected value of $x$ given $H_0$, corresponding to small p-values, are those which favor $H_0$, if $H_1$ and $H_0$ are the only hypotheses in hand, as shown in the bottom plot of the same figure. Now, in the light of these examples, I simply re-propose you the following sentence from the first principle of the ASA's statement ``The smaller the $p$-value, the greater the statistical incompatibility of the data with the null hypothesis, if the underlying assumptions used to calculate the $p$-value hold.''[2] As you can now understand, it is not a matter of assumptions concerning $H_0$, but rather on whether alternative hypotheses to $H_0$ are conceivable and, more important, believable!