Frequentistic coverage

Another prejudice toward Bayesian inference shared by practitioners who have grown up with conventional statistics is related to the so-called `frequentistic coverage'. Since, in my opinion, this is a kind of condensate of frequentistic nonsense,^8.13 I avoid summarizing it in my own words, as the risk of distorting something in which I cannot see any meaning is too high. A quotation^8.14taken from Ref. [68] should clarify the issue:

``Although particle physicists may use the words `confidence interval' loosely, the most common meaning is still in terms of original classical concept of ``coverage'' which follows from the method of construction suggested in Fig. ... This concept is usually stated (too narrowly, as noted below) in terms of a hypothetical ensemble of similar experiments, each of which measures $m$ and computes a confidence interval for $m_t$ with say, 68% C.L. Then the classical construction guarantees that in the limit of a large ensemble, 68% of the confidence intervals contain the unknown true value $m_t$, i.e., they `cover' $m_t$. This property, called coverage in the frequentistic sense, is the defining property of classical confidence intervals. It is important to see this property as what it is: it reflects the relative frequency with which the statement, `$m_t$ is in the interval $(m_1, m_2)$', is a true statement. The probabilistic variables in this statements are $m_1$ and $m_2$; $m_t$ is fixed and unknown. It is equally important to see what frequentistic coverage is not: it is a not statement about the degree of belief that $m_t$ lies within the confidence interval of a particular experiment. The whole concept of `degree of belief' does not exist with respect to classical confidence intervals, which are cleverly (some would say devilishly) defined by a construction which keeps strictly to statements about $P(m\,\vert\,m_t)$ and never uses a probability density in the variable $m_t$.

This strict classical approach can be considered to be either a virtue or a flaw, but I think that both critics and adherents commonly make a mistake in describing coverage from the narrow point of view which I described in the preceeding paragraph. As Neyman himself pointed out from the beginning, the concept of coverage is not restricted to the idea of an ensemble of hypothetical nearly-identical experiments. Classical confidence intervals have a much more powerful property: if, in an ensemble of real, different, experiments, each experiment measures whatever observables it likes, and construct a $68\%$ C.L. confidence interval, then in the long run $68\%$ of the confidence intervals cover the true value of their respective observables. This is directly applicable to real life, and is the real beauty of classical confidence intervals.''

I think that the reader can judge for himself whether this approach seems reasonable. From the Bayesian point of view, the full answer is provided by $P(m_t\,\vert\,m)$, to use the same notation of Ref. [68]. If this evaluation has been carried out under the requirement of coherence, from $P(m_t\,\vert\,m)$ one can evaluate a probability for $m_t$ to lie in the interval $(m_1, m_2)$. If this probability is $68\%$, in order to stick to the same value this implies:

• one believes $68\%$ that $m_t$ is in that interval;
• one is ready to place a $\approx 2:1$ bet on $m_t$ being in that interval and a $\approx 1:2$ bet on $m_t$ being elsewhere;
• if one imagines $n$ situations in which one has similar conditions (they could be different experiments, or simply urns containing a 68% proportion of white balls) and thinks of the relative frequency with which one expects that this statement will be true ($f_n$), logic applied to the basic rules of probability imply that, with the increasing $n$, it will become more and more improbable that $f_n$ will differ much from $68\%$ (Bernoulli theorem).

So, the intuitive concept of `coverage' is naturally included in the Bayesian result and it is expressed in intuitive terms (probability of true value and expected frequency). But this result has to depend also on priors, as seen in the previous section and in many other places in this report (see, for example, Section ). Talking about coverage independently of prior knowledge (as frequentists do) makes no sense, and leads to contradictions and paradoxes. Imagine, for example, an experiment operated for one hour at LEP200 and reporting zero candidate events for zirconium production in ${\rm e^+e^-}$ in the absence of expected background. I do not think that there is a single particle physicist ready to believe that, if the experiment is repeated many times, in only 68% of the cases the 68% C.L. interval $[0.00,\, 1.29]$ will contain the true value of the `Poisson signal mean', as a blind use of Table II of Ref. [60] would imply.^8.15 If this example seems a bit odd, I invite you to think about the many 95% C.L. lower limits on the mass of postulated particles. Do you really believe that in 95% of the cases the mass is above the limit, and in 5% of the cases below the limit? If this is the case, you would bet $5 on a mass value below the limit, and receive $100 if this happened to be true (you should be ready to accept the bet, since, if you believe in frequentistic coverage, you must admit that the bet is fair). But perhaps you will never accept such a bet because you believe much more than 95% that the mass is above the limit, and then the bet is not fair at all; or because you are aware of thousands of lower limits, and a particle has never shown up on the 5% side...