PROBABILITY OF PROBABILITIES (AND OF ODDS AND OF BAYES FACTORS)

The issue of `probability of probability' has already been discussed above, but in the particular case in which the second `probability' of the expression was indeed a propensity [and I would like to insist on the fact that whoever is interested in probability distributions of the Bernoulli parameter $p$, that is in something of the kind $f(p\,\vert\,I)$, is referring, explicitly or implicitly, to probabilities of propensities]. I would like now to move to the more general case, i.e. when we refer to uncertainty about our degree of belief. And, again, I like to approach the question in a pragmatic way, beginning with some considerations.

The first is that we are often in situations in which we are reluctant to assign a precise value to our degree of belief, because “we don't know” (this expression is commonly heard). But if you ask “is it then 10%?”, the answer can be “oh, not that low!”, or “not so high!” depending on the event of interest. In fact it rarely occurs that we know absolutely nothing about the fact,and in such a case we are not even interested in evaluating probabilities (why should we assign probabilities if we don't even know what we are talking about?).

The second is that the probability of probability, in the most general sense, is already included in the following, very familiar formula of probability theory, valid if $H_i$ are all the elements of a complete class of hypotheses,

$$P(A\,\vert\,I) = \sum_iP(A\,\vert\,H_i,I)\cdot P(H_i\,\vert\,I)\,.$$ (16)

We only need the courage to read it with an open mind: Equation (16) is simply an average of conditional probabilities, with weights equal to probabilities of each contribution. But in order to read it this way at least $P(H_i\,\vert\,I)$ must have the meaning of degree of belief, while $P(A\,\vert\,H_i,I)$ can represent propensities or also degrees of belief.

Probability of probabilities could refer to evaluations of somebody's else probabilities, as e.g. in game theory, but they are also important in all those important cases of real life in which direct assessments are done by experts or when sensitivity analysis leads to a spectrum of possibilities. For example, one might evaluate his/her degree of belief around 80%, but it could be as well, perhaps with some reluctance, 75% or 85%, or even `pushed' down to 70% or up to 90%. With suitable questionsit is possible then to have an idea of the range of possibilities, in most cases with the different values not equally likely (sharp edges are never reasonable). For example, in this case it could be a triangular distribution peaked at 80%. This way of modeling the uncertainties on degrees of belief is similar to that recommended by the ISO's GUM (Guide to the expression of uncertainty in measurement (19)) to model uncertainties due to systematic effects. After we have modeled uncertain probabilities we can use the formal rules of the theory to `integrate over' the possibilities, analytically or by Monte Carlo (and after some experience you might find out that, if you have several uncertain contributions, the details of the models are not really crucial, as long as mean and variance of the distributions are `reasonable'). The only important remark is to be careful with probabilities approaching 0 or 1. This can be done using log scale for intensities of belief, for the details of which I refer to (20) [in particular Sections 2.4, 3.1, 3.3 and 3.4 (especially Footnote 22), and Appendix E] and references therein.

Once we have broken the taboo of freely speaking (because in reality we already somehow do it) of probabilities of probabilities, it is obvious that there is no problem to extend this treatment of uncertainty to related quantities, like odds and Bayes factor, i) as a simple propagation from uncertain probabilities; ii) in direct assessments by experts. For example, direct assessments of odds are currently performed for many real-life events. Direct (`subjective') assessments of Bayes factors were indeed envisaged in Ref. (20).