Assessing subjective degrees of beliefs - virtual bets

A good way to force experts to provide the initial beliefs they have formed in their minds, elaborated somehow by their `educated intuition' (see Appendix C), is to propose them a virtual lottery, in which they can choose the event on which to bet to win a rich prize. One is the event of interest (let us call it $A$), the other one is a simpler one, based on coins, urns, dice or card games. The latter can be considered a kind of `standard', or a `reference' (as it is done in measurements to calibrate instruments), whose probability is the same for everyone. We can ask ourselves (or the experts), for example, if we (or they) prefer to bet on $A$ rather than on head resulting from a regular coin; or on white extracting a ball from a box containing 100 balls, 90 of which white; and so on.

Obviously, none can state initial odds with very high precision.¹⁹ But this does not matter (table 1 can help to get the point). We want to understand if they are of the order of 1 (equally likely), of the order of a few units (one is a bit more likely than the other one), or of suitable powers of 10 (much more or much less likely than the other one). If one has doubts about the final result, one can make a `sensitivity analysis', i.e. vary the value inside a wide but still believable range and check how the result changes. The sensitivity (or insensitivity) will depend also on the other pieces of evidence to draw the final conclusion. Take for example two different evidences, characterized by Bayes factors of $H_1$ versus $H_2$ very high (e.g. $10^4$) or very small (e.g. $10^{-4}$), corresponding to $\Delta$JL's of $+4$ or $-4$, respectively (for the moment we assume all subjects agree on the evaluation of these factors). Given these values, it is easy to check that, for many practical purposes, the conclusions will be the same even if the initial odds are in the range $1/10$ to $10$, i.e. a JL between $-1$ and $+1$, that can be stated as JL$_{1,2}(E_0)=0\pm 1$. Adding `weights of evidence' of $+4$ or $-4$, we get final JL's of $4\pm 1$ or $-4\pm 1$, respectively.²⁰

The limit case in which the Bayes factor is zero or infinity (i.e. $\Delta$JL's $-\infty$ or $+\infty$) makes the conclusion absolutely independent from priors, as it seems obvious.