Probability of complete classes

These formulae become more interesting when we consider a set of propositions $H_j$ that all together form a tautology (i.e., they are exhaustive) and are mutually exclusive. Formally

$$\cup_i H_j = \Omega$$ (7)

$$H_j \cap H_k = \emptyset \quad \mbox{if }\ j \ne k \, \,.$$ (8)

When these conditions apply, the set $\{H_j\}$ is said to form a complete class. The symbol $H$ has been chosen because we shall soon interpret $\{H_j\}$ as a set of hypotheses.

The first (trivial) property of a complete class is normalization, that is

$$\sum_j P(H_j) = 1\, ,$$ (9)

which is just an extension of Eq. (6) to a complete class containing more than just a single proposition and its negation.

For the complete class $H$, the generalizations of Eqs. (6) and the use of Eq. (4) yield:

$$P(A) = \sum_j P(A, H_j)$$ (10)

$$P(A) = \sum_j P(A\,\vert\,H_j) \, P(H_j) \,.$$ (11)

Equation (10) is the basis of what is called marginalization, which will become particularly important when dealing with uncertain variables: the probability of $A$ is obtained by the summation over all possible constituents contained in $A$. Hereafter, we avoid explicitly writing the limits of the summations, meaning that they extend over all elements of the class. The constituents are `$A, H_j$,' which, based on the complete class of hypotheses $\{H\}$, themselves form a complete class, which can be easily proved. Equation (11) shows that the probability of any proposition is given by a weighted average of all conditional probabilities, subject to hypotheses $H_j$ forming a complete class, with the weight being the probability of the hypothesis.

In general, there are many ways to choose complete classes (like `bases' in geometrical spaces). Let us denote the elements of a second complete class by $E_i$. The constituents are then formed by the elements $(E_{i},H_j)$ of the Cartesian product $\{E\}\times\{H\}$. Equations (10) and (11) then become the more general statements

$$P(E_i) = \sum_j P(E_i, H_j)$$ (12)

$$P(E_i) = \sum_j P(E_i\,\vert\,H_j) \, P(H_j)$$ (13)

and, symmetrically,

$$P(H_j) = \sum_i P(E_i, H_j)$$ (14)

$$P(H_j) = \sum_i P(H_j\,\vert\,E_i) \, P(E_i)\,.$$ (15)

The reason we write these formulae both ways is to stress the symmetry of Bayesian reasoning with respect to classes $\{E\}$ and $\{H\}$, though we shall soon associate them with observations (or events) and hypotheses, respectively.