Insufficient reason and maximum entropy

The first and most famous criterion for choosing initial probabilities is the simple Principle of Insufficient Reason (or Indifference Principle): If there is no reason to prefer one hypothesis over alternatives, simply attribute the same probability to all of them. This was stated as a principle by Laplace^3.13 in contrast to Leibnitz' famous Principle of Sufficient Reason, which, in simple words, states that ``nothing happens without a reason". The indifference principle applied to coin and die tossing, to card games or to other simple and symmetric problems, yields to the well-known rule of probability evaluation that we have called combinatorial. Since it is impossible not to agree with this point of view, in the cases for which one judges that it does apply, the combinatorial ``definition'' of probability is recovered in the Bayesian approach if the word ``definition'' is simply replaced by ``evaluation rule''. We have in fact already used this reasoning in previous examples.

A modern and more sophisticated version of the Indifference Principle is the Maximum Entropy Principle. The information entropy function of $n$ mutually exclusive events, to each of which a probability $p_i$ is assigned, is defined as[40]

$$H(p_1, p_2,\ldots p_n) = - K\sum_{i=1}^np_i\ln{p_i},$$ (3.25)

with $K$ a positive constant. The principle states that ``in making inferences on the basis of partial information we must use that probability distribution which has the maximum entropy subject to whatever is known[41]''. Note that, in this case, ``entropy'' is synonymous with ``uncertainty''[41]. One can show that, in the case of absolute ignorance about the events $E_i$, the maximization of the information uncertainty, with the constraint that $\sum_{i=1}^np_i=1$, yields the classical $p_i=1/n$ (any other result would have been worrying $\ldots$).

Although this principle is sometimes used in combination with the Bayes formula for inferences (also applied to measurement uncertainty, see Ref. [23]), it will not be used for applications in these notes.