Inference from a data set and sequential use of Bayes formula

In the elementary examples shown above, the inference has been done from a single data point $d$. If we have a set of observations (data), indicated by ${\mbox{\boldmath$d$}}$, we just need to insert in the Bayes formula the likelihood $p({\mbox{\boldmath$d$}}\,\vert\,\theta ,I)$, where this expression indicates a multi-dimensional joint pdf.

Note that we could think of inferring $\theta$ on the basis of each newly observed datum $d_i$. After the one observation:

$$p(\theta \,\vert\,d_1,I) \propto p(d_1 \,\vert\,\theta,I) \, p(\theta \,\vert\, I)$$ (45)

and after the second:

$$p(\theta \,\vert\,d_1,d_2,I) \propto p(d_2 \,\vert\,\theta,d_1,I) \, p(\theta \,\vert\,d_1,I)$$ (46)

$$\propto p(d_2 \,\vert\,\theta,d_1,I) \, p(d_1 \,\vert\,\theta,I) \, p(\theta \,\vert\, I)\,.$$ (47)

We have written Eq. (47) in a way that the dependence between observables can be accommodated. From the product rule in Tab. 1, we can rewrite Eq. (47) as

$$p(\theta \,\vert\,d_1,d_2,I) \propto p(d_1,d_2 \,\vert\,\theta,I) \,p(\theta \,\vert\, I)\,.$$ (48)

Comparing this equation with (47) we see that the sequential inference gives exactly the same result of a single inference that properly takes into account all available information. This is an important result of the Bayesian approach.

The extension to many variables is straightforward, obtaining

$$p({\mbox{\boldmath$\theta$}}\,\vert\,{\mbox{\boldmath$d$}}, I) \propto p({\mbox{\boldmath$d$}}\,\vert\,{\mbox{\boldmath$\theta$}},I) \, p({\mbox{\boldmath$\theta$}}\,\vert\,I) \,.$$ (49)

Furthermore, when the $d_i$ are independent, we get for the likelihood

$$p({\mbox{\boldmath$d$}} \,\vert\,{\mbox{\boldmath$\theta$}},I) = \prod_i p(d_i \,\vert\,{\mbox{\boldmath$\theta$}},I)$$ (50)

$${\cal L}({\mbox{\boldmath$\theta$}}; {\mbox{\boldmath$d$}}) = \prod_i {\cal L}({\mbox{\boldmath$\theta$}}; d_i) \, ,$$ (51)

that is, the combined likelihood is given by the product of the individual likelihoods.