Unfolding an experimental distribution

If one observes $n(E)$ events with effect $E$, the expected number of events assignable to each of the causes is

$$\widehat{n}(C_i) = n(E)\, P(C_i\,\vert\,E)\,.$$ (7.2)

As the outcome of a measurement one has several possible effects $E_j$ ( $j=1, 2, \ldots, n_E$) for a given cause $C_i$. For each of them the Bayes formula () holds, and $P(C_i\,\vert\,E_j)$ can be evaluated. Let us write () again in the case of $n_E$ possible effects^7.1, indicating the initial probability of the causes with $P_\circ (C_i)$:

$$P(C_i\,\vert\,E_j) = \frac{P(E_j\,\vert\,C_i)\, P_\circ (C_i)} {\sum_{l=1}^{n_C} P(E_j\,\vert\,C_l)\, P_\circ (C_l)}\, .$$ (7.3)

One should note the following.

• $\sum_{i=1}^{n_C} P_\circ (C_i) = 1$, as usual. Note that if the probability of a cause is initially set to zero it can never change, i.e. if a cause does not exist it cannot be invented.
• $\sum_{i=1}^{n_C} P(C_i\,\vert\,E_j) = 1$. This normalization condition, mathematically trivial since it comes directly from (), indicates that each effect must come from one or more of the causes under examination. This means that if the observables also contain a non-negligible amount of background, this needs to be included among the causes.
• $0 \le \epsilon_i \equiv \sum_{j=1}^{n_E} P(E_j\,\vert\,C_i) \le 1$. There is no need for each cause to produce at least one of the effects. $\epsilon_i$ gives the efficiency of finding the cause $C_i$ in any of the possible effects.

After $N_{obs}$ experimental observations one obtains a distribution of frequencies $\underline{n}(E) \equiv \{n(E_1), n(E_2),$ $\ldots , n(E_{n_E})\}$. The expected number of events to be assigned to each of the causes (taking into account only the observed events) can be calculated by applying () to each effect:

$$\left.\widehat{n}(C_i)\right\vert _{obs} = \sum_{j=1}^{n_E}n(E_j)\, P(C_i\,\vert\,E_j) \,.$$ (7.4)

When inefficiency^7.2 is also brought into the picture, the best estimate of the true number of events becomes

$$\widehat{n}(C_i) = \frac{1}{\epsilon_i} \sum_{j=1}^{n_E}n(E_j)\, P(C_i\,\vert\,E_j) \hspace{1. cm}\epsilon_i \ne 0\,.$$ (7.5)

From these unfolded events we can estimate the true total number of events, the final probabilities of the causes and the overall efficiency:

$$\widehat{N}_{true} = \sum_{i=1}^{n_C} \widehat{n}(C_i)\, ,$$

$$\widehat{P}(C_i) \equiv P(C_i\,\vert\,\underline{n}(E)) = \frac{\widehat{n}(C_i)}{\widehat{N}_{true}} \, ,$$

$$\widehat{\epsilon} = \frac{N_{obs}}{\widehat{N}_{true}}\, .$$

If the initial distribution $\underline{P_\circ} (C)$ is not consistent with the data, it will not agree with the final distribution $\underline{\widehat{P}}(C)$. The closer the initial distribution is to the true distribution, the better the agreement is. For simulated data one can easily verify that the distribution $\underline{\widehat{P}}(C)$ lies between $\underline{P_\circ} (C)$ and the true one. This suggests proceeding iteratively. Fig. shows an example of a bidimensional distribution unfolding.

More details about iteration strategy, evaluation of uncertainty, etc. can be found in Ref. [56].

Figure: Example of a two-dimensional unfolding: true distribution (a), smeared distribution (b) and results after the first four steps [(c) to (f)].

I would just like to comment on an obvious criticism that may be made: ``the iterative procedure is against the Bayesian spirit, since the same data are used many times for the same inference''. In principle the objection is valid, but in practice this technique is a ``trick'' to give to the experimental data a weight (an importance) larger than that of the priors. A more rigorous procedure which took into account uncertainties and correlations of the initial distribution would have been much more complicated. An attempt of this kind can be found in Ref. [57]. Examples of unfolding procedures performed with non-Bayesian methods are described in Refs. [54] and [55].

Note added: A recent book by Cowan[58] contains an interesting chapter on unfolding.