Peelle's Pertinent Puzzle

To summarize, when there is an overall uncertainty due to an unknown systematic error and the covariance matrix is used to define $\chi^2$, the behaviour of the fit depends on whether the uncertainty is on the offset or on the scale. In the first case the best estimates of the function parameters are exactly those obtained without overall uncertainty, and only the parameters' standard deviations are affected. In the case of unknown normalization errors, biased results can be obtained. The size of the bias depends on the fitted function, on the magnitude of the overall uncertainty and on the number of data points.

It has also been shown that this bias comes from the linearization performed in the usual covariance propagation. This means that, even though the use of the covariance matrix can be very useful in analysing the data in a compact way using available computer algorithms, care is required if there is one large normalization uncertainty which affects all the data.

The effect discussed above has also been observed independently by R.W. Peelle and reported the year after the analysis of the CELLO data[48]. The problem has been extensively discussed among the community of nuclear physicists, where it is currently known as ``Peelle's Pertinent Puzzle''[50].

Recent cases in High Energy Physics in which this effect has been found to have biased the result are discussed in Refs. [51,52].

Note added: the solution outlined here is taken from Ref. [47], and it has to be considered an ad hoc solution. The general (of course Bayesian) solution to the $\chi^2$ paradox has been worked out recently[53], and it will be published in a forthcoming paper.