Probably a discovery: Bad mathematics means rough scientific communication¹

[ arXiv:1112.3620v2 ]

G. D'Agostini

Università ``La Sapienza'' and INFN, Roma, Italia

(giulio.dagostini@roma1.infn.it, http://www.roma1.infn.it/~dagos)

[Automatically translated by latex2html: for a printable version see [ arXiv:1112.3620 ]]
 
For additional links see here

Abstract:

According to the media, in spring of this year the experiment CDF at Fermilab had made most likely (``this result has a 99.7 percent chance of being correct''[1]) a great discovery (``the most significant in physics in half a century''[2]). However, since the very beginning, practically all particle physics experts did not believe that was the case. This is the last of a quite long series of fake claims based on trivial mistakes in the probabilistic reasoning that can be sketched with the following statements, understandable by everybody: the probability of a senator to be a woman is not the same as the probability of a woman to be a senator; a free neutron has only $3\times 10^{-4}$ probability to decay after two hours, but, if we observe a neutron decaying after such a time, this is not an indication of an anomalous behavior of such a particle; the fact that the probability of a Gaussian random generator with $\mu=0$ and $\sigma=1$ to produce a number, rounded to three decimal digits, equal to 3.000 is $4.2\times 10^{-6}$ does not allow us to say that, once this number has been observed, there is only $4.2\times 10^{-6}$ probability it comes from that generator, neither that $4.2\times 10^{-6}$ is the probability that 3.000 is a statistical fluctuation; and not even, still considering the latter numerical example, we can say that the probability of 3.000 to be a statistical fluctuation is $1.3\times 10^{-3}$, `because' this is the probability of such a generator to produce a number larger or equal than the observed one. The main purpose of this note is to invite everybody, but especially journalists and general public, most times innocent victims of misinformation of this kind, to mistrust claims not explicitly reported in terms of how much we should believe something, under well stated conditions and assumptions. (A last minute appendix has been added, with comments on the recent news concerning the Higgs at LHC.)

• Introduction
• The facts
  • The (filtered and processed) data
  • The statistically motivated claim
  • How the claim was explained to the general public (and perhaps even what some particle physicist thought)
  
  
• Where is the problem?
• Why there is such a problem?
  • ``The essential problem of the experimental method''
  • A curious ideology
  • The mismatch
  
  
• The mathematics of beliefs
  • Stating the strength of ``pragmatic beliefs'' by odds
  • Coherent virtual bets
  • Belief Vs imagination, beliefs Vs wish, subjective Vs arbitrary: the role of the coherent (virtual) bet
  • Updating beliefs
  
  
• Conclusions
• Bibliography
• Appendix: `???' at Fermilab Vs Higgs boson at CERN
• About this document ...

Additional related links^(*) (besides those of the bibliography)

• A discrepancy published by CDF (Georgio's blog)
• Of course it was not new physics (Georgio's blog)
• Citare i blog (Rangle)
• The USC Auger group blog
• Quanto è significativo un risultato scientifico (E. Presanti)
• I'd put a tenner – but not a ton – on the Higgs-Boson existing (Philip Ball, Guardian)
• skepsisfera.blogspot.com (Bayes and Particle Physics)
• Philip Strange Science Writing
• L. Maiani, Quando abbiamo chiuso Lep. Storia della caccia al Bosone di Higgs in attesa dell'annuncio da Ginevra
• Other recent, related examples of p-values erroneously turned into probabilities:
  • “Con tale procedura per l’eccesso osservato si ottiene una significanza di 3.6 deviazioni standard, che si riduce a 2.3 deviazioni standard... (in altre parole, la probabilità che l'eccesso osservato sia dovuto ad una fluttuazione del fondo è solo dell'1%)” (Atlas/INFN, 13 Dicembre 2011).
  • “Physicists from the CDF and DZero collaborations found excesses in their data that might be interpreted as coming from a Higgs boson with a mass in the region of 115 to 135 GeV. In this range, the new result has a probability of being due to a statistical fluctuation at level of significance known among scientists as 2.2 sigma.” (Fermilab press release, March 7, 2012).
  • “Researchers on the Tevatron see an excess of events produced in the machine’s proton–antiproton collisions that could be caused by a Higgs with a mass between 117 and 131 GeV. The excess had a statistical significance of 2.6 sigma, meaning there is about a 0.5% probability that the result is due to chance.” (Nature, 07 March 2012).
  • “The Fermilab physicists have found a broad hump in their data in the same region, between 115 billion and 135 billion electron volts. Those results came from combining the data from two detectors operated on the Tevatron: the Collider Detector at Fermilab, and DZero. The chances of this signal being the result of a random fluctuation in the data were only about 1 in 100, the group said.” (New York TimesMarch 7, 2012).
  • “Physicists claim evidence of a new particle only if the probability that the data could be due to a statistical fluctuation is less than 1 in 740, or three sigmas. A discovery is claimed only if that probability is less than 1 in 3.5 million, or five sigmas.T” (The Daily Galaxy, March 07, 2012; same as Science Daily, March 07, 2012).
  • “Il loro livello di confidenza statistica, cioè la probabilità che quanto osservato non sia frutto del caso, è ancora troppo basso. I fisici misurano questa probabilità con una misura detta “sigma”: i dati del Tevatron hanno un sigma di 2,6, che significa che c'è uno 0,5 per cento di possibilità che quanto osservato siano fluttuazioni casuali. Quelli del CERN erano a 3,1 sigma e 2,2 per CMS e ATLAS rispettivamente. Una rapida combinazione statistica dei dati di tutti gli esperimenti, europei e americani, porterebbe secondo alcuni ricercatori a un valore di 4 sigma per un bosone di Higgs in quella “finestra” di massa. ” (Treccani.it, 13 marzo 2012)
• Similar claims just before the 4 July 2012 announcement:
  • “ «I dati confermano la soglia dei 5 sigma, vale a dire una probabilità di scoperta pari al 99,99994 per cento» spiega Gian Francesco Giudice, teorico del Cern.” (Corriere della Sera, 3 July 2012).
  • “ Scientists at the Large Hadron Collider are expected to say they are 99.99 per cent certain it has been found - which is known as 'four sigma' level. ” (Daily Mail, 1 July 2012).
  • “ the Tevatron team says that the observed Higgs signal in the combined data from the CDF and DZero experiments in the bottom-quark decay mode has a statistical significance of 2.9 sigma. This means there's only a 1-in-550 chance that the signal is due to a statistical fluctuation. ” (TG Daily, 3 July 2012).
  • “ Based on two experiments, the Fermilab team found that there is only a one-in-550 chance that the signal was a statistical fluke. ” (The Wall Street Journal, 3 July 2012).
  • “Ahead of the expected announcement, the journal Nature reported "pure elation" Monday among physicists searching for the Higgs boson. One team saw only "a 0.00006% chance of being wrong," the journal said. ” (USA Today, 2 July 2012).
  • “Mi riservo un post più lungo su questo argomento se la scoperta del bosone di Higgs venisse annunciata, cioè nel caso in cui il segnale raggiunga una significanza di 5 sigma, che corrisponde a dire che vi è solamente lo 0.00006% di probabilità che il segnale sia una fluttuazione statistica!!
    ...
    In tutto, questi due esperimenti vedono una significanza locale di 2.9 sigma (ricordate che per acclamare la scoperta abbiamo bisogno di 5 sigma), ovvero c'è una probabilità su 550 che questo risultato sia dovuto a fluttuazioni statistiche, per una massa dell'Higgs intorno a 120 GeV (più precisamente non possono escludere la regione di massa compresa tra 115 e 135 GeV). ” (The Gravity Room, 3 July 2012).
  
   
  → I stopped this game the 3th of July 2012, but you might continue yourself e.g. searching the following key words (or something similar) on the Google:
  • “0.00006% chance of being wrong”;
  • “higgs cern 99.99994%”;
  • “higgs 99.9999”.
  Latest check (July 2014, in occasion of the 2014 Hasco Summer School): the keywords higgs cern ’99.99994%' gave about 1.5 million results! (Two-three years after the number has decreased, due to some Google policy to 'forget' old links.)
   
  
• Other links after July 4 (but not any longer simply on trivial misunderstandings of p-values):
  • An interesting thread on Bayesian.org forum initiated by Tony O'Hagan (somehow triggered by Dennis Lindley).
  
   
• More links, added after June 2017, just to remind that the problem is far from being settled.
  • "Al momento il sigma dell'esperimento, un indice di probabilità che va da 1 a 5, è attestato esattamente a metà, ossia 2,5: un valore che non è sufficiente per trarre una conclusione definitiva - nonostante altre ricerche abbiano già suggerito qualcosa del genere.", Luigi Bignami, Focus, Scienze, 20 aprile 2017.
  • "We trust the existence of the Higgs particle, which is today the weakest of the confirmed theories, with a 5-sigma reliability, namely a Bayesian degree of confidence of 99.9999%.", Carlo Rovelli, arXiv:1609.01966v1 (local copy).
  • "Nel caso dell'eccesso sullo spettro delle coppie di fotoni, se uno prende il grafico di ATLAS in cui la montagnola è più prominente, la probabilità che questa sia dovuta a una casualità è due su 10.000, dunque piuttosto piccola. Quando però consideriamo il fatto di aver cercato montagnole un po' dappertutto, allora questa probabilità aumenta a due su 100. I numeri di CMS sono persino più grandi, indicando una probabilità ancora più grande che si tratti solo di una fluttuazione del rumore di fondo.", Marco Dalmastro, Le Scienze, 19 dicembre 2015
  • "When all the statistical effects are taken into consideration, Dr. Cranmer said, the bump in the Atlas data had about a 1-in-93 chance of being a fluke — far stronger than the 1-in-3.5-million odds of mere chance, known as five-sigma, considered the gold standard for a discovery.", Dennis Overbye, The New York Times, 15 December 2015.

^(*) Note: all these web sites were visited for the first time after the post of the paper.