5-sigma does not prove Higgs

Economics professor Stephen T. Ziliak writes:

I want to believe as much as the next person that particle physicists have discovered a Higgs boson, the so-called “God particle,” one with a mass of 125 gigaelectronic volts (GeV). But so far I do not buy the statistical claims being made about the discovery. Since the claims about the evidence are based on “statistical significance” – that is, on the number of standard deviations by which the observed signal departs from a null hypothesis of “no difference” – the physicists’ claims are not believable. Statistical significance is junk science, and its big piles of nonsense are spoiling the research of more than particle physicists.

I’m an economist. So don’t trust me with newfangled junk bonds or the fate of the world financial system. But here is something you can believe, and will want to: Statistical significance stinks. In statistical sciences from economics to medicine, including some parts of physics and chemistry, the ubiquitous “test” for “statistical significance” cannot, and will not, prove that a Higgs boson exists, any more than it can prove the reality of God, the existence of a good pain pill, or the validity of loose monetary policy. ...

I show in a book I wrote on the subject with Deirdre N. McCloskey, The Cult of Statistical Significance (2008), that the null hypothesis test procedure – another name for statistical significance testing – produces many such errors, with tragic results for real world economies, law, medicine, and even human life. ...

Consider again the illogic of the physicists’ procedure. The signal in the data which has been observed over and above background noise (denoted as being at 5 sigma) is possibly a Higgs boson – that is true. But in sober moments – when flash-bulbs fade and fizzy drinks fall flat – those same particle physicists admit that the jury is still out – that the statistically significant bump could be “consistent with” other plausible hypotheses, not specified by their models – just like Mrs. Smith could have died of something other than cramp, and probably did.

He has a point, if the 5-sigma significance were the main evidence for the Higgs boson. There is danger in relying on p-values too much.