Last week I attended a special webinar on “Statistical Significance and p-values” presented by the European Network of Business and Industrial Statistics (ENBIS). To my relief, none of the speakers called for abandoning the use of p values. Though I feel that p’s should not be a statistic to solely rely on for deeming results significant or not, when used properly they certainly reduce the risk of pressing ahead with spurious outcomes. It was great to get varying perspectives on this issue.

Here are a couple of fun quotes on that I gleaned from this ENBIS event:

• “Surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?” – Rosnow, R.L. & Rosenthal, R. “Statistical procedures and the justification of knowledge in psychological science”, American Psychologist, 44 (1989), 1276-1284.
• “My definition of a statistician is ‘one who prefers true doubts to false certainty’.” – Stephen Senn (Statistical Consultant, Edinburgh, Scotland, UK)

If you have a strong stomach for stats, see this Royal Society review article: The reign of the p-value is over: what alternative analyses could we employ to fill the power vacuum? It includes discussion of an alternative to p values called the “Akaike information criterion” (AIC). This interested me, because, as a measure for goodness of model-fit, Stat-Ease software provides AICc—a version of this statistic that corrects (hence the appendage “c”) for the small sample sizes of industrial experiments (relative to large retrospective scientific studies).

As the headline article for their November 17 Business section, my hometown newspaper, the St. Paul Pioneer Press, picked up an alarming report on p-values by Associated Press (AP). That week I gave a talk to the Minnesota Reliability Consortium*, after which one of the engineers told me that he also read this article and lost some of his faith in the value of statistics.

“One investment analyst reacted by reducing his forecast for peak sales of the drug — by $1 billion. What happened? The number that caused the gasps was 0.059. The audience was looking for something under 0.05.”

– Malcom Ritter, AP, relaying the reaction to results from a “huge” heart drug study presented this fall by Dr. Scott Solomon of Harvard’s Brigham and Women’s Hospital.

As I noted in this May 1^st blog/, rather than abandoning p-values, it would pay to simply be far more conservative by reducing the critical value for significance from 0.05 to 0.005. Furthermore, as pointed out by Solomon (the scientist noted in the quote), failing to meet whatever p-value one sets a priori as the threshold, may not refute a real benefit—perhaps more data might generate sufficient power to achieve statistical significance.

Rather than using p-values to arbitrarily make a binary pass/fail decision, analysts should use this statistic as a continuous measure of calculated risk for investment. Of course, the amount of risk that can be accepted depends on the rewards that will come if the experimental results turn out to be true.

It is a huge mistake to abandon statistics because of p being hacked to come out below 0.05, or p being used to kill projects due to it coming out barely above 0.05. Come on people, we can be smarter than that.

* “Know the SCOR for Multifactor Strategy of Experimentation”