In today’s Wall Street Journal, The Numbers Guy (Carl Bialik) reports on a unanimous ruling by the Supreme Court that companies cannot hide behind statistical significance (lack thereof in this case) as an excuse for nondisclosure of adverse research. He passes along this practical advice:
“A bigger effect produced in a study with a big margin of error is more impressive than a smaller effect that was measured more precisely.”
— Stephen Ziliak, economics professor
However, this legal analysis of the ruling cautions that statistical significance remains relevant for assessing materiality of an adverse event.
Given all this, we can be certain of only one thing – more lawsuits.
#1 by Carpenter on June 7, 2011 - 4:20 pm
Concept illustrated here: http://xkcd.com/892/
#2 by Eric Kvaalen on June 27, 2011 - 1:00 pm
I disagree with the quote from Carl Bialik. It depends on the lower limit of the confidence interval. The study that has a higher lower limit (and positive!) for the effect is more meaningful even if the nominal value of the effect (the middle of the confidence interval) is lower than in the other study.
That’s why a study that gives a confidence interval going from negative to positive is rightly considered insignificant. (I may be disagreeing with the Supreme Court here!)
This is the reason I advocate giving confidence intervals instead of just significance level or the nominal effect size.