“Most results, including those that appear in top-flight peer-reviewed journals, can’t be reproduced.”
This is a “dirty secret” revealed by the Wall Street Journal’s Gautam Naik in this December report. It cites statistics from Bayer that nearly two-thirds of published studies could not be replicated. Naik blames the complicated nature of experiments nowadays along with the “positive bias” researchers driven to produce results. Glenn Begley, vice president of research at Amgen, a biotechnology company, suggests that “academic scientists, like drug companies, should perform more experiments in a ‘blinded” manner to reduce any bias toward positive findings.”
Meanwhile, Duncan Watts, author of Everything is Obvious: *Once You Know the Answer says
“When you do the experiment properly [randomized and controlled], all the numbers go down.”
He’s speaking on the bias of marketing executives toward their own sensibilities, which often do not reflect those of the population being sold to. See what the Financial Times “undercover economist” Tim Harford says about this here. Unfortunately, in my experience, those (the analysts) who know better than to extrapolate from small, non-representative sample of opinions from the ‘powers-that-be’ (often n=1, that is—the Boss) get very little support for spending money to put these assertions to the test. Even though you know the top dogs might be barking up the wrong tree it’s easiest just to go along with the pack and press ahead. To do otherwise risks suffering a painful bite-back. Yes, I am a cynic.