The amazing persistence of biased scientific results—Popeye’s spinach found fraudulent
Posted by mark in science, Uncategorized on December 5, 2012
I recently completed a series of webinars on using graphical diagnostics to deal with bad experimental data.* The first thing I focused on was avoidance of confirmation bias – hearing what you want to hear, for example in the persistence of the possibilities of cold fusion. See more cases of confirmation bias in this detailing by Peter Bowditch in Australasian Science.
I came across another interesting example of the persistence of wished-for results in a review** of Samuel Arbesman’s new book on The Half-Life of Facts. It turns out that spinach really does not delivery the amount of iron that my mother always believed would make it worth us eating this horrible food. She was a child of the 1930’s, at which time it was widely believed that the edible (?) plant contained 35 milligrams of iron, a tremendous concentration, per serving. However, the actual value is 3.5 mg—the chemist who first analyzed it misplaced the decimal point when transcribing the data from his notebook in 1870! In 1937 this error was finally corrected, but my mom never got the memo, unfortunately for me and my six younger siblings. ; )
*“Real-Life DOE” presentation, posted here
** The Scientific Blind Spot by David A. Shaywitz in the 11/19/12 issue of Wall Street Journal
Speak softly but carry a big statistic
I heard on CBS Radio radio today this play on Teddy Roosevelt’s famous words. It was quoted by U.S. Senator Amy Klobuchar as her secret weapon (statistics, that is) for women politicians. Searching internet I think it originated from Anne E. Kornblut in her book Notes from the Cracked Ceiling in a section dedicated to Klobuchar. She (the Senator) figures on making an impact on the impasse over the coming “fiscal cliff”. I have no doubt that Senator Klobuchar and scores of other politicians, male and female, will be slinging a lot of statistics during this debate on how to avert financial disaster for us taxpayers. It will take some work to ferret out what’s really true out all the partisan hyperbole.
Statistician mines poll results to come up with odds-on fav for President
CBS News this morning reported the prediction by New York Times statistician Nate Silver on who will be our next President. OK, now that you know (presuming you could not resist following the link), how sure are you that it’s accurate? After all Silver is the author of The Signal and the Noise: Why So Many Predictions Fail – But Some Don’t—published only a month or so ago. My hunch is that Silver does as well as anyone—given so many unknowns that cannot be known, not the least of which is the fickle nature of undecided voters who might en masse switch allegiance the day of the election. Anyways, I am viewing his prediction the same as a weather forecast two days out, that is, with a good deal of skepticism but, nevertheless, appreciation for the science behind the modeling.*
PS. A friend asked me this week whether averaging polls is really valid. I suppose so based on Silver doing it. See how he does it at this detailing by him in his “538” blog (538 is the number of electors in the United States Electoral College).
*For example, within 72 hours of a hurricane’s landfall, meteorologists now predict the bulls-eye within a 100-mile radius—compared to 350 miles 25 years ago. They did really well forecasting Sandy as reported here by The Washington Post.
Probability of vote being pivotal is so small it’s not worth voting
That was the view of 2nd-year PhD student Douglas VanDerwerken up until this Presidential election. He abstained on the basis of the lack of return on investment for spending the time to vote when it really cannot make a difference. VanDerwerken lays it all out for statistics magazine Significance in an article for their current (October) issue.* According to his reckoning, there is less than one chance in a million (4.5×10^-7 to be precise) of any person’s vote having an impact. This would be a situation where the voter lives in a swing State and the election comes to a dead heat.
Fortunately (in my opinion—being one who views it as a civic duty) VanDerwerken had an epiphany based on moral reasons, so he shall vote. Thank goodness!
“If you think about it, voting in a large national election – such as the US Presidential election – is a supremely irrational act, because the probability that your vote will make a difference in the outcome is infinitesimally small.”
– Satoshi Kanazawa, rational choice theorist**
* “The next President: Will your vote decide it”
**See Kanazawa’s three-part series on “Why Do People Vote” for his blog “The Scientific Fundamentalist” hosted by Psychology Today. Start with Part 1 posted here and continue on to the end for the answer.
Time to lighten up on homework?
Posted by mark in pop, Uncategorized on October 21, 2012
The Wall Street Journal’s Market Watch this Friday posted the data shown in this chart. For the 11 countries shown** you can see why WSJ seconds the call by French President Hollande to ban all homework.
Students would party hearty but this laissez-faire approach will not fly with those blessed with ambitious parents. Nevertheless the call for less homework, fueled by new data from the National Center for Education Statistics, reinforces other studies going back at least a decade.
It will be interesting to see what emerges as a consensus for a the happy medium on amount of homework assigned. Four hours per night seems way too much, especially at the 8th grade level.
Meta-analysis of hundreds of studies done on the effects of homework shows that the evidence supporting the practice is, at best, modest. Homework seems to be most useful in high school and for subjects like math. At the elementary school level homework seems to be of marginal or no academic value.
– Malcolm Gladwell
*See the report here
**I took out Saudi Arabia, whose result of 34% below average, given 11% being assigned over 4 hours of homework per night, fell far below even these very off-putting predictions–an outlier statistically.
Acknowledgment: Thanks to Devan Govender for alerting me to this issue.
USA unemployment statistic creates a sensation
“Unbelievable jobs numbers..these Chicago guys will do anything..can’t debate so change numbers.”
– Jack Welch
Thursday morning I attended a briefing on the economy by an expert from Wells-Fargo bank. Looking over the trends in USA unemployment rates he noted that no incumbent since World War II has achieved re-election when joblessness exceeded 8 percent. Friday the Bureau of Labor Statistics (BLS) announced that the the national unemployment rate is now 7.8%, an improvement from 8.1% last month. How accurate is this number and is it precise enough that a 0.3% difference can be considered significant? I agree with the conclusion of this critique posted by Brookings Institution that “a large part of monthly unemployment fluctuations are spurious.” So, really, all this fuss about it being 8.1 versus 7.8 percent is really silly from a statistical point of view. However, it is entertaining!
Rock on with algorithms?
Posted by mark in pop, Uncategorized on October 2, 2012
I started off my career as an experiment designer before the advent of cheap calculators. Paying $400 for an HP unit that (gasp!) did logarithms went far beyond my wherewithal in 1974. That was roughly the tuition for one college quarter at University of Minnesota if memory serves. I managed to cover that cost plus room and board by working 24 hours a week washing pots and pans at a hospital kitchen. Those were the days!
Calculating effects from the two-level factorial designs I did that summer as an intern at a chemical research lab required a lot of hand calculations—many numbers to add and subtract. Thankfully a fellow named Yates developed an algorithm after these experiments were invented in the 1930s. Following his directions one could tally things up and even do check sums without having to think much. That’s what algorithms do—provide a recipe for solving problems.
As an engineer I have a healthy respect for algorithms, but my wife, who works as a preschool teacher, thinks this is geeky. For example, I admired the nerdy professor in the TV show “Numbers” that aired a few years ago. But every time he expounded on some algorithm that ingeniously saw the pattern of a serial criminal, she just laughed. Ironically she is now hooked on a show called “Person of Interest” that is based on predictive policing, that is, using algorithms to calculate a crime to come. That scares me!
According to a new book by Christopher Steiner titled Automate This: How Algorithms Came to Rule Our World (see this Wall Street Journal review) all of us had best be on our guard against seemingly clever ways to systematically solve problems. It seems that the engineers, mathematicians, programmers and statisticians who come up with these numerical recipes invaded Wall Street. They became known as the “Quants”—dominating the way stocks now get traded.
The problem with all this (even I have to admit) is that these systematic approaches to things take all the fun out of making choices. Do we really want algorithms to pick our soul mates, invest our money, etcetera? I am up for algorithms like Yate’s that quickly solve mathematical problems. A good example of this is the first known algorithm recorded on clay tablets in 2500 B. C. that helped Sumerian traders divvy up a given amount of grain equally to a varying number of recipients. However when things become capricious with many unknowns that are unknowable being thrown into the mix, I’d rather make my own decisions guided by wise counsel.
There is an elephant in the room whenever it comes to discussing computer algorithms, particularly highly automated ones. Almost all such algorithms are inaccurate. They are inaccurate for many reasons, the most important of which is that human behavior is fickle. The inaccuracy could be shockingly high.
– Kaiser Fung, author of Numbers Rule Our World
I really shouldn’t bring this up, but do you suppose certain politician might be spending a lot of money on algorithmic solutions to how they can win election? Do these algorithms have any qualms about turning their protagonists into nabobs of negativism? I do not believe that an algorithm has any heart, unfortunately. An algorithm is like Honey Badger—it just don’t care.
How to better your brain to make it think and retain
Buried in my file of fodder for blogs I re-discovered a heads-up from the New York Times on 1/21/11 that giving yourself a quick quiz after studying something once works better than going over and over it. The test-triggered active retrieval promoted meaningful learning by half in terms of how much students remembered a week later.
If you know something, or if you have stored information about an event from the distant past, and never use that information, never think of it, your brain is functionally equivalent to that of an otherwise identical brain that does not “contain” that information.
— Cognitive neuroscientist Endel Tulving quoted in this publication in Current Directions in Psychological Science of research on active retrieval by Jeffrey D. Karpicke of Purdue University
Coincidentally I just read this passage in “Brenner and God” by Wolf Haas, three time winner of the German Thriller Prize, which struck a chord about how the mind works in mysterious ways: “…just like a light that’s too bright can be bad for the eyes, so, too, can a mind that’s too awake be not at all good for thoughts…a half-asleep person can always outmatch an awake person by a long shot, no discussion.”
This happens with me when I am really wrapped up in a writing project or dealing with a very tough problem. Then I cannot sleep well as thoughts keep winding through my head. Often as I am nearly into a dream an answer comes to me. Then the only thing is to get up and write it down in the hopes that next morning it still makes sense. In any case, if I do not make a note, I then cannot sleep for fear of forgetting it. But surprisingly these ideas do usually hold up to the light of day, albeit not always terribly brilliant.
Does this happen to you too?
The next bubble that’s bound to burst: college tuition
I am glad to have graduated the youngest of my 5 children—now self-sustaining in Ohio State University’s biochemistry PhD program. Even taxpayer-subsidized state-school students like her can easily pile up $10,000s in debt for ever-growing tuition. Those going to private institutions are likely to end up with a lot more money to pay back after they complete their studies.
One year ago my high-school classmate Mark Perry, now a professor of economics and finance in the School of Management at the Flint campus of the University of Michigan, warned about a Higher Education Bubble. Under the bombshell blurb “That’s a jump of 1,120%” [from cost of college in 1978], the latest (August 27) issue of Bloomberg Businessweek extends the Bureau of Labor Statistics in Perry’s scary chart to 2012. Given what happened in housing, this is becoming extremely alarming!
Students are paying less and less of direct college costs, relying more on government grants and loans. That has encouraged universities to jack up tuition expenses, fueling a vicious circle reminiscent of the housing bubble.
– David Hogberg, Investors Business Daily
A more graphic illustration is provided via this glimpse at a Broadside by Glenn Reynolds. View his video and weep if you have children heading for college. It’s hard to imagine that this can go on (graduates paying many $100s per month for debt) for much longer. The meager educational-returns on massive investments (loaded into huge debts) just do not seem sustainable.
Random thoughts
Posted by mark in Basic stats & math, design of experiments, Uncategorized on August 26, 2012
The latest issue of Wired magazine provides a great heads-up on random numbers by Jonathan Keats. Scrambling the order of runs is a key to good design of experiments (DOE)—this counteracts the influence of lurking variables, such as changing ambient conditions.
Designing an experiment is like gambling with the devil: only a random strategy can defeat all his betting systems.
— R.A. Fisher
Along those lines, I watched with interest when weather forecasts put Tampa at the bulls-eye of the projected track for Hurricane Isaac. My perverse thought was this might the best place to be, at least early on when the cone of uncertainty is widest.
In any case, one does best by expecting the unexpected. That gets me back to the topic of randomization, which turns out to be surprisingly hard to do considering the natural capriciousness of weather and life in general. When I first got going on DOE, I pulled numbered slips of paper out of my hard hat. Then a statistician suggested I go to a phone book and cull numbers from the last 4 digits from whatever page opened up haphazardly. Later I graduated to a table of random numbers (an oxymoron?). Nowadays I let my DOE software lay out the run order.
Check out how Conjuring Truly Random Numbers Just Got Easier, including the background by Keats on pioneering work in this field by British (1927) and American (1947) statisticians. Now the Australians have leap-frogged (kangarooed?) everyone, evidently, with a method that produces 5.7 billion “truly random” (how do they know?) values per second. Rad mon!