Extracting Sunbeams from Cucumbers
Posted by mark in Basic stats & math on January 8, 2012
With this intriguing title Richard Feinberg and Howard Wainer draw readers of Volume 20, Number 4 into what might have been a dry discourse: How contributors to The Journal of Computational and Graphical Statistics rely mainly on tables to display data. Given that “Graphical” is in the title of this publication, it begs the question on whether this method of for presenting statistics really works.
When working on the committee that developed the ASTM 1169-07 Standard Practice for Conducting Ruggedness Tests, I introduced the half-normal plot for selecting effects from two-level factorial experiments. Most of the committee favored this, but one individual – a professor emeritus from a top school of statistics – resisted the introduction of this graphical tool. He believed that only numerical methods, specifically analysis of variance (ANOVA) tables, could support objective decisions for model selection. My comeback was to dodge the issue by simply using graphs and tables – this need not be an either/or choice. Why not do both, or merge them by putting number on to graphs – the best of both worlds?
“A heavy bank of figures is grievously wearisome to the eye, and the popular mind is as incapable of drawing any useful lessons from it as of extracting sunbeams from cucumbers.”
— Economists (brothers) Farquhar and Farquhar (1891)
In their article which can be seen here Feinberg and Wainer take a different tack (path of least resistance?): Make tables look more like graphs. Here are some of their suggestions for doing so:
- Round data to 3 digits or less.
- Line up comparable numbers by column, not row.
- Provide summary statistics, in particular medians.
- Don’t default to alphabetical or some other arbitrary order: Stratify by size or some other meaningful attribute.
- Call out data that demands attention by making it bold and/or bigger and/or boxing it.
- Insert extra space between rows or columns of data where they change greatly (gap).
Check out the remodeled table on arms transfers which makes it clear that, unlike the uptight USA, the laissez faire French will sell to anyone. It would be hard to dig that nugget out of the original data compilation.
Pace yourself with willpower to accomplish 2012 resolutions
Today’s Science Friday radio show made it known that willpower is a very limited resource. That’s the bad news. The good news is that you can build up your willpower by proper exercises, just like you do for toning up muscles. This is the premise of Florida State University psychology professor Roy F. Baumeister, whom Science Friday host Ira Flatow interviewed. Baumeister and co-author John Tierney reveal all in their book Willpower: Rediscovering the Greatest Human Strength, chosen as one of the Best Books of 2011 by Amazon’s editors.
Here are a few pointers I picked up from Science Friday that will prevent depletion of your willpower reservoir:
- Do not start in on all of your New Year’s resolutions all at once – pace yourself: Work on one resolution at a time.
- Pick off the easiest resolutions first, such as making your bed every morning or taking your dog for a walk daily.
- Exercise and build up your willpower with trivial things such as sitting up straighter and using your non-dominant hand to mouse around the computer.
For an inspiring demonstration of supreme willpower see this video of the Stanford Marshmallow Experiment.
http://www.youtube.com/watch?v=6EjJsPylEOY
Unfortunately, according to Professor Baumeister, testing your willpower on something as compelling as a sugary snack can be very perilous, because once you fail it goes downhill from there. That’s why my goal is to first eat all the Christmas cookies before giving up sweets.
Happy New Year — best wishes for achieving all your resolutions…
Why you should be very leery of forecasts
Check out this blog by statistician William Briggs that gives the heads up on how Hurricane Predictors Admit They Can’t Predict Hurricanes. Years ago as a chemical engineer working on process development I would be encouraged by plant personnel to crank old data through a regression analysis to model the operation, thus avoiding any work on their part to run designed experiments. The joke was that we got very good at predicting what would happen last month.
In this case the issue is hurricanes. As Briggs explains, the top experts can fit past data very well (r = 0.79 for 50-year period the last half of 20th century). He refers to this as a ‘hindcast’. But, as the hurricane forecasters themselves admit, these models predict so poorly (r = 0.04) that you may as well just use an average — what I call the ‘mean’ model as a double meaning (ha ha) because it is so disappointing for the analyst.
What it boils down to is that any forecasts on hurricanes this early before the coming season will really just be a lot of hot air, despite impressive statistics from models fit to prior years. The same goes for long-term outlooks on other natural phenomena.
Favorite posts from three rings in the 2011 Management Improvement Blog Carnival (2 of 3)
Posted by mark in leadership, Wellness on December 19, 2011
For the 2011 Annual Management Improvement Blog Carnival,* I reviewed the Unfolding Leadership blog (the second of three looked over by me and my son Hank). In his meme** inspired SMALL WORLD entry Dan Oestreich explains how he began blogging in 2004 as an outlet for creative expression, not only in words, but also photography. The pictures are what caught my eye and made Unfolding Leadership stand out from the pack. This blog of October 7 provides an example of Oestreich’s eloquence in words and visuals.
I found the mid-year blog ON HYPOCRISY AND SELF-PROTECTION very thought-provoking because of its forgiving dissection of why so many leaders say one thing but do the other. Oestreich provides an inside look at just such a fellow who happens to be in charge of a technical group. From my experience, competence in science and engineering correlates inversely with people skills. However, if one is willing to put himself under a microscope and be open to change, improvement is possible. I urge anyone in a leadership position, especially those with technical backgrounds, to read this blog.
Going back to the first Unfolding Leadership blog of 2011 titled SCAR one finds a very poignant story of a woman who had the courage to speak up at work, but, unfortunately, it created a very bad outcome. I really like the accompanying photograph and found it very apropos. Read the comments for further insights on this issue of trust.
* For all the 2011 blogs see this list http://curiouscat.com/management/carnival_2011.cfm. Background on the carnival itself can be found here http://management.curiouscatblog.net/about/.
**(An idea that is spread from blog to blog – see this page http://thedailymeme.com/what-is-a-meme/ for the provenance)
Favorite posts from three rings in the 2011 Management Improvement Blog Carnival (1 of 3)
Posted by mark in Consumer behavior on December 17, 2011
For the 2011 Annual Management Improvement Blog Carnival, hosted by John Hunter, we (my son Hank and I) will pick our favorite posts from these three blogs:
See us hosts and the blogs we’ve chosen to review at this site coordinated by John and background on the carnival itself here.
I will start off our part in this Carnival with my take on Edge Perspectives. The author is John Hagel — a fellow with a background in law and management who caught my fancy for his passion about embracing change. I live by Apostle Paul’s advice (1 Thes. 5:14-22) to “Test all things; hold fast what is good.” However, as Hagel explains in his blog on Cognitive Biases in Times of Uncertainty uncertainty breeds fear for new technologies that offer big benefits. It’s amazing to see how much people vary in interest/aversion to new technology. For example, I recently made an impulse purchase of Sony’s Internet TV Blu-ray disc player. When I asked Hank to help me set it up in our master bedroom, he got really keyed up about how this device streams content controlled by a nifty remote with all the functions of a full-size keyboard. But then my wife tried to turn the TV on – not a good scene. She just needs a lot more time to embrace technological change than Hank or me.
A number of the 2011 blogs in Edge Perspectives provide in-depth reviews of books by other big thinkers offering their prescription for how Americans can regain their edge. To be honest, I am just too busy dealing with everyday life to work up the necessary energy to get a grasp on these huge issues. My attention gets re-engaged when Hegel delivers thoughts from the heart such as his Revolution from the Edge entry. Here he reports back from TED, nonprofit devoted to “Ideas Worth Spreading” that draws top thinkers from around the globe, one of whom is invited to turn the world inside out – aided by a $100,000 award. Hagel focuses his thoughts on the uprisings in the Middle East and North Africa, where a well-educated younger generation broke free of old ways enforced by rigid and repressive dictators. Having just come away from India, where democracy fosters technological innovation (if only the infrastructure could catch up!), I am keen on Hagel’s call for those in developed nations to do everything possible for removing the barriers of corruption, debt and environmental degradation that stand in the way of further progress by emerging countries.
“As a man changes his own nature, so does the attitude of the world change towards him. … We need not wait to see what others do.”
Mahatma Gandhi
Formula for Happiness
Spiritual teacher Jaya Row, a microbiologist by training, provides a simple formula for happiness in this article on Timeless Truths which I saw in The Times of India while vacationing in Aurangabad last month. Here it is: Happiness Quotient (HQ) equals the number of desires fulfilled divided by the number of desires entertained. She advises that we focus more on the denominator than the numerator of this HQ ratio. I interpret this as trying to be happy with less, rather than being greedy for more.
While in Aurangabad I came by the tomb of the Mughal emperor Aurangazeb. His father, Shah Jahan, built the Taj Mahal – the most beautiful building in the world, in my opinion.* Aurangazeb was an ascetic who followed a particularly austere from of Sufism. At the ripe old age of 88 the emperor was buried according to his wishes in a simple tomb purchased with money he earned himself by stitching caps. Sometimes the apple does fall far from tree. One wonders who led the happier life – Aurangazeb or his father. Now I think that the Taj Mahal really symbolizes the desires that never can be fulfilled in anyone’s life, that is, a monument to unhappiness.
Ending on a happier note, I offer up this photo of monkeys scampering along the road to Chand Minar, the Tower of the Moon, built by the Mughal conquerors of what came to be known as Aurangabad.
*Another story: While working for General Mills in the mid-1980s as a purchaser of Indian agricultural products, my agent bribed the guard at this monumental set of tombs in Agra to enable a private tour for my wife and I under a full moon and candlelight. Priceless!
Pareto distribution makes a handy ‘screw driver’
Posted by mark in Consumer behavior on October 30, 2011
“What scientific concept would improve everybody’s cognitive toolkit?” This was the 2011 question posed by sociologist Nicholas Christakis in his annual survey of science and technology gurus via the online salon Edge. Internet scholar Clay Shirky chose the Pareto distribution as a tool that could be put to better use for reducing income disparities. When I led manufacturing improvement teams, I graphed cumulative losses by cause and used this Pareto chart* to focus my engineering colleagues on the 20 percent that caused 80% of the problems. It seems that Christakis knew something when he put forward the Pareto early this year because, since then, the 99 Percent Project has taken aim at the 1 percent of folks from Wall Street who purportedly control all of our money. Being a technical type I am not interested in getting into issues of elitism. I’d rather experiment to identify the vital few factors that affect a system of interest.
Meanwhile, the United Nations declared this summer that tomorrow, October the 31st, the 7 billionth baby will be born. They may be overly precise on the timing, yet this is a very compelling statistic. Breaking down the world’s wealth by this population will continue to keep economists busy, but you can be sure it will maintain a “predictable imbalance” as observed by their colleague Pareto. Along those lines, I suggest you measure how Halloween candy gets distributed to your neighborhood population of trick-or-treaters. If a few big kids don’t take the lion’s share, I will eat my monster mask..
*The American Society of Quality (ASQ) provides a detailing of the Pareto Chart here.
Wrong more often than right but never in doubt
Posted by mark in Consumer behavior, Uncategorized on October 23, 2011
The New York Times Magazine provided a great readout on the “Surety of Fools”* in today’s issue. The author, psychologist Daniel Kahneman, starts by providing a real-life example of WYSIATI – “What you see is all there is.” Read his story to learn more about this, but basically it means that many times what you observe does not provide any meaningful information for predicting future behavior. Cocky Wall Street brokers are hit very hard, especially the males, who “act on their useless ideas significantly more often than women.” Ouch!
Kahneman examined the illusion of skill in a group of investment advisors who competed for annual performance bonuses. He found zero correlation on year-to-year rankings, thus the firm was simply rewarding luck. What I find most interesting is his observation that even when confronted with irrefutable evidence of misplaced confidence in one’s own ability to prognosticate, most people just carry on with the same level of self-assurance.
The bottom line is that you shouldn’t swallow everything said by assertive and confident people who advise on highly-variable systems such as financial markets. Heeding what an experienced physician suggests is one thing, but going with the most boastful money manager is another.
“True intuitive expertise is learned from prolonged experience with good feedback on mistakes.”
— Daniel Kahneman
“It’s what you learn after you know it all that counts.”
— John Wooden
* Posted earlier under a different title here.
Trouble with math & stats? Blame it on dyscalculia.
Posted by mark in Basic stats & math, Uncategorized on October 16, 2011
According to this article in Journal of Child Neurology “dyscalculia is a specific learning disability affecting the normal acquisition of arithmetic skills, which may stem from a brain-based disorder. Are people born with this inability to do math in particular, but otherwise mentally capable – for example in reading and writing? Up until now it’s been difficult to measure. For example, my wife, who has taught preschool for several decades, observes that some of her children progress much more slowly than other. However, she sees no differential in math versus reading – these attributes seem to be completely correlated. The true picture may finally emerge now that Michèle M. M. Mazzocco et al published this paper on how Preschoolers’ Precision of the Approximate Number System Predicts Later School Mathematics Performance.
Certainly many great minds, particularly authors, abhor math and stats, even though they many not suffer from dyscalculia (only numerophobia). The renowned essayist Hillaire Belloc said*
Statistics are the triumph of the quantitative method, and the quantitative method is the victory of sterility and death.
I wonder how he liked balancing his checkbook.
Meanwhile, public figures such as television newscasters and politicians, who appear to be intelligent otherwise (debatable!) say the silliest things when it comes to math and stats. For example a U.S. governor, speaking on his state’s pension fund said that “when they were set up, life expectancy was only 58, so hardly anyone lived long enough to get any money.”** One finds this figure of 58, the life expectancy of men in 1930 when Social Security began, cited often by pundits discussing the problems of retirement funds. Of course this was the life expectancy at birth, in times when infant mortality remained a much higher levels than today. According to this fact sheet by the Social Security Administration (SSA), 6.7 million Americans were aged 65 or older in 1930. This number exhibits an alarming increase. The SSA also provides interesting statistics on Average Remaining Life Expectancy for Those Surviving to Age 65, which show surprisingly slow gains over the decades. I leave it to those of you who are not numerophobic (nor a sufferer of dyscalculia) to reconcile these seemingly contradictory statistical tables.
*From “On Statistics”, The Silence of the Sea, Glendalough Press, 2008 (originally published 1941).
**From “Real world Economics / Errors in economics coverage spread misunderstandings” by Edward Lotterman.
Clickers allow students to vote on which answer is right for math questions
Posted by mark in Basic stats & math on October 12, 2011
Yesterday I attended a fun webinar on Interactive Statistics Education by Dale Berger of Claremont Graduate University. Because I was multitasking (aka “continuous partial attention” — ha ha) at work while attending this webinar my report provides just the highlights. However, you can figure out for yourself what they (the stats dept at Claremont) have to offer by going to this web page offering WISE (Web Interface for Statistics Education) tutorials and applets.*
After the presentation a number of educators brainstormed on interactive stats. David Lane of Rice U (author of many stat applets) suggested the use of “interactive clickers” – see this short (< 2 min.) newscast, for example. I wonder what happen when a majority vote for the wrong answer? For some teachers it might be easiest just to declare the most popular response as the correct answer. That would be consistent with the way things seem to be going in politics nowadays. ; )
*Just for fun try the Investigating the Central Limit Theorem (CLT) applet (click the link from the page referenced above or simply click here). This would be a good applet to provide when illustrating CLT using dice (such as is done in this in-class exercise developed by two professors from De Anza College). In this case, pick the uniform Population and sample size 2. Then Draw a Sample repeatedly, and, finally, just Draw 100 samples. Repeat this exercise with sample size 5 a la the game of Yahtzee (a favorite in my youth). Notice how as n goes up the distribution of averages becomes more normal and narrower. That’s the power of averaging.