Praise for

THE FLAW OF AVERAGES

"Statistical uncertainties are pervasive in decisions we make every day in business, government, and our personal lives. Sam Savage's lively and engaging book gives any interested reader the insight and the tools to deal effectively with those uncertainties. I highly recommend The Flaw of Averages."

--William J. Perry, former U.S. Secretary of Defense

"Enterprise analysis under uncertainty has long been an academic ideal. . . . In this profound and entertaining book, Professor Savage shows how to make all this practical, practicable, and comprehensible . . . the Distribution String . . . represents a major breakthrough in the communication of risk and uncertainty."

--Harry Markowitz, Nobel Laureate in Economics

"This is a book written for laymen with enough interesting insights to engage even the most scholarly professional."

--Douglas Hubbard, author of How to Measure Anything

"Sam Savage is the Edward Tufte of risk."

--Matthew Raphaelson, Executive Vice President, Wells Fargo

A GROUNDBREAKING MUST-READ FOR ANYONE WHO MAKES BUSINESS DECISIONS IN THE FACE OF UNCERTAINTY

In The Flaw of Averages, Sam Savage--known for his creative exposition of difficult subjects--describes common avoidable mistakes in assessing risk in the face of uncertainty. He explains why plans based on average assumptions are wrong, on average, in areas as diverse as finance, healthcare, accounting, the war on terror, and climate change. Savage refers to anachronistic statistical jargon as Red Words, which he defines as things that may not be uttered in a singles bar. Instead, he presents complex concepts in plain English (Green Words), backed up by interactive simulations at www.FlawofAverages.com, which connect the seat of the intellect to the seat of the pants.

Savage also presents the emerging field of Probability Management aimed at curing the Flaw of Averages through more transparent communication of uncertainty and risk. Savage argues that this is a problem that must be solved if we are to improve the stability of our economy, and that we cannot repeat the recent mistakes of applying "steam era" statistics to "information age" risks.

Foreword xv

Preface xvii

Acknowledgments xxi

Introduction Connecting the Seat of the Intellect to the Seat of the Pants 1
You cannot learn to ride a bicycle from a book,and I claim the same is true for coping with uncertainty. Paradoxically, this book attempts to dowhat it claims is impossible.

Foundations

Part 1 The Big Picture 9

Chapter 1 The Flaw of Averages 11
In planning for the future, uncertain outcomes are often replaced with single, so-called average numbers. This leads to a class of systematic errors that I call the Flaw of Averages, which explains among other things why forecasts are always wrong.

Chapter 2 The Fall of the Algebraic Curtain and Rise of the Flaw of Averages 22
The electronic spreadsheet brought the power of business modeling to tens of millions. In so doing, it also paved the way for an epidemic of the Flaw of Averages.

Chapter 3 Mitigating the Flaw of Averages 26
New technologies are illuminating uncertainty much as the lightbulb illuminates darkness. Probability Management is a scientific approach to harnessing these developments to cure the Flaw of Averages.

Chapter 4 The Wright Brothers Versus the Wrong Brothers 34
The success of the Wright Brothers' airplane was the result of carefully constructed models that they tested in their wind tunnel. Analogous models ca help us manage uncertainty and risk, but as we saw in the financial crash of 2008, models can also be used to obfuscate.

Chapter 5 The Most Important Instrument in the Cockpit 40
The proper use of models, like the instruments in an airplane, is not obvious.

Part 2 Five Basic Mindles for Uncertainty 45

Chapter 6 Mindles are to Minds What Handles are to Hands 49
Just as industrial designers develop handles to help us grasp the power of physics with our hands, informational designers develop Mindles (first syllable rhymes with "mind") to help us grasp the power of information with our minds. Section 2will provide some important Mindles for grasping uncertainty.

Chapter 7 Mindle 1: Uncertainty Versus Risk 52
These two concepts are often used interchangeably but they shouldn't be. Uncertainty is an objective feature of the universe, whereas risk is in the eye of the beholder.

Chapter 8 Mindle 2: An Uncertain Number Is a Shape 55
Even graduates of statistics courses have a hard time visualizing uncertainty. A shape in the form of a simple bar graph, called the histogram, does the trick. Try running a simulation in your head or better yet, on your web browser at FlawOfAverages.com.

Chapter 9 Mindle 3: Combinations of Uncertain Numbers 67
When uncertain numbers are added or averaged, the chance of extreme events goes down. I cover a case study in the film industry.

Chapter 10 I Come to Bury Sigma, Not to Praise it 78
Just as the height and weight of a criminal suspect have been superseded by surveillance videos and DNA samples, sigma is pushing obsolescence.

Chapter 11 Mindle 4: Terri Dial and the Drunk in the Road 83
A banking executive discovers the Strong Form of the Flaw of Averages: Average inputs don't always result in average outputs. Designing an incentive plan around your average employee is systematically erroneous.

Chapter 12 Who Was Jensen and Why Wasn't He Equal? 91
The Nuts and Bolts of the Strong Form of the Flaw of Averages
This chapter shows how to identify the Flaw of Averages before it occurs by understanding your options and restrictions.

Chapter 13 Mindle 5: Interrelated Uncertainties 98
Interrelated uncertainties are at the heart of modern portfolio theory. They are best understood in terms of scatter plots.

Part 3 Decisions and Information 109

Chapter 14 Decision Trees 111
Decision trees are a powerful Mindle for thinking through decisions in the face of uncertainty.

Chapter 15 The Value of Information Because There Isn't Anything Else 118
The flip side of decision trees. Information is the complement of uncertainty. What is it worth to find things out?

Part 4 The Seven Deadly Sins of Averaging 127

Chapter 16 The Seven Deadly Sins of Averaging 129
So here are the Seven Deadly Sins, all eleven of them. And the twelfth deadly sin is believing we won't discover even more tomorrow.

Chapter 17 The Flaw of Extremes 133
Viewing uncertainties solely in terms of nonaverage outcomes also leads to devastatingly wrong answers and policy decisions.

Chapter 18 Simpson's Paradox 139
Imagine a weight loss treatment that makes people lose weight on average, unless they are either male or female, in which case it makes them gain weight on average.

Chapter 19 The Scholtes Revenue Fallacy 142
Suppose you have various product lines with different unit sales. The average unit sales times the average profit per unit might be positive while your average profit might be negative.

Chapter 20 Taking Credit for Chance Occurrences 147
If you execute a marketing campaign and make a bunch of sales, how do you know the increase wasn't just by chance?

Applications

Part 5 The Flaw of Averages in Finance 155

Chapter 21 Your Retirement Portfolio 157
If your retirement fund will last you 20 years given average returns, then you are as likely as not to suffer financial ruin before you get there.

Chapter 22 The Birth of Portfolio Theory: The Age of Covariance 163
Harry Markowitz started a revolution in finance in the early 1950s by explicitly recognizing risk/return trade-offs.

Chapter 23 When Harry Met Bill(y) 169
Bill Sharpe extended the work of Markowitz and brought it into widespread practice.

Chapter 24 Mindles for the Financial Planning Client 175
How the pros explain this stuff to their clients.

Chapter 25 Options: Profiting from Uncertainty 181
Options allow us to exploit uncertainty through an understanding of the Strong Form of the Flaw of Averages.

Chapter 26 When Fischer and Myron Met Bob: Option Theory 192
The theory of three economists led to the trillion-dollar derivatives industry.

Chapter 27 Prices, Probabilities, and Predictions 200
The new phenomenon of prediction markets is changing the way we perceive and report uncertain events, such as political races.

Part 6 Real Finance 213

Chapter 28 Holistic Versus Hole-istic 215
When people invest in portfolios of oil exploration sites, they often use the hole-istic approach. That is, they rank the places to drill hole by hole, then start at the top and go down the list until they run out of money. This ignores the holistic effects of portfolios.

Chapter 29 Real Portfolios at Shell 222
For several years, Shell has been using Probability Management to manage its portfolios of petroleum exploration sites in a more holistic manner.

Chapter 30 Real Options 228
An example of a real option is a gas well in which you have the choice of whether or not to pump depending on the price of gas.

Chapter 31 Some Gratuitous Inflammatory Remarks on the Accounting Industry 236
You can't rely on accountants to detect risks because generally accepted accounting principles are built on the Flaw of Averages.

Part 7 The Flaw of Averages in Supply Chains 245

Chapter 32 The DNA of Supply Chains 247
The inventory problem introduced in Chapter 1 is at the heart of all supply chains.

Chapter 33 A Supply Chain of DNA 254
When stocking out is not an option.

Chapter 34 Cawlfield's Principle 257
A manager at Olin creates a simulation to get two divisions of his organization to work as a team and discovers a general principle in the process.

Part 8 The Flaw of Averages and Some Hot Button Issues 263

Chapter 35 The Statistical Research Group of World War II 265
The exciting environment in which my father became a statistician.

Chapter 36 Probability and the War on Terror 272
Two inescapable statistical trademarks of the war on terror are the problem of false positives and implications of Markov chains.

Chapter 37 The Flaw of Averages and Climate Change 289
The earth's average temperature may actually be going down, not up, but you won't be happy when you find out why. The Flaw of Averages permeates this issue.

Chapter 38 The Flaw of Averages in Health Care 299
Treating the average patient is not healthy.

Chapter 39 Sex and the Central Limit Theorem 307
Women have a diversified portfolio of two X chromosomes, whereas men have only one. Apparently it makes a difference.

Probability Management

Part 9 Toward a Cure for the Flaw of Averages 317

Chapter 40 The End of Statistics as You Were Taught It 319
The ninetheenth-century statisticians confirmed their theories by simulating uncertainty with dice, cards, and numbered balls. Today, computerized dice, cards, and balls are bypassing the very theories they were trying to confirm.

Chapter 41 Visualization 324
Visual statistics provides a window into distributions. You need to see it to appreciate it.

Chapter 42 Interactive Simulation: A New Lightbulb 328
Imagine simulating 100,000 rolls of a die before your finger leaves the Enter key. A new technology does for probability distributions what the spreadsheet did for numbers.

Chapter 43 Scenario Libraries: The Power Grid 332
New data structures allow the results of simulations to be added together like numbers, providing a more practical approach to enterprisewide risk models.

Chapter 44 The Fundamental Identity of SLURP Algebra 341
This looks like math. Feel free to skip it.

Chapter 45 Putting It into Practice 343
The technology surrounding Probability Management is improving fast, and recent breakthroughs promises to make it more accessible than ever.

Chapter 46 The CPO: Managing Probability Management 354
The CPO must strike the correct balance between transparency of presentation, data collection, and statistical rigor.

Chapter 47 A Posthumous Visit by My Father 364
Some comments from the hereafter.

Red Word Glossary 367

Notes 371

About the Author 382

Index 383

SAM L. SAVAGE is a Consulting Professor of Management Science and Engineering at Stanford University, and a Fellow of the Judge Business School at the University of Cambridge.