The Philosopher's Cornered
Dealing with Randomness
by Dan Wolaver
was amazed when the temperature reached 68° on January 6, 2007.
And then the temperature went down to –6° on March 6th! It felt
like something weird was happening, and I was a little concerned.
In fact, extremes like that are not unusual. It's just that our
mental model for complicated things (like the weather) is usually too
simple. Just taking the Earth's tilt into account we expect the
high temperature for the day to be around 85° on August 1st. It
gradually descends to about 25° on February 1st, and then begins its
rise toward 85° the next August 1st. Taking the Earth's rotation
into account, the low for the day is about 20° below the high.
This simple model is shown in the right-hand graph here.
Of course, such a simple model doesn't take into account the thousand other factors that affect the weather, and these factors are more random than the Earth's tilt or rotation. Occasionally a number of these factors happen to come together to produce a "perfect storm," and we get extremes in the weather—in the temperature, in the precipitation, in the wind. This can be seen in the plot of the actual temperatures for 2007 in the left-hand graph here. So the left-hand plot is really more normal, and it's the right-hand plot that would be weird if it happened.
But we're uncomfortable with randomness. We don't understand it, and we can't plan for it. So we turn to the weather experts that are able to take many factors into account and give us a reasonable prediction ten days out. We study centuries of precipitation records and build our house high enough to survive the 100-year flood level (the "perfect storms"). If a 500-year flood level happens in our lifetime, we rely on insurance.
Another complex system is the economy. The Dow-Jones Industrial Average acts erratically, but over the long term it increases about 9% annually, doubling every eight years. So a simple model based on investment return and inflation would give the red plot here. But "perfect storms" can create the bubble of the late 1990s and the great recession of the late 2000s. It's risky to try to take advantage of the random ups and downs (the blue curve) because they're unpredictable—there are too many factors in the model. But professional investment advisors have better models than most of us. So how do we deal with the randomness? Invest for the long term and get professional advice. Some resort to astrology or "lucky numbers"—desperate acts in the face of the unknown!
The are some simple measures of randomness that can help us understand it. The number of customers buying a mattress from a merchant depends on many factors—the whim of thousands of potential customers. If the merchant gets an average of 64 sales a week, he can expect the weekly sales to vary by about ± 8, where the standard deviation 8 is the square root of the average 64. Sales are within the standard deviation 68% of the time, so four weeks in six the sales will be within 64 ± 8, one week in six will have more than 64 + 8 = 72 sales, and one week in six will have less than 64 – 8 = 56 sales. But occasionally (one week in fifty) the sales will exceed 80 (two standard deviations above 64). The tendency is to look for a special cause for the unusually number, but it's just normal randomness.
We get an understanding of randomness if we play around with it and and observe what's typical. Suppose we shake up 400 white marbles and 400 black marbles in a bag and pour them into a tray. It will look something like the pattern here. Our first reaction is that something has caused white and black clumps to form—the marbles aren't mixed well enough. But random distribution is clumpy—not even. Betting on the winning horse three times in a row is just randomness, not the result of "being on a roll."
In the midst of randomness we sometimes long for a simplicity that is predictable and safe. Rather than trying to insulate ourselves from the complex world with its surprises, we should understand it and embrace the excitement of the adventure.
And that's my philosophy.