When discussing historical investing data one of the more interesting points that inevitably arises is the question of just how applicable past results are to current events and future investing decisions. Some people reject all historical data as completely irrelevant because the future will never look exactly like the past, while others hold up mathematical evidence-based investing as some sort of scientific principle that one would be foolish to question. I imagine one might expect me to fall into the latter camp, but frankly I think it’s more complicated than that. Good data speaks for itself but unless you’re speaking the same language you can easily get the message completely wrong.
Rather than simply dismissing all doubts as the ramblings of analysis Luddites, I believe many concerns about backtesting are in fact very valid points raised by true students of financial history with a keen awareness of how even the best data can be highly misleading. For example, I often hear concerns about how a dataset since 1970 might distort the returns of portfolios including gold because of how the repeal of Bretton Woods instigated a one-time run on gold in the early 70’s. Similarly, lots of people mistrust portfolio data including high percentages of bonds because of the 20-year run of falling interest rates back to historical norms after the rate bubble burst in the early 80’s. Both are very fair points that deserve investor attention.
While we’re at it, I think it’s telling that the same people who distrust gold and bonds never seem to question US stock data. If we’re being fair about data biases, I’d argue that the same skepticism can be applied to the historic stock run in the 80’s and 90’s that defined the concept of irrational exuberance. The fact that greedy investors see this period as the norm to be expected rather than an unsustainable bubble to be downplayed says a lot about how straightforward numbers can mean completely different things to different people.
Whether it’s gold in the early 70’s, bonds after 1981, or stocks leading up to 2000, all three of these could rightly be classified as unsustainable asset bubbles by neutral investors only interested in the truth. And the effect of these bubbles on long-term averages couldn’t be more pronounced. Check out the average annual inflation-adjusted returns for each asset (from a US perspective), and pay special attention to just how much bubbles can skew the average return.
Gold
Since 1970: 6.0%
Bubble (1972-1974): 52.2%
Since 1970, excluding bubble: 3.0%
Bonds
Since 1970: 3.2%
Bubble (1982-2002): 7.0%
Since 1970, excluding bubble: 0.3%
Stocks
Since 1970: 7.6%
Bubble (1982-1999): 14.7%
Since 1970, excluding bubble: 3.5%
Remove the bubbles from the data, and a lot of what passes for common knowledge in investing circles suddenly gets turned on its head. Gold loses its luster, bonds spoil their reputation as a dependable source of income, and stocks sacrifice more than half of their touted average return. Stop for a moment and let that sink in. The stock surge lasted 18 years, which seems like a long time in isolation but is comparatively small in a dataset spanning 49. Way more often than not stocks were pretty ordinary, and one could argue their modern high-flying reputation is based largely on a single 18-year bubble that made a few Wall Street guys fabulously rich and forever skewed the average return but popped decades ago.
So when I look at data like that, I certainly can’t blame investors for being extremely skeptical of all backtesting. At some point it feels like a little child gleefully chasing one bubble after another with nothing ever lasting long enough to catch on your own investing timeframe. How on earth is an investor supposed to make wise decisions based on historical averages so skewed by unrepeatable events that they’re essentially meaningless?
For that reason it never surprises me when I get questions about excluding data from the calculations. Usually people want to crop out the early 70’s to eliminate what they see as an unfair advantage for gold. I choose not to do that in part because while removing the early 70’s may indeed properly handicap gold in many averages, the side effect is that it also removes some of the very worst years for stocks. Ignoring the upside for one asset could arguably be a conservative choice, but simultaneously ignoring the downside for another asset is a recipe for lots of poor decisions.
Surgically removing data only for specific years of individual assets is also a problem, as if one extends the exercise to every potential historical bubble the resulting returns database would look no more solid than a giant block of Swiss cheese. Is that really the “unbiased” historical record you’d like to use to choose a portfolio for your life savings? I doubt it. Most people intuitively understand that any analysis based on information that hollow will inevitably have just as many holes.
So while I don’t believe selectively ignoring data you don’t like is a good idea, I do concede that looking at long-term averages full of asset bubbles is a very real problem. And as a result, a lot of investing research today really is skewed by rosy data that may not apply to you. From my perspective, however, the core issue really isn’t with backtesting but with backtesters. What if I told you that you’ve just been looking at the data the wrong way?
Just like how describing the experience of riding a roller coaster by only looking at the long-term average speed is incredibly silly, depending so heavily on a single long-term average return to quantify portfolio performance is objectively shortsighted. But luckily that’s not the only way to study historical data, and not every method is equally susceptible to the skewing potential of asset bubbles. So for those worried about bubbles and interested in how they might avoid them when studying their own portfolios, let’s look at a few Portfolio Charts tools and talk about how they’re different.
Seeing the big picture
The quintessential chart on the site is also one of the most powerful when it comes to looking past asset bubbles. Rather than showing a single long-term average since 1970, the Heat Map displays more than a thousand compound averages spanning every start and end date we have data for. Let’s take a look at the Heat Map of a simple portfolio of 100% Gold.
Investors concerned about the data-warping effect of the repeal of Bretton Woods might normally believe that any analysis containing data back to 1970 must surely skew all results for portfolios including gold. But that’s not the case here. See those gray boxes at the very top? Those are the only data points affected by the initial gold spike ending in 1974. Any row starting in 1975 and below skips the questionable data entirely and the brutal sea of red speaks for itself. Rather than hiding the poor post-bubble returns behind a deceptive average, the Heat Map highlights them in all their glory.
Next, let’s study the Heat Map for stocks with and without the bubble that too many people take for granted. Isolating bubble years right in the middle of a dataset is a little trickier, but the data is all there if you know where to look.
The dark gray triangle represents the full set of investing timeframes with start and end dates located solely within the stock bubble, while the light gray field shows other datapoints that are influenced by the bubble. Sit back for a moment and evaluate the overall color balance of each image. How might looking past the stock bubble affect the overall desirability of stocks?
Not to be left out, here’s the same chart for a portfolio of 100% intermediate bonds.
When people talk about bonds, it’s common for someone to express concern about two situations — rising rates and perpetually low rates. Look in the right spots, and the Heat Map contains both. The triangular corner of dark red at the top-left represents the collection of start and end dates solely contained between 1970 and 1981 as interest rates rapidly spiked into double-digits. And the area at the bottom represents the more recent era of perpetually low rates. Both sets are completely untouched by the favorable blue bubble starting in 1982, and I personally like to evaluate the performance of any bond-heavy portfolio by looking at those two areas on the chart.
Freedom from start date bias
While the Heat Map is one of the best examples of a big-picture view for investing, it’s not the only tool on Portfolio Charts to offer that type of perspective. In fact, they all do! Each chart is densely packed with data from every possible start and end date, and the goal is to provide convenient ways to evaluate the best times to invest, worst times to invest, and everything in between all in a single image. I call that concept “start date independent” backtesting, and I believe it’s critical to understanding a very underrated idea in asset allocation — portfolio consistency.
As an example, the Portfolio Growth chart may look like a Monte Carlo simulation but it’s really not. Instead, each line represents the real-world compound portfolio growth that actual investors experienced in their bank accounts. The only difference is that each line starts on a different start date.
Clearly quoting the average return doesn’t tell the whole story! Each of those high peaks represent portfolio values during the stock bubble and those fabulous returns obviously didn’t last very long before plummeting back down to the pack. Personally I focus on the mass of lines closer to the bottom for much more realistic expectations.
To be fair, let’s also look at a chart for a portfolio heavy in gold to see how much the early outsized gold returns affected the spread of results.
The Permanent Portfolio contains 25% gold, and lots of people expect at least a few lines on that chart beginning in the early 70’s to have massive spikes just like the visual for stocks. But eliminate the 5 oldest (darkest) lines to ignore the early gold run before 1975, and the chart would look virtually the same. Not only are some analysis methods less susceptible to bubbles, but some portfolios are as well.
A fresh approach to historical data
I could go on for hours explaining each chart and how to interpret them, but the core concept is this:
Bubbles do skew static averages, but analysis methods that properly account for uncertainty and show all outcomes in context allow you to effectively learn from history rather than cherry-pick reality.
Granted, it’s a complicated thing to model and difficult to understand without serious coaching, so I don’t blame people for falling back on more familiar approaches. But it’s game-changing when it’s done right, and Portfolio Charts is my effort to help bring sophisticated ideas to everyday investors. Every chart and metric has this concept in mind, and while some are more adept than others at completely avoiding unrealistic hot streaks in the historical data, all are designed to help you plan conservatively and avoid the temptation to expect bubble-level returns with your own money. From safe withdrawal rates and drawdowns that look at worst-case scenarios to baseline returns that calculate realistic conservative returns expectations without trying to predict the future, my goal is to keep your head out of the clouds and your feet planted on solid empirical ground.
So while bubbles are very real problems in any backtest of historical data that seeks to help people make wise choices for the future, I believe analysis that highlights them in historical context and steers investors away from unrealistic expectations is actually far more effective than trying to selectively rewrite history. Circling back to the original dire averages with and without asset bubbles, removing the questionable source data may be tempting but visualizing where those returns stood in the grand scheme of things is way more empowering. One approach hides the historical reality, while the other shines the light on the information required to make a fully informed decision.
Keep your eyes on the goal rather than lost in the weeds, and that perspective will give you just the right mindset to succeed.
Did this help you see backtesting in a new light?