As an engineer by training, I’ve always been comfortable with the use of historical numbers in decision making. From empirically tested yield strengths that inform design specifications to segmented sales numbers that inspire new product ideas, making wise choices is all about putting good data to productive use. And as one can see by perusing the deep data focus of Portfolio Charts, I naturally apply that same mindset to investing.
So one of the things I admit I’ve had to adjust to over the years is something that I never really experienced in my engineering career. When it comes to finance, some people are just really dismissive of historical data.
It’s common in some circles for people to quickly shut down any discussion of past returns as the foolish discussion of unrepeatable events. Some reflexively point to specific timeframes that surely skewed the numbers in deceptive ways. Others explain how current events are very different than the past and proclaim that the old results are ancient history. And a surprising number have developed an outright distrust of math and statistics as the tools of unsophisticated investors and financial snake oil salesmen.
To be fair, these opinions are often born of legitimate examples of data abuse. Lots of people misuse statistics all the time or naively apply them out of context, and I totally understand how backtesting gets a bad name. But in my experience, the problem is not with backtesting but with backtesters. It’s super valuable when done correctly, but you have to know what you’re doing and approach it in the right way.
So to cut through the financial cynicism, I’d like to share three short stories about real-world situations where the use of historical data is widely accepted. By exploring how data is used well, perhaps we can learn a thing or two about how to also apply it effectively it in our personal portfolio decisions.
/// Lessons Learned ///
The Job Interview
Sitting alone at the unfamiliar conference table for a few moments while the interviewing manager made his way to the room, the butterflies in Joan’s stomach were palpable. She had been unhappy at her old job for ages, and finding a new place to work that appreciates her unique talents and offers new opportunities for career growth had been an ongoing challenge. So after months of applications and weeks of phone interviews, this was her time to shine.
Joan is organized and thorough — confident in getting results but perpetually anxious about the process of getting there. One might even argue that nagging voice in her head always pointing out room for improvement is one of her greatest strengths, but it’s not always easy to deal with in the moment. In her last few minutes to prepare, she nervously flipped through her portfolio of work and game planned how the conversation might go.
George walked in the room, introduced himself, and pulled up a seat across the table. Loud, confident, and disarmingly laid-back, he reclined the chair and put his hands behind his head.
“I’m glad you’re here, Joan. We can definitely use some help. Tell me about yourself.”
In this moment, the stress melted away and the clarity set in.
“Well, sir, I graduated from college ten years ago and have been working in the industry ever since. Here’s a copy of my resume. As you can see, I have a really good track record of delivering profitable results in similar projects that you do here. And my managers all speak very well of….”
George’s face flashed a thinly disguised look of someone who just caught you saying something stupid. “Let me stop you right there. None of that really matters.”
Caught off guard, Joan demurred. “I don’t understand.”
“I’ve been in this business a long time”, George replied. “And one thing I’ve learned is that nothing you can tell me about your past will guarantee that you will succeed in the future. Sometimes the people I like best in interviews don’t work out, and sometimes the people I reluctantly hire are the best performers. So your education and work history are irrelevant.”
Perplexed but weirdly intrigued, Joan relaxed. “Then how do you select who to hire?”
“Well, I like that you’re average. No real strengths and no major weaknesses. In this business journal I read, you’re basically the definition of ordinary that all other employees are measured against.”
George pulled out a magazine from his briefcase and excitedly flipped the pages. “Check this out. You even look just like the illustration! The hair, the clothes — you have to admit the resemblance is pretty uncanny. Since I have no idea how good of a fit you will be, it’s only logical to hire the average employee and take what the market provides.”
Joan: “So you don’t care about my strengths and weaknesses?”
George: “Nope! You’ll probably cherry-pick them, anyway.”
Joan: “And you’re interested because someone like me is the benchmark employee in an article you read.”
George: “Absolutely. It’s a reputable journal, so they know what they’re talking about.”
Joan: “Do you plan to put me on projects where you think I’ll succeed?”
George: “Why bother? Someone else might be better. It will all work out on average.”
Joan: “Do you even do performance reviews?”
George: “Chasing out-performance is a fools errand. But if you have one bad year, you’ll probably be replaced.”
Joan: “I think we’re done here.”
As George explains it today, Joan stormed out in a huff because she had an outsized opinion of her own talents. But as you know from talking to your fellow CEO down the street who later hired Joan, she is a tremendous worker who lives up to the billing and consistently meets or exceeds all expectations. Her references were spot on — she was and is exactly the right person for the job.
But that was then and this is now. George is once again casually reclining in the chair across from your desk, fiddling with your things while bragging about his management prowess. You’re in charge of his performance review.
What do you tell him?
With the benefit of editing out the expletives, I might say something like this which applies to hiring people and portfolios alike:
When it comes to evaluating new employees, it’s a commonly accepted fact of life that history matters. While it’s true that even the best resume will never be able to guarantee success in a future job, the reason managers look at your work history is that it is an important measure of your track record. What are your strengths and weaknesses? How did you adapt to unexpected challenges? What types of employers were thrilled with your results, and in which roles did you struggle to meet expectations? When seeking a good fit for the unique challenges of a new job, those things matter.
Choosing a portfolio is similar to interviewing a job candidate. They all have strengths and weaknesses. Times when they thrive and others when they struggle. So when I hear investors take the approach of brushing all of history aside as unrepeatable and talking up the superiority of blindly accepting the average, I can’t help but see a bit of George in that attitude. Even with no guarantees, there’s still no denying that history is a helpful guide to identifying and prioritizing the portfolio characteristics that matter to you most. While it’s always a good idea to evaluate any claims on a resume with a skeptical eye, be smart about it and think constructively. Don’t be a George.
The Turbulent Flight
I’ve flown a lot over the years. Too much, really. A big part of being a young mechanical engineer in my line of business involved working with Asian suppliers, and regular trips to China were a required part of the job description.
As you might imagine with many hundreds of cumulative hours of flight time over the Pacific ocean, not all of those flights were particularly comfortable. Even on the biggest planes, turbulence over the arctic circle is no joke. You eventually get used to a little bumpiness. But when the floor drops, the wings bounce, and families start screaming in fear, all you can do is to hold on tight and pray for the best.
In those lowest moments, one of the things that would often go through my mind was the words of my favorite engineering professor. On top of his primary gig teaching eager but exhausted undergraduates and sharing crazy old stories from his own working youth, he also happened to be an expert in aircraft accident reconstruction. That’s one of the things that made him such a good teacher in fracture mechanics. And for better or worse, those first-hand anecdotes really did a number on how I think about airplanes.
One fact that I still remember is something that really freaked me out as an engineering student. Did you know that all airplanes have cracks? And I’m not talking about microscopic manufacturing flaws, but long visible cracks in critical components like the wings. In fact, during regular maintenance the technicians are trained not to ground a plane for cracks but to measure them. Only a few inches long? No big deal. It’s only when they reach a critical propagation length that it’s time to replace the part.
Most people naturally think that sounds insane. Why on earth would they take such risks with components that are clearly flawed? Isn’t the responsible thing to replace all cracked parts as soon as you see them? How can the airline executives sleep at night?
Well, that’s where good engineering steps in.
Airplane parts with cracks are comfortably safe because they have been tested way more than you know. The bolts have been studied in a lab to understand their shear strength. The wings have been bent to failure. The cracks that worry sleepless engineering students have been empirically proven to follow repeatable fatigue rules. Maintenance personnel diligently track every update to make sure things are always performing within spec. The parts are designed to bend rather than break and to withstand even the most turbulent storms. And systems are built with multiple redundancies so that even if one thing fails, others are there to save the day.
With each larger bump comes more adrenaline. It’s just human nature. But that is quickly followed with images in my head of those countless tests and thoughtful planning that I know protect us all.
Smart portfolio design works like the interconnected systems on an airplane. Nobody thinks about them when the sky is blue and the air is smooth. But once the storms hit, the fear can be enough to make even the most experienced investor sweat and desperately want off that plane. Look closely enough, and you’re also guaranteed to find plenty of cracks in some important parts! But that’s no reason to panic. Because while absolute perfection is impossible to maintain, reliable safety is not.
Counter-intuitively, the key to that safety is not to change a component of your portfolio the moment it shows the first sign of cracking. That will only drive up costs with no extra benefit and leave you always chasing an unachievable sense of perfection. Instead, proper historical testing and robust design strategies can serve to protect you no matter what happens. Don’t be a stressed-out repair man constantly avoiding failure with patchwork fixes. Be a confident and battle-tested engineer using good data to design for success.
The Game Winner
Time out!
As an exhausted player tosses the basketball to the refs on the way to the bench, the coach looks up at the scoreboard. It’s a playoff elimination game. They’re down three points with two seconds left. After advancing the ball to half court for the inbounds play, there’s just enough time for one dribble and a shot to tie. Make it and you go to overtime. Miss and you go home.
This is the moment every player dreams of from the time they are kids shooting turnaround jumpers at the playground to the sound of an imaginary crowd. All the workouts, practice, sweat, and tears come to this. It’s the type of play that heroes are made of.
As the stadium roars in nervous anticipation, the players turn to the coach for the play call. Who gets the shot? While he does his best to exude calm confidence on the outside, he’s sweating bullets. There’s a huge amount of weight in that decision because the entire season is on the line. So he goes through his options. Luckily it seems like a no-brainer.
In a moment like this, everybody knows you put the ball in the hands of the point guard. They’re accustomed to the pressure and are on average some of the top shooters in the league. And his own point guard is one of the best in the business, a #1 pick and 3-time all-star who is huge, highly skilled, and seemingly born for this moment. Without hesitation, the coach draws up the play. It’s his shot to make or miss.
The ref hands the ball to the inbounder and blows the whistle.
Time slows to a crawl. This is it.
Pass, catch, turn, shoot.
Airball.
Season over.
Did I mention the point guard is Ben Simmons?
NBA fans reading this are probably wincing and yelling at the screen right now, as they immediately understand the issue. But even for those who don’t know what I’m talking about, perhaps a few images will bring you up to speed. Here is heat map that illustrates the location and make frequency of every shot Simmons took in the 2017-2018 season. Notice the problem?
To say Simmons is a poor 3-point shooter really doesn’t do it justice. To be bad at something you first have to try it, and Simmons almost never even attempts 3-pointers. Regardless of his immense talent in so many other areas and in spite of his point guard label, Ben Simmons is perhaps the last person in the league you’d want taking that shot. For a comparison, even though very few people would normally call the same play for a center, here’s the same chart for his frontcourt teammate Joel Embiid.
Would giving the ball to Embiid instead of Simmons guarantee a made shot? Of course not. But it would greatly increase your odds of success. Any fan with two eyes knows that, and any coach worth his salary does as well. But the thing is, you don’t have to be an avid basketball watcher to figure that out. You just need access to the right historical data.
The charts above are just one offshoot of the groundbreaking work of a man named Kirk Goldsberry. With a PhD in geography and a dissertation in real-time traffic maps, Goldsberry has a natural talent for data and visualizations. He also has a huge personal interest in the game of basketball, and eventually those two worlds collided. Goldsberry realized he could apply the same visualizations that he used to map traffic to also map shooting patterns on a basketball court, and he had the foresight to also understand just how valuable that information could be in evaluating a player’s offensive game. For example, here’s a chart he created illustrating the clear shot preferences of one of the greatest shooters of all time — Dirk Nowitzki.
Now I’m no Kirk Goldsberry. But I do have my own unique background in engineering, design, and finance that offers a similar opportunity for the cross-pollination of ideas. And you may notice a passing similarity between those basketball heat maps and my own portfolio versions.
Like using a heat map to understand Nowitzki’s most effective spots on the floor, a data-driven investor can use a heat map to understand a portfolio’s most effective economic environment. If one is worried about times of low real returns, look no further than the high-inflation 1970’s and turbulent 2000’s. For all if its accolades, maybe the total US stock market isn’t the right player to call upon in that situation. Something like the Permanent Portfolio could be the right asset allocation for the job.
Does that mean the Permanent Portfolio is the best choice for every situation? Of course not! But calling the right play isn’t always about handing the ball to the most famous star player. Your job as a coach is to understand the big picture and match the player to the unique needs of the team. And the right historical data, presented clearly and used judiciously, can be a huge help in making that choice with no regrets.
Three Stories, One Theme
Now none of this is rocket science. In many cases it’s just common sense. When talking about jobs, planes, or basketball, most people intuitively understand the value of historical data. So why do people have such a hard time applying the same ideas to investing? In my experience, I think the problem comes down to a matter of goals. To put it succinctly — when using past data to make current decisions, what are you optimizing for?
Investors almost invariably seek to maximize returns because everyone wants to get as rich as possible. I get it. But I’d argue that the obsession with raw returns over other important factors is actually the core source of the problem. In fact, the reason that backtesting is more widely accepted in other fields may simply be that the raw maximization of a single stat is more clearly unsustainable and not always even that desirable.
Hiring the fastest-working employee isn’t so great when they make mistakes and the quality suffers. Insisting on the fastest supersonic plane has undesirable tradeoffs in cost and safety. Maximizing points per game may sound like a no-brainer until you realize your team can’t guard an empty paper bag. There’s more to life than speed to the finish line.
And yes, maximizing your portfolio for future average returns not only has a lot of tradeoffs of its own but is also virtually impossible to sustain using past data. Backtesting often seems broken for portfolio management not because the data is bad, the methodology is flawed, or the timeframe is inapplicable. Think beyond your own myopic goals, and perhaps it’s because you’re using the data for a problem it is not well-suited to solve.
So if the best use of backtesting is not simply to maximize returns moving forward, what exactly is it good for? As an alternative, l think it’s helpful to think about a common theme in all of the stories we’ve talked about — consistency. While there are never any guarantees, the best choice is usually to hire someone who consistently gets the job done, to design a plane that consistently navigates the worst conditions even with a few cracks, and to hand the ball to a player who consistently makes big shots when they’re needed most. The currency of a happy society is trust. And that applies to portfolios as well.
The Heat Map is just one example of a creative visualization of portfolio data, and Portfolio Charts offers lots of ways to study the historical numbers. But they all share one thing in common. Just like how a Goldsberry heat map tracks not just the last shot but all of them, the charts on the site buck normal convention by showing not just the most recent investing timeframe but all of them simultaneously. And for that reason, they’re particularly good at evaluating portfolio consistency.
So yes, people abuse portfolio backtesting all the time. But the problem is not the data but the individual. Historical data is used effectively all over the place in everyday life, and by studying those situations we can learn an important lesson that too many analytical investors lose sight of in their quest to perpetually beat the markets.
Are you frustrated that past data can’t predict the future? Prioritize consistency over specific predictions, and a lot of those problems melt away. Reframe your mindset, and you’ll finally be ready to take the next step towards using good data to invest with confidence.
Help me share more stories of investing success.