Trust the Math?
In 2021, The New York Times ran a story titled "This Book Is Not About Baseball. But Baseball Teams Swear by It." The book was Thinking, Fast and Slow by Daniel Kahneman. In his book, Kahneman uses psychological experiments to demonstrate how our quick, intuitive judgments are often prone to biases and errors. This insight has become deeply embedded in MLB front office culture.
In many respects, that development has been a productive one for the industry. Front offices have adopted analytical frameworks to reduce the influence of human bias and make more objective decisions throughout their organizations.
However, over time, within the industry I have noticed a subtle shift toward treating statistical model outputs as more certain than they actually are. In doing so, front offices have developed a different problem: mistaking data-backed decisions for good decisions.
To be crystal clear, this is not an argument against analytics.
Data-driven decision making is a crucial component in a modern Major League operation. Organizations that fail to leverage data in scouting, player development, roster construction, and in-game strategy are certain to fall behind.
Having said that, treating outputs from a statistical model as the definitive answer rather than useful estimates is the opposite of intellectual rigor.
The danger I see at times is that analytical recommendations carry a level of perceived authority that exceeds their actual certainty. Models produce best estimates based on historical comparisons, not absolute truth. Yet within organizations this distinction can become blurred.
Once that happens, model-backed recommendations are safer to defend professionally than those rooted in judgment or scouting opinion, because the process itself is viewed as rigorous and the industry has largely accepted it as the modern way to make decisions. The result is a subtle form of groupthink in which challenging the model requires more personal and professional risk than following it.
The question is no longer whether a model is right, but if anyone is willing to stake their reputation within the front office on disagreeing with it. This concern does not appear unique to my own experience in the industry.
Theo Epstein, arguably the most successful executive in the analytics era (and maybe ever), recently cautioned on the Dirt From the Dugout podcast that the industry’s current state is “a little bit out of balance” when it comes to its use of analytics:
“Data is important and analytics are important, technology is important, and you can use those tools to help you make good predictions about how players are going to perform in the future. But if it's done at the exclusion of the human element and…getting to know other people, understanding what makes them tick, developing a connection with them, understanding them better that way, and putting them in a position to succeed, then it's not worth it.”
Theo's observation gets at the core issue. Models are enormously valuable, but their purpose was never to have the only say.
The best executives I’ve seen use models as inputs, not substitutes for holistic decision making and human evaluation. They seek out disagreement and welcome opinions that don’t always align with the math. Great organizations also consider factors such as makeup and whether a player’s skill set fits the organization’s scouting philosophy, player development strengths, and overall strategy.
I do not believe that the takeaway of Thinking, Fast and Slow should be that numbers are always right. The lesson was that we as humans are often more confident in our judgments than the evidence justifies.
In MLB front offices, it seems we have become appropriately skeptical of our intuition. In some corners of the industry, however, I fear we have mistaken statistical analysis for certainty and have replaced one source of overconfidence with another.