Why You Can't Simply Trust a Number
I was asked to answer a simple question: is Santa Fe truly the creative capital of America? There were rankings that said yes. I looked at the criteria and found opinions dressed up as measurement. No real science underneath. So I went and built something I could actually stand behind. What I learned in the process is what I wish every person who's ever forwarded a statistic had known first.
Last year Owen Lipstein and Maggie Fine, the creators of Santa Fe Magazine, came to me with a simple question: Is Santa Fe truly the Creative Capital of America? There were rankings that said yes. I looked at the criteria behind those rankings and found something I've seen a hundred times in forty years. Readers were paying close attention to the results. Nobody was paying attention to the method. The method turned out to be opinion. Informed opinion, maybe, but opinion dressed up as measurement, with no real science underneath it.
So I went and built something I could actually stand behind. I found three national studies conducted by different organizations using different methodologies measuring different things over a sixteen year span. All three reached the same conclusion. Santa Fe ranked first. When three independent measures point at the same answer from three different angles, that's not a ranking anymore. That's evidence. That's triangularization in practice and it's the only reason I put my name on the report.
Here's the thing about numbers that nobody tells you when you're learning to trust them. They feel like facts. "Inflation is down." "Crime is at a forty-year low." "This drug reduces risk by 50 percent." The decimal place alone signals that someone measured carefully and you can rely on the result. That feeling is the problem. A number is actually the end product of a long chain of decisions about what to measure, how, who to ask, when, what to count, and what to leave out. By the time it reaches you, all of those decisions are invisible. You just see the answer.
And right now, with politicians on every side of every issue citing numbers like they're reading scripture, and institutions publishing studies that contradict each other weekly, the stakes of getting this wrong have never been higher. I'm not saying don't trust institutions. I'm saying even the good ones deserve a second look before you forward their findings to your entire contact list.
Here are five ways numbers mislead us. Some of these are accidents. Some aren't.
The first is measuring the wrong thing and it's usually accidental. When a politician tells you crime is down, ask which crime. Violent crime? Property crime? Reported crime? In which cities, measured how, compared to which year? "Crime is down" is a number. Whether you're actually safer is a different question entirely, and the number doesn't answer it.
The second is that the number reflects how the question was asked which is rarely accidental. "Do you support protecting our communities from violent crime?" gets very different results than "Do you support increased policing?" Same underlying question. Survey design's a form of authorship, and the person who writes the questions shapes the answers. The ones who do it on purpose know exactly what they're doing.
The third is that numbers capture a moment, not a truth. The unemployment rate doesn't count people who stopped looking for work, people working part-time who need full-time, or the person who took a job three levels below their experience just to keep the lights on. Every politician who's ever said "unemployment is the lowest in decades" was technically right and practically incomplete. The snapshot's real. The story it tells often isn't.
The fourth is a bad baseline. "GDP grew 3 percent" sounds healthy until you notice the previous quarter shrank 4. "Drug X reduced hospitalizations by 50 percent" sounds extraordinary until you learn the base rate was so low that fifty percent of almost nothing is still almost nothing. A number without the right comparison isn't information. It's a press release.
The fifth is distortion in translation, and this one's gotten significantly worse. A finding about correlation becomes a headline about causation. A study with a wide margin of error becomes settled fact by the time it hits your feed. A peer-reviewed paper that took three years to produce gets summarized in thirty seconds by an AI tool that has no idea what it left out, and that summary gets shared ten thousand times before anyone reads the original. The Santa Fe rankings I mentioned at the top were a version of this. Real places. Real people. Opinions that looked like data.
Darrell Huff wrote about most of this in 1954 in How to Lie with Statistics. Jerry Muller covered the institutional damage in The Tyranny of Metrics in 2018. Smart people have been raising these flags for seventy years. The flags aren't working. We need a habit, not a warning.
I call mine the triangularization of a number. Researchers have used triangulation for decades to stress-test findings before drawing conclusions. I'm just arguing we should all be doing a version of it before we act on whatever number just landed in our lap or our feed. It's what I did in Santa Fe. It's what I wish I'd done earlier in my career more consistently than I did.
It's three things and you really do need all three.
The first is just asking where the number came from. Not analyzing it, not challenging it, just asking. Who collected it? Who was asked? When? What was left out? You don't need any expertise for this. You need thirty seconds and a mild willingness to look slightly annoying in a meeting. If nobody can answer, that's your answer right there.
The second is finding a different number that's measuring the same thing from another angle. One number is somebody's version of the truth. Two independent numbers pointing in the same direction start to feel like evidence. Three independent studies using different methodologies all reaching the same conclusion, like I found in Santa Fe, and you've got something worth putting your name on.
The third is the one people skip most often because it feels unscientific. Check the numbers against what you've actually seen with your own eyes. Not what you've read. What you've seen. Your experience of your neighborhood, your customers, your industry, your own body, is data too. It's the oldest kind we have and the one no platform can manufacture. A number tells you what happened. Your experience tells you whether it makes sense.
Here's what skipping all three looks like right now, today.
Last year, multiple studies were published claiming that remote work either dramatically reduced productivity or dramatically improved it. Both sets of studies were real. Both were peer-reviewed. Both were cited confidently by executives making policy decisions that affected millions of people's lives. The numbers pointed in opposite directions because the studies measured different things, used different populations, asked different questions, and defined productivity in ways that had almost nothing in common. The executives who triangularized made better decisions. The ones who grabbed the study that confirmed what they already wanted to do made the news for the wrong reasons.
Numbers can lie, measure the wrong thing, and be interpreted badly by people who should know better. I've done all three, sometimes in the same week. None of that makes them useless. It means they deserve about sixty seconds of scrutiny before you stake something on them.
Ask where it came from. Find a second one. Check both against what you know. It works better than any fact-checking app ever built, because it grows your own judgment instead of borrowing someone else's.
One more thought before I leave this alone. You might be thinking, fine, reported statistics are slippery, but what about behavioral data? What people actually do, not what they claim. Clicks, purchases, movement. Surely that's objective. Surely that's real.
It's real. And it still misleads. A retailer I worked with once built a confident story about why a product was outselling everything else in certain stores. Demand patterns, local preference, demographic fit. Turned out someone had shelved it at eye level by mistake. The behavior was captured perfectly. Our explanation for it was completely wrong. Behavioral data tells you what happened. It's got no idea why. And that gap is where the next trap lives.
That's next.
The Great Zandini Sees:
Ask where the number came from. Find a second one. Trust neither until your own experience weighs in. That's not doubt. That's judgment.
