The Statistics That Come Out of Nowhere
The annual cost of global corruption probably isn’t $2.6 trillion. Bareheaded people likely won’t lose 80 percent of their body heat.
This winter, the university where one of us works sent out an email urging employees to wear a hat on particularly cold days because “most body heat is lost through the top of the head.” Many people we know have childhood memories of a specific figure—perhaps 50 percent or, by some accounts, 80 percent of the heat you lose is through your head. But neither figure is scientific: One is flawed, and the other is patently wrong. A 2004 New York Times column debunking the claim traced its origin to a U.S. military study from the 1950s in which people dressed in neck-high Arctic-survival suits were sent out into the cold. Participants lost about half of their heat through the only part of their body that was exposed to the elements. Exaggeration by generations of parents got us up to 80 percent. (According to a hypothermia expert cited by the Times, a more accurate figure is 10 percent.)
This rather trivial piece of medical folklore is an example of a more serious problem: Through endless repetition, numbers of dubious origin take on the veneer of scientific fact, in many cases in the context of vital public-policy debates. Unreliable numbers are always just an internet search away, and serious people and institutions depend on and repeat seemingly precise quantitative measurements that turn out to have no reliable support.
For years, the three of us have been tracking the origins of numbers that claim to measure illicit activities, which are by their nature hard to measure. You may have heard that more than $1 trillion in bribes is paid each year, or that corruption costs the world economy $2.6 trillion annually. The $1 trillion figure comes from a set of extrapolations from a handful of surveys conducted by the World Bank and the World Economic Forum in the early 2000s in a variety of countries. These calculations produced a wide range of estimated annual-bribe payments—from about $600 billion to $1.7 trillion. The $1 trillion figure is roughly the midpoint of that range. The problem with taking just the average is that doing so strips the data of the enormous uncertainty in already-questionable estimates. And yet, the figure keeps resurfacing—the World Bank’s website, for example, cited it as recently as 2020—as if the annual amount of bribery were constant.
The $2.6 trillion corruption estimate, meanwhile, traces back to a one-sentence bullet point in an advocacy brief from a group of respected organizations, including the World Economic Forum and Transparency International. The brief cited no source, and, as far as we can tell, the number was likely based on a careless misreading of an earlier study. But the figure was later cited by the heads of prominent international bodies, including the United Nations and the Organization for Economic Co-operation and Development.
These numbers are what we might call “decorative statistics.” Their purpose is not to convey an actual amount of money but to sound big and impressive. That doesn’t keep them from being added, subtracted, divided, or multiplied to yield other decorative statistics. Some organizations and news outlets combine the bribery and corruption estimates and declare that the planet experiences $3.6 trillion in graft year after year.
Unfortunately, the very pervasiveness of meaningless numbers undercuts the credibility of statistics more generally, even when the numbers never make it into anyone’s financial calculations.
We recently came across a study by two respected researchers that put the scale of illegal bets placed each year at $1.7 trillion. Where did such a precise figure for hard-to-measure, clandestine activities come from? Their paper cited a document published by the UN Office on Drugs and Crime. That document, however, actually gives a range of $340 billion to $1.7 trillion, cites no source, and rightly warns about the inherent difficulty of measuring the underground economy. But the $1.7 trillion figure has taken on a life of its own.
The plague of decorative statistics goes back decades. In one of a series of 1991 speeches about America’s supposed decline in global economic competitiveness, Vice President Dan Quayle remarked that the United States had too much litigation and too many lawyers, as evidenced by the fact that 70 percent of the world’s lawyers were American—a number that was then repeated by authority figures across the political spectrum. But as the law professor Marc Galanter calculated at the time, America’s share of lawyers was probably more like 25 to 35 percent, roughly in line with the U.S. share of global GDP in the early 1990s. Quayle also claimed that lawsuits (and the threat of lawsuits) cost Americans $300 billion a year. That equally alarming estimate—also widely quoted in discussions of tort reform—came from a Forbes magazine article that quoted a back-of-the-envelope calculation by a corporate-defense lawyer who used as his foundational cost estimate an offhand, unsourced assertion that a CEO had made at a roundtable discussion. The news that the civil-justice system costs the country $300 billion annually was, in Galanter’s memorable phrasing, “news from nowhere.”
Our suspicion is that junk statistics have only proliferated in recent years. The internet should make debunking them easier; the sort of painstaking detective work that Galanter went through to trace the origin of Quayle’s figures can now be done quickly. However, even putting aside the obvious problem of deliberate falsehoods spread online, the web is an endless bazaar of unreliable source materials. When numbers are so readily available, they’re also easy to combine in various permutations to come up with attention-grabbing new statistics.
Audiences should be skeptical of numbers that get thrown around without sufficient explanation of their provenance. But the greater responsibility for addressing this problem lies with journalists, scholars, government departments, reputable civil-society groups, international organizations, and everyone else whom citizens and policy makers rely upon for basic facts. The press has a particular responsibility—unlike, say, politicians or advocates, journalists operate under a code of ethics that demands the accurate reporting of factual statements.
For writers and speakers who might feel obliged to adorn their argument with numbers, we have some advice: First, before quoting a statistic, work back to the original source rather than citing a downstream source that references something else (which may reference something else, which references something else). Phrases such as studies have shown that or it has been estimated that should be red flags. Citing a source for a statistic isn’t good enough; you need to track down the original source.
Also, beware of what we might call “statistics laundering”: A less-than-trustworthy source makes a questionable quantitative claim that a more respectable person or organization, either opportunistically or carelessly, then recycles in some official speech or document. Later on, that secondary source is cited as authority for the statistic, which gives the number a veneer of reliability. The $300 billion cost of lawsuits is a case in point: An offhand remark by a corporate executive becomes the basis of a lawyer’s cost calculation, which is then cited in a magazine column and then picked up and repeated by the vice president of the United States—and then by many others.
Third, don’t put too much faith in a prestigious name. An estimate by a Harvard professor is not a “Harvard estimate.” A statistic that appears in an unpublished World Bank working paper, perhaps authored by an outside consultant, is not “the World Bank’s calculation.” Even some statistics officially issued by well-known institutions turn out to be groundless, but suggesting that an organization has endorsed statistical claims that it has not carefully vetted increases the likelihood that bad information will spread.
Finally, recognize that, in the game of broken telephone that occurs when a quantitative estimate migrates into public discourse, numbers get rounded up, and then rounded up again. Estimates that originally take the form of a broad range get turned into a single number; estimates that concern a fairly narrow and specific domain are treated as if they apply much more broadly. Important caveats drop away. Sometimes researchers make a good-faith effort to measure a difficult-to-quantify phenomenon only to have a distorted version of their findings later presented as truth.
You can make the case for what matters without using made-up numbers to imply a certainty you don’t really have. You don’t need to pretend that bareheaded people will lose 50 or 80 percent of their body heat in winter. Just wear a hat.