The Limits of Official Statistics
By Jena Barchas-Lichtenstein (Knology)
she/her/they/them
This piece was submitted on November 11, 2020
Official statistics are “data and diverse information products [made] available to keep policy-makers, various user groups, and the general public apprised of the current economic and social situation” (Gal and Ograjenšek 2017, 86). They include data from censuses and government social surveys, as well as a great deal of data collected for administrative or legal reasons, such as unemployment or marriage statistics (Bulmer 1980). Put differently, official statistics are backed by both the authority and resources of a government.
Statisticians Iddo Gal and Irena Ograjenšek (2017) have argued that nonspecialist adults need not just a general statistical literacy but a specific literacy with official statistics. And the last eight months—since the WHO declared COVID-19 a global pandemic—have only served to illustrate this point.
I am co-PI of a four-year National Science Foundation grant (a collaboration between Knology and PBS NewsHour) looking at how statistical information is presented in the news media. We started the grant in July 2019. And by February 2020, when we were collecting a landscape set of news articles, COVID-19 was everywhere. It felt like every quantitative story was about COVID-19—number, time, and location of cases; infection fatality rate; case fatality rate; you name it. Numbers upon numbers, graphs upon graphs. I didn’t come into this project planning to look at COVID-19, but it has occupied a lot of my time.
My understanding of COVID-19 numbers is also informed by my own practices and experiences. As a New Jersey resident, I followed the first US peak through my governor’s daily numbers—and through the traffic outside my new home office giving way to ambulances and birdsong.
In a piece that connects the COVID-19 pandemic and the challenges to public numeracy, sociologist Joel Best (2020) observes that behind every measurement and every comparison is a series of human decisions. Glossing over those human decisions takes away the ability to critically assess what is happening. [1]
Some of these decisions are fairly abstract and harmless, at least at first glance. Statisticians debate the best way to identify outliers, which are data points that are exceptional in some way with regard to the rest of the data.
But even the definitions rely on human decisions. As Best reminds us, calculating the crime rate or the unemployment rate requires defining what counts as a crime and who counts as unemployed. And even if everyone agreed on those definitions, the “true” number of crimes or unemployed people would be unknowable. Instead, quantification relies on proxies, measurable figures that bear some relationship to the phenomena to be quantified. The crime rate and unemployment rate are based, respectively, on the number of crimes known to law enforcement agencies (Best 2020) and a complicated monthly interview methodology (BLS, n.d.). [2]
Both of these sentences are true as of November 11, 2020:
There have been about ten million confirmed cases of COVID-19 in the United States. [3]
We don’t know how many people have—or have had—COVID-19 in the United States.
“Confirmed cases” of COVID-19 is actually a reference to positive test results. But not everyone who is sick has a positive test result, and not everyone who has a positive test result is sick.
Not everyone has equal access to testing. Among other reasons, different states and cities have different criteria for who qualifies for a test (Conner 2020), and there are racialized disparities in the location of testing centers (Vann et al. 2020). [4] There are policies in place that reward counties for low numbers of tests (CMS 2020), which incentivizes restrictions.
Not everyone trusts the tests—and they have good reason. The US government has a long history of perpetuating medical violence against racialized bodies and communities through such acts as deliberately infecting people (Rodríguez and García 2013), withholding treatment (CDC, n.d.), and forced sterilization (Ko 2016). And public health surveillance is regularly used to stigmatize racialized communities (Lieberman 2020; Sangaramoorthy 2012).
There are pressures on people to get tested—or not to get tested—that have nothing to do with exposure and risk. Currently, the number of tests is being driven up by repeated testing of individuals who are not experiencing symptoms, like professional basketball players (Wojnarowski 2020) and college students (Hubler 2020). At the same time, people who don’t have paid leave from their jobs (Desliver 2020)—which includes millions of health care workers (Long and Rae 2020), who are relatively likely to be exposed (Lu 2020)—might avoid getting tested because they can’t afford to miss work.
The numbers also have different meanings over time. There have been variable delays in getting the results (Volz and Galewitz 2020), and today’s tests might reflect people who were infected two weeks earlier. Relying on the numbers might mean identifying outbreaks too late. And that’s before considering all the shifting dynamics mentioned above, which mean that the relationship between positive tests and cases is not constant.
There is the complicated question of who eventually displays symptoms and who does not. But even if everyone who was infected became symptomatic, it would still be true that not everyone with a positive test is sick: no medical test is perfectly sensitive or perfectly specific. A sensitive diagnostic does not say that infected people are not infected; a specific diagnostic test does not give positive results to people who are not infected (Street and Kelly 2020). Yes, there is reason to believe that current COVID-19 tests are fairly specific (Schmerling 2020), but no test is perfect.
All of those are questions to think about any time you see a number. How is that number generated? What does that number actually mean? What’s not going into that number? What don’t you know? And what is unknowable?
A confession: I am what psychologist Ellen Peters calls a “statistics stalker.” But I had a nightmare last week that I was alone on a roller-coaster, stomach always just behind the rest of my body. The first two downhill lurches were over quickly, but then came a climb I could not see the top of.
In this dream, I was in two places at once: inside my body and far away, watching the car as it hurtled up the hill. From a distance, it was clear that the tracks were being built just ahead of me.
Gal and Ograjenšek (2017) list six critical types of knowledge that adults need to make sense of official numbers.
First, adults need to understand the system of official statistics and the principles that underlie it. They need to understand that there are shared principles and methods, and that official data collection procedures are frequently designed for comparability across times and places.
The US public health system flouts nearly all these principles. It is decentralized and fragmented at every level—insurance and access to care, surveillance principles and practices, reporting pipelines. (And in fact, the patchwork nature of the US health-care system is largely by design; see Yong 2020.)
By the time “the numbers” reach us, filtered through news media and government dashboards, they appear uniform. But the truth is that case counts are tallied and reported differently in each state. Writing about HIV/AIDS, Thurka Sangaramoorthy (2012, 294) observes that meaningful variation is made invisible in the name of standardization:
Data collection technologies do not standardize data as intended. . . . These practices and technologies of standardization enable seemingly fluid and self-evident translations of complex structural and social factors into comparable and discrete categories, while simultaneously claiming that these standardized data variables represent “real” and “factual” maps of the epidemic.
Official statistics are beset by all the same problems as other types of numbers. Governments are not immune from sampling error, from nonresponse error, from social-desirability bias. But governments have a massive infrastructure available to them, which makes their numbers—at least in theory—the gold standard.
Second, according to Gal and Ograjenšek (2017), adults need to understand five general characteristics of statistics about society. These statistics are typically multivariate rather than interesting in isolation, they are aggregated rather than reported individually or in raw form, they are collected periodically and thus typically designed for comparison, they are typically presented to the public in texts that take the form of government reports or journalism, and they may be presented in a wide range of complicated visualizations.
Third, the social indicators the public is used to seeing are rarely raw variables. They may be aggregates, ratios, or weighted according to complex formulas.
Fourth, citizens need to understand—at least in the abstract—the techniques and transformations that official statistics undergo. They need to understand the purpose of rolling averages and other data-smoothing techniques, as well as weighting, stratified sampling, and other ways of accounting for demographic variation. Gal and Ograjenšek suggest that the mathematical particulars are not necessary, but the ability to assess numbers critically requires an understanding of the big picture.
By way of example, consider the following image—a month old as I write this, and older still as you read it:
From the Washington Post, October 15, 2020.
These figures are not only presented as aggregates but are population-adjusted to allow for comparison between states and the country as a whole. Each figure includes daily data points collected over a seven-month period, but they call attention not to any one day’s data but to a more general trend over time. And each figure uses bolding to draw the eye to a curve that attempts to smooth variation caused by weekend lags and data anomalies.
Fifth, the public needs a conceptual understanding of research methods and data sources.
Sixth, the public needs to understand that they can access the same reports and databases that journalists rely on. Doing so requires both digital literacy to navigate the websites and a higher level of technical literacy to parse the metadata and methodological information listed there.
I would add to this list a seventh type of necessary knowledge: the public needs to recognize that numbers have no monopoly on truth.
Such a critical stance is doubly important because journalists—who so often mediate public access to official statistics—struggle with this question, too. Tony Van Witsen (2018), who worked as a science journalist before returning to graduate school, conducted semi-structured interviews with journalists at a range of outlets to understand how they make decisions about statistics. His interviewees largely said that statistics are more credible and objective than other types of information:
Many subjects believe, however incompletely, that numbers have a special epistemic status simply by virtue of being numbers. Even though all subjects recognized individual numbers could be problematic, the idea persisted that numbers provide direct access to a kind of truth not available from live sources or eyewitness descriptions. (6)
Take a minute and click your way through the CDC dashboard. You might prefer Johns Hopkins, which is global in scope, or the COVID Tracking Project,[5] which compiles information from US state and local governments. Or maybe you prefer your own state’s Department of Health, since you’d like to know what’s happening close to you.
Whichever one you choose, ask yourself: Which numbers are easy to find? Which are not? Can I compare between yesterday and today? Between last month and today? Between rapid antigen tests, PCR tests, and antibody tests? Between those who were tested due to exposure and those who were tested to fulfill a bureaucratic requirement? Between genders, races, age groups, or employment status—leaving aside all the problems with how this information is collected in the first place (Gillborn et al. 2018; Zuberi 2000)?
What can you find by looking deeper? And what is fundamentally unknowable?
Remind yourself with every click that behind every number—and behind every dashboard (Mattern 2015)—is a series of human choices.
Statistics do not eliminate uncertainty. They simply allow us to quantify it.
“For anthropologists,” Jean Segata wrote in April 2020, “numbers, cases, statistics or prevalence have faces, embodied trajectories, and biographies. . . . Pandemics are not only metrics. They must be considered from a perspective of situated lives and sensibilities. Pandemics are also embodied experiences. And each experience counts.” The story of this pandemic cannot be told purely in numbers. It will be told as millions of individual stories, about Donald Trump and James Brooks (Samuels 2020). And it will also be told as a story about scale: the story that requires us to count, however imperfectly.
There are other less official ways of counting. Embodied ways of counting. How many ambulance sirens do you hear in an hour? How quickly do people cross the street to get away from each other? How long can you hold your breath at the grocery store? How many Zoom funerals have you attended?
In an interview that aired on the BBC on May 10, statistician David Spiegelhalter described the UK government’s statistical communication about COVID-19 as “number theater” that doesn’t take into account the “strengths and limitations” of the data. Much like the UK government, various agents and agencies of the US government use this data performatively: for communicating and constituting control, knowledge, and “business-like discipline and accountability” (Erikson 2012, 367).
Quantification is “a technology of distance” (Porter 1992) that allows for accountability at scale. And indeed, states operate as what I call quantificational regimes—through public health, but also through taxation, censuses, mapping, and many other “techniques of enumeration” (Sangaramoorthy and Benton 2012, 288). The United States is not so much an imagined community as it is a quantified one (cf. Anderson 1983). To the extent that there is an “us,” it’s only because we can be counted (cf. Lieberman 2020).
Acknowledgments. This essay was written as part of Meaningful Math, a research project funded through National Science Foundation Award #DRL-1906802. The opinions, findings, and conclusions or recommendations expressed are those of the author and do not necessarily reflect the views of the National Science Foundation. I am also grateful for feedback from my colleague John Voiklis and the Public Anthropologies editors.
REFERENCES CITED
Anderson, B. 1983. Imagined Communities: Reflections on the Origin and Spread of Nationalism. London: Verso.
Best, J. 2020. “COVID-19 and Numeracy: How about Them Numbers?” Numeracy 13 (2): article 4.
Briggs, C. 2003. “Why Nation-States and Journalists Can’t Teach People to Be Healthy: Power and Pragmatic Miscalculation in Public Discourses on Health.” Medical Anthropology Quarterly 17 (3): 287–321.
Bulmer, M. 1980. “Why Don’t Sociologists Make More Use of Official Statistics?” Sociology 14 (4): 505–23.
Conner, K. 2020. “Can Anyone Get Tested for Coronavirus Now? Here’s Who Qualifies.” CNET, June 7. https://www.cnet.com/health/can-anyone-get-tested-for-coronavirus-now-heres-who-qualifies/.
Desliver, D. 2020. “As Coronavirus Spreads, Which U.S. Workers Have Paid Sick Leave—and Which Don’t?” Pew FactTank, March 12. https://www.pewresearch.org/fact-tank/2020/03/12/as-coronavirus-spreads-which-u-s-workers-have-paid-sick-leave-and-which-dont/.
Erikson, S. L. 2012. “Global Health Business: the Production and Performativity of Statistics in Sierra Leone and Germany.” Medical Anthropology 31 (4): 267–84.
Gal, Iddo, and Irena Ograjenšek. 2017. “Official Statistics and Statistics Education: Bridging the Gap.” Journal of Official Statistics 33 (1): 79–100. DOI: 10.1515/JOS-2017-0005.
Gillborn, D., P. Warmington, and S. Demack. 2018. “Quantcrit: Education, Policy, ‘Big Data,’ and Principles for a Critical Race Theory of Statistics.” Race, Ethnicity, & Education 21 (2): 158–79.
Hubler, S. 2020. “Colleges Learn How to Suppress Coronavirus: Extensive Testing.” The New York Times, October 2. https://www.nytimes.com/2020/10/02/us/colleges-coronavirus-success.html.
Ko, L. 2016. “Unwanted Sterilization and Eugenics Programs in the United States.” PBS, January 29. https://www.pbs.org/independentlens/blog/unwanted-sterilization-and-eugenics-programs-in-the-united-states/.
Lieberman, E. 2020. “Risk for ‘Us,’ or for ‘Them’? The Comparative Politics of Diversity and Responses to AIDS and Covid-19.” SSRC Items. https://items.ssrc.org/covid-19-and-the-social-sciences/democracy-and-pandemics/risk-for-us-or-for-them-the-comparative-politics-of-diversity-and-responses-to-aids-and-covid-19/.
Long, M., and M. Rae. 2020. “Gaps in the Emergency Paid Sick Leave Law for Health Care Workers.” Kaiser Family Foundation, June 17. https://www.kff.org/coronavirus-covid-19/issue-brief/gaps-in-emergency-paid-sick-leave-law-for-health-care-workers/.
Lu, M. 2020. “These Are the Occupations With the Highest COVID-19 Risk.” World Economic Forum, April 20. https://www.weforum.org/agenda/2020/04/occupations-highest-covid19-risk/.
Mattern, S. 2015. “Mission Control: A History of the Urban Dashboard.” Places Journal. https://doi.org/10.22269/150309.
Mattern, S. 2020. “Andrew Cuomo’s Covid-19 Briefings Draw on the Persuasive Authority of PowerPoint.” Art in America, April 13. https://www.artnews.com/art-in-america/features/andrew-cuomo-covid-briefings-powerpoint-slideshow-authority-1202683735/.
Peters, E. 2020. “Is Obsessing Over Daily Coronavirus Statistics Counterproductive?” New York Times, March 12. https://www.nytimes.com/2020/03/12/opinion/sunday/coronavirus-statistics.html.
Porter, T. M. 1992. “Quantification and the Accounting Ideal in Science.” Social Studies of Science 22:633–52.
Rodríguez, M. A. and R. García. 2013. “First, Do No Harm: The US Sexually Transmitted Disease Experiments in Guatemala.” American Journal of Public Health 103 (12): 2122–26.
Sangaramoorthy, T. 2012. “Treating the Numbers: HIV/AIDS Surveillance, Subjectivity, and Risk.” Medical Anthropology 31(4): 292–309.
Sangaramoorthy, T., and A. Benton. 2012. “Introduction: Enumeration, Identity, and Health.” Medical Anthropology 31 (4): 287–91.
Segata, J. 2020. “Covid-19: Scales of Pandemics and Scales of Anthropology.” Somatosphere. http://somatosphere.net/2020/covid-19-scales-of-pandemics-and-scales-of-anthropology.html/.
Shmerling, R. H. 2020. “Which Test is Best for COVID-19?” Harvard Health Blog. https://www.health.harvard.edu/blog/which-test-is-best-for-covid-19-2020081020734.
Street, A., and A. Kelly. 2020. “Counting Coronavirus: Delivering Diagnostic Certainty in a Global Emergency.” Somatosphere. http://somatosphere.net/forumpost/counting-coronavirus-diagnostic-certainty-global-emergency/.
US Bureau of Labor Statistics (BLS). “How the Government Measures Unemployment.” https://www.bls.gov/cps/cps_htgm.htm.
US Center for Disease Control (CDC). “US Public Health Service Syphilis Study at Tuskegee.” https://www.cdc.gov/tuskegee/index.html.
US Centers for Medicare & Medicaid Services (CMS). 2020. “CMS Updates COVID-19 Testing Methodology for Nursing Homes.” https://www.cms.gov/newsroom/press-releases/cms-updates-covid-19-testing-methodology-nursing-homes.
Vann, M., S. R. Kim, and L. Bronner. 2020. “White Neighborhoods Have More Access to COVID-19 Testing Sites: Analysis.” ABC News, July 22. https://abcnews.go.com/Politics/white-neighborhoods-access-covid-19-testing-sites-analysis/story?id=71884719.
Van Witsen, A. 2018. “How Journalists Establish Trust in Numbers and Statistics: Results from an Exploratory Study.” In Understanding the Role of Trust and Credibility in Science Communication. https://doi.org/10.31274/sciencecommunication-181114-8.
Volz, M., and P. Galewitz. 2020. “As Long Waits for Results Render COVID Tests ‘Useless,’ States Seek Workarounds.” Kaiser Health News, July 23. https://khn.org/news/states-search-for-ways-to-deal-with-covid-19-testing-backlogs/.
Wojnarowski, A. 2020. “NBA Tweaks Coronavirus Testing Policy to Allow Players to Return Quicker.” ESPN, August 3. https://www.espn.com/nba/story/_/id/29588865/nba-tweaks-coronavirus-testing-policy-allow-players-return-quicker.
Yong, E. 2020. “America’s Patchwork Pandemic is Fraying Even Further.” The Atlantic, May 20. https://www.theatlantic.com/health/archive/2020/05/patchwork-pandemic-states-reopening-inequalities/611866/.
Zuberi, T. 2000. “Deracializing Social Statistics: Problems in the Quantification of Race.” Annals of the American Academy of Political and Social Science 568:172–85.
NOTES
[1] Ironically, Best himself elides the question of who glosses over the decisions, writing that “numbers appear in discussions of public issues and public policy.” A complicated apparatus of government and media actors (cf. Briggs [2003] on public discourse about health) contributes to the reification of these numbers, which in turn reproduces the power structures behind them.
[2] Further complicating the picture, many news organizations also report on the number of people applying for unemployment benefits.
[3] As of April 14, 20201, there were nearly 31.5 million reported cases of COVID-19 in the United States.
[4] While it is largely beyond the scope of this piece, it’s important to acknowledge that the dynamics of racialization are playing out in extraordinarily complicated ways across the United States. East Asian Americans, Black and Latinx Americans, Indigenous Americans, and Orthodox Jews have variously been blamed for the pandemic’s spread, had their illnesses and deaths treated as insignificant, and experienced a rise in hate crimes—even while the racialized experiences of white Americans largely continue to be treated as the unmarked default and go unexamined.
[5] The COVID Tracking Project stopped collecting data on March 7, 2021.
CITE AS
Barchas-Lichtenstein, Jena. 2020. “The Limits of Official Statistics.” American Anthropologist website, April 14.