Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are

This country, long soiled by slavery and Jim Crow laws, seemed finally to have stopped judging people by the color of their skin. This seemed to suggest that racism was on its last legs in America. In fact, some pundits even declared that we lived in a post-racial society.

In 2012, I was a graduate student in economics, lost in life, burnt-out in my field, and confident, even cocky, that I had a pretty good understanding of how the world worked, of what people thought and cared about in the twenty-first century. And when it came to this issue of prejudice, I allowed myself to believe, based on everything I had read in psychology and political science, that explicit racism was limited to a small percentage of Americans—the majority of them conservative Republicans, most of them living in the deep South.

Then, I found Google Trends.

Google Trends, a tool that was released with little fanfare in 2009, tells users how frequently any word or phrase has been searched in different locations at different times. It was advertised as a fun tool—perhaps enabling friends to discuss which celebrity was most popular or what fashion was suddenly hot. The earliest versions included a playful admonishment that people “wouldn’t want to write your PhD dissertation” with the data, which immediately motivated me to write my dissertation with it.*

At the time, Google search data didn’t seem to be a proper source of information for “serious” academic research. Unlike surveys, Google search data wasn’t created as a way to help us understand the human psyche. Google was invented so that people could learn about the world, not so researchers could learn about people. But it turns out the trails we leave as we seek knowledge on the internet are tremendously revealing.

In other words, people’s search for information is, in itself, information. When and where they search for facts, quotes, jokes, places, persons, things, or help, it turns out, can tell us a lot more about what they really think, really desire, really fear, and really do than anyone might have guessed. This is especially true since people sometimes don’t so much query Google as confide in it: “I hate my boss.” “I am drunk.” “My dad hit me.”

The everyday act of typing a word or phrase into a compact, rectangular white box leaves a small trace of truth that, when multiplied by millions, eventually reveals profound realities. The first word I typed in Google Trends was “God.” I learned that the states that make the most Google searches mentioning “God” were Alabama, Mississippi, and Arkansas—the Bible Belt. And those searches are most frequently on Sundays. None of which was surprising, but it was intriguing that search data could reveal such a clear pattern. I tried “Knicks,” which it turns out is Googled most in New York City. Another no-brainer. Then I typed in my name. “We’re sorry,” Google Trends informed me. “There is not enough search volume” to show these results. Google Trends, I learned, will provide data only when lots of people make the same search.

But the power of Google searches is not that they can tell us that God is popular down South, the Knicks are popular in New York City, or that I’m not popular anywhere. Any survey could tell you that. The power in Google data is that people tell the giant search engine things they might not tell anyone else.

Take, for example, sex (a subject I will investigate in much greater detail later in this book). Surveys cannot be trusted to tell us the truth about our sex lives. I analyzed data from the General Social Survey, which is considered one of the most influential and authoritative sources for information on Americans’ behaviors. According to that survey, when it comes to heterosexual sex, women say they have sex, on average, fifty-five times per year, using a condom 16 percent of the time. This adds up to about 1.1 billion condoms used per year. But heterosexual men say they use 1.6 billion condoms every year. Those numbers, by definition, would have to be the same. So who is telling the truth, men or women?

Neither, it turns out. According to Nielsen, the global information and measurement company that tracks consumer behavior, fewer than 600 million condoms are sold every year. So everyone is lying; the only difference is by how much.

The lying is in fact widespread. Men who have never been married claim to use on average twenty-nine condoms per year. This would add up to more than the total number of condoms sold in the United States to married and single people combined. Married people probably exaggerate how much sex they have, too. On average, married men under sixty-five tell surveys they have sex once a week. Only 1 percent say they have gone the past year without sex. Married women report having a little less sex but not much less.

Google searches give a far less lively—and, I argue, far more accurate—picture of sex during marriage. On Google, the top complaint about a marriage is not having sex. Searches for “sexless marriage” are three and a half times more common than “unhappy marriage” and eight times more common than “loveless marriage.” Even unmarried couples complain somewhat frequently about not having sex. Google searches for “sexless relationship” are second only to searches for “abusive relationship.” (This data, I should emphasize, is all presented anonymously. Google, of course, does not report data about any particular individual’s searches.)

And Google searches presented a picture of America that was strikingly different from that post-racial utopia sketched out by the surveys. I remember when I first typed “nigger” into Google Trends. Call me na?ve. But given how toxic the word is, I fully expected this to be a low-volume search. Boy, was I wrong. In the United States, the word “nigger”—or its plural, “niggers”—was included in roughly the same number of searches as the word “migraine(s),” “economist,” and “Lakers.” I wondered if searches for rap lyrics were skewing the results? Nope. The word used in rap songs is almost always “nigga(s).” So what was the motivation of Americans searching for “nigger”? Frequently, they were looking for jokes mocking African-Americans. In fact, 20 percent of searches with the word “nigger” also included the word “jokes.” Other common searches included “stupid niggers” and “I hate niggers.”

Seth Stephens-Davidowitz's books