Just before the South Carolina primary last week, there was a report from the Public Religion Research Institute (PRRI) on religion in the state. I couldn’t believe what was reported. It took some digging to find the source of the problem: it is very difficult to find valid surveys of states, and it’s even harder to find ones that can be compared over time.
Here’s what PRRI reported. Since Pew surveyed in 2007, the percentage of white born-again Christians dropped from 36 to 26 percent. Other white Protestants fell from 24 to 17 percent. This means that white Protestants lost nearly a third of their members in just eight years. White Protestants have gone from being a majority (60 percent) to a minority (43 percent).
If these results were true, then it was big news. Really big. Sure, South Carolina is becoming more diverse, but did white Protestants drop 800,000 in less than a decade? Such a change is incredible (literally).
That was my gut reaction. Figuring out whether my instinct was right took some digging. It took some time, but I found the problem.
First, you need to understand how surveys like this work. Both Pew and PRRI are claiming that their results represent South Carolina. That’s not quite true. Both of them are conducting national surveys (Pew conducted one big survey, PRRI aggregates dozens of smaller surveys). They each then weight their results so that their surveys match demographic data in the national population. This is all standard practice and valid.
The problem comes when a survey takes a look at a state. Even with a large number of respondents, South Carolina residents in a national survey may or may not be representative of their state because of how polling firms order their questions, select their samples, weight respondents, and other technical issues.
In this case, there is a problem was that there were too many white, non-Hispanic people in their sample of South Carolina. The Pew 2007 survey was 73 percent white, non-Hispanic. According to the U.S. Census, this should have been about 66 percent. PRRI’s 2015 survey had 61 percent white, non-Hispanic, which is about three points lower than the Census estimates. The over and under representation of white, non-Hispanics is the real reason that there looks like the bottom has fallen out of the white Protestant demographic in South Carolina. Pew over-estimated white, non-Hispanics, and PRRI slightly under-represented this same group (small enough that PRRI could be statistical variation or rounding). Together, the surveys show a change six times the actual shift in white, non-Hispanic residents.
While PRRI isn’t to blame for Pew’s over-estimate, they took Pew’s estimates and ran with it. The result was a report that focused on “The Declining Proportions of White Christians in South Carolina.” This isn’t the case, and a survey research firm reporting on changing demographics should know that there hasn’t been that dramatic difference in the state.
Using Pew’s other state data and results in PRRI’s 2015 survey, I estimated a statistical model to see how much the inaccurate Pew data affected the results. Given South Carolina’s large white born-again population and the size of Pew’s overestimate of white, non-Hispanics, I estimate that the drop in white born-again Christians from 2014 and 2007 is about half of what was reported. White born-again Christians have likely dropped from about 31 percent–not 36 percent–to 26 percent.
A five point drop is still sizable, but realistic. Part of the decline is the rise of the so-called nones. Part is the small increase of minorities in the state. And part is age—asking if people are “born-again” made more sense in earlier generations but is less reliable than other measures used to identify evangelicals.
Bottom-line: The decline in white Protestants was exaggerated by differences between the polls, which is always a risk when comparing state surveys from different years.