Culture The 'Splainer

Religion survey babble confuses 103% of readers. Here’s why

"The Tower of Babel" oil painting from 1604.

Photo courtesy of Abel Grimmer [Public domain], via Wikimedia Commons

“The Tower of Babel” oil painting from 1604.

The ‘Splainer (as in “You’ve got some ‘splaining to do”) is an occasional online feature in which Kimberly Winston and other RNS staff give you everything you need to know about current events to hold your own at a cocktail party.

(RNS) Reading religion surveys can seem like confronting the Tower of Babel: stacked questions, confusing terms, unscientific methodology.

It gets even crazier when results are contradictory. How does that happen?

Let us ‘splain.

Some surveys lean like the Tower of Pisa

The Pledge of Allegiance is a perfect example.

There’s almost always a flap over how many Americans do — or don’t — want the words “under God” kicked out of the Pledge of Allegiance. Indeed, on Wednesday (Nov. 19) a court in Monmouth, N.J., will hear the case of the American Humanist Association battling the Matawan-Aberdeen Regional School District to have schools edit out mention of God.

The humanists claim 34 percent of Americans agree with their view. But, wait. What about a survey conducted earlier this year by LifeWay Research, a Christian research agency? It found that only 8 percent would cut God from the Pledge.

Why four times the difference? Look to the poll language.


Photo courtesy of American Spirit via Shutterstock

A crowd recites the Pledge of Allegiance at a Kerry campaign rally in 2004.

LifeWay asked: “Should the words ‘under God’ be removed from or remain in the Pledge of Allegiance to the United States of America?” That’s a straight-up question with no preface.

The humanists’ survey, however, began with a bit of pointed Pledge history — before getting to the (loaded) question:

“For its first 62 years, the Pledge of Allegiance did not include the phrase ‘under God.’ During the Cold War, in 1954, the phrase ‘one nation, indivisible … ‘ was changed to read ‘one nation, under God, indivisible … ‘. Some people feel this phrase in our national pledge should focus on unity rather than religion.

“Do you believe the Pledge of Allegiance should:

  1. Return to the unchanged version: ‘one nation, indivisible … ‘
  1. Continue with the changed version: ‘one nation, under God, indivisible … ‘”

This is not kosher poll methodology, say experts.

“Always ask yourself why this group sponsored this survey,” advised David Kinnaman, president of the Christian research company Barna Group. “Read the questions and see if the responses are prompted. What is the information asking me to fear or to love? Are they trying to elicit one of those emotions from me?”

Watch the labels

Researchers on religion and politics are fascinated with the evangelical vote. Is it growing? Shrinking? Trending X or Y direction?

But “evangelical” is one of the slipperiest words out there. Since every survey group sets its own definition, results can confuse more than they enlighten.

This has long been true. In 1998, Gallup asked people if they were “evangelical or born again” and came up with 47 percent, says survey research veteran Conrad Hackett, a demographer at the Pew Research Center’s Religion & Public Life Project.

But University of Notre Dame sociologist Christian Smith came up with 7 percent that same year. Smith counted as evangelical only Protestants who attend church regularly or say religion is extremely important in their lives and who choose evangelical from a list of possible identities including fundamentalist, mainline, liberal or something else, said Hackett.

Pinning down religious identity in an era when 20 percent are “nones” — people who say they have no particular faith brand — is like trying to climb a greased pole. Neither can you assume that a label reflects reality, or that identity, belief and actions align. Oy!

Speaking of “Oy”… Pew Research dealt with the complex question of “who is a Jew” by allowing people to define themselves by religion, culture and family ancestry. However, there’s no parallel spot for cultural Christians who have little or no commitment to Christian theology or religious practice.

No survey has a “Christian Lite” category. Maybe they should. When Kinnaman at Barna Group looked at the mix of belief and behavior and church involvement, his organization’s survey found that 38 percent of self-labeled Christians were essentially like nones in their political and cultural outlook.

People fib

Raise your hand if the weight listed on your driver’s license is correct — or ever was. Well, we fib on surveys, too.

When Philip Brenner, a University of Michigan research fellow with the Institute for Social Research, examined hundreds of surveys and time diaries, he found Americans over-report their church attendance by 10 percent to 18 percent.

Why? We give answers that fit our self-image, Brenner said. We reframe the question to be: “Are you the sort of person who attends religious services?” Sure we are.

Methodological madness

We’re all bombarded with online opportunities to answer surveys. Fun — and totally unscientific. Put no credence in the results because they’re in no way representative of anyone except people who are online (no surveying the Amish) and who may have a point of view to promote.

An old rotary phone.

Photo courtesy of Billy Brown, via Wikimedia Commons

An old rotary phone.

4chan, the anonymous online forum that delights in provoking mischief, recently upended Time magazine’s fourth annual “word banishment” online poll by encouraging people to hate on the word “feminist.”

The magazine editor later apologized for including the word “feminist” — but not for employing a survey method that’s a gateway to troll heaven.

The old-fashioned randomly dialed phone survey is biting the dust. Why? For one, consider whether you even use your smartphone for phone calls anymore. And if you’re under 30, start by Googling “landline.”

So major research firms are moving to elaborately devised panels of people drawn randomly to represent American diversity who are willing to reply online or by mail to surveys. Pew Research devised an American Trends Panel, carefully assessed so everyone isn’t the same age or inclination.

There’s a hitch, however. You can’t track change over time in surveys if the researchers changed methodology, too.

Pew Research recently released a panel-based survey on online and offline religion that found 46 percent of U.S. adults say they saw someone sharing “something about their faith” on the Internet in the last week.

Is that a greater number than five years ago? We can’t tell. Earlier surveys about religion and online behavior were phone surveys.


About the author

Cathy Lynn Grossman

Cathy Lynn Grossman specializes in stories drawn from research and statistics on religion, spirituality and ethics. She also writes frequently on biomedical ethics and end-of-life-issues


Click here to post a comment

  • The entirety of statistics and surveys is corrupt to the core. Statistics is simply the process of torturing numbers until they agree with you.

  • “This is not kosher poll methodology, say experts.”

    Of course. We wouldn’t want to solicit informed opinions instead of uninformed opinions.

  • “The humanists claim 34 percent of Americans agree with their view. But, wait. What about a survey conducted earlier this year by LifeWay Research, a Christian research agency? It found that only 8 percent would cut God from the Pledge.”

    Perhaps it is important that the persons responding to the poll know something about the history of the subject in question? I don’t see how it’s essentially problematic for a poll question to be prefaced with a bit of history. (Thus, those who are taught — or who are reminded — that “God” did not always exist as part of the pledge would be more likely to cut it from the pledge.) Now, how that history is framed may be up for debate.

  • Actually, for a poll, its VERY important that the opinion is uninformed. The poll is supposed to represent what everyone as a whole believes about a certain subject, and does so through random selection of people.

    If you convince someone of your opinion before they give their answer, it’s not a valid result, unless you are testing to find out “how many people agree with removing the words ‘under God’ from the pledge when presented with the following information”.

    Giving the history lesson means that the poll finds four times the people support the measure then the amount that actually do, because the pollsters have changed the opinions of the people that they polled (WITHOUT changing the opinions of the people who were not polled), therefore it no longer represents what the populace believes.

  • Polls on religion get into deep spiritual and psychological realms that really are almost unpollable and clearly easily inaccurate. What else can account for the mainstream Protestant churches doing virtually everything polls say they should do or are in supposed agreement with polls yet seem to be in the worst trouble.

  • There’s no point in getting informed opinions only on an opinion poll, since not everyone the poll supposedly represents is informed. If the poll’s goal is to assess only informed opinions, it succeeds, but since the poll was not billed as such, it instead misleads.

  • Many claim to be born again as the percentage of people shows but the
    Bible says otherwise. The Bible says the gate is narrow and many will try
    to enter but won’t be able to/very few find it! Broad is the road to destruction
    is what the Bible says and many are on it. Bible says man shall perish cause
    of their lack of knowledge! We see it everywhere with people that don’t know
    what the Bible says. Their lifestyle shows cause it hasn’t changed one bit.
    Bible says all drunkards go to hell yet I meet person after person that still
    gets drunk because most people only want to talk about gay marriage or
    abortion so they don’t have to face their own sin. 1 Corinthians 6:9-12 lists
    many sins along with homosexuality so all sin is bad!. We all must Repent!

    The wine that Jesus made was new wine/diluted and it was made for
    symbolic reasons not to get drunk plus the Bible also says don’t get
    drunk on strong wine for it’s debauchery so wine drunkards go to hell
    too/as well unless they all Repent! Pride,saying something mean then
    laughing after like that makes it okay/sharp tongues,coveting,jealousy,
    gambling,greed,gossip,takin the Lords name in vain are also wrong!
    Many today sleep around then get married thinkin that they covered
    it up but never Repent/agree premarital sex/sleepin around is wrong!
    Bible says Repent and believe the Gospel to be saved! We must Repent!

  • Lol, really?? Even in a piece on data and polling as the subject matter you find a way to segway to your copypasta??

    Karla, you are simply IMPRESSIVE!!


  • Fourth Valley-Amen! Thank you for all of your feedack. I always appreciate
    you and that you do the Bible studying. God bless.

  • Reliable surveys publish the details, questions, and the demographics, and geographic locations and number of respondants. It is always a good way to find out how slanted the poll is.

  • It’s even worse than that for some of us non-Christians. For example, I have yet to take a mainstream religious survey that didn’t assume that I believed in at most one deity. As a polytheist, it was impossible for me to give an accurate answer.

  • Omorka- Read Jesus among other gods by Ravi Zacharias. There is only
    one God…the God of the Bible. Case for Christ by Lee Strobel is good too.
    Psalm 22:16-18 and Isaiah 53:3-5 are Bible prophecy about Jesus Christ.

  • Polls are bogus! People who view things differently are never counted. I’ve never in my life taken a poll. I have an opinion that is never heard. Sucks having to live with the crap that weird people choose that makes life terrible.

  • I agree emphatically that informed opinions are better than uninformed options. The extra background information makes the poll question better than the stripped version without any background. However …

    I take the point that one should not steer the answer. The background information on “under God” and the Pledge of Allegiance was factual and pertinent, and rather less egregiously steering than many poll question setups.

    To be more neutral, but still informative, there should have been concise statements pro and con (e.g., to counter the comment about unity, one could simply state that “Many Christians do not believe in separation of church and state, and think that including ‘under God’ is more important than emphasizing unity”).

    Fundamentalists and securlarists would not change their answer. I frankly don’t know how many moderate Christians would answer the question differently with the pro and con statements, but I do think that it would legitimate the poll methodology.

    Adding information does change our answers — that is why the concept of deliberative democracy is so powerful. It isn’t steering, if both sides of the argument are presented. It would be sad indeed if all that pollsters were interested in is measuring our current level of ignorance.

2019 NewsMatch Campaign: This Story Can't Wait! Donate.