The Elusiveness of "Public Opinion"

An excerpt from The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls

Ever since the inception of modern polling, survey researchers have struggled with the Achilles' heel of their measuring instrument: the frequently vague meaning of survey questions. Over half a century ago, Cantril and Fried called it an "important and neglected problem." Stanley Payne, who preaches the virtues of brevity, simplicity, and specificity in his classic work The Art of Asking Questions (1951), warns his fellow pollsters, "the penchant of many respondents for answering questions which have no meaning for them poses a major problem for public opinion researchers." Nearly thirty years later, Norman Nie and his colleagues wrestled with the same problem in another guise when they described the vicissitudes of understanding "the changing American voter": "Even if the same question is asked at two different times, is it really the same question? The fact that times change may mean that the meaning of the question undergoes change." Writing on the future of public opinion research in the fiftieth-anniversary issue of Public Opinion Quarterly, sociologist James Davis lists first among the three biggest problems facing the field "Validity: What do the questions mean? We know respondents answer reliably and carefully, but we do not really know what they mean when they tell us about 'communism' or 'happiness' or 'performance on domestic issues.'" Summarizing a body of research on measurement errors associated with the meaning of survey questions, survey methodologist Robert Groves puts the issue more precisely when he talks about language as the medium of survey measurement: "Although the language of the survey questions can be standardized, there is no guarantee that the meaning assigned to the questions is constant over respondents." As Fowler reminds us at the outset of his research on the measurement effects of ambiguous survey questions: "It has been axiomatic at least since Stanley Payne in 1951 wrote his classic book, The Art of Asking Questions, that survey questions should be clear. Ideally, they should mean the same thing to all respondents, and they should mean the same thing to respondents as to the researcher, the person who will interpret the answers."

Cognitive psychologists Herbert Clark and Michael Schober probably identify the heart of the theoretical problem best in their discussion of how respondents answer vaguely worded survey questions in idiosyncratic ways: "Respondents… make the interpretability presumption: 'each question means what it is obvious to me here now that it means.'" If it is true that respondents interpret questions like this, their doing so violates a cardinal assumption in survey measurement: that the question should mean the same thing to all respondents. And if this assumption cannot be made, then valid comparisons across respondents become extremely difficult, if not impossible. Aggregating individual responses to a question to summarize public opinion on a given topic thus becomes a questionable, if not meaningless, exercise.