Opinion Polls: A Guide for the Critical Consumer

By Adam J. Berinsky

An excerpt from Silent Voices: Public Opinion and Political Participation in America

What can producers and consumers of polls do to uncover and account for exclusion bias? One possibility would be for pollsters to design survey items and interviewing practices that discourage respondents from offering don't know responses. [T]he proportion of nonresponses to survey questions is clearly affected by the format of the questions. Items that explicitly offer respondents the don't know option produce much higher rates of nonresponse. Perhaps, then, the solution is to turn the survey researcher's old saw around. Instead of encouraging citizens to take the don't know option when they are unsure of their opinion, we should be discouraging them from doing so.

But such a strategy creates problems of its own. A don't know response, after all, tells us something about the respondent's views-namely, that providing an answer is difficult from a cognitive standpoint, a social standpoint, or both. Pressuring citizens to give answers to all survey questions might increase the quality of information collected in some cases, but it could also exacerbate the problems. After all, a citizen who is uncomfortable providing an answer to a socially difficult question will not easily collapse and say precisely what is on his or her mind, simply because of pressure to give an answer. More likely he or she will again try to abstain or-if he or she does give an answer-it is unlikely to be a fully truthful response.

Similarly, respondents who have a difficult time linking their predispositions to a particular political controversy will not suddenly find their political voice when pressured to give an answer. The gap between their underlying wants, needs, beliefs, and desires and the stakes of that controversy runs deeper than the surface. Given the complexities of the political world, the opinions of some respondents are more firmly grounded in their interests than others.

A better solution, I think, is to try to gain a greater understanding of what the don't know response is telling us about public opinion.

The typology of issue hardness presented in [the figure below] can help us identify those cases where a relatively high rate of question abstention could conceal certain types of opinions-those sentiments that run counter to particular social norms.

On these issues, opinion polls might provide a poor guide for future public response. For example, polls concerning the Equal Rights Amendment (ERA) in the mid-1970s showed majority support for the amendment, but typically 14-18 percent of the respondents said they had no opinion on the matter. These nonrespondents could have been genuinely uncertain or ambivalent, but it is also possible that some held their tongue rather than give an answer that could be construed as sexist. From this perspective, the failure of the ERA to attain passage in the late 1970s and early 1980s would not be surprising. Furthermore, questions concerning cognitively difficult issue areas that engender high rates of no opinion responses could miss underlying sentiment, or "latent opinion." The early years of the Vietnam War provide an example of such a case. But other instances of the obscurement of latent opinion behind don't know responses are possible. For example, questions concerning specific economy policies could be difficult for some individuals to answer in the short run. But once their long-term effects on the economy become clear, a clear sentiment might develop. This is not to say that we should abandon polls as a way to measure the public will. [W]e should, [however,] cast a cautious eye on the enterprise of survey research.

Not all opinion polls should be regarded with equal suspicion. Consider the case of public opinion concerning spending on education. This issue area is easy in both a cognitive and a social sense. It has been ingrained in the political culture over such a long period of time that respondents can answer questions on this topic with negligible cognitive costs. Furthermore, the social costs associated with expressing support or opposition to increased funding are minimal. Accordingly, it is not surprising that questions concerning school funding yield low don't know rates-on average about 2 percent. There might be reason to question how citizens would trade off school spending against higher taxes, but in the case of school funding, we can take polls at their face.

The key to identifying areas of potential bias is to look for survey questions on cognitively or socially difficult topics, where the number of people who do not offer responses is unusually large relative to the other items in the survey. It is important here to compare don't know response rates within a given survey. Rates of question abstention vary across polls depending on whether given survey organizations encourage or discourage no opinion responses. Furthermore, many of the polls carried out for media and politicians are conducted using only responses of registered voters or "likely" voters-the most engaged and involved segment of the electorate. In these polls, many of the citizens who would ordinarily gravitate to the don't know response at the opinion formation stage have been purged from the sample of poll respondents.

Of course, even with a high don't know rate, the presence of bias is not assured. But such questions should raise warning flags. For example, consider opinion concerning the proper means to stimulate a weak economy. In September 2001, Princeton Survey Research Associates conducted a poll for Newsweek asking, "Which of the following [programs], if any, would you personally favor to try to stimulate the economy and avoid a long recession?" When asked about a program that was relatively easy in a cognitive sense, "Major new government spending for security and infrastructure improvements, and to help New York City," only 3 percent of respondents abstained. However, when asked about a more complex program, "Cutting capital gains taxes on profits from the sale of real estate, stocks and other investments," 14 percent of the respondents did not offer a response. [T]he informed consumer should regard the second question-a more cognitively complex topic-with some suspicion.

In addition, a vigilant consumer of polls should carefully interpret questions concerning socially difficult issues. Throughout the last two decades of the twentieth century, for instance, popular portrayals of homosexuals in American society changed greatly. Social norms concerning discussion of issues of homosexuality showed a corresponding change. This is not to say that all individuals felt compelled to keep antihomosexual views to themselves. But almost certainly, for a portion of the public, questions concerning gay rights became highly sensitive topics. Like questions concerning policies designed to ensure racial equality, we should therefore be suspicious of questions with high rates of question abstention. In May 2000, a poll conducted by International Communications Research (ICR) for the Associated Press poll asked one-half of its 1,012 national adult sample, "In general, do you thinks gays and lesbians should or should not be allowed to be legally married?" The poll showed that a majority of Americans opposed the practice. But. the rates of question abstention suggest that underlying opposition might be even higher: 11 percent of respondents said they did not know where they stood, and another 3 percent outright refused to answer the question. The other half of the sample on that same survey was asked, "In general, do you think gays and lesbians should or should not be allowed to form a domestic partnership that would give the same-sex couple the same rights and benefits as opposite-sex marriage?" While a plurality of respondents opposed the measure, 11 percent claimed to have no opinion, and another 3 percent balked at providing any response. Given the emerging social difficulty of the issue, it could be that polls are not capturing the full extent of anti-gay marriage sentiment.