Behind the Controversy By Mark Lindeman and Rick Brady Part II page3

Even if interviewers manage to approach voters strictly at random, they cannot compel them to comply uniformly, nor to refuse strictly at random. Differential nonresponse is a known problem—and, in some respects, a measurable and partially tractable problem. Interviewers are instructed to tally the race, sex, and approximate age of their refusals and misses. (During the day, they call in three times to report current vote tallies, nonresponse tallies by demographic, and a sample of all exit poll responses.) These tallies are used in weighting the data to account for nonresponse. The differences can be substantial. According to interviewer tallies, the completion rate for voters ages sixty and over in 2004 was only 43 percent, compared with over 55 percent for younger voters.

Survey complexity also influences response rates and, in some experiments, response bias. The 2004 exit poll survey contained thirty political and demographic questions filling both sides of an 8½" by 11" sheet of paper. Warren Mitofsky reported in 1991 [and discusses again in this issue of Public Opinion Pros] an experiment with three sizes of questionnaires, ranging from the full size used in 2004 to a quarter-sheet form that asked only about vote choice. The shortest form had the highest response rate but also the highest bias, consistently favoring the Democratic candidate. A half-page questionnaire yielded lower bias rates.

But the most troublesome forms of differential nonresponse are those that cannot be seen until the official returns are available. As Martin Plissner and Warren Mitofsky noted as early as 1982, “There is no guarantee that voters who respond to the poll are like those who refuse to answer”—and if their candidate preferences are different, then exit poll projections will be systematically worse than those based on vote returns alone. John Brennan has noted that if nonrespondents have different preferences than respondents, weighting for observable nonresponse may actually increase the bias.

Mitofsky attributed bias observed in the 1988 presidential election exit poll to “people who refuse to be interviewed.” Again, after the 1992 presidential election, Mitofsky attributed an apparently consistent Democratic bias to “a difference in response rates of supporters of one candidate versus the other," adding, "There’s no possibility for it to be anything else and be this systematic.” (Ironically, Mitofsky’s history of speaking out on this issue did not prevent uninformed critics from claiming that he had “invented” the possibility of differential nonresponse to explain away the 2004 results.)

Actual misreporting of votes can also mar exit poll results. In the 1989 Virginia gubernatorial election, one exit poll estimated a ten-point victory for Democrat L. Douglas Wilder. Wilder actually defeated his opponent by only two-tenths of a point, becoming the first elected African-American governor in U.S. history. This error has been speculatively attributed to misreporting by whites reluctant to admit their unwillingness to vote for the black candidate. Differential nonresponse is unlikely to explain the error because the same exit poll gave an accurate estimate in the contest for attorney general.

Bias due to differential nonresponse is not necessarily a function of response rates, per se. Merkle and Edelman’s 2002 study of nonresponse in exit polls, conducted from 1992 through 1998, revealed “very little or no correlation” between response, refusal, and miss rates and exit poll bias. Similarly, Brennan observed that, despite better response rates in 1992 than 1988, bias was larger in 1992. (Interestingly, in a “parallel election” conducted during a 2005 San Diego mayoral election, the apparent bias in the results favoring the Democratic candidate was greater in precincts with larger completion rates.) Brennan expressed concern that nonresponse was becoming “more selective” or characterized by “partisan coloration.” Perversely, one experiment with providing incentives produced moderately higher response rates, but also higher bias. The authors of this study postulated that the prominent media logos on the folders that were offered as incentives were perceived differently by Republicans and Democrats, explaining the differential nonresponse rates.

The tendency to refuse participation has been shown to correlate with age, race, gender, and partisanship of the interviewer. Less is known about interviewer effects on differential nonresponse. According to Edelman and Merkle, in 1992, interviewer partisanship was associated with absolute error rates (that is, no results for signed error were reported); no other interviewer effects were statistically robust. In 2004, not only did young interviewers attain lower response rates, but their error rates (absolute and signed) were generally larger. Some research indicates that interviewer race could influence results in some polls.