Behind the Controversy By Mark Lindeman and Rick Brady Part II page4

The debate (such as it is) over whether the 2004 exit polls evince fraud has exposed some widespread misconceptions. One recurrent myth is that the exit polls are inherently simple and practically foolproof exercises in marble-counting. Therefore, anyone equipped with a calculator or a copy of Excel can figure a “margin of error” and determine that over one hundred thousand survey responses point ineluctably to a Kerry victory—and to fraud. People who hold this view perceive alternative explanations of the exit poll discrepancy as unscientific obscurantism. As one observer put it, if Bush won, “Half a century of polling and centuries of mathematics must be wrong.”

This perception of exit poll inerrancy may seem almost willfully perverse to survey researchers, who spend much of their time pondering all the different ways in which polls can go wrong. Then again, given how often “pollsters” are derided as charlatans who undermine democracy, there is some novelty in being savaged as infallible scientists who destroy it by concealing the truth—though, of course, not all the criticisms have been so far beyond the pale.

The 2004 exit polls raised complex issues of confidentiality and transparency. On election night, CNN.com and other sites posted preliminary exit poll tabulations from many states within minutes of the scheduled poll closing times—soon after 7:30 p.m., in the crucial case of Ohio—as well as interim national tabulations. Many observers quickly became aware of these results and readily determined that they indicated a Kerry lead. Unfortunately, most did not realize that Edison/Mitofsky would update both its projections and its tabulations based on official vote counts as those became available, and that media sites would follow suit.

As Kerry’s lead in the Ohio and national “exit polls” (tabulations) transmogrified into a several-point deficit, some observers believed that they were witnessing Orwellian historical revisionism in real time. (Some still believe that the “true results” would have been suppressed had not a quick-thinking activist saved screen shots of the initial tabulations.) We believe that media outlets were right to post these preliminary analytical results expeditiously once the polls closed. Failing to do so would only have intensified the suspicions, since, inevitably, copies of the NEP tabulations would become available. In retrospect, however, we wish that these sites had prepared their readers for the “results” to change. CNN and NBC both posted informative explanations, but only assiduous readers would have understood the implications.

Many critics have complained that Edison/Mitofsky and the NEP should be more open in sharing the exit poll data. Most notably, Representative John Conyers (D-MI) wrote to Warren Mitofsky in December 2004 requesting that Mitofsky release “the ‘raw data’ from your exit polls,” including precinct identifiers—a request that Mitofsky politely rebuffed. Edison/Mitofsky did quickly release data from individual questionnaires; ironically, some critics saw this as an effort to forestall releasing the more important precinct identifiers. Edison/Mitofsky also released its extensive evaluation report to NEP, and collaborated with the Election Science Institute to “blur” data from the forty-nine Ohio exit poll precincts so that 2004 results could be matched with 2000 results, enabling further analysis. In these and other respects, the 2004 exit polls have been the most transparent in history—which perhaps is not setting the bar very high.

We do not say that NEP has revealed enough. Much remains to be learned about the sources of bias in the 2004 exit polls. While the January 2005 evaluation report presents evidence on some correlates of bias, it does not discuss some potentially important correlates, nor does it present multivariate analyses. Inquiring minds, including ours, want to know. To be sure, Conyers’s initial request does raise important privacy concerns. Mitofsky has pointed out that the code of professional ethics and practices subscribed to by members of the American Association for Public Opinion Research (AAPOR) pledges researchers to “hold as privileged and confidential all information that might identify a respondent with his or her responses.” If the precinct identifiers were publicly released, at least some respondents could be identified from their demographic profiles. Nevertheless, surely more analytical results could be released without jeopardizing confidentiality.

Yet we cannot share the belief of some observers that the exit polls hold the key to unlocking conclusive evidence of fraud, past or future. In particular, even if extensive and decisive fraud occurred in the 2004 election (which we doubt), precincts with large discrepancies are not especially likely to yield the proof. Anecdotally, at least some large discrepancies appear to owe to interviewers’ failure to follow the sampling protocol at busy, chaotic polling places. Nonresponse bias is possible under any circumstances, but it is probably more likely when random sampling subtly gives way to respondent voluntarism. Comparison with other election returns (as in Walter Mebane and Michael Herron’s report) is probably more reliable for detecting concentrated fraud. More diffuse fraud, on the order of a few percentage points in many precincts, cannot be reliably detected by current exit polls, even if nonresponse bias could be ruled out.

In summary, the 2004 exit poll raises fascinating issues of survey methodology and best practice in its own right. Further, it has given rise to a small but vigorous sect that merits the attention of anyone interested in the murky confluence of science, politics, and mythopoeia. And, perhaps obliquely, it spotlights rationally defensible doubts about election integrity—doubts that, regrettably, survey research probably is unable to resolve.

Mark Lindeman is an assistant professor in the political studies program at Bard College, and Rick Brady is an M.C.P. candidate in the School of Public Administration and Urban Studies at San Diego State University.