After George W. Bush narrowly defeated John Kerry in the 2004 U.S. presidential election, some observers became firmly convinced that Kerry had won—and that the exit polls showed it. Early tabulations based on the National Election Pool (NEP) national exit poll indicated that Kerry had approximately a three-point advantage in the popular vote. For those who believed that the exit polls had a record of "uncanny accuracy"—until recent elections—the inference of massive fraud was obvious. There are good reasons to doubt that the 2004 exit polls accurately measured the outcome. (My personal favorite is the exit poll projection that Kerry won New York by over thirty percentage points. My home state may be blue, but not that blue.) Still, some people firmly believe that the NEP pollsters, Edison Media Research and Mitofsky International, are withholding telltale evidence of massive election fraud.
On election night 2006, a gap once again emerged between initial exit poll tabulations on the one hand and official returns on the other. An initial tabulation based on the national House exit poll, posted a bit after 7 p.m. EST, indicated that Democratic candidates had outpolled Republican candidates by about 11.4 percentage points. This figure overstates the official Democratic margin by perhaps 3.5 to 4.5 points (depending on how uncontested races are treated). Statewide Senate and governor's races had similar "red shifts"—that is, Republicans on average did better in the vote counts than in the exit polls.
Given widespread doubts about the integrity of vote counts—underscored by the startling discovery of over 18,000 "undervotes" in a congressional race in Sarasota County, Florida—I understand the impulse to conclude that the exit polls are more reliable. And indeed, the Election Defense Alliance (EDA) promptly issued a report arguing that the 2006 exit polls evinced a "landslide denied"—millions of votes stolen from Democratic candidates around the country. Here we go again.
The EDA report, authored by exit poll debate veterans Jonathan Simon and Bruce O'Dell, acknowledges doubts about the polls' accuracy—or, in their words, "the campaign that has long been waged to discredit [their] reliability." However, they argue that the 2006 exit poll contains "an intrinsic and objective yardstick" to check its accuracy: a question about whom respondents voted for in the 2004 presidential election. According to the initial (7 p.m. EST) House tabulation, 47 percent of voters said they had voted for Bush in 2004, and 45 percent said they had voted for Kerry—a two-point margin similar to the official 2004 results. In the final tabulation, weighted to match 2006 vote counts, the gap widens to six points (49 percent Bush, 43 percent Kerry). Without evidence that Bush voters turned out in 2006 at a much higher rate than Kerry voters, this six-point gap seems impossible. Simon and O'Dell propose that "the valid exit poll was the unadjusted exit poll." They believe that the adjusted exit poll had to be padded with Bush voters in order to match an inaccurate vote count. Ergo, millions of Democratic votes were stolen around the country.
Unfortunately for Simon and O'Dell, their exit poll evidence contradicts itself. As they observe in a footnote, if the 2004 exit polls were "as accurate as the 2006 exit polls have proven to be," then Kerry received many more votes than Bush. Why, then, aren't there more Kerry voters than Bush voters in the initial 2006 tabulation? The mystery deepens if one believes, as Simon and O'Dell apparently do, that Kerry voters turned out in 2006 at a higher rate than Bush voters. And indeed, a second footnote duly suggests that the likely Democratic margin in the 2006 House races was "more than 20 percent (61 percent-38 percent)." If so, then the initial tabulation understated Democratic performance by over eleven points—about twice as large as the 2004 exit poll discrepancy! So the EDA defense of exit poll reliability seems to be highly selective.
Still, what about the apparent excess of Bush voters in 2006? Steve Freeman and others have pointed to a similar phenomenon in 2004. In 2004, the weighted national exit poll tabulation indicates that 43 percent of voters had voted for Bush in 2000 and only 37 percent had voted for Gore. It is not even possible for 43 percent of the 2004 electorate to have voted for Bush in 2000. Freeman argues that the exit pollsters entered what he facetiously calls a "Wonderland of numbers," in effect inventing millions of Bush 2000 voters to provide Bush's winning margin in 2004. However, the argument assumes that exit poll respondents correctly report their past presidential votes. Do they?
Well, no. The evidence indicates that exit polls (and other surveys) tend to exaggerate the previous winner's vote share. Of the twenty presidential exit polls archived in the Inter-university Consortium for Political and Social Research (ICPSR) collection, eighteen give the previous winner a larger margin than in the vote count. These results are displayed in Figure 1. The average change was a 7.7 percent increase in margin at the midterm, and a 11.0 percent increase in margin at the next election. These changes are influenced by differences in turnout, but the trend probably owes to outright misreporting of past votes. The change varies substantially from year to year, for reasons that are not entirely clear. Incumbent presidents with high job approval ratings tend to do better; among the losers, former presidents tend to do better than less familiar candidates. Even relatively unpopular incumbents generally do better in retrospect: It appears that some voters simply forget having voted for the loser. (I discuss this phenomenon more thoroughly elsewhere, although that 2005 analysis does not include the midterm exit poll results.)
Only twice did the incumbent's retrospective winning margin fall below the margin in the official returns. In 1994, self-described Bill Clinton voters outnumbered George H. W. Bush voters by 5.3 percent, about 0.3 points less than the official 1992 margin. And in the 1982 ABC exit poll, Ronald Reagan voters outnumbered Jimmy Carter voters by 5.8 percent, about 3.9 points less than the official 1980 margin. Note that in both cases, the losing candidate was an incumbent.
The 1982 case poses a further riddle: Reagan did worse retrospectively in the ABC exit poll, but ten points better in the CBS poll. Why the difference? Two possibilities appear. One is question order. In the ABC survey, the previous-vote question comes after a slew of questions evaluating Reagan's job performance and other political issues. These questions may have primed respondents to consider their current views of Reagan—or simply to think more about presidential politics in general—when they answered. In the CBS survey, the previous-vote question comes near the beginning. The second possible reason is that in the ABC survey, Reagan's name comes after Carter's; in the CBS survey, Reagan's name comes first! Compare 1984: Neither survey had many evaluation questions, Carter's name came first on both surveys, and the results were similar.
In short, the past-vote question is anything but "an intrinsic and objective yardstick"—and the 2006 surfeit of self-reported Bush voters is hardly surprising.
Simon and O'Dell cite some other figures from the weighted table that they regard as anomalous, but no more convincingly. For instance, they note that Bush's 43 percent job approval rating in the weighted tabulation was higher than in other surveys; CNN put it at 35 percent using the same question. However, Bush's approval rating in the unweighted tabulation is 42 percent—just one point lower. (To obtain a Bush approval rating in the mid-30s would require truly astonishing weights: Would you believe that the Democrats won the House by over thirty points? Apparently, the authors had abandoned exit poll accuracy entirely by this point in their analysis, but if so, they could have told us.)
Why is Bush's job approval rating so high in the exit poll? The response categories probably make a difference. Whereas CNN asks respondents whether they approve or disapprove, the exit poll offers four categories—"strongly approve," "somewhat approve," "somewhat disapprove," and "strongly disapprove." The "somewhat approve" category may attract some voters who would otherwise opt for outright disapproval. Indeed, Scott Rasmussen reports that in split-sample experiments, offering all four choices has yielded approval ratings three to four points higher than a two-way choice. The survey population may matter, too. Most approval ratings are based on surveys targeted at all adults, not just likely (or actual) voters. Charles Franklin noted in September that the differences in approval across these populations are unpredictable and typically small. However, the pollingreport.com approval trend table at least hints that the gap may have been wider in November. Whatever the reason(s), attributing Bush's high approval rating to fraud-concealing weights looks like a nonstarter.
Similar arguments apply to Simon and O'Dell's second anomaly: that congressional approval seems too high in the exit poll. Their third anomaly is a supposed excess of "born-again or evangelical" Christians, but actually the 2006 exit poll figures—weighted or unweighted—seem consistent with past exit polls and other surveys.
What about the preelection "generic" polls that showed Democrats leading by double digits? The generic polls ask whether respondents would vote for the (unnamed) Democratic or Republican candidate in their own district. Some fraud-minded observers have selectively quoted Bafumi, Erikson, and Wlezien's statement that the "generic polls turn out to be very good predictors" of the actual vote. But Bafumi et al. do not mean what these observers want them to mean. On the contrary, they argue that the polls "perform poorly as point estimates," and must undergo further analysis to "discount the exaggerated sizes of the generic poll leads." In fact, if we use Bafumi et al.'s model with Simon and O'Dell's estimate of the generic Democratic lead, the model projects an actual vote margin of about 7.8 points, close to the official returns. Bafumi et al. also report that their margin of error (95 percent confidence interval) for vote share is about 3.7 points—which means that their margin of error for vote margin is over 7.0 points. A predicted Democratic margin of "eight points plus-or-minus seven" hardly supports suspicions of massive fraud.
While generic polls on average tend to overstate Democratic margins, final Gallup polls have (on average) been more accurate. David Moore and Lydia Saad reported that for midterm elections from 1950 through 1990, the final Gallup poll had an average absolute error under 1.3 points on vote share—a record that continues through 2006. For what it is worth, in 2006, the final Gallup poll projected a 7.0-point Democratic margin, again close to the official returns.
Thus, generic polls actually don't support Simon and O'Dell's inference of massive vote miscount. Nor do polling results in individual races, where the polls name the candidates. Among all House races for which pollster.com reported poll results, the median Democratic vote margin was about 0.3 points larger than the pollster.com five-poll average. Limiting the analysis to competitive races yields similar results. In Senate races, the median Democratic candidate did 1.7 points better than the pollster.com average. Of course miscount is perfectly possible in individual races. Indeed, the evidence for miscount or some other frustration of voter intent in Sarasota County is very strong. But survey-based evidence of a "landslide denied" is hard to descry.
Warren Mitofsky took a dim view of various people who cited the 2004 exit polls as evidence of fraud. He wrote to me in December 2005 that, in his view, they "do not want to know the truth. They are on a 'cause.' They are advocates, not scholars and nothing anyone says will interfere with their mission." At the time, I argued with him; now I might not bother. Still, I have no warrant to doubt the subjective sincerity of many of the analysts, much less the people who have accepted their arguments as authoritative. I do worry that such weak arguments detract from far more credible concerns about election integrity.
Apart from the new (yet strangely familiar) round in the "exit poll debate," the 2006 election also saw the advent of an effort to use exit polls systematically in election auditing. Steve Freeman and Ken Warren piloted "election verification exit polls" in twenty-eight selected precincts in two Pennsylvania congressional districts. Other exit polls and "parallel elections" took place around the country, and several—including Freeman's—elicited apparent discrepancies with the vote count. At first glance, the results I have seen could readily be attributed to poll bias rather than miscount. Nonetheless, Freeman notes that the discrepancies may indicate "possible election fraud," and, of course, this is true. Exit polls ought to register massive vote miscount if it occurs. Unfortunately, their false-positive rate seems exorbitant.
I have little enthusiasm for the task of using exit polls systematically to audit elections. I see no reliable way to eliminate participation bias in exit polls, especially if they have the explicit purpose of detecting election fraud (which, as Freeman notes, presently worries Democratic voters more than Republicans). So I offer this modest proposal: To verify election results, let us focus on adopting secure, verifiable, and trusted voting systems. Despite my morbid fascination with exit poll debates, I would just as soon skip the next one.
Mark Lindeman is an assistant professor in the political studies program at Bard College.
1976 (CBS/New York Times): In 1972, For Whom Did You Vote?
Nixon / McGovern / Someone Else / Did Not Vote
1978, 1980 (CBS/New York Times): In 1976, For Whom Did You Vote?
Carter / Ford / Someone Else / Did Not Vote
1982 (CBS/New York Times): How Did You Vote In The 1980 Election For President?
Did Not Vote For President in 1980 / Reagan / Carter / Anderson / Someone Else
1982 (ABC/Washington Post): Back in 1980, for whom did you vote for president?
Jimmy Carter / Ronald Reagan / John Anderson / Someone else / Didn’t vote
1984 (CBS/New York Times): Who Did You Vote For In The 1980 Presidential Election?
Carter / Reagan / Anderson / Someone Else / Did Not Vote
1984 (ABC/Washington Post): Back in 1980, for whom did you vote for president?
John Anderson / Jimmy Carter / Ronald Reagan / Someone else / Didn’t vote
1986, 1988 (CBS/New York Times): Who Did You Vote For In The 1984 Presidential Election?
Mondale / Reagan / Someone Else / Did Not Vote
1986, 1988 (ABC/Washington Post*): Back in 1984, for whom did you vote for president?
Walter Mondale / Ronald Reagan / Someone else / Didn’t vote
1990 (VRS**): Who Did You Vote For In The 1988 Presidential Election?
George Bush / Michael Dukakis / Someone else / Didn’t happen to vote in 1988
1992 (VRS): Who Did You Vote For In The 1988 Presidential Election?
George Bush (Rep) / Michael Dukakis (Dem) / Someone else / Did not vote in 1988
1994 (VRS): Who did you vote for in the 1992 presidential election?
Bill Clinton (Dem) / George Bush (Rep) / Ross Perot (Ind) / Someone else /
Did not vote in 1992
1996 (VNS): In the 1992 election for president, did you vote for:
Bill Clinton (Dem) / George Bush (Rep) / Ross Perot (Ind) / Someone else /
Did not vote for President
1998, 2000 (VNS): In the 1996 election for president, did you vote for:
Bill Clinton (Dem) / Bob Dole (Rep) / Ross Perot (Ref) / Someone else / Did not vote
(1998: Did not vote for President in 1996)
2002 (VNS): In the 2000 election for president, did you vote for:
Al Gore (Dem) / George W. Bush (Rep) / Ralph Nader (Gre) / Someone else /
Did not vote
2004 (NEP): Did you vote in the presidential election in 2000?
No, I did not vote / Yes, for Al Gore / Yes, for George W. Bush /
Yes, for another candidate
*The Washington Post did not participate in the 1988 ABC News poll
** VRS conducted the 1990 exit poll with CBS News and the New York Times, but this question appeared only on the VRS questionnaire.