The ABCs of the 2004 Preelection Polls

By V. Lance Tarrance, Jr.

Despite living in the Information Age with its seemingly instant precision, the survey research and campaign industries learned another hard lesson on election night of November 2004: Sample surveys of political opinion are really just the ABC science at work—that is, the science of "Almost Being Certain." The reality is that random sample preelection surveys are indeed precise information tools, but not too distant from those used by military intelligence in a fast-paced war—as if we are "pretty sure that WMDs are there, but we are not absolutely certain."

The known statistical error of probability sampling and the unsystematic error unknown in the administrative processes of polling combine to give us a calculated yet approximated margin of error that yields results of "almost being certain"—hence, the axiom that all political polling is simply a "projection" at one point in time and not a "prediction." Yet, contrary to that axiom, many of the polling firms venture such a prediction on the eve of each election with the allocation of undecided voters in nonstandard ways, all of which complicates the known universe ever more. The well-regarded bipartisan Battleground Poll this past election yielded, interestingly enough, two totally different "final predictions" (obviously using different procedures for allocating undecided voters) from the same data set that was jointly collected.

But there are yet more error-prone areas to factor into any estimate. This brings us to the art, and not the science, of identifying so-called likely voters. Each polling firm today is forced to develop its own likely-voter "models" because of the one out of every two eligible citizens who does not participate, or the one in four registered voters who does not show up at election time. This recalibration is not science, as any impartial inspection would testify.

Gallup has had one of the more reliable determinations of likely voters, as was demonstrated especially vividly during the 1994 midterm elections, when the organization discovered the late "Contract with America" surge to the polls. Gallup has more or less set the industry standard by using seven questions on their preelection surveys to determine who is likely to turn out to vote. They attempt to measure campaign interest, knowledge of the neighborhood polling place, various past-vote histories, intention to vote, registration, and commitment to participate. It should be noted that a slightly inflated estimate of intention to vote can skew survey results just enough to produce a subtle swing in vote preference. Other organizations have their own approaches, equally prone to having their determinations thrown off by small misestimates. With such a variety of techniques trying to identify probable voters and screen out the least likely ones, projections of the likely vote are not foolproof.

So, let's see—Bush achieved his popular-vote victory with a plus-3 percent margin, and each of the national surveys reported a margin of error of also plus- or minus-3-4 percent. The average spread among fourteen polling firms from October 27 until November 1 was one-half of Bush's eventual margin (plus-l.5 percent), an estimate well within the margin of error. The "outcome direction" was also correct—Bush was ahead when all of the reported polls were averaged and, indeed, won.

It should also be recalled that there was a "tectonic shift" in August (between the party conventions) from Kerry to Bush (Kerry had been the lead for some four months prior to that!), and a consistent lead (albeit a small one) after that for Bush during all the ups and downs of September and October. These two factors largely account for why the Bush campaign team were "cautiously optimistic" on election day and so shocked at the early exit-poll results, which showed Kerry ahead.

Let's look at these fourteen organizations and their preelection estimates. Ten of the firms had their likely-voter results in the correct direction; that is, about three-quarters had it "mostly right" for Bush. The most notable of the firms that were at variance with the Bush outcome were FOX News's Opinion Dynamics poll (plus-2 percent for Kerry) and Gallup's CNN/USA Today poll (only tied). However, the Gallup pollsters, before they allocated the undecided vote, had the election results nearly dead-on, with a plus-2 percent margin for Bush, and they just adjusted wrongly. If these two organizations had been of lesser stature or influence no one would have cared; however, missing the direction of the outcome is usually considered a cardinal sin in the expensive world of campaign professionals.

It should be pointed out (if not forever remembered) that there were two organizations that did get the outcome precisely right. They were CBS News/New York Times and the Pew Research Center, each with a 3 percent lead for Bush, even though their timing and sample sizes varied considerably.

Since battleground states (read, the electoral college) were the featured item of the '04 presidential election, most national polling firms were forced to expend new energy and more money in many of them (so as to augment their "popular vote" estimates to their clients). This is where the performance record becomes murkier. Florida and Ohio were absolutely first-tier in importance, while Wisconsin, Iowa, and New Mexico were mostly second-tier, with Minnesota-Pennsylvania-Michigan third-tier (and their importance indeed fading away on election night.)

Let's first consider Florida, a state that eventually went for Bush by five percentage points. Of the nine or so polling organizations among the fourteen discussed above that delved into the quirky politics of Florida, less than half had estimates with the correct direction, while three were "almost certain" that Kerry was in front. To be fair, the average of preelection polls in Florida showed at best plus-1 percent for Bush, so all the estimates were going to be tricky no matter what the outcome. Nevertheless, two notable organizations were in the "missed-it" category—FOX News had Kerry winning by five percentage points (a swing of ten points) in Florida, while Gallup also had Kerry winning by three percentage points (a swing of eight points) there. (Both firms also inaccurately had Betty Castor winning the U.S. Senate seat in Florida.) There were two other firms, however, who had it "mostly correct"—the Rasmussen and Mason-Dixon polls estimated a margin of victory of three to four percentage points for Bush, a good mark in a state becoming more and more complicated to measure.

And what about Ohio? This was the most critical subnational area in the entire election, and one in which the politics were not as unstable as in Florida. Bush won Ohio by 2 percent of the total vote, and the average of all the preelection polls was also about 2 percent, a much better correlation than in Florida. Of the eight organizations of our fourteen conducting work in Ohio, 90 percent had it "mostly correct" in their polling. Unfortunately for Gallup's entry into the Ohio contest, theirs was the only estimate in the wrong direction (plus-4 for Kerry), which means they missed the trend in both of the first-tier battleground states.

What does all this really tell us? First of all, polling is still an "almost being certain" science, and partisan overreach must be curtailed. Second, nationally skilled polling organizations may not always be as reliable for individual state estimates, as they may lack awareness of many of the "local knowledge rules" that are at play in each of them. Third, because contemporary elections are so hotly contested, over-canvassed, and fraught with negative images ("The wrong war in the wrong place at the wrong time"), likely-voter models can be slightly off.

Survey estimates should always be considered an important part of our political environment, but not the environment itself. After all, we all work with a very tricky political map nowadays, as well as with a science that can only produce ABC estimates, not certitudes.

Lance Tarrance, Jr., was a director of the Gallup Organization from 1987 to 1993 and the president of Gallup China from 1993 to 1996. He was president and founder of The Tarrance Group (1977-93) and is a member of the American Political Science Association.

Readers who wish to respond to this or other articles appearing in Public Opinion Pros, or to contribute commentary of their own in 800 words or less, should consult our author submission guidelines and editorial policies under "Letters to the editor and op-ed articles." Op-ed submissions must be received at least two weeks before the first of the month for the issue in which they are to appear.