From the Field: Presidential Approval Ratings in Perspective By Larry Hugick, Jonathan Best, and Stacy DiAngelo page 3

Using the analysis of variance (ANOVA) technique on a subset of the database created for this study, Jonathan Best put differences among the five media polls to a statistical test. Although not large enough to meet the criteria for statistical significance in the ANOVA, some consistent differences from quarter to quarter were seen in average approval ratings for the five media polls. Figure 2 shows graphically how the various polls tracked with one another over the five years that were used for the ANOVA. Almost without exception, Newsweek's approval ratings tended to be the lowest for both presidents. During the Bush years at least, CBS tracked pretty consistently as second lowest to Newsweek. ABC/Post tended to be at the opposite extreme during the Bush years, with the highest approval ratings.

While Newsweek tended toward a lower percentage approving, its percentage disapproving tended to be in the middle. As shown in Figure 3, ABC/Post and FOX were most likely to be at the extremes for disapproval, with ABC/Post registering the highest and FOX the lowest. Again, although these differences were not large enough to be statistically significant in the ANOVA, their consistency suggests they bear further study.

Differences among the five polls in the "Don't know/Refused" ratings were statistically significant, with Figure 4 showing the clear separation between the "high DK" and "low DK" polls. From quarter to quarter, the DK level for ABC/Post and CNN was always lower than those for CBS and Newsweek. ABC/Post tended to be lowest; Newsweek the highest. While FOX appeared to be a "high DK" poll, consistently different from ABC/Post and CNN, its DK level seemed different in the Bush data than for Clinton. During most of the Bush years, FOX had the highest DK level, often reaching 10 percent or higher. In contrast, FOX was never higher than both CBS and Newsweek in quarterly averages for DK levels in the Clinton data.

How did the DK affect comparisons of the quarterly averages for approval and disapproval across the five polling organizations? To find out, we created an "approval ratio" that removed the DK from the denominator. The approval ratio was defined as

Approval Ratio = Approval ÷ (Approval + Disapproval)

Figure 5 shows differences in the approval ratio among the five polls over time. At least for the most recent years studied, the four polls with a conventional methodology—ABC/Post, CBS, CNN, and Newsweek—moved even closer together when the approval ratio replaced basic approval as the statistic used for evaluation. The FOX results, however, did not move toward the others, and followed a somewhat different path. FOX's approval ratio tended to be higher than those of the other four polls for most of the quarters analyzed—those for Bush post-9/11, and Clinton's during the middle two years of his second term, when all five polls were active.

Are presidential approval ratings from the major media polls truly interchangeable? Our analysis indicates they are not. Ratings from the five polls studied here did, indeed, track very closely with one another, especially when the focus was narrowed to the most widely reported statistic—percent approve—rather than the full set of results, including the percentages disapproving and those saying "don't know." For the period 1997-2003, the typical difference across all five polls in average annual approval was just four percentage points, and their trend lines moved up and down largely in concert. However, there were some differences that followed a consistent pattern over time, and even one difference—in the level of "don't know"—that was statistically significant.

To put the approval results of these five polls in perspective, our study indicates they should be divided in two ways. First, we can separate the four polls that employ a conventional telephone survey methodology with a full demographic weighting from the one poll that takes a different methodological approach. The ABC News/Washington Post poll, the CBS News poll, the CNN/USA Today/Gallup poll, and the Newsweek poll all use a conventional methodology. While they differ in some specific aspects of methodology—such as length of a typical interviewing period and respondent selection procedures—they have enough in common that in the few ways their results seem to differ, they do so in a predictable and consistent manner.

FOX News/Opinion Dynamics, the newest of the five polls, differs from the other four in basic methodology, using a registered voter base and only limited demographic weighting of its sample. The FOX poll maintains demographic balance through sample controls rather than weighting, does not do much to control for respondents' education levels, and does not have a general policy of placing the approval question at the front of the interview.

The second important way to divide the five polls is by their typical level of "don't know" responses to the approval question. Two of the polls with a conventional methodology are "low DK" (ABC/Post and CNN) and two are "high DK" (CBS and Newsweek). FOX has been a high DK poll, and has shown an average percent approval remarkably close to CBS and Newsweek in recent years. However, there are enough differences between FOX and these two polls on other measures to suggest it needs to be kept separate.

ABC/Post and CNN consistently get a lower level of DK to the approval question than the other two conventional polls, CBS and Newsweek, and FOX. What's more, lower DKs tend to drive approval ratings higher while leaving disapproval largely unaffected. For each one-point reduction in the level of DK, approval rises by eight-tenths of a point, but disapproval is statistically unchanged. All this suggests that poll watchers should exercise greater caution when interpreting approval ratings from polls in a different DK category. Newsweek approval ratings, for example, are more comparable to those of CBS than to those of ABC/Post.

The FOX/Opinion Dynamics poll has tracked remarkably closely to the four conventional polls—particularly those in the high DK category—in percent approve during recent years. Nonetheless, analysis identifies differences below the surface in the FOX poll results that set FOX apart from the others. During the years studied, FOX's level of DK varied more than the others between presidents, often reaching 10 percent or higher double-digits during the Bush years but tracking well below 10 percent during the Clinton years. The trend lines of the four conventional polls tended to converge when the DK was factored out of the equation, but FOX did not join the pack. FOX seemed to register lower disapproval ratings, not only during the Bush years but during the Clinton years as well. The FOX poll's high point in Bush approval came a month after other polls showed his rating at a peak, a lag-time that is not easily explained. For all these reasons, it seems prudent to assume that the FOX poll approval ratings are not interchangeable with those of the conventional polls, at least until it establishes more of a track record.

Historical comparisons of current presidential approval ratings with those of past presidents often draw on data from the in-person Gallup poll. In a separate analysis, we found that the differences in results by phone versus in-person methodology were relatively small, but that approval ratings from in-person polls were slightly closer to those from high DK telephone polls like CBS and Newsweek than from low DK polls like ABC and CNN. FOX is about as close as the other high DK polls to Gallup in-person polling in percent approve, but tends to register lower disapproval and higher don't know.

Larry Hugick is chairman, Jonathan Best is senior methodologist, and Stacy DiAngelo is assistant study director, Princeton Survey Research Associates International. The authors owe a debt of gratitude to their colleagues at other organizations whose polls were selected for inclusion in this study, and whose cooperation in answering questions and disclosing the details about their methods and practices added immeasurably to this research effort. They would like to thank Gary Langer of ABC News, Kathleen Frankovic of CBS News, David Moore and Jeff Jones of the Gallup Organization, Dana Blanton of FOX News, and John Gorman of Opinion Dynamics Corporation for their help.