Differences in Polling Methodology

To compare the methodologies of the five media polls included in this study, the researchers who conduct the polls were asked to respond to a short survey. Information made available by the various polling organizations through the internet and other sources was also examined. A review of the methods shows FOX, the new kid on the block, differing most in its procedures. Differences were apparent in five areas.

• Interviewing methods and practices. FOX and Newsweek have the quickest turnaround time and shortest field periods—two days. The other three polls tend to have longer field periods, although CNN and CBS, in particular, conduct a fair number of two-day polls. All five polls make callbacks to noncontacts, and ABC, CBS, and Newsweek also make refusal conversion attempts as a general rule. Also as a general rule, CNN makes refusal conversions only for polls in the field more than three days; FOX does not attempt them. Only one poll—CBS—is conducted in-house by the media organization sponsoring the poll, but each of the other four polls relies on one field house, or limits its use of a different field house to rare occasions.
• Sampling and weighting. All five polls use RDD samples for their telephone polls rather than list samples. FOX, which reports results based on registered voters instead of the total population of adults ages eighteen and over, also differs from the other four polls in the way it deals with demographic variation in its sample. Opinion Dynamics weights the FOX poll sample only by race and ethnicity, and reports using sample controls to achieve demographic balance by gender, age, and region. Apparently, aside from race weighting, FOX pollsters take no specific steps to deal with the tendency of unweighted samples to include too many better-educated respondents and too few respondents with low levels of formal education.
  The other four polls take a more conventional approach to demographic weighting. They all weight their obtained samples by gender, race and ethnicity, age, and education. CBS, CNN, and Newsweek also weight by geographic region. ABC does not specifically weight by region, but achieves geographic balance through other means. CBS also weights by the number of adults and the number of phone lines in the household. CNN also weights by the number of phone lines, but not by the number of adults. Interestingly, none of the five media polls is routinely weighted by party identification, a technique commonly used in polls done for political candidates by partisan pollsters.
  Typical sample sizes for the five media polls are quite similar. CNN and Newsweek usually poll about 1,000 adults, although Newsweek's standard sample size in polls with approval ratings was 750 adults prior to 2001. ABC's typical sample size is 1,200 adults, and FOX polls generally have a sample of about 900 registered voters. CBS's sample size varies from poll to poll, ranging from about 700 to 1,200 adults.
• Respondent selection. The five media polls use a variety of methods to select a respondent within each household. Newsweek asks to speak with the youngest male or oldest female who is at home, a method CNN also used until late 2000. CNN now uses the "most recent birthday" method, as does FOX. ABC uses a variation of the most recent birthday method adjusted so interviewers ask to speak with a male 75 percent of the time and a female 25 percent of the time. CBS has the most rigorous approach, first determining the number of adults in the household and selecting one at random, whether or not the person is at home, and making an appointment, if necessary, to interview the selected respondent.
• Questionnaire issues. Four of the five media polls in our study reported placing the approval question very high in the questionnaire. CBS, CNN, and Newsweek reported putting the approval ratings question first, except in election years, when trial heats or other election-related questions might take precedence. ABC reported asking the approval question first almost always. FOX is the one poll that does not have a standard policy of placing the approval question at the front of the interview, reporting that placement varies from poll to poll.
  Specific interviewer instructions about whether to probe for a response among those who do not say they approve or disapprove vary from poll to poll. CBS and CNN indicated that no probing is done, and such responses are simply recorded as "don't know." Interviewers working on the Newsweek poll are specifically instructed to limit probing to cases where a respondent says "it depends." ABC and FOX reported using a full probe, re-asking the approval question of all respondents who do not say they approve or disapprove.
• Changes in methodology during the study period. Representatives of the polling organizations were asked if any major methodological changes were made to their polls during the period of this study, 1997 to 2003. ABC reported no major changes. The four other polls reported some changes, none of which seems truly "major." CBS reported adding callbacks to not-in-service numbers in the past two years. As noted earlier, CNN changed to the "most recent birthday" respondent selection method in late 2000. FOX reported experimenting with different CATI systems that vary respondent selection or callback procedures. The Newsweek poll increased its sample size from 750 to 1,000 in January 2001.