Alert the Media: Teach Journalists About Polls

By Frederic I. Solop and Kristi K. Hagen

A serious problem in the relationship between public opinion polling and the public is an ongoing disconnect between the conducting of social research and the reporting of social science findings by journalists and news editors, whose interpretations of polling research are often missing critical information and loaded with biases and inaccuracies.

In 1999, we first presented a new model of professional development for reporters and editors to teach them how to interpret and report survey research data properly. The need for such a model was made apparent by an analysis of all polling stories appearing in the Arizona Republic over a three-year period from 1996 to 1998, and continued monitoring indicates that this need is as pressing as ever today.

The Arizona Republic is the largest daily newspaper in the state. It was founded in 1890 and was purchased recently by Gannett. Based in Phoenix, the newspaper has a statewide circulation base of almost half a million people (577,000 on Sunday). A total of 609 polling stories were found by searching an archived version of the newspaper using the keywords "survey" and "polling." Individual news stories, excluding opinion pieces, commentary pieces, and letters to the editor, were printed and examined to determine the presence or absence of fifteen conditions related to the use of polling data.

We found, first, that polling headlines all too often do not accurately reflect story contents. This problem occurred in almost one-third (30 percent) of all polling stories. This is a very serious issue for a number of reasons. For one thing, people who read only the headline are left with a faulty understanding of the issues covered in the article, seemingly legitimized by the mention of scientific polling data. Those who do continue on to read the entire story may have their understanding of it biased by the headline that "frames" it.

Nearly one-quarter (23 percent) of the stories themselves included content that did not accurately reflect the findings of the research being reported. In others, it was difficult or impossible to judge the accuracy of the reporting because of a dearth of information. While an overwhelming proportion-90 percent-of stories did include basic frequencies from research, 10 percent did not. Eighty-eight percent lacked crosstabulated information, including demographic breakouts, and virtually none (less than 1 percent) included multivariate statistics.

Finally, we were troubled by the quotes that appeared-or failed to appear-in these stories. While a slight majority (54 percent) of stories did offer perspectives from informed sources, principal investigators of studies were directly quoted only 23 percent of the time. Particularly worrying was a trend toward including respondent quotes in polling news stories. Overall, 8 percent included respondent quotes. This concerned us for two reasons: First, quotes from individual respondents can never adequately reflect the aggregate-level insights upon which polling stories rely. Second, the inclusion of such quotes suggests to readers that there has been a breach of the confidentiality promised to survey respondents, as it is not always clear whether or not respondents were asked if their statements and identifying information could be published in the newspaper. This practice has the potential to create a chilling effect on future survey participation.

A final problem found in the Arizona Republic stories was the frequent omission of basic methodological information. Margins of error were included in less than half (48 percent) of the polling stories reported, and confidence levels in only two percent. More than one-quarter (27 percent) did not identify the population surveyed, and almost a third (31 percent) failed to say how many people were included in the study. Only about half of the polling stories (47 percent) even included dates when the survey took place. Twenty percent failed to identify sponsors whose funding of the surveys might have biased the results, and fielding agencies, often the same entities as sponsors, were identified in only 61 percent.