Switched-On Data: A Need for Standards in Secondary Survey Research

By Thomas R. Marshall

In recent years the public opinion profession has become steadily more focused on improving the standards of survey research. Most of this focus is understandably aimed at practitioners who are actively engaged in conducting and analyzing surveys--that is, primary research. Yet a significant part of the survey industry is based on secondary research--the reanalysis and comparison of poll results from the past, usually originally collected by other researchers.

The experiences and needs of secondary survey research are quite different from those of primary research. Because the secondary survey researcher works with polls that are already completed, perhaps years or even decades previously, it is impossible to control for important issues such as question wording, question order effects, question context effects, sample size, or the interview mode. The researcher cannot arrange timely split-ballot questions or innovative experiments to improve the quality of the results. In many cases, even determining how critical methodological decisions were made is impossible, since neither the original researchers nor their records are still available.

These issues in secondary research are not slight, and they are rightly a concern within the survey research industry. The growing availability of computerized databases help to make historical survey data more widely available, and the steady expansion of survey research into areas such as judicial behavior or crossnational polling increases the number of those who use secondary research. Other trends that encourage secondary research include the demand that studies incorporate historical and over-time comparisons, the growing use of historically-based polling methods, such as pooled, cross-sectional data, better graduate training and the demands of academic publication, and the much lower costs of analyzing other people's polls as compared to conducting one's own. Not least of all, the simple passage of time means that the number of years since the advent of modern polling techniques in the 1930s and the number of historical polls that are available for study are steadily increasing as well.

How widespread is secondary survey research in public opinion and the social sciences, and what standards do practitioners now apply to it? To identify the current extent of such work, I reviewed the last three years (2003, 2004, and 2005 to date) of eight major social science journals that commonly report survey research: Public Opinion Quarterly, American Political Science Review, American Journal of Political Science, Journal of Politics, Political Research Quarterly, American Sociological Review, American Journal of Sociology, and Social Forces.

Because Public Opinion Quarterly is especially important in survey research, all articles in each issue were examined. For the remaining seven social science journals, all articles in a randomly selected three issues were classified over this time period. The selected articles included 58 in Public Opinion Quarterly and 234 in the remaining social science journals, for a total of 292. All articles on survey research were counted as either primary or secondary survey research, or as neither. Primary survey research included research for which the author(s) were the survey investigators. Secondary research included articles in which the author(s) used a survey conducted by other researchers. Survey research was that based on sampling from a larger population, typically although not always persons.

The importance of secondary survey research is immediately apparent when Public Opinion Quarterly is separated from the remaining seven social science journals in Figure 1. Secondary research is slightly more common than primary research, by a small margin (45 percent to 40 percent) in Public Opinion Quarterly and by a two-to-one margin (28 percent to 14 percent) in the remaining social science journals.

In Figure 2 we see that, overwhelmingly, both primary and secondary survey research are based on statistical analysis of the actual individual records or interviews from the surveys in question, not simply the aggregate results. The only significant body of articles based on aggregate poll results is "The Polls" section of Public Opinion Quarterly. Elsewhere, survey research based on aggregate poll results is now quite rare, at least in these journals.

Figure 3 shows that while primary research is seldom based on more than a single survey, secondary survey research is usually based on multiple surveys, either with or without counting "The Polls" section of Public Opinion Quarterly. This suggests that secondary research now focuses on patterns among surveys and trends across time in a way that primary research seldom does. Secondary research is also much more likely to encounter the problems inherent to comparing different polls, such as differences in question wording, contact mode, or house effects.

Finally, Figure 4 compares primary and secondary survey research in terms of the lag (in years) from the date of the first poll cited to the date of publication. Again, striking differences appear. For Public Opinion Quarterly, the first poll in primary survey research was conducted only about four years prior to journal publication--a very short time indeed, given delays in article writing, submission, and printing. For secondary research the first poll cited averaged eighteen years prior to publication (excluding "The Polls"). For other social science journals, the difference was an eight-year lag (for primary research) versus twenty-one years (for secondary research).

(Where more than one survey was reported, the lag in Public Opinion Quarterly from the first to the last survey averaged 1.6 years for primary survey research and 16.5 years for secondary survey research. In the remaining social science journals, the lag from the first to the last survey averaged 2.1 years for primary survey research and 9.4 years for secondary survey research.)