Perhaps the simplest remedy is to identify these possible sources of error. A transparency approach is fairly minimal and might involve a requirement that any major concern be identified in the first table or figure or text in which it appears, or perhaps in a reference or appendix. This approach might be the more feasible if recommended by professional associations like the American Association for Public Opinion Research (AAPOR) and would of necessity be imposed by editors and eventually insisted upon by reviewers, possibly following guidelines like those issued by AAPOR for response rate calculations, or Standards and Best Practices.

In the absence of uniform standards, editors might do well to send article submissions based on multiple secondary surveys to reviewers familiar with these issues. Transparency would at least alert readers and reviewers to the possibility that measurement artifacts are at hand, while requiring a minimum of convention paper, journal, report, or book space.

A second approach would be to control for measurement artifacts statistically. Two-survey comparisons would not benefit from this approach, since the effect of measurement artifacts could not be separated from time, events, or site. However, it is feasible when numerous surveys have been combined in pooled, cross-sectional surveys or meta-analyses, when large samples per survey are available, or when numerous surveys are combined in a database, for example in voter studies or cross-national surveys or lengthy over-time comparisons. A vexing issue is the growing number of studies that combine numerous surveys into indices without controlling for measurement artifacts, and the subsequent use of these databases or indices in other research as independent or dependent variables. In such instances, question wording changes, interview mode switches, and house effects could be included as dummy variables and examined for their statistical effects. Controlling for measurement artifacts would allow researchers (or readers) to examine their possible effects.

Secondary survey research based on surveys collected at other times by other authors is now common, and will likely become more common in the future. To some degree it may be disappointing that these issues were widely recognized a quarter-century ago (for instance, in a 1983 study by Elizabeth Martin) but remain largely unaddressed even today. A set of standards for secondary survey research based on transparency and statistical controls would help to avoid many of them. At the very least, the findings reported here call for closer scrutiny.

Thomas R. Marshall is a professor of political science at the University of Texas at Arlington.