Having said this, I want to propose a more meaningful set of disclosure principles. My interpretation of proper disclosure would still permit a researcher to tailor a survey to the purpose at hand. It also would obviate the need for performance standards. The disclosure principles I am about to propose would shed light on the interpretation of the results. It would not permit surveys to be cloaked in the pseudoscientific aura bestowed by following the present disclosure standards. And finally, I believe this proposal is more consistent with the purposes spelled out in the preamble and the professional practices section of the AAPOR Code.

Along with reporting the items contained in the minimal disclosure section, the researcher should go further. If there are biases in the research design—for good reason or not—the researcher conducting the study will be the one to make these design flaws and their consequences known to the consumer of the research. This to me is proper disclosure. I would like to repeat what I just said: If there are biases in the research design—for good reason or not—the researcher conducting the study will be the one to make these design flaws and their consequences known to the consumer of the research.

If there is a sample selection bias it should be disclosed. If there is a weighting bias it should be disclosed. If there is a question order effect known to the researcher it should be disclosed. And further more, the limitation of any conclusions should be spelled out if there are such biases.

It is absolutely the obligation of the researcher to spell out the limitations of his or her survey. Let me be even clearer about what I want to see disclosed. When I say limitations, I want to know how the design affects conclusions based on that survey. Specific statements about what can and cannot be concluded should be part of the disclosure.

For example, under my proposal, nonprobability samples are all right, provided the bias of the method is properly highlighted and there are proper cautions against generalizations from the sample to some larger population. If there are no call-backs or if only respondents at home are interviewed, the limitations they impose on the interpretation of these results should be made clear.

If weighting for the probabilities of selecting one adult in a household is not part of the estimation procedure then I want to be told in the disclosure statement about any relationship between household size and characteristics correlated with this variable. I should also be told that people in households with more than one adult are underrepresented due to the lack of weighting for this selection probability.

If the sampling error is not reported, then the disclosure statement should have different information. It should say that the sampling error is not reported either because the design is not random at all stages and therefore a sampling error computation is inappropriate, or the researcher thinks reporting sampling error would be misleading, given all the other possible sources of survey error. Either would be all right, provided the disclosure statement made it clear that there can be no justifiable conclusions drawn from the survey about any change over time in any characteristic obtained in the survey, or that there could be no meaningful interpretation about the differences between two subgroups in the same survey on some opinion item.

If there are question biases I want to know what they are. In surveys about racial attitudes, I want to know whether there is an interaction between the race of interviewers and respondents and whether the conclusions are affected by this interaction.

To me, this type of disclosure would come closer to “truth in packaging” of survey research. What we have now does not do it. This type of disclosure would reduce the tension about methods. I want to conclude by offering another proposal that could reduce tensions and shed light on the various reasons for using alternative survey methods. Rather than ignoring our differences, or having one-sided discussions about them, we should bring them into an open forum for public discussion. My proposal is for a series of debates about methods to take place at our annual conferences. Last year at AAPOR, we had a public discussion about the reporting of sampling error when a survey is made public. It was a useful beginning. However, I would like to take it a step or two further.

These debates should have formal rules. Both sides should be prepared to produce a written report of the arguments and counter arguments. These reports could be published in Public Opinion Quarterly or issued as separate papers by AAPOR. I can imagine lively discussions, useful to the participants, to AAPOR members, and perhaps even beneficial to the consumers of our work. I would even hope for the amelioration of the issues. Perhaps, the reduction of tensions.

I offer these thoughts today in the hope of improving the quality of work produced by members of our association. In recent years the credibility of survey research has been attacked. Most of the work by members of this association has been of high quality, professionally conducted and accurately reported. But just often enough, the caliber of the surveys reported to the public is suspect. I know raising standards involves increased cost, but I believe we must pay the price. If polls are a public utility, as Julian Woodward said, then we owe it to the public as well as ourselves to raise the level of our performance. And we must do it through our own initiative, and not because of pressure from outside.

"Presidential Address: Methods and Standards: A Challenge for Change," by Warren J. Mitofsky, appeared in Proceedings of the Forty-Fourth Annual Conference of the American Association for Public Opinion Research, in Public Opinion Quarterly 53 (Autumn 1989):446-53. Reprinted by permission of Oxford Journals, Oxford University Press.