For many years I was an advocate of more severe performance standards. I wanted a code that would define the parameters of a scientific poll and standards that would prohibit biased techniques. These standards would allow us to answer a journalist’s inquiry about the quality of someone’s poll with a forthright declaration. At last we could say: “That poll was not performed in a manner that meets AAPOR’s standards.” If we have criteria for evaluating the quality of a poll, then the press and other consumers of public opinion research would be able to distinguish research of high quality from survey salesmanship.

With great reluctance, I have changed my mind. As much as I believe that in most cases it is possible to write performance standards for specific survey research projects, I no longer believe that it is possible to codify performance criteria that are general enough for most needs.

For example, in my own work at CBS News, under most circumstances I could not imagine sampling precincts for the purpose of making an election projection by any means other than random selection of precincts where the probabilities are used in the estimator. This was the method used when we selected a sample in the Philippines for the Marcos election in 1986, just as it has been for every state sample in the United States that we have selected during the last 22 years.

However, had I been in Panama two weeks ago, I could have envisioned a circumstance where probability sampling would not have been feasible, but a quota sampling of voting precincts would have made sense. I could have had very useful information about fraud in the election even with a purposive sample. I could have obtained the vote counts tabulated by the local election officials and compared them to the counts released by General Noriega’s government. I would not have been able to quantify the election result or make an estimate of the magnitude of the fraud, nor could I have specified a sampling error for the data I did have. But I could have shown a systematic distortion of the counting in the precincts I selected. If most differences had favored the Noriega-backed candidate, it would seem reasonable to conclude that there had been election fraud.

A different example of an unscientific but informative exit poll occurred in Moscow a few weeks ago. The magnitude of the answers to some of the opinion questions showed a level of feeling that was a clear indication of strong voter sentiment. This conclusion was responsible even though there was no measure of precision possible for the results. Moreover, the fact that Muscovites answered the questions was as important as any substantive finding.

What I have tried to do with these slightly atypical examples is make it clear that different surveys have different purposes. Defining standard methodological practices when the purpose of the survey in unknown does not seem practical. Some surveys are conducted under circumstances that make probability methods impossible. These special circumstances require caution against unjustified or unwarranted conclusions, but frequently legitimate conclusions are possible and sometimes those conclusions are important.

Several years ago, an American Statistical Association review of survey practices by Bailar and Lanphier (1978) contained many useful conclusions. It was based on a survey of surveys and was about the way they were conducted. It identified both good and bad survey practices. One of the conclusions in the report was that all surveys should be based on probability selection methods. It said this even though the surveys examined for the purpose of making this recommendation did not constitute a probability sample of all surveys.

Now I am not suggesting some rationalization for conducting nonprobability surveys for most of the situations public pollsters deal with. If CBS News or some other organization were conducting a national survey of public opinion in the United States, and there was a reasonable amount of time to conduct the survey, and the results were to be reported as representing the views of all adult Americans, I still think a probability selection at all stages of the design is the only reasonable practice.

But I am now willing to accept alternative methods, methods that I personally would not use under the circumstances I just described. Even methods that many people may consider biased. I think it is reasonable for a researcher to conduct his or her research using any design that fits the problem, provided there is proper disclosure of their methods and its limitations.

But now we have come to the catch. To do this, I believe we need to change the disclosure requirements contained in the AAPOR Code. The code is wholly inadequate as it is now written. Let me explain.

The preamble to the code says in part: “Our goals are to support sound and ethical practice in the conduct of public opinion research.” I emphasize the word “sound.” It then says in the code under “Principles and Professional Practice in the Conduct of Our Work,” “we shall exercise due care” in our designs so we can “assure the reliability and validity of results.” The word I want to highlight here is “validity.” The code goes on to say, we will not produce “misleading conclusions,” or “knowingly make interpretations of research results... that are inconsistent with the data available.”

In section three, the code calls only for minimal disclosure. It asks for identification of the sponsor, question wording, dates and method of interviewing, and details about the sample design and its precision.

My interpretation of these words, taken as a whole, is that the code, including the preamble, is ambiguous. It is quite rigid in its insistence on the use of sound scientific methods and that we not mislead the public. I assume that means we should not mislead each other also. But then it is quite weak when it asks for only minimal disclosure. It implies that a researcher can be relieved of the burden of sound scientific methods as long as there is disclosure conforming to the minimal requirements of the code.

It is not satisfactory, in my view, for the researcher just to disclose the items that are called for in AAPOR’s “Standard for Minimal Disclosure.” That information is not informative enough for most consumers of survey research, certainly not for most members of the news media and other members of the public. It is barely useful to trained survey researchers. What we have done with our minimal disclosure is to place the burden on our users for making sense of the limitations of our surveys. We have provided users with a few technical details called for by the code and then we tell them either to hire a professor to interpret these details or get trained themselves in survey research.

That is wrong. It is the responsibility of the researcher to spell out the limitations of his or her own survey when there are limitations. That is the clear implication of the section on principles of professional practice. If the disclosure section were consistent with the requirements contained in the section on the principles of professional practice, we would disclose much more than is presently called for in the disclosure section.