Forgetting People: What’s Wrong with Public Opinion Research Today, and How to Fix It

By Steve Farkas  

Today’s public opinion researchers are grappling with serious challenges facing their profession—declining response rates, cell phones, push polls, inappropriate use of internet surveys. While they are very right to be concerned about threats to quality, their focus is misplaced. The biggest menace facing this industry is not borne of response rates or the challenges of technology. The real problem is with the very quality of our surveys—the questions we ask, how we describe results, and how we allow results to be used.

There is a bitter irony here, because we have more surveys than ever, yet our society’s understanding of the public’s mind on key issues is impoverished. In his 2003 presidential address to the American Association for Public Opinion Research (AAPOR), Mark Shulman called for “a rebalancing of our profession so that we recognize the limitations of ‘response rates’ and other simple measures of survey precision and quality, while, more importantly, placing more weight on our profession’s insights and contributions to social and public policy decision making.” The good news is that the problems—most troubling with respect to surveys that deal with policy issues—are identifiable, and the solutions are within our grasp.

Let us begin with the problems.

Surveys commissioned by public interest groups on policy issues are a particular weak point in the industry.

Nominally nonpartisan, public interest groups commission opinion studies on their respective issues, first because they want to generate news coverage to publicize their cause. Such surveys typically find that the issue that is of uppermost concern to the organization is also uppermost in the public’s mind. What’s more, the surveys show that when it comes to whatever the cause is, the public is not only deeply concerned but is willing to pay more in taxes. When elections are near, the surveys will also show that the public will vote against politicians who don’t take the requisite stand. Added up, all these polls show us that citizens care about everything, will happily pay tax rates that surpass 100 percent, and will vote against all politicians if they fail to take any stand.

Unless we’re ready to say the public is completely irrational, the inescapable conclusion is that something is wrong with what we’re doing. Too often, we don’t ask respondents to establish priorities, we don’t put their attitudes toward one topic in context with others, we don’t get their responses to real tradeoffs in public policy—in short, we give voice to a fantasy public, not a real one. But policymakers need to know just how troubled people really are by a problem, how excited they are by a proposal—indeed, how much they have thought about the proposal. We are getting dangerously close to a point where surveys are merely guaranteed commodities whose results any interest group can purchase to show its cause has the public’s support. 

The industry’s questionnaires have “a give them everything” mentality.

Surveys shy away from forcing people to prioritize their preferences and choose among choices—everything is a problem, anything can be a solution. In the survey world, little attention is paid to the intensity of people’s opinions and sentiments. The “very’s” and “somewhat’s” are very often combined, so the vast majority believes something. There’s a shying away from putting the issue in context—how does federal aid to the arts stack up in priority to education or defense?

In the real world, the one that policymakers have to grapple with, choices have to be made. Not all problems get the same level of attention, not all solutions merit attention, and resources are limited. In the real world, solutions have consequences and tradeoffs, but survey questions rarely raise the consequences of a choice.

The most important distinction survey research routinely fails to make is when the public is engaged and thoughtful enough to have something meaningful to say about solutions—and when it’s not. Some years ago, when welfare reform was preoccupying Washington, D.C., Public Agenda did a study on public attitudes toward the issue. The level of engagement was extremely high, so much so that people were able to trace the structure of what they thought reform ought to look like with impressive detail and thoughtfulness. Certain education issues also reveal this level of consideration. But when it comes to evaluating solutions and proposals for many other specific policy initiatives, our surveys often assume laughably high levels of attention from the public. Dutifully reporting the percentage of the public supporting a solution is not doing our duty when we know—or should know—that the public hasn’t given it much thought.