Survey Do’s and Survey Don’ts

By Steve Farkas

Ohio has long been an epicenter for conflict around education reform—be it schemes to “fix” school funding, efforts to expand school choice, or attempts to ratchet up the level of rigor in classrooms. Such conflict is reflected in the sheer number of surveys of Ohioans popping up lately, some of which report different, even contradictory, results. For rank and file Ohioans, inconsistent reports of what they “favor” or “oppose” may simply breed confusion and further pessimism about the chances of substantially improving education in the Buckeye State. Yet it is possible to draw some commonsense conclusions—and precautions—about where the Ohio public really stands, developing in the process some do’s and don’ts that can help us make sense not only of all those Ohio data, but of many other polling results on the subject of education in America.

Don’t believe surveys that show Ohioans are critical only of other people’s schools, not their own.

Buckeye residents don’t think their local public schools are doing a great job. In a 2007 survey conducted by the FDR Group for the Fordham Foundation, more than four in ten Ohioans (43 percent) thought that a high school diploma from their local public schools was no guarantee that the typical student had learned the basics. Only 41 percent of Ohio parents said they would send their own children to traditional public schools if money were not an issue. These ratings point to too many unhappy customers.

Be skeptical of surveys that show Ohioans are willing to pay more in tax money for the schools.

Ohioans are polite, and they value education. So it’s hard for them to disagree when nice interviewers ask about spending more for something as important as the public schools. But if they believe the schools are educating ineffectively and spending their money ineffectively, do they really mean it? Respondents to the survey thought—by a 71 percent to 21 percent margin—that if more money were spent on the public schools, “the money would actually get lost along the way” rather than “get to the classroom and improve education.” And recently, nearly 70 percent of all new school operating and capital levies on their state’s ballot were voted down by the Ohio electorate. Since few educators or leaders bother to update voters on what happens to their money when they’re not asking for it, citizens are left wondering what the schools are doing with the taxes they’ve been paying all these years.

Don’t expect surveys to resolve the debate over reform of school system funding.

Most Ohioans are not closely monitoring the minutiae of new funding schemes and haven’t thought about how to restructure the school financing system. But the public does have some convictions that are relevant to the debate. In our survey, for example, 43 percent said it was the school districts themselves—not the state board of education (named by 18 percent)—whom they “trust most to make decisions about how to spend tax money allocated to Ohio’s public schools.” This is not an ideological stance, but a pragmatic calculation that the closer decision-making is to the playing field the better the decision. And in focus groups people clearly evinced another conviction: a desire to help the schools of poorer communities without preventing wealthier communities from spending more on theirs.

Be aware of surface support for a proposal when the public has not given much thought to an issue.

Lacking any independent knowledge, people try their best to answer by looking to the question for clues. Thus, much will depend upon survey question wording: what language is used and whether it is even understandable, which concepts and values are highlighted, which players are associated with the proposal. In short, respondents are going on instinct. For example, in our survey, 63 percent of Ohioans supported a concept called “weighted-school funding,” described in the question as “a proposal to make the amount of money that Ohio spends on children’s education differ according to each child’s individual needs and special circumstances, and to have all of that money follow children to the schools they attend.” It’s a complicated thought and a new one for the vast majority of people to consider. The fact that most favored it means that people liked what they heard in the question, and that there was initial support worth exploring—but not that the issue was resolved in their minds. 

Carefully scrutinize polls that show the public abandoning a policy they previously supported.

Once they commit to a policy, people just don’t change their minds easily or capriciously, and they often have a lot more patience than leaders and elites. Instead of drawing conclusions from a particular poll, pay attention to trend data—that is, results from survey questions that are repeats of previously asked questions. For example, the FDR/Fordham survey showed that support for charter schools was stable, with 52 percent in favor of them in 2007 and 51 percent in favor in 2005. When substantial shifts in opinion occur, it’s typically in response to something real that’s occurred in the world.

Finally, be suspicious of surveys that give the public a chance to reach for simplistic solutions to funding dilemmas or a chance to reach for the proverbial free lunch.

Taxpayers will, of course, like proposals that offer to cap their property taxes, to exempt whole groups from levies, to take money from magical places like tobacco settlements. A survey may skip the part where the public has to wrestle with policy tradeoffs and costs; responsible elected officials will not be able to.

Many observers believe that the public is easy to manipulate. As it turns out, it’s opinion polls that are easy to manipulate.

Steve Farkas is president of the FDR Group.

Readers who wish to respond to this or other articles appearing in Public Opinion Pros, or to contribute commentary of their own in 800 words or less, should consult our author submission guidelines and editorial policies under "Letters to the editor and op-ed articles." Op-ed submissions must be received at least two weeks before the first of the month for the issue in which they are to appear.