Today's Non-Useful Non-Data: Third Way and the Case of the Bogus Focus Groups

Our strenuously moderate friends at Third Way made the mistake of inviting the very astute political reporter Molly Ball to accompany them on one leg of their listening tour of the Real America--basically a bunch of meet-ups and focus groups convened in swingish areas around the country. Third Way released a report on that leg of the trip--to Wisconsin's 3rd Congressional District--and Ball has written a devastating piece comparing her experience on that trip to what the Third Way folks wrote up in their report. Short version: "What they wrote up sure ain't what I saw and heard!"

The report is short, covering only three big takeaways from the seven listening sessions Third Way conducted. The first is the importance of hard work; the second is the need for a strong workforce. The third, described in a section entitled “Just Get the Hell Out of My Way,” is locals’ purported antagonism to big government. “Whether the question is about immigration or banks, taxes or welfare, the people we spoke to generally felt that government policies were irrelevant to their daily lives,” it states. This view is made to sound like one that was broadly expressed, but in fact, we mostly heard it in just one session—the group of curmudgeonly farmers. Almost all of the quotations in this section are drawn from that group. There are no quotations from the people we met who were pro-government, such as the teachers and laborers and activists, who voiced concern that local, state, and federal government ought to be doing more to take care of people.

According to the report, the community’s “biggest frustrations” are “laggard government and partisan squabbling.” “The idea that such bickering can be tolerated in D.C. is appalling to most,” it states. The good people of western Wisconsin, Third Way found, wanted nothing so much as a society where people could put aside their differences. The report quotes a man who said, “We come together on projects and solve problems together.” It doesn’t quote any of the Wisconsinites we met who expressed partisan sentiments or questioned the prospect of consensus.

The researchers had somehow found their premise perfectly illustrated. Their journey to Trump’s America had done nothing to unsettle their preconceptions.

I'm not generally a big fan of focus groups precisely because they are so susceptible to observer bias, leading the witness and cherry-picking. But even by the modest standards of focus group practitioners, this is pretty egregious. And there's more. As David Atkins reports over at Washington Monthly's excellent Political Animal blog, their whole methodological approach was suspect from the get-go and violated basic, simple standards people typically use to conduct this kind of research. 

1) Never reveal the sponsor of the research. Third Way’s recruiters apparently told respondents not just during the groups but apparently in the screening process, which means that respondents were at liberty to research the group, its motives and its funding in advance. The potential for problems is enormous: respondents will game their answers to fit the organization’s interests, or conversely come in hostile to the organization. 

2) Don’t have someone internal to the organization actually collect and analyze the data. Increasingly, organizations are ignoring this for cost-saving reasons, but best practice is to have an independent firm handle data collection and reporting so that it’s clear that the organization’s own internal pressures aren’t biasing the data or the conclusions. From Ms. Ball’s reporting, it seems that Third Way staff designed, executed, collected and analyzed the findings internally. That might be marginally acceptable if you’re quickly trying to figure out which car ad or refrigerator features to go with, but not for a $20 million study to set the direction of one of the world’s two most powerful political parties after a devastating defeat.

3) Do not bias respondents toward your objectives at the beginning of the groups. This is by far the worst of the sins apparently committed by Third Way. It’s so egregious that one is left to wonder whether the malpractice was intentional or simply incompetent. Let’s review how the groups began:

Hale or her colleague Luke Watson, Third Way’s deputy director of strategy, began each Wisconsin focus group with a variation on the same refrain.

“We are a think tank that deals with what the plurality of Americans are thinking about—in other words, we don’t spend a lot of time on the ideological edges,” one of the two would explain. “It has started seeming like the far left and the far right were the only voice in America, but we know that’s not true. We focus on the 70 percent in the middle, because we think most of us, as Americans, are there.”

Even for a non-research professional, the red flags here should be obvious. Moderators have enormous power of persuasion in a focus group room, and the psychological tendency of respondents in a group setting is to adapt their answers to the moderator’s suggestions. Moderators need to be extremely careful in how they phrase even the innocuous questions to avoid pushing respondents toward one direction or another. They certainly should not be telling respondents what the organization’s own beliefs are right out of the gate.

Yikes! Well, as they say, garbage in, garbage out.