@sozobe,
A propos the PPP poll about trust in news broadcasters, there's been some critical back and forth since.
Gary Langer from ABC News, which didn't come out very well in the poll,
objected sharply to elements of the poll's methodology. He brings up a number of valid sounding complaints, and some more far-fetched sounding ones, about the poll's selection of respondents and phrasing of questions. He also, however, derides the value of automated polls, which tend to be used by the new kids on the block of polling (like Survey USA, Rasmussen and PPP) altogether. He writes that "in our ABC News polling standards we don’t regard autodialed, pre-recorded polls as valid and reliable survey research."
This had both PPP's
Tom Jensen and
Scott Rasmussen responding in turn.
Both noted that you can go through any poll and find details worth nitpicking over, especially when it comes to the demographics of those found by the poll - random dialling will always lead to some variety in the make-up of those reached, after all. To highlight an example in point, Jensen picks on Langer's complaint that only 47% of the PPP poll's respondents reported having voted for Obama in 2008, when he received 53% of the vote. Considering margins of errors, that does seem a rather silly point. Jensen replies that "we're not going to weight every poll for the next four years to the self reported 2008 vote" and points out that a recent CBS/NYT poll, which does use Langer's preferred live interviewer methodology, had respondents reporting to have voted for Obama by a 23 point margin.
The real heat of both men's responses, however, is targeted at Langer's rejection of automated polling per se. In his dismissal of such polls, Langer also rejected references to how accurate such polls have proven to be in recent elections, positing that "accurate modeling is not an adequate stand-in for good polling [..] what matters are sound methods [and] substantive measurements, not horse-race bingo." This in particular has the automated pollsters fuming, and Jensen's critique in particular is strident in tone. Rasmussen makes roughly the same case, but more articulately and calmly and with some more meat. Their point, basically: what the hell are polls good for if they can not be proven to measure public opinion with some accuracy? Election races provide pollsters with an opportunity to do so, and both pollsters point out that their polling turned out to assess voters' sentiment in Massachusetts this month much better than polls conducted with traditional, live interviewer methods.
The pollsters of ABC and the other major media Langer approvingly cites, meanwhile, chose not to poll the race at all, focusing on measures like job approval that can not be verified immediately by election outcomes, Jensen argues. "One of the ironic things about ABC and similar polling operations is that they rarely poll anything they could be held to public account for. For instance the Massachusetts Senate race was undoubtedly of huge national significance but none of the outfits that pass Langer's litmus test of reliability conducted a pre election poll there that could have shown that they do- or don't- know what they're doing."
The cultural chasm that comes to the front in both Langer's and Jensen's and Rasmussen's posts is enormous, and it is that between old media and new media, between the traditional pollsters and the daring new kids, between blue chip polls that have gone on for decades and experimental polls that aren't as established, but argue that they're actually better at getting the numbers right. Although Rasmussen's post will suggest otherwise, this is not a leftwing MSM versus rightwing outlaws thing. PPP is a Democratic firm, after all, and Survey USA, which also uses automated methodologies and faces the same scepsis among traditional media types, is hardly a conservative pollster either.
It's a pity that all the heat and passion that comes with that chasm - Langer's studied condescension, Jensen's heated derision and Rasmussen's triumphalist rebel shtick - makes it kind of hard to establish your own opinion about the issue without being guided by your gut reaction to their tones. Do you associate more with Jensen and Rasmussen as bold outsiders, loud and aggressive as they may be, or with the earnest Langer, condescensing as he may be taken to be? Hard to ignore your instinctive response to that kind of thing when reading through this stuff (Langer is more my type, obviously), which is a pity because both do have good points. It's always worth digging through the particulars of a poll, especially if it comes up with a surprising or even unlikely result. But it's also true that if a new generation of pollsters has more than held their own in polling actual races for a few years now, the unwillingness of some of the traditional, "old" media to even take their results into consideration or report on them is bullheaded.
I'd say always look at all the polls on an issue, if more are available, don't rely on any particular one or category of pollsters, and don't think any one poll will necessarily have things right. Whether the results of this particular poll can be trusted, meanwhile, has only become less clear for me. Let's see what the next one says. ;-)
