DJ Drummond of the blog Stolen Thunder has an interesting piece on polls.

As I was riding the Park n Ride to work one morning last week, the bus passed a fortune-teller’s store, which had gone out of business. The place had been doing poorly for a very long time. As I pondered the painted word ‘psychic’ on the side of the building, I wondered why the medium did not know this was going to happen. For all of that, there are many fortune-tellers who make good money, telling folks stories which sound good, which is what the customers really want.

[...]

The NCPP has published a guide for journalists to use when discussing the report from an opinion poll. Some of the questions they say a journalist should ask include

Who did the poll?
Who paid for the poll and why was it done?
How many people were interviewed for the poll? (This, by the way, is why the state polls are less statistically valid than the nation polls. They usually involve a much smaller poll of respondents than the national polls use)
How were the people chosen to take part in the poll?
What area were those people from?
What was the response rate to contact attempts? (almost no one answers that one)
How does the poll describe the results?
What questions were asked, and how were they worded?
What order was used for questions? (this is an old, old trick – if a controversial issue is asked just before a certain candidate is named, often this creates a false connection between the candidate and the controversy in the mind of respondents)
What events are likely to have impacted the poll results?

The plain fact is, that almost no media outlet or polling group reveals the answers to all of these kinds of questions. And that damages their credibility as valid reflections of the public’s opinion.

The AAPOR warns that polls almost always fall into one of seven categories of sponsorship:

1. Academic institutions
2. Federal, State and Local Governments
3. Media Organizations
4. Non-profit groups or Foundations
5. Special interest groups
6. Businesses and Corporations
7. Political campaigns, consultants, and candidates

They also warn about many of the same problems in poll reporting as the NCPP, and the AAPOR goes further to warn that many of these sponsors may be biased, even when they claim to be objective. The first question, in any poll result, is ‘who benefits’?

This brings me to Rasmussen. In an earlier article, I rather harshly said that Rasmussen was in it for the money. I have always asked the questions recommended by the NPCC/AAPOR, and in doing so I have noticed how some major polls refuse to answer important questions. I learned a long time ago that if you had enough information, you could work out the mechanics of most major conclusions. I found Gallup, AP-Ipsos, and CBS News very open about their methodology and the internal data of their polls. But I have always been disappointed in the secretive and sometimes disingenuous behavior of Zogby, NBC, ABC, and Rasmussen. As it happens, Scott Rasmussen has enjoyed a lot of success at his business, but if you want internals it will cost you, and if you want to reverse-engineer his polls to test for bias, he simply won’t allow you a way to have all the information needed. That, again being blunt, strikes me as both mercenary and essentially dishonest. And therefore, no matter what his polls say I will always see a little red flag next to his published claims. My rule of thumb is very simple – it does not matter whether the poll says what I’d like to hear, it matters whether all the supporting data is available.

Another thing that is troubling, is the desire for instant gratification. For example, Sarah Palin clearly beat Joe Biden in the debate they had last week (although ardent democrats are just as sure Biden triumphed), and so there are people on both sides looking at the current polls for signs of their guys big win. But there are two reasons why you will not see that, at least right now. First, even the invalid daily tracking polls take their results from three days of polling, which means the first poll with full post-debate reaction will not be released until Monday. And I have to warn you that the polls will not reflect an immediate valid effect – if they show a sudden jump for either side it is far more likely to mean that the polls have – once again – changed the party affiliation weights, a practice as dishonest as putting one foot on the floor while you weigh yourself. The reason is because while people do change their minds, historically it has always been a few people at a time and in small steps. One proof of that is that as popular as Presidents Reagan and Clinton were, neither saw a major shift in party identification during his Administration. People supported the man, not necessarily the party, and in this election it would foolish indeed to imagine that people who supported Biden were also becoming more pro-democrat than before, or that those who supported Palin were becoming more pro-republican than before. After all, the biggest statistical group of undecideds consider themselves to be independents, and while they will come over to one candidate or the other, that does not lock them in with the whole party. Party identification should be a static measure for the duration of a campaign, and manipulating party affiliation is simply cheating the results to create a story out of imagination.

And, from PollingReport.com regarding online polls.

What Is “Scientific”?
The belief that traditional telephone polls are “scientific” depends on your definition of science

AWB

If you find this article informative? Consider donating any amount you choose.

No related posts.

This entry was posted on Monday, October 6th, 2008 at 12:05 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.