About Rationally Speaking

Rationally Speaking is a blog maintained by Prof. Massimo Pigliucci, a philosopher at the City University of New York. The blog reflects the Enlightenment figure Marquis de Condorcet's idea of what a public intellectual (yes, we know, that's such a bad word) ought to be: someone who devotes himself to "the tracking down of prejudices in the hiding places where priests, the schools, the government, and all long-established institutions had gathered and protected them." You're welcome. Please notice that the contents of this blog can be reprinted under the standard Creative Commons license.

Monday, January 14, 2008

On polling, New Hampshire, and all that jazz

It is often amusing to follow debates in the media, especially on the rare occasions in which it criticizes itself. But few controversies in recent memory rise to the level of ridiculousness of the frenzy about “what went wrong” with voter polling before the recent Democratic primary in New Hampshire.

Nothing went wrong with the polling. Polling is a statistical tool, and as such it is neither infallible (that’s what it means when statisticians talk about the “probability” of a given outcome), nor can it be expected to be unaffected by all-too human phenomena, like changing one’s mind at the last minute, or simply lying to the pollsters. Both of which clearly happened in NH -- and will likely happen again, possibly during this very same Presidential campaign.

First, changing one’s mind. Voters are fully entitled to do it any time they want, including after they walk into the voting booth (as long as it is before they cast their vote). New Hampshire is well known as a state famous for sporting a lot of independent-minded voters, sometimes to the point of childishness, as in “you tell us to do X? We’ll do Y, even though X would clearly be better for us. There.” (It is also known for the ridiculous motto “Live Free or Die,” but that’s another story.) Moreover, Hillary Clinton had her now well known “emotional moment” just the day before the primary. Much speculation has been done on whether Clinton’s (very mild) choking up in response to a voter’s question about how she can manage the harshness of the campaign was genuine or calculated. It doesn’t matter, it clearly affected a significant number of voters -- especially women -- in New Hampshire, and probably contributed to the discrepancy between the poll’s predictions and the actual outcome of the vote.

Second, lying. It is well known that a recurring problem for every social science researcher (and pollsters are applied social scientists) is that the subjects -- unlike most other animals and all plants and bacteria -- are characterized by (more or less) rational thinking, including the ability to figure out what the social scientist is trying to do and purposefully sending said scientists off track. In polling, this is called the “Bradley effect,” after the 1982 mayoral campaign in Los Angeles, which was narrowly lost by Tom Bradley (a black candidate) to George Deukmejian (white), despite pre-election polls giving a significant margin to Bradley. What happened then, and likely what happened this year in NH, is that a good number of white voters felt uneasy at telling pollsters that they were not going to vote for Obama, fearing being labeled as racists. So they said instead that they were either going to vote for him, or that they were undecided. Once in the voting booth, however, their true color, so to speak, translated into a vote for Hillary. This may be irritating behavior on the part of said voters, but there is no law that says that one should answer truthfully to a pollster.

Moreover, as I mentioned earlier, polls are statistical tools. They are based on a complex sampling procedure, with a small number of people included in the sample. Many assumptions go into shaping that procedure, and failure of any of these assumptions -- like the actual representativeness of the sample in terms of gender, socio-economic layer, race, and a good number of other parameters -- may result in an inaccurate prediction. But notice that pollsters have very, very often gotten it right, and that a certain number of failures is in fact predicted by statistical theory.

Now for the blaming game. It is amusing to hear the media “apologizing” for having gotten the predictions wrong, not to mention blaming their own pundits and calling for more “humility” in the future. Debates are even taking place on whether pre-election polls should be banned altogether, on the ground that they affect the election’s outcome.

OK, to begin with, one ought to make a distinction between pollsters’ predictions and punditry. As much as I despise most (but not all) of the “talking heads” who freely opinionate on radio and TV about everything and everybody (just like I do on this blog!), there is no reason to expect that someone’s opinions be correct or predictive. They are, after all, opinions. So, for once, no reason to pour the ashes on the pundits’ heads (I’ll gladly make an exception for Rush Limbaugh and Ann Coulter, though, just because). Furthermore, the idea that social scientists (the pollsters) ought to be humbled because they got it wrong this time is based on a fundamental misconception about science. Science is not about absolute and infallible truths, so when a scientist gets something wrong, she simply takes note, tries to understand why, and attempts to do better the next time. There is no shame in getting it wrong, because science -- especially social science -- can make progress only by trial and error. Doing a “scientific” poll is not synonymous with reading the word of God directly from an infallible Holy Book.

Finally, banning polls on the ground that they affect the election is nonsense on stilts. Every comment in the media, every column in the newspapers, every post on a blog may affect the outcome of an election. And so it should be, if we are to live in a democracy. Polling means that voters have an idea of what others think before a single vote is cast. That is valuable information that may or may not be used by the voter, at her discretion. For instance, during the 2000 elections, many people voted for Ralph Nader, the independent candidate, only in states were they knew (because of polls) that Gore would easily win over Bush. They wanted to send a message to the Democratic party, without risking giving the country to a nutcase. Unfortunately, it didn’t work out that way, but that’s also another story.


  1. fyi -- My organization, the American Association for Public Opinion Research (AAPOR) announced today an ad hoc committee to evaluate the methodologies of the pre-primary polls and to archive the data -- announcement is here: https://www.aapor.org/aaporannouncesadhoccommitteetoevaluatenewhampshireprimarypolls

  2. I have decided to become a post election analyst. I feel that I am eminently qualified due to my past experience as a Coastguardsman and most recently, a retired truck driver.

    Here's my take on how support is shaping up for the Democratic candidates: Obama is winning among white voters with wine birthmarks on their left buttocks, rural Asian voters with two hole out houses, and fundamentalist snake handlers who have been bitten twice on the ass and survived (There seems to be a synergy between these and the birthmark folks - has something to do with their butts I suppose). In the meantime Hillary is ahead among redheaded African American gynecologists, gay truck drivers and bisexual Marine Corps NCO's, as well as cross dressing smoke-jumpers fighting based at McMurdo Station, Antarctica.

  3. To see some of the extensive evidence that George Bush and Karl Rove stole the election of 2004 take a look at THE ELECTION JUSTICE CENTER at the Solar Bus web site. They have articles,analysis,updates, and original reporting on what really went on. Or read Mark Crispin Miller's feature NONE DARE CALL IT STOLEN on the HARPERS web zine. Or just read the Wikipadia entry. IT'S A SLAM DUNK BUSHIE!

  4. I have heard a lot of explanations of why the polls were wrong. But few realize another possibility is that the polls were correct.

    And if you want to know what the Rational way to look at the situation contrasted to the irrational way, read Brad Friedman's latest.


  5. My original thought on the disparity between the pre-election polls and the final results was that it was what I'll call the "Avis syndrome". In other words, the supporters of the second place candidate (Clinton) "Tried Harder". They were motivated by reports that they were losing ground. At the same time, the supporters of the leading candidate were lulled into a (false) sense of security. Especially those who were casual supporters. "Hey, it's cold outside, I have to spend two boring hours at the caucus, my guy's going to win anyway and my favorite show is on TV. I guess I'll just stay home."
    Of course this is speculation on my part but if true means that the polls have an effect on the results.
    Now comes the alternate speculation that the polls were correct, the voting went along the lines of the polls, but the votes were miscounted. Maybe. As the Brad Blog says, you won't know unless you count.

  6. I keep thinking about H. Clinton's "choke up" factor, and it kind of sounds like a bad one. Did anybody ask why people voted for her (or not)? Would people even recognize why they did, or didn't?

    If she had been beaten there, the meedja would be saying that her acting (or weakness) backfired, and that Americans want a strong President in times of war, blah, blah... And it will all make sense also.

    Maybe it's just too early to tell...

    Oh, and since the US president does whatever he (she soon?) pleases in the world, I guess we all should be allowed to have a vote in deciding who's gonna be shafting us next. :O)

  7. j wrote: " I guess we all should be allowed to have a vote in deciding who's gonna be shafting us next."

    You could do it the American way... invade, and depose the current head of government. Although I might warn you, the outcome isn't always quite what you expect. :-) There's something about a 'law of unintended consequences'.

  8. And more on topic...

    The Florida primaries are coming up. I won't be able to vote since only Dems and Reps are included and you have to vote within your declared party. Of course Florida is being punished by the major parties for moving its primary date so campaigning here has been at a minimum. My informal observations are:
    In the area of public displays/support - I have seen hundreds of Ron Paul signs in yards, along streets, on bumpers. I have seen one Mitt Romney sticker and two Kucinich stickers. That's it.
    In TV advertising I have seen several Giuliani spots (highlighting the Islamic terrorism threat) and nothing else. You wouldn't know that the primary is 14 days away and if you didn't watch or read national news you wouldn't know who was running.

    As for the N.H. results, I'm hard pressed to think that Clinton's human emotion moment created that big a difference. I'm more inclined to attribute it to my "Avis Syndrome" or to the "subconcious racism" hypothesis.

  9. On the ill sought pursuit of charisma and elect ability

    If the maj of the populace would vote for what is not just necessarily pleasing to themselves or what they happen to think constitutes elect-ability, either party could have what might be a generally decent leader for our nation. My picks would be Bill Richardson or Fred Thompson. For their respective parties they both seem to be stable and experienced politicians. But who cares, right? That is not how this game is qualifying "successful" at this point.

    "Winning" for either party is currently the only goal. And that REALLY is bad for whatever political persuasion you might be. With this in mind, all either party will win essentially is a figurehead, probably not overly endowed with brains or guts.

    Americans, along with a lot of the rest of the world, are becoming empty headed little things anymore. And likely, we will get the candidate not only of our supposed choosing, but most certainly the one we deserve.

    Honestly now, are any one of us as Pure, Transparent and Just as we would want the ideal candidate to be...or do we just want to win?
    That's simply not good enough, you know.

  10. The use of polling has gone beyond the statistical value of what information it contains. It is troubling to see how the polling results generally show errors of +/- 3% to +/- 5%. These numbers suggest the assumption of unrestricted random sampling, even when that clearly has not been done. The continued use of phone calls selected by a computer has introduced bias in the samples because of Caller ID/blocking, and multiple phones (I have three, one of which isn't even in my home town). Even worse is the presumption (by Zogby International) that using a database of interested-participants for internet surveys can be anything but a faulty sample. The way the news media cite defective polling is manipulative. Ignorance of statistics is not an acceptable excuse. We have problems enough with communication without formalized deception.


Note: Only a member of this blog may post a comment.