On polling, New Hampshire, and all that jazz

It is often amusing to follow debates in the media, especially on the rare occasions in which it criticizes itself. But few controversies in recent memory rise to the level of ridiculousness of the frenzy about “what went wrong” with voter polling before the recent Democratic primary in New Hampshire.

Nothing went wrong with the polling. Polling is a statistical tool, and as such it is neither infallible (that’s what it means when statisticians talk about the “probability” of a given outcome), nor can it be expected to be unaffected by all-too human phenomena, like changing one’s mind at the last minute, or simply lying to the pollsters. Both of which clearly happened in NH -- and will likely happen again, possibly during this very same Presidential campaign.

First, changing one’s mind. Voters are fully entitled to do it any time they want, including after they walk into the voting booth (as long as it is before they cast their vote). New Hampshire is well known as a state famous for sporting a lot of independent-minded voters, sometimes to the point of childishness, as in “you tell us to do X? We’ll do Y, even though X would clearly be better for us. There.” (It is also known for the ridiculous motto “Live Free or Die,” but that’s another story.) Moreover, Hillary Clinton had her now well known “emotional moment” just the day before the primary. Much speculation has been done on whether Clinton’s (very mild) choking up in response to a voter’s question about how she can manage the harshness of the campaign was genuine or calculated. It doesn’t matter, it clearly affected a significant number of voters -- especially women -- in New Hampshire, and probably contributed to the discrepancy between the poll’s predictions and the actual outcome of the vote.

Second, lying. It is well known that a recurring problem for every social science researcher (and pollsters are applied social scientists) is that the subjects -- unlike most other animals and all plants and bacteria -- are characterized by (more or less) rational thinking, including the ability to figure out what the social scientist is trying to do and purposefully sending said scientists off track. In polling, this is called the “Bradley effect,” after the 1982 mayoral campaign in Los Angeles, which was narrowly lost by Tom Bradley (a black candidate) to George Deukmejian (white), despite pre-election polls giving a significant margin to Bradley. What happened then, and likely what happened this year in NH, is that a good number of white voters felt uneasy at telling pollsters that they were not going to vote for Obama, fearing being labeled as racists. So they said instead that they were either going to vote for him, or that they were undecided. Once in the voting booth, however, their true color, so to speak, translated into a vote for Hillary. This may be irritating behavior on the part of said voters, but there is no law that says that one should answer truthfully to a pollster.

Moreover, as I mentioned earlier, polls are statistical tools. They are based on a complex sampling procedure, with a small number of people included in the sample. Many assumptions go into shaping that procedure, and failure of any of these assumptions -- like the actual representativeness of the sample in terms of gender, socio-economic layer, race, and a good number of other parameters -- may result in an inaccurate prediction. But notice that pollsters have very, very often gotten it right, and that a certain number of failures is in fact predicted by statistical theory.

Now for the blaming game. It is amusing to hear the media “apologizing” for having gotten the predictions wrong, not to mention blaming their own pundits and calling for more “humility” in the future. Debates are even taking place on whether pre-election polls should be banned altogether, on the ground that they affect the election’s outcome.

OK, to begin with, one ought to make a distinction between pollsters’ predictions and punditry. As much as I despise most (but not all) of the “talking heads” who freely opinionate on radio and TV about everything and everybody (just like I do on this blog!), there is no reason to expect that someone’s opinions be correct or predictive. They are, after all, opinions. So, for once, no reason to pour the ashes on the pundits’ heads (I’ll gladly make an exception for Rush Limbaugh and Ann Coulter, though, just because). Furthermore, the idea that social scientists (the pollsters) ought to be humbled because they got it wrong this time is based on a fundamental misconception about science. Science is not about absolute and infallible truths, so when a scientist gets something wrong, she simply takes note, tries to understand why, and attempts to do better the next time. There is no shame in getting it wrong, because science -- especially social science -- can make progress only by trial and error. Doing a “scientific” poll is not synonymous with reading the word of God directly from an infallible Holy Book.

Finally, banning polls on the ground that they affect the election is nonsense on stilts. Every comment in the media, every column in the newspapers, every post on a blog may affect the outcome of an election. And so it should be, if we are to live in a democracy. Polling means that voters have an idea of what others think before a single vote is cast. That is valuable information that may or may not be used by the voter, at her discretion. For instance, during the 2000 elections, many people voted for Ralph Nader, the independent candidate, only in states were they knew (because of polls) that Gore would easily win over Bush. They wanted to send a message to the Democratic party, without risking giving the country to a nutcase. Unfortunately, it didn’t work out that way, but that’s also another story.