Do Polls Work?

In an article posted on June 17, 2011 on the iPolitics website, Frank Graves, head of the polling firm EKOS , attempted to explain why his final poll failed so badly in predicting the Conservative majority in the 2011 Federal Election. The EKOS poll predicted the Conservative popular vote as 33.9%, almost 6% lower than its actual popular vote of 39.6% on Election Day. While raising the question of whether polls worked as a barometer of public opinion, he concluded that the poll was in fact “accurate”, but it applied to all eligible voters, not merely those who voted. His lengthy defence of EKOS methodology did little to answer the obvious question of why measure a group from which we know many will not vote, and therefore will not be a good basis for predicting the election outcome.

The point of these election polls as Mr. Graves himself notes, is to predict the winner of the horserace.  That is why the public is interested in reading stories that predict election results. That is why newspapers commission polls and publish stories that predict election results. And that is why pollsters use every trick in the book to make sure their polls are as close as possible to the election results. Nobody (and perhaps this is a failing) is interested in election polls for the purpose of “modeling the population of all eligible voters”. It’s all about bragging rights.

But to be fair to Mr. Graves, it’s not just EKOS that blew it. All the polls did. In spite of methodological differences between different pollsters, in spite of different demographic weighting schemes to ensure coincidence with the actual vote, in spite of different methods to deal with undecided voters, in spite of different techniques to identify the likelihood that a respondent will actually vote, they all blew it. This suggests the polls were missing something big, something happening on a societal scale and not the result of a statistical technicality in how the polls were conducted.

This “something big” is the development of public opinion on party preference, and how opinion polls combined with stories in the media influenced this development. For pollsters, the regrettable reality is that sometimes polls can measure this, and sometimes they cannot. In situations they cannot, it is because the public chooses not to share their opinions with the pollster. If so, there is not much that Mr. Graves, or anyone else for that matter, can do to get a reliable prediction of voters’ intentions. By rejecting the possibility of this situation as “implausible”, Mr. Graves is led to what I believe is an even more implausible conclusion — that his poll was measuring the wrong group.

In his analysis, Mr. Graves dismisses the possibility of polls influencing the election by noting that only 21% of voters indicated that polls had either a “moderate” or “great extent” in influencing their decisions. But surely this understates the problem. Poll results are often embedded in press stories that discuss broader political issues such as differences in party platforms and leadership styles. Polls also influence the writers of the stories, even when they are not referenced directly. Also, some would find the question insulting as it trivializes their decision process which may well be influenced by polls.   A more relevant question would be whether stories in the press about the election affected the public’s final decision on Election Day. There is little doubt that the numbers agreeing on the influence of the press would be substantially higher than 21%.

A sociological theory corroborated experimentally that explains the relationship between the development of public opinion and how that opinion is measured by polls, was propounded by Dr. Elisabeth Noelle-Neumann in her book “The Spiral of Silence: Public Opinion – Our Social Skin “. The theory describes specific situations in which survey respondents are willing to share their opinions as well as situations in which they are not. This is exactly what happened into 2011 Federal Election and has been described in some detail in my earlier post.

Finally, it should be noted that evidence mounted by Mr. Graves from the follow-up survey (after the election) conducted by EKOS was entirely consistent with the spiral of silence theory in which certain segments of the Canadian population revealed their true opinions on party preference, while others did not due to their  concern about social consequences. Specifically, of those who indicated they voted in the May 2 election, 37.9% said they voted Conservative. This figure is much closer to the actual vote of 39.6% compared to the final pre-election poll of 33.9%. Clearly, after the election, the Conservative majority was public knowledge and as predicted by the theory there was little disinclination on the part of those voters who lied from revealing their true party preferences.

Secondly, Mr. Graves notes that 21% of voters admitted to shifting their voting preference from the beginning of the election. That’s about as close as one can hope for in getting voters to admit that they lied in the pre-election polls. The 21% makes the 6% difference in Conservative popular vote between pre-election polls and actual vote quite reasonable. To this point it’s worth noting that the Conservative majority resulted in only a 2% increase in popular vote from the 2008 election when they were handed a minority government.

So we have come full circle to Mr. Graves initial question: Do polls “work”?  If they do work, the process as I have sketched out is far more complex than pollsters would have you believe. Secondly, polling has matured although it must be said, not well.

Long ago the idea of public opinion polling was a noble one, with a generally accepted understanding that this was an instrument to reduce the democratic deficit. Today, like television, radio and other innovative technologies, its practice has brought it into disrepute. Instead of a tool to enhance democracy it has become simply one of an arsenal of propaganda tools used by corporations and governments to promote their interests. These end users of polls know full well how effectively public opinion can be manipulated through the agency of media. Yet somehow neither the polls nor the media seem to be willing to confront this essential truth. By hewing to this fiction they have dangerously narrowed the scope of public debate by allowing polls to misinform the public, often treating ignorance as opinion.

Perhaps in today’s time, the right question to ask is not whether polls work, but who are they working for.

This story was originally posted on the iPolitics site on July 10, 2011

About Oleh Iwanyshyn

Oleh Iwanyshyn has been involved professionally with surveys from the mid-70s when he started as a methodologist at the Institute for Behavioral Research at York University. Later, while at the Canadian Broadcasting Corporation his focus shifted to media and election surveys. He now runs his own survey research company, ViewStats Research, established in 1997 and specializing in online surveys. Public opinion surveys is an essential communication tool between a democratic society and its leaders. Unfortunately, surveys can be very easily manipulated. Revealing such manipulations and their consequences is the raison d'être of the poll stuff blog.
This entry was posted in Polling and tagged , , , , , , , , , , , , , , , , . Bookmark the permalink.

One Response to Do Polls Work?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.