Lately, contradictory election polls seem to be as common as contradictory political messages during election campaigns.
Contradictory polls
The recent provincial election in Ontario is a good example. During the campaign poll results were all over the map . In one instance, an Abacus Data poll published by Sun Media on Tuesday, September 15 showed the Progressive Conservatives 9% ahead of the Liberals, while a few days earlier, on Friday, September 9, a Harris/Decima poll published by the Globe and Mail showed the Liberals 11% ahead of the Progressive Conservatives. These massive differences cannot be explained by sampling error.
While the public expects contradictory political messages, contradictory polls are unsettling. Aren’t polls supposed to be a scientifically accurate measurement of the popularity of candidates? With their ubiquitous reference to the 95% confidence interval, accuracy is a polling virtue the industry assiduously fosters.
So if polls conflict, the public naturally wonders which ones are right and which ones are wrong.
Or are any of the polls correct?
In polling the 2011 Canadian Federal election, no poll predicted a majority Conservative government.
These conflicts tend to undermine the trust that the public has in prediction polls, and, by association, any media accounts referencing such polls.
Concerned about the damage this may cause to the election process and the credibility of the political debate, some have called for the regulation of the polling industry to ensure greater accuracy and consistency among polls.
Regulating polls to improve accuracy
Regulating polls would require some sort of agreement on standards for the different methodologies used by pollsters. Unfortunately all of the commonly used methodologies have substantial deficits that impact their accuracy.
- Household telephone polls miss many respondents who have cell phones but no household telephone. The latter group is estimated to be between 20 and 25% of the public and is growing. Household telephone polls need to and occasionally are supplemented by a representative subsample of cell phone only users. For the longest time this methodology has been the gold standard for election polls.
- Robo polls have a high rate of nonresponse because, well… people don’t like talking to robots. It raises the question of who do the respondents represent. Playing loosely with the laws of probability produces polls based on what more correctly should be called quota samples rather than probability samples. In the absence of a probability sample, concepts like sampling error used to ascertain accuracy become relatively meaningless. The undisputed virtue of this methodology is its low cost.
- Internet polls are based on massive respondent pools that are supposed to represent the entire voter population. The problem is that many in a pool are not randomly selected but volunteer simply to make a buck (self-selection). This undermines the representivity of the pool in relation to the population. Representivity of Internet polls is also limited by a lack of Internet accessibility for approximately one third of the population. The undisputed virtue of this methodology is its quick turnaround. Results can be produced overnight.
Weighting results for better accuracy
However, the irony here is that even if a methodology, due to its flaws, badly misrepresents the target population of voters, pollsters can still achieve very accurate predictions by weighting the raw data.
The weighting algorithm is generally regarded by pollsters as a trade secret. Based on years of experience that have provided pollsters with empirical evidence on how to best allocate the undecideds among the competing candidates, and how to create an index that calculates the numerical probability that a respondent will actually vote on election day, the weighting algorithm can literally turn polling dross into polling gold.
The actual algorithm can be quite complex. Since nearly half of all eligible voters don’t vote on election day, pollsters in essence have to figure out which half of the sample is the half that does.
In the recent provincial election in Ontario, the pretenders to the throne i.e., polls with relatively untested methodologies, managed to get numbers closer to election day results than polls using the household telephone methodology.
This so aggravated some pollsters employing the more traditional and expensive household telephone methodology that they have publicly challenged the rest of the industry about using second-rate methodologies. While there is merit in their criticisms, the rest of the industry has basically shrugged it off as sour grapes.
It may well be that the prediction poll is a very specialized instrument, like a racing thoroughbred, that is good for one thing and one thing only. The new methodologies and statistical tricks used to make election predictions may make the measurement of other opinions less accurate. Presently there is insufficient comparative research between these methodologies from which to draw any conclusions.
However, so long as the public’s primary interest in election polls is discovering who’s winning the race, methodological concerns will be given short shrift. In the prediction business, the ends justify the means.
Scientific versus subjective nature of polling
The truth is, all this hue and cry is simply a consequence of the fact that an election poll is an exercise affected as much by the art of polling as by the science. The weighting algorithm includes subjective choices on the part of pollsters as to which variables will be instrumental in influencing voting decisions. The art and the science combine to produce the algorithm. The algorithm can be thought of as a black box that converts raw polling numbers to published predictions. It is these black boxes that ultimately separate pollsters into winners and losers.
It follows that if accuracy is as much a function of subjective decisions as the application of scientific statistical principles, then deciding on a set of methodological standards to regulate the industry is next to impossible.
In fact, I would argue regulation is entirely the wrong way to go.
Benefits of competition among pollsters
Rather than focus on regulating polls, democracy would be better served if the number of election polls were increased during campaigns. Even going to the expense of offering polling companies tax breaks as inducements for mounting election polls would be worth considering given the stakes.
Increasing competition between pollsters would be beneficial for a number of reasons. Most importantly:
- The increased availability of polls would reduce the influence of outliers e.g., inaccurate, biased, or rogue polls.
- Media would have a much wider choice of data points by which to formulate their stories on the public’s voting intentions. No doubt the task of writing an accurate account of what the polls say would be more difficult, but the challenge would inspire good journalism.
- Greater competition would open up fresh thinking among participants, leading to more accurate polling data and a better understanding of what moves Canadians to vote as they do.
While it seems counterintuitive, increasing competition would not lead to chaos.
The reason?
Election polls are a very different animal from other forms of public opinion polling. There is a judgment day with predictions. Election Day results determine which predictions were accurate and which were not. It holds pollsters accountable for these predictions.
In that sense, competitive process is self regulating — it ensures the survival of the fittest and limits the proliferation of those that cannot make the grade.
Accuracy matters
It’s easy to dismiss the debate about the need for better accuracy from polls as some kind of petty food fight within the polling community. After all, there is a mountain of social research revealing little consensus on any direct linkage between election polls and election outcomes.
But that completely misses the point.
Impact on the campaign narrative
Few would argue that when polls are combined with broad and intensive media coverage, they have a strong influence on the development of the public narrative during campaigns. That narrative determines the nature of public debate during election campaigns and influences how the public thinks and votes. The process can be quite complex as there are many factors that interact to produce voting decisions.
Crafting that narrative is the ongoing back-and-forth between polls, media, and the campaigns. Pollsters need to accept their responsibility in this engagement.
Yet in Canada, the polling community is still debating if polls influence electoral outcomes. In doing so, it hides behind the façade of a scientific model that pretends polls are simply a tool that measures public opinion but does not influence its essential nature.
This was the thinking in physics a century ago when physicists realized that scientific model simply didn’t work. It was replaced by quantum physics which understood that the act of measuring an activity changes the essential nature of the activity, the so-called Heisenberg uncertainty principle.
The time is long overdue for polling to make the leap into modernity. Pollsters need to address the complex process of how polling impacts voting intentions and behaviour through media dissemination.
Making sense of puzzling election results
The 2011 Canadian Federal election is a good example of just how complex this process can be.
In that election, the polls missed all the big stories. Pollsters had a hard time explaining why the NDP caught fire in Québec and decimated the Bloc Québecois. They also completely missed the magnitude of the Liberal collapse in Ontario that gave the Conservatives their majority government.
When respondents lie
Explanations for these politically seismic events can be found if one allows for the possibility that polls, through the agency of media, can influence voter intentions, and that voters don’t always reveal these intentions to pollsters.
These social dynamics suggest that the NDP wave would never have happened if the polls were not reporting its surge through various media during the campaign.
They also explain how the polls missed the depth of voter disaffection with the Liberals in Ontario. Many Liberal voters in traditional Liberal strongholds secretly rebelled but kept their decision to vote Conservative to themselves to avoid disapproval from those around them. It’s not unlike the situation in Italy where polls consistently underreported Berlusconi’s popularity because many respondents were “too embarrassed” to publicly admit their approval of him.
In failing to predict the outcome of the 2011 Federal election, the polls revealed that they were not only measuring the effectiveness of the political campaign, they had become part of it. And as part of it, they failed to fully factor in their role, with the help of media, in influencing voter opinion.
Lastly, having made the argument for greater competition, there is little argument needed for greater transparency.
The press and the public are entitled to a comprehensive disclosure of methodological practices with published surveys. A fair bit of this is in place already but it could stand improvement, particularly in the areas of sample coverage as it relates to different methodologies, the differences in response rates between these methodologies, weighted versus unweighted marginal counts, the exact wording of poll questions, and a complete list of poll sponsors.
However, while such data may be useful to the trained eye in revealing methodological concerns, they offer no guarantee for revealing the accuracy of predictions.
Role of the press in contributing to poll accuracy
Make no mistake. As the public narrative develops over the course of a political campaign, accuracy of election polls cannot be separated from how well the press interprets these polls.
Polls, regardless of how accurate the methodology, cannot survive an inaccurate interpretation in the press, just as the press, regardless of how well intentioned, cannot survive inaccurate polls. The two are joined at the hip.
In this context, regulating polls to ensure accuracy implies regulating the press for the same end. It is not only technically impossible to do, it goes against one of the most venerated rights of our democracy — a free press.
Instead of asking if polls should be regulated, the more relevant question is: How can the public debate during political campaigns be improved to more accurately reflect public sentiment?
To that end, regulation would be a poor choice of solutions. The answer lies with more competition, more transparency, and a more discerning media.
This article was originally posted on December 6, 2011 in the iPolitics special feature “78 Ways to fix the way we do Politics” under the title “Should election polls be regulated?“
