Should you trust polls when it comes to Toronto's mayoral byelection? It's complicated

It's a mayoral race unlike any we've seen in recent memory. 

And the byelection to fill Toronto's top job has pollsters in the field trying to map out public opinion on a weekly basis. Over two dozen public opinion polls have been conducted since former mayor John Tory resigned in February, all gauging where the large field of potential replacements stands in the eyes of voters.

So, should you trust their findings? The answer is complicated. 

Robert Hutton, Mission Research's director of insights and the former chief operating officer of Pollara Strategic Insights, said voters should approach all public polls with some skepticism at this point. He has concerns about the method used to gather many of the polls popping up in the race.

Hutton said the polls have the potential to miss some key voter groups. But he's also cautioning against ignoring them altogether, he said.

Those polls, conducted by interactive voice response (IVR), are essentially robo-calls. A computer dials a number with a pre-recorded survey and gathers responses. The trouble is people tend to ignore robo-calls or they're simply filtered out as spam, Hutton said.

"The problem with that is that there's only a certain segment of the population that don't screen out those calls," he said. "It misses a whole lot of people. And if those people that it misses have certain common characteristics, perhaps common voting characteristics or attitudes, it'll misrepresent those people in the pool."

So, why do pollsters use IVR? Hutton said it's cost-effective and can be deployed relatively quickly.

But it's not the only way polling companies gather their data. Many research firms use a mix of landline and cell calls with live operators and online polling to conduct surveys. Those methods take a bit more time and can cost a minimum of $20,000. But Hutton says they allow for stronger results and a more representative and comprehensive sample.

Despite that, he's not eager to throw IVR out the window.

"We should look at it and say, is there a legitimate trend here?" he said. IVR polls in this race have consistently found Olivia Chow is the frontrunner, Hutton says, which he believes is accurate because that's been the finding made by all of the pollsters.

But he stops short of drawing any conclusions about the pack behind her, noting field placement has varied week-to-week, depending on the poll.

"Now, does that tell me that we have great accuracy on IVR polls further down the list? In the second, third, fourth, or fifth place candidates? My suggestion is probably not."

John Tabone, the Chief Executive Officer of the Canadian Research Insights Council (CRIC), the industry association for research agencies in Canada, said there's nothing wrong with IVR polls, but people should demand transparency from the pollster. That means being able to have a clear look at the sample taken, the demographic breakdown of their numbers and the unweighted sample.