Will 'More or Less' give James O'Brien a fisking?

If you recall, I found last Friday's Newsnight - the one with James O'Brien - to be deeply biased. 

David Keighley at News-watch doesn't just agree, he's also put a huge stick of dynamite under the programme's deeply dodgy use of statistics. 

It's a must-read post. 

Newsnight and Ipsos MORI had co-operated on a post-Referendum poll and that night's Newsnight gave its take on the findings. 

Here's an extract from James O'Brien's introduction:

And what about buyer’s remorse? All those voters who supposedly want to change their minds? Well, maybe not. 92% of respondents said they would definitely vote the same way. But of them, 5% of Leave voters did say they would now change their vote, compared to just 2% of Remain voters. 

Note the 'maybe' in "maybe not" there and the big "But". And note the statement that "5% of Leave voters did say they would now change their vote, compared to just 2% of Remain voters" - a 'finding' highly relevant to the programme's later discussions about overturning the 23 June referendum, perhaps with a second referendum.

Newsnight used the following graphic to illustrate the point: 

Here's how David outlines the problem: 

Third – and even worse, perhaps – two separate statistics of polling information were conflated so as to overemphasise the numbers who said they would change their vote. 

92% of the Ipsos Mori respondents said they would not change their minds if asked to vote in a second referendum (with 4% saying they would change their vote, 3% saying they didn’t know, and 1% saying they wouldn’t vote) 

Newsnight presented this 92% figure in the graphic shown above. However, O’Brien then introduced an additional statistic: that 5% of Remain voters and 2% of Leave voters said they would now change their vote. Two smaller circles were duly placed on the chart to reflect this, despite these numbers having no direct correlation to the initial 92% figure. Therefore, the graphics and commentary suggested 7% wishing to change their votes, whereas the Ipsos Mori data itself had given a figure of just 4%. 

Further, the two smaller circles of 5% and 2% cannot even be fairly compared to each other, given that more voted to Leave in the referendum than voted to Remain. The only way to have fairly reflected this difference would have been to have introduced a second chart, showing the overall numbers of Leave and Remain voters, and how potential shifts in voting intention might have affected the totals. 

A closer inspection of the Ipsos Mori data also reveals that, to produce the 5% and 2% figures, two responses were combined: those who would ‘definitely’ change their vote, and those who ‘probably’ change their vote. 

Had Newsnight focused only on those who were certain to change their votes, then the chart and commentary would have been even less striking: only 1.1% of those polled would definitely change their Leave vote, and just 0.4% would definitely change their Remain vote – a far less dramatic statistic than the one selected. 

Put another way – bringing in the unweighted sample size of 935 voters who were actual consulted to reach these findings –  only FIVE  people told Ipsos Mori that they would definitely change their mind from ‘remain’ to ‘out’ and two people said they would definitely switch from ‘remain’ to ‘leave’. On that highly tenuous basis, Newsnight told its viewers, in effect  that 5% of total ‘leave’ vote of 17.4m was considering changing sides. This was a preposterous extrapolation.

Please read the polling results for yourselves. You will see that David is correct. 

Q3 is specific. As you can see for yourselves, only 1% of Leave voters said they "would definitely change their vote". 

So when James O'Brien told Newsnight viewers that "5% of Leave voters did say they would now change their vote", he wasn't giving them an honest representation of the poll's actual results. 

What he should have said is:

And what about buyer’s remorse? All those voters who supposedly want to change their minds? Well, almost certainly not. 92% of respondents said they would definitely vote the same way, 4% said they would change their vote, 3% said they didn't know and 1% said they wouldn't vote. And a separate question revealed that just 1% of Leave voters would definitely change their vote and 0% of Remain voters would change their vote. 

Once you've read those Ipsos Mori findings for yourself, wouldn't you agree that my re-write of James O'Brien's spin on the results is much closer to the reality of those results?

******

And another thought that struck me is that - were Newsnight biased in the other direction - they could (and with more justification) have spun the findings of Q2. which asked, "And as you may know the United Kingdom voted to leave the European Union. Do you think that was the right decision or the wrong decision for the United Kingdom?" 

The curious result of those findings is that 4% of Remain voters now say that it's the "Right decision" and 3% Leave voters now say it's the "Wrong decision" - a result that could very easily be spun to show that it's actually Remain voters who are suffering buyer's remorse.

******

Was the conflating of 'apples and pears' in that graphic merely sloppy or proof of bias?

Was the hyping and distorting of the 'buyer's remorse' findings merely sloppy or proof of bias?

Was the failure to point out the curious finding of Q2 merely sloppy or proof of bias?

All I'd say is that I've heard and seen how James O'Brien spun it on Newsnight and I've read the very careful Ipsos Mori presentation of the findings and, to my mind, Newsnight presented the findings in a reckless fashion. 

On top of the extraordinary levels of anti-Brexit bias recorded on this very blog last week, I'm far less inclined to give them the benefit of the doubt that I might have been before. 

This is shoddy journalism either way though.