Picking the right horse for the right course

 

Robert Cuffe

BBC News's head of statistics, Robert Cuffe, was on Newswatch this week. Here's a transcript: 

Samira Ahmed: Well, the man whose job it is to guide and advise the BBC's journalists on questions like these is Robert Cuffe, who is head of statistics for BBC News, and he joins me now from our newsroom camera upstairs here at New Broadcasting House. Thank you for coming on Newswatch. Can we start with all the different numbers being cited in stories about testing? There are the numbers of tests that can be done in theory, the number of tests being done, and the number of tests done actually processed. Which is the most important one to use in new stories, do you think? 

Robert Cuffe: Well, at the moment the story really is about the gap between demand and supply, and we don't really have good stats on demand. We know the supply. We know the number of tests that are reported everyday that come back with results, and we know the capacity that the labs say they have, and those two are pretty much bumping one up against the other. So it suggests that the system is pretty tight. And when not all tests arrive evenly spread through the week or across the country, that is going to lead to some of the problems that we have been seeing. But we can't tell just how bad the problem is. Like a lot of the coronavirus stories we are trying to piece together what feels to be honest like a jigsaw with pieces from many different boxes. 

Samira Ahmed: OK. Now, we have seen a significant rise in the number of positive cases of the virus, and that is partly because of much more testing. But doesn't that mean that the graphs that we have been using for months tracking the rise and fall of infection numbers since March [have] become a bit meaningless? 

Robert Cuffe: They can be if they are not used correctly. It is certainly true to say that we have gone through three phases. If you think back to March or April, where testing was limited to people maybe only in hospital, it was very, very severely constrained, and then we move through the summer and testing was largely available to anyone who asked for it and everyone was encouraged to go for tests, through the situation we have been reporting this week when we are starting to see constraints come through again, albeit not quite as, nowhere near as tight as they were earlier in the year. Those are three different phases, and to say that the number of confirmed cases on a given day means the same thing in those phases is probably not helpful, and that is why in the charts that we use to display those figures we are putting a break down the middle to illustrate those different eras that we have been through. But I still think it is useful to show the full history. 

Samira Ahmed: Another issue that viewers raise, in August the government changed the way that it calculates deaths from Covid-19. So now it is counting all deaths within 28 days of a positive test. But this fails to include those of course who might have died of Covid-19 more than 28 days after a test, and, on the other hand, it does include those who died in unrelated ways, who happen to have had a positive test. So again, it doesn't seem to be a very satisfactory statistic, does it? 

Robert Cuffe: I think every statistic has weaknesses, and we've just got to pick the right horse for the right course. So if you want to have a more nuanced judgement of whether a death was caused by coronavirus you could look at the death certificate and whether the registering doctor thought that was one of the underlying causes, or the direct cause of the death. But that takes up to two weeks before those numbers come out. Now, we do report them, so we are reporting both sets of figures, and there is a big difference between the totals. So there have been about 42,000 of the deaths within 28 days of a positive test, and about 56,000 mentions of coronavirus on death certificates, so those are big gaps. It is clear that the 28 day number, the number we hear every day, is missing a large part of the death toll but it is much more useful for telling us in a timely way what the trends are. So you don't want to wait when we think about an epidemic that can grow very, very quickly, you don't want to wait two weeks before you start to make decisions about the deaths that are helping you to understand what else is going on elsewhere. 

Samira Ahmed: One of the things you have made clear is that you feel BBC journalists need to contextualise, and that often means using different statistics and then explaining why they are different. But, of course, in headlines in particular, and short summaries, it can feel like we're giving a very simple story, things are going up or things are going down. It is not satisfactory, is it? 

Robert Cuffe: It is very difficult to summarise the coronavirus epidemic in a simple number. That is certainly the case. But I'm not sure that should be surprising. I'm not sure anything in life boils down to one question that can be answered very simply. It is really across the news and within the details of a story that we can really tell the whole detail of what is going on and put that picture together. 

Samira Ahmed: What is really interesting, Robert, is that clearly the BBC is coming up with some of its own statistics based on the data out there. Some viewers feel that the BBC has been taking government data too much on trust and we should do less of that and more of coming up with the BBC's own calculations. 

Robert Cuffe: I don't think that you need to build data from scratch in order to analyse, interpret or challenge the use of particular statistics. Certainly where the story justifies it, we will scrape the website, we will do the survey, we will send in the Freedom of Information request, but I don't think it makes sense for the BBC to start trying to calculate GDP or to conduct a census. So when we get in those statistics that come from government, it is our job really to understand what is the strongest thing that can be said with them, what is the correct use to which they could be put. 

Samira Ahmed: Robert Cuffe, thank you so much.