How not to lose 16,000 COVID-19 test results: a data scientist’s view

A critical piece of the UK Test and Trace infrastructure failed hard this week. All contacts of almost 16,000 COVID-19 infected people were allowed to circulate unknowingly for an entire seven days in the community. That’s about 50,000 people.

I’m not going to complain about Public Health England (PHE) using excel to merge the test results from each test centre. That was obviously wrong.

This is about something far more worrying: I don’t understand why there wasn’t proper monitoring in place. This is a shameful failure of technical leadership. I’m not calling for excel to be replaced; I’m calling for the NHS Test and Trace leadership to be replaced by people who understand data.

Monitoring is basic data science. If your team can’t perform at this level then you shouldn’t be handling any important data, certainly not the data that our national pandemic strategy depends on. The firm I work at only captures data for financial institutions; nobody’s life is in our hands. Yet we’d never deploy a data pipeline without proper monitoring.

Why isn’t this basic skill at the core of the NHS Test and Trace system? The UK government have spent £10B for the programme, most of which has gone to external consultants. I guess £10B doesn’t buy you a serious technical culture, where throwing things together in excel without proper monitoring would be a point of professional embarrassment.

Experienced data scientists would put (at least) two simple monitors on this process:

• A plot showing the total number of new positive tests each day
• A plot showing the number of tests from each test centre

Let’s take these in turn.

Plot of the total number of new positive tests each day

This plots looks like this when there is a problem in data collection:

A plot like this just looks weird. Why has the trajectory suddenly flattened off? Does this reflect a trend in the underlying data or is it an artefact of the data collection process? This needs to be investigated.

This is where the second plot comes in.

Plot of the counts of tests from each test centre

This plot looks like this if data ingestion has failed:

The counts are bunched up at the right-hand side (implying lots of files have the same number of rows). This is a dead give-away that some artificial limit has been hit. Naturally occurring counts never look like this.

A Data Science Culture

Each test centre should speak briefly with PHE on a daily basis, to verify that their data has been collected properly. This week we collected X positive samples from Test Centre Y. Yes, that’s the correct number.

This, the world’s most boring meeting, will seem like a complete waste of time until one day somebody says ‘we collected 65,536 rows of data’ and a sharp junior scientist says ‘wtf!’. (65,536 is the maximum number of rows allowed in older versions of excel)

Using these two simple plots and a regular meeting we have created an early warning system for data collection issues. Please don’t tell me ‘we’re in a pandemic crisis, there just wasn’t time to set this up’. There was time, because we’re in a pandemic, and because people will die because this process has failed.

I’m chair of the organisation representing the professional interests of data scientist in the UK, the Data Science Section of the Royal Statistical Society. Many of our members complain they are prevented from being effective because they don’t have a technical manager who understands their work. This kind of manager would be deeply troubled until basic data monitoring was in place.

High quality data science work requires an organisation-wide understanding of data. I do mean organisation-wide: I know companies where the CEO herself checks over data ingestion figures every day.

An experienced data science leader should be installed into NHS Test and Trace immediately, and given authority to run the data pipeline to a professional standard. A culture of rigour and discipline will prevent another catastrophic error. Unfortunately, you can’t buy a culture like this, even (or perhaps especially) by hurling billions of pounds at large consultancy companies.

Sign-up for future updates, news & events from us here:

Thanks to Piers Stobbs and Ryan Procter for reading a draft of this post.