Data Science Needs Technical Managers

We believe a lack of data scientists in leadership roles is a major reason why organisations don’t get value from data science. Our recent survey has shown that data scientists with a manager from a data science background are more likely to feel they’re delivering value for an organisation.

Last year we conducted a survey of 150 practicing data scientists, to understand the issues they face and to find out how the RSS can help. One of the most interesting results was that half of the respondents didn’t feel their work brought significant value to their employer. We were interested to understand why this seemed to be happening fairly universally across a range of industries and organisations.

In general these data scientists seemed less worried about the quality of their data, budget, skills or the technology available to them. Instead the biggest challenges they face are social. Their organisation doesn’t understand them, or doesn’t support them, or both when it comes to delivering value from data science projects. When we asked them what their main obstacle was, their top two answers were “lack of strategy” and “limited support from senior managers”.

In the survey we asked a set of freeform questions to help us get some deeper insight. When respondents talked about “lack of strategy”, one root cause was a lack of common language between data scientists and senior management. For example, respondents said that they were facing a “lack of … appreciation for what data science actually is” and “projects … that have little grounding in technical reality”. It seems clear that there has been a failure to find an overlap between business goals and the things the data science team can practically deliver.

We also found that for data scientists with a non-technical manager, 45% felt their work delivered significant value. This rose to 66% when a data scientist had a manager from a data science or academic research background (see figure below). This difference was statistically significant (p < 0.05). We believe that to improve the value you get from your data science team you need to ensure your data science team is backed up by a senior manager who understands their work.

It’s worth noting that our respondents are primarily based in the UK and affiliated with the RSS so may not be totally representative of all industries. But we suspect this is an opportunity for many real organisations today. And we believe this backs up the RSS Data Science Section approach of trying to promote the career development of data scientists as a way to improve the value organisations get from data science.

One of our goals at the Data Science Section of the RSS is to help advance the careers of data scientists. One way to address this “communication gap” in organisations and get more value from data science is to bring more people with data science experience into the high-level conversations about business strategy. The field is relatively young, but we believe that as more people with data science experience progress into senior management roles this will go some way to resolve the challenges data scientists are facing today.

Thank you to everyone who contributed to the survey, the results have helped direct our efforts and we’ll be publishing more articles from the results over the coming months. Please join our mailing list to keep up to date with the work of the RSS Data Science Section.