Dark Data: Why what you don’t know matters

Emeritus professor David Hand launched his latest book on the universe of information at Imperial College London.

In the era of big data, it is easy to imagine that we have all the information we need to make good decisions. However according to David Hand, Emeritus Professor of Mathematics and Senior Research Investigator at Imperial College London, the data we have are never complete and may only be the tip of the iceberg.

Understanding what we don’t know

Now thanks to David Hand we can begin to understand some of what we don’t know – the dark data that we need to be aware of to ensure that we make good decisions. President Alice Gast Imperial College London

Staff, students and alumni gathered at the College to celebrate the launch of the book, Dark Data: Why What You Don’t Know Matters. Opening the event, President Alice Gast said: “If you think that you’re totally sure of something, think again. I’m sure after this evening, you’ll be thinking again.

“If you are in doubt of the veracity of things you read, perhaps your doubts are justified. We know what we know, we don’t know what we don’t know, but now thanks to David Hand we can begin to understand some of what we don’t know – the dark data that we need to be aware of to ensure that we make good decisions.”

Dark data

In this latest book, Professor Hand explores the importance of missing – or ‘dark’ – data, and its impact on our decision-making. The data we cannot see has the potential to skew stats and mislead us, potentially with disastrous consequences. Professor Hand shows how and why this happens, how this can be avoided and how counter-intuitively hiding data can sometimes lead to deeper understanding and better decisions.

Elephant powder

During the launch, Professor Hand discussed that although the power of data is undeniable, problems arise when it is missing. According to him, and quoting Donald Rumsfeld, there are fifteen types of dark data, including data we know are missing – known unknowns – and data we don’t know are missing – unknown unknowns.

Professor Hand gave the story of elephant powder as an example of this dark data. A man who is putting little piles of ‘elephant powder’ down the middle of a road says it is effective because there are no elephants. What this man is missing is information, or dark data, on what would happen if he didn’t put the elephant powder down – he may discover that there are no elephants after all.  

Another example of dark data is around Hurricane Sandy, a physical storm that was accompanied with a ‘Twitter storm’ of 20 million tweets. Twitter tells you what is happening, where it’s happening, when it’s happening and to whom it is happening – in other words, where to send the emergency services. However a community with no power or phone masts would not be sending tweets, so the authorities wouldn’t know that that’s really where they need to send the emergency services – they’re missing a crucial part of the data, the dark data.

Recognising dark data

Through the book, Professor Hand hopes to outline the dark data that exist and the situations in which they can arise, so that we can learn to recognise and control for them.

Professor Gast added: “Many of us have enjoyed his books and they have changed the way we look at the world, and certainly the way we look at numbers, data and coincidences. I can think of nobody else who can make statistics so interesting, understandable and pertinent.”

Prolific writer

David J. Hand is an Emeritus Professor of Mathematics and Senior Research Investigator at Imperial College London. He is a Fellow of the British Academy, and an Honorary Fellow of the Institute of Actuaries, and has served twice as President of the Royal Statistical Society.

Professor Hand is a prolific writer, having published some 300 scientific papers and 29 books, including The Improbability Principle: Why Coincidences, Miracles and Rare Events Happen All the Time, Information Generation: How data rule our world and The Wellbeing of Nations.

Image credits: Danny Fitzpatrick - photographer

Image of Hurricane Sandy: "Hurricane Sandy Over Virginia (MODIS)" by jalbertbowdenii is licensed under CC BY 2.0 .