Three Thoughts About Data, Disruption, and Living Through a Pandemic

Naturally he refused to play – but was persuaded to relent and to try his best with the hastily-repaired instrument. The result: not a catastrophe, but a masterpiece. The faults of the piano refocused Jarrett and forced him to explore new ways to improvise. The concert recording, The Koln Concert, is one of the most popular jazz records in history.

This isn’t just a matter of jazz genius: a few years ago, when the London Underground suffered a partial shutdown for 48 hours because of a strike, Londoners re-routed their commutes, sometimes using buses or overground trains. After the strike was over, an analysis of the electronic turnstile showed that tens of thousands of people had changed their route and then never changed back. The brief disruption indicated they’d been doing the commute wrong all their lives.

I talk through these examples, and much more in the latest episode of the Data Brilliant podcast with Qlik’s Chief Data Officer, Joe DosSantos. When thinking about the above examples of how disorder can actually lead to innovation, we cannot help but draw parallels to the awful pandemic we’re all facing. Nothing will compensate for the hundreds of thousands of lives lost – nor the devastation wreaked on people’s livelihoods – but there is ample historical precedent for the old saying that necessity is the mother of invention.

The Promise of Data

As Joe and I are both data people at heart, we talked a lot about the promise of data. In particular, we discussed how too often people are willing to shrug and to dismiss statistical claims as inevitably distorted and misleading. That’s a tragedy: in a complex world there are truths we can perceive with statistics that have no other way of knowing. I’ve been championing the idea that statistics are a vital tool for seeing details about the world that we otherwise could not – like a telescope, a microscope or an X-ray. We simply need to be curious and open-minded – a healthy scepticism about surprising claims should not dissolve into knee-jerk cynicism.

While there are many important technical skills for a data scientists or analyst, that spirit of curiosity is as important as any of them. We need to be willing to look deeper, and broader – and to ask what our categories mean. Inevitably, a statistical view means drawing sharp distinctions where in reality there are shades of grey. That’s okay – but we should be conscious of the fact that messy reality does not always fit into the tidy boxes we create.

Learning From Mistakes

As we all know, it’s hard to get things right first time, whether we’re talking about a forecast, a statistical analysis, a business venture or a golf swing. What matters is the ability to notice and correct your errors at the earliest possible opportunity. That’s why Joe and I discussed Philip Tetlock, the psychologist and forecasting expert, who writes approvingly of “foxes” and “actively open-minded thinkers” – the point here being a willingness to test your ideas, seek alternative views, and change your mind.

That’s good. But as Joe pointed out during the episode there are other approaches – for example, rapid iteration, or the use of A/B testing, or experimenting with prototypes. What all of these approaches have in common is the central realization: I could be wrong. We don’t like staring at that uncomfortable fact, but the sooner we recognize it the easier it will be to learn from our own errors.

Tune in to Data Brilliant now to listen to Tim and Joe’s discussion in full.