How’s Your Plumbing? The Growing Value of Data Pipelines

This topic on everyone’s lips was preempted by great thought leadership blog posts from Dan Vesset at IDC. He notes that in becoming a data-informed and intelligent enterprise CEOs have moved beyond lip service: “Data and analytics investments – and their benefits – are proof points on earnings calls, with specific technologies cited.” He notes that many novel analytical techniques are appearing, which, combined with better data synthesis, will be key to achieve the goal of enterprise intelligence.

I was very happy to see that emerge, as it aligned well with the theme of Qlik’s 2020 Trends (i.e., that analytics alone is no longer enough – you also need synthesis). If you missed it earlier in the year, I will present it again at the upcoming QlikWorld 2020. The analogy that IDC uses is that data is actually not like oil but rather like water. If you can capture, cleanse and then irrigate it, it will lead to a significant change in your productivity and harvest. Staying with the water analogy, the outcomes are only as good as the plumbing and the pipeline. You want to channel it, prevent leakage, avoid corrosion and eventually bring it to the right place at the right time.

The executive summary from IDC establishes pretty strongly what we suspected – that organizations which think holistically around data and analytics as a pipeline get better outcomes. Leaders (those with the highest data-to-insight score) perform better on a number of dimensions. I recommend waving this piece of research in front of any skeptical decision-maker.

So, what in the data surprised me, and what is the low-hanging fruit? Together with my colleague Michael Bienstein, I dove into the data and wanted to bring you some nuggets, outside of IDC’s executive summary, that I consider notable.

• Survey respondents say they capture a big proportion of all possible data. Most people think they can catch 70-90% of all relevant data. But, overwhelmingly, organizations are trying to make sense of their own data, created by their own business systems. My POV: The perceived high proportion of data captured shows, on one hand, that “big data” is just, from respondents’ perspectives, today’s data, i.e., size isn’t a constraint. On the other hand, I suspect many respondents are setting their expectations too low (you don’t know what you don’t know), which can lead to underestimating the potential to capture other data sets, such as external and partner-generated data. Interestingly, the top 20% of leaders (as identified in the IDC survey sample as those with the highest data-to-insight scores) say they have more challenges than average when it comes to finding the relevant data. Do you have the topology and “aqueducts” in place to capture that? Organizations should map out their internal and external data estate that can potentially bring value to the organization. An information catalogue can serve as connective tissue.

• It becomes apparent from the survey that the main barriers to success are NOT finding the relevant data, but rather investing in the relevant technology and assessing the ROI to justify budget. It shouldn’t be. To justify ROI is always hard, but, in these COVID times, to succeed you need to start small and ramp up usage and spend with the value you see from your project. An agile project methodology combined with a SaaS business model helps to keep projects ongoing, even if they are smaller and more tactical. In some cases, water can now often be drawn straight from the tap.

• Almost half of respondents say that they have data quality issues. On the surface of it, this doesn't look too good, but my point of view is that it isn’t necessarily bad news. Don’t let that stop you. The only way to improve data quality is to actually identify the problems through shining a light on the data. Through frequent usage, iterations and then applying the right filter, it will gradually improve. Some of it may not be suitable for drinking but may find uses elsewhere. Locking data down until you get to 100% quality is not the right answer, because it will become stale, and you’ll never get there, anyway – data is too fluid and changes all the time.

• Two-thirds of respondents have less than 80% of data automated. This means lots of manual processes. Do you have people passing buckets of water to each other still? This is the low-hanging fruit part, because there most definitely are tools (data movement, change data capture, data warehouse automation, catalogues, etc.) and methodologies (DataOps) that can bring agility and automation to data management, just like self-service has for analytics.

• In terms of AI/ML, under 20% of knowledge workers are able to use such technologies today. There is a hope to improve this over the next few years, but it remains aspirational. Here, it’s incumbent on the vendors in this space to simplify as much of the complexity as possible, through utilizing these technologies within your staple data management and analytics tools. That makes customers' lives easier and helps a build vs. buy decision.

• When assessing the success of a project, “improving operational efficiency” came out as most often achieved. I suspect that success metric may grow further in importance and priority. “Never waste a good crisis,” as Churchill said, and business leaders will look toward business process re-engineering. Make sure to choose a tool that can be embedded into your operations, moments and processes.

In normal times, during rain season, when order books were full, organizations may have been excused for having leakage in their pipes. But, that is no longer a luxury that can be afforded. The demand backdrop is getting tougher, and so is the supply side. As IDC points out, it’s now a C-suite level issue amongst your competitors. Organizations need to make it imperative to maximize their data-to-insight score. Synthesizing data, cataloguing it and channeling it out for analysis and insight is the plumbing for the intelligent enterprise. Does it sound overwhelming? Attend QlikWorld 2020 next week, where IDCs’ findings, Qlik’s solutions and customers’ best practices will be surfaced and scrutinized further.