Pre-Aggregation is the Enemy of Insight

When we realize our technological limits, are we accepting defeat or rising to a new challenge?

Neil deGrasse Tyson was on CNBC’s Squawkbox a few days ago (fast forward to the 2:49 mark) and he argued “given our technology, and our knowledge of the fabric of the space time continuum,” we will never travel outside of our own solar system.

He further went on to say that “we have no capacity to do it technologically.” I originally thought that this position was depressing and short-sighted but in fact, the recognition that existing technology has extreme limitations turns out to be uplifting. As soon as you recognize your limitations, you start down a path to overcome them.

Does your technology have extreme limitations too?

As far as business analytics in the Financial Services is concerned, we have been working around technological limitations for decades. As evidence:

  • In the 1990s it was widely understood that our data was way too big to deal with so a technique called pre-aggregation, or the building of hierarchical cubes was implemented. This is still widely thought of (incorrectly I might add) as the only way to handle large datasets while maintaining the performance demanded by the business users.
  • Our competitors’ customers often complain about spinning circles and are counseled by performance “doctors” to pre-aggregate and simplify what they are trying to accomplish. This must be what Gartner meant when warning of not being able to deal with complex business problems as described in the Magic Quadrant.
  • Every time I open up a spreadsheet with more than 100k records I see an hour glass and start working on something else while it loads. Then if I put a complex formula to join that table with another one, I’m hosed.
  • Even wildly popular cloud based solutions for CRM (Salesforce), Travel & Expense(Concur) and HR (Workday) have taken a traditional approach to reporting by asking users to manage dozens of summary level reports or worse, export the detail data into a spreadsheet.

Is your organization willing to rise to the challenge that all this disparate data creates?

There are a couple of common threads amongst these examples. First, these techniques were born in a time where hierarchical cubes and pre-aggregation were the only possible way to handle the large data volumes and still achieve acceptable performance. Second, they presume someone could design a report that answers all of the business questions, which is misleading and profoundly false.

The elephant in the room

Pre-aggregation is the enemy of insight. There is always a need to dive deeper into a KPI or a variance and the limitations of this technique is glaring. So what did we (as business users) do? We built a shadow IT environment that stores all the detailed transactions. Come on… admit it, we all did this (and many still do). Spare me the lecture on data governance when your users are working around the inflexibility of your traditional query tools by exporting the details on their desktop and/or sending details through email.

My point is that this isn’t rocket science, we do have the capacity to “solve this business problem technologically and Qlik’s unique approach, combined with the powerful Qlik engine, is un-matched in the marketplace.

As Dean Sutherland (Regional Director of Sales in the Mid-West and 11-year veteran at Qlik) said on a call when he was whiteboarding this concept for a customer, “It is easy for us to be passionate about this because it is true.”

If you would like to find out more about Qlik and how we are transforming business analytics in the Financial Services industry, I would like to invite you to read my blog called “Financial Services Compass.”


In this article:

Keep up with the latest insights to drive the most value from your data.

Get ready to transform your entire business with data.

Follow Qlik