So, How DO You Humanize Big Data?
Sure, it all sounds great, but what are the best ways to make it happen?
In my last blog, I talked about big data – how to not just visualize it, but “humanize” it using Qlik’s approach, On Demand App Generation (ODAG). This month, I wanted to share a story where Qlik Consulting assisted a customer in implementing ODAG.
Although I had quite a few examples to pick from, I chose this one because it demonstrates how ODAG can be a solution for more than just typical big data.
The story begins with Sean Donovan, a Senior Consultant with Qlik Consulting. He’s been working with the Qlik product suite for over five years, and has extensive experience connecting to and modeling various types of data sources, as well as building applications in both QlikView and Qlik Sense. Sean recently was assigned to work with a customer that, due to some unique requirements, chose to leverage ODAG. I spoke with Sean so I could share why we advised to use ODAG, how it was implemented, and how long it took to set up.
The first question was, of course, “How big is their data? Why did we recommend ODAG?” Sean’s answer goes to the heart of the 3 V’s in defining big data – Volume, Velocity and Variety. The situation involved the following:
- The data was coming from a couple hundred data sources, and each data source was made up of a couple hundred tables. Altogether, over 40,000 tables added up to terabytes of data.
- More data sources would be dynamically added, so the solution needed to connect to new data sources without requiring development effort every time.
- For security and auditing purposes, some data tables could NEVER be in a single data model for some users. Because of different user roles, there was the potential of multiple combinations of data tables. So, using static, pre-defined data models was going to limit the capability they needed.
In this case, the VOLUME of the data and quantity of tables was not the deciding factor even though it accounted for terabytes of data. Rather, the VELOCITY and VARIETY of dynamically adding new data sources and the need for a higher level of security, drove the decision to leverage ODAG.
Sean developed an application using Online Demand App Generation over a subset of tables to prove out the solution. The result was a landing page where, based on the user’s security rights, they could select the data sources – existing and new – and data tables, then simply press a button to “Generate My App”. ODAG executed queries against those data sources and tables, built out the data model and generated the Qlik Sense application. Within seconds, the user had an application for the specific production data they requested, and the management team had the assurance the user could only see data they were allowed to see. The result? We had “humanized” their data in a way they never could before!
The final question for Sean, and the one most people want to ask, is: “How long did it take to build this first version?” His answer: only one week, because he had the in-depth skillset AND the tools/techniques provided with ODAG. Without either of those, this project could have easily taken the customer weeks or months. There’s a bit more work to include more data tables and deploy it out to all end users, but the work of Sean and ODAG has set the stage for the customer to realize the full potential of their data.
If you want to learn more about On Demand App Generation, or how Qlik Consulting can help you humanize your big data, please read our data sheet, or contact us at [email protected].