The Human Side of Big Data
Don’t just visualize your big data – humanize it!
Big data. By far, one of the hottest topics right now in the world of data and analytics. But what is it, really, and why is everyone talking about it?
How does it impact analytics? And what do you need to do to turn big data into useable insights?
To help me provide the best answers to these questions, I recently met with Radx Radhakrishnan, an expert on big data from the Qlik Consulting team. Radx has been working with data and analytics for over 10 years, and is currently enrolled in the Big Data Specialized Studies program at the University of California Irvine.
First things first, Radx told me, it’s important to understand the properties that define big data, and how it impacts analytics. Big data is not just about size, there are in fact “3 V’s”.
- Volume. Yes, this is the most obvious one, and what most people think about when they hear big data – terabytes of data, thousands of tables or files, and billions of rows. Companies can now capture data from most everything, including data that no one used to care about.
- Velocity. Size alone does not mean you are dealing with big data. Speed of capturing data and making it available to act upon is also important. Even when the data set is smaller, it may still be big data because the company using the data wants to capture and react to it as close to real-time as possible.
- Variety. Data is no longer stored only in a structured format. To capture data at near real-time speed requires doing so in its original format, which for some data sources such as website and machine data – i.e., sensor readings, machine performance, security threats – there is no pre-defined structure. Systems exist, such as Hadoop, for the sole purpose of capturing any kind of data, in any format, at high rates of speed. However, raw data is rather useless to us mere humans. So, once data is captured, it needs to be transformed into a human readable form – such as HBase or Hive.
Now, according to Radx, this is when the real fun begins! Many analytical platforms will help you “visualize” your big data – which is a fancy way of saying they will help you see it in a static format. If you have questions, then you need to ask for another “visualization” of a smaller set of data in a different way, and so on. Not a very fast method for acting on what’s happening right now!
Qlik, as you’ve probably heard, prefers to let you ask questions AND get answers back from your data at the same time. So, we developed an approach that works like humans think.
No matter the depth or breadth of your data, as humans, we naturally break things down because that’s the only way we can truly digest the information. So, whether it’s by geography, by business line, by product category, or whatever makes sense to you – the first step is identifying what part of your big data you want to analyze. Then, automatically generate an analytical application over that data set that can be used to discover insights at the most detailed level. Qlik calls this approach “Online Demand App Generation” which was specifically developed to help humans analyze big data.
In my next post, I will be talking to a Qlik Consultant who recently implemented “Online Demand App Generation” for a customer and learn from him what it takes to deploy a solution that turns big data into useable insights.
For more information on how Qlik Consulting can help you get the most out of your big data, read our data sheet, or contact us at [email protected]