Big data offers a new style of data analysis which is different from traditional business intelligence (BI). Let me explain. Traditional BI projects required business users to know the questions they want to ask before they started a project. Their questions drove the data model for the data warehouse and determined how the data would be stored. The data model also determined which data was collected by which mechanism. Creating the enterprise data model this way could be a lengthy process. In many cases, it resulted in a system that was slow to adapt to changes within the business.
Now, with big data, data analysis is happening from the bottom up. Organizations are collecting as much data as they can from many sources without knowing beforehand exactly what questions they are going to ask about data collection. This means that they don’t need to transform data into the standard data model of the corporate data warehouse at the point of data collection. Instead, their data is stored in the form in which it was originally captured and only given an appropriate structure by the analysis process using it. This flexible approach leads to a dynamic approach to data analysis that lets them react quickly to the rapid changes within their business.
As they look at creating a data lake, the challenges for real-time analytic processing requires five rules of big data: