I’m sure you will be shocked (shocked!) to learn that this integration was not as simple as the Sales person thought. In fact, this work internally cost us over $40K to develop. And no, the sales person did not help with the coding.
This story came to mind as I’ve gotten to know Attunity, Qlik’s most recent acquisition. It’s enabled Qlik to introduce our Data Integration Platform, a combination of Attunity products and Qlik Data Catalyst.
There is a lot more to Data Integration then one might think. For example, IDC says that “data integration software employs a wide range of technologies. These include, but are not limited to, data connectivity; extract, load, and transform (aka ELT); change data capture (CDC), format and semantic mediation, data federation and virtualization, data quality and profiling, master data management, metadata management, and self-service data preparation and integration technologies.”1
Whew! That’s a lot more than simply moving the data from one place to another!
So how does this definition from IDC compare with the capabilities of Qlik’s Data Integration Platform:
- Data connectivity - Qlik’s Data Integration Platform supports connectivity to over 100 different data sources and all of the major technology stacks for the public cloud vendors.
- Extract, load, and transform (aka ELT) - Qlik’s Data Integration Platform can automatically extract, onboard and transform the data
- Change data capture (CDC) - Qlik’s Data Integration Platform includes real-time CDC that is agentless and log-based so there is minimal impact to production systems.
- Format and semantic mediation - Users can define a workflow and rules within the Qlik Data Integration Platform to automatically transform data based on certain formats or patterns.
- Data federation and virtualization - Qlik’s Data Integration Platform can scan the data to define the associated metadata and build a complete profile without transferring the data itself.
- Data quality and profiling - When data is onboarded into Qlik’s Data Integration Platform, data elements can be profiled to understand and report on the cleanliness of the data.
- Metadata management - Building a Smart Data Catalog that documents every aspect of your data is a key part of the Qlik Data Integration Platform. This include the management of technical, business, and operational metadata to make each data element understandable and actionable.
- Self-service data preparation and Integration technologies - Users can create graphical data preparation flows to generate new datasets.
So, the next time you hear the term Data Integration, keep in mind that it’s a lot more than just moving the data. And be careful who you ask when estimating the effort.
Interested in learning more about Qlik’s Data Integration Platform? Check out this page.
1 Data Integration and Integrity Software https://www.idc.com/getdoc.jsp?containerId=IDC_P31...