Customer & Partner Spotlights

Empowering Real-time Data Replication: Unleashing the Potential of Qlik Replicate and Amazon MSK

Discover how the powerful combination of Qlik Replicate and Amazon MSK revolutionizes real-time data replication, driving insights and unleashing the full potential of your enterprise data.

Blog promotional image with text that reads "Empowering Real-time Data Replication: Unleashing the Potential of Qlik Replicate and Amazon MSK"

In the current data intensive world we live in many customers deal with heavy volumes of data that reside in databases and streaming systems. There are many ways to move data from one cloud platform to another, but for efficient migration, ease of use/development, and near zero downtime, a tool that does continuous near real-time data movement and CDC (change data capture) is needed. For Customers who want to migrate the data seamlessly to many targets including streaming systems, this becomes particularly challenging. There are various ways to carry out this with full load, but there are few tools available that support ongoing replication and change data capture (CDC). In this blog post we will look at how to use Qlik, specifically Qlik Replicate to effectively migrate from MySQL to Amazon Aurora for PostgreSQL and also publish changes to Amazon MSK near real time.

The Need to Stream Data Changes to Various Targets

Initially, Qlik customers used CDC tools to migrate databases or backup data to cloud databases, but they have since expanded their use cases significantly. Distributed cloud environments offering a variety of services have made it possible to separate workloads for the correct environment. In the old-world developers waited for data to arrive at a central database and wrote applications on top of the central database. The latency of the transactions and complexities made it difficult. Let’s imagine a scenario where the corporate transaction database could detect new orders and update cloud database and mobile applications as changes occurred. With this type of capability businesses can proactively alert customers of orders and drive customer satisfaction. For born in cloud companies this type of integration is available, but how about those companies where the data is stored on older on-premises database. Using Qlik Replicate for CDC and Amazon MSK for messaging queue this could be achieved.

Amazon MSK and Qlik Replicate Use Cases

Highly available and secure – Amazon MSK creates an Apache Kafka cluster and offers multi-AZ replication within an AWS Region. Amazon MSK continuously monitors cluster health and automatically replaces any failed components. Amazon MSK provides multiple levels of security for Apache Kafka clusters including Amazon VPC network isolation, AWS Identity and access management (AWS IAM) for control-plane API authorization, encryption at rest, TLS encryption in-transit, TLS based certificate authentication and supports Apache Kafka Access Control Lists (ACLs) for data-plane authorization.

Fully Managed – Amazon MSK lets customers focus on creating their streaming applications without having to worry about the operational overhead of managing Apache Kafka environment. Amazon MSK manages the provisioning, configuration, and maintenance of Apache Kafka clusters and Apache ZooKeeper nodes.

We will focus on Qlik Replicate and Amazon MSK working together seamless to deliver changes. These complimentary technologies offer a range of powerful use cases, including real-time data integration, data replication for disaster recovery, seamless migration to the cloud, enabling hybrid analytics, and helping efficient data streaming for advanced analytics and machine learning applications. Here are some of the use cases we observe in the marketplace.

Event-driven architectures: Using Qlik Replicate with Amazon MSK, businesses can build event-driven architectures, where events are produced by Qlik Replicate’s CDC engine and consumed by different systems or applications via Amazon MSK. This allows for decoupling of components, scalability, and real-time processing of events to data sub systems.

Data replication and synchronization: Qlik Replicate Amazon MSK can be used for data replication and synchronization across different systems or datacenters. This is particularly useful for maintaining data consistency, disaster recovery, and creating geographically distributed architectures. This design pattern allows data producers to subscribe via topics architecture.

Machine learning and real-time processing: With its ability to handle high-throughput data streams, using Qlik Replicate with Amazon MSK can serve as a backbone for real-time data processing and machine learning workflows. It allows for continuous data ingestion, transformation, and model training or inference in real-time pipelines.

Real-time data streaming: The combination of Qlik Replicate and Amazon MSK enables organizations to stream and process high volumes of real-time data from various sources, such as IoT devices, application logs, and clickstream data. This data can be used for real-time analytics, monitoring, and decision-making.

Architecture Overview

For the blog we will be looking at the following scenario: A homegrown application with a MySQL Database backend. We will setup two Qlik Replicate Tasks. Task 1 will move data into Amazon Auroa Postgres and perform CDC. Task 2 will do a full load and continuously publish the changes into Amazon MSK.

Following Diagram Illustrates the setup in AWS:

Diagram illustrating the the setup in AWS.

Task 1 for MySQL to Amazon Aurora Postgres

Diagram illustrating the Task 1 for MySQL to Amazon Aurora Postgres

Diagram illustrating the Task 1 for MySQL to Amazon Aurora Postgres

Diagram illustrating the Task 1 for MySQL to Amazon Aurora Postgres

Task 2 for MySQL to Amazon MSK

Diagram illustrating the Task 2 for MySQL to Amazon MSK.

Diagram illustrating the Task 2 for MySQL to Amazon MSK.

Diagram illustrating the Task 2 for MySQL to Amazon MSK.

Customer Benefit

In summary, by using Qlik Replicate an enterprise can enable not only moving data from a local database to a cloud database but also to a streaming platform like Amazon MSK. This flexibility allows customers to build applications and stream data in real time. Qlik Replicate and Amazon MSK can enable real-time data replication, driving insights, powering downstream applications and unleashing the full potential of your enterprise data.

Conclusion and Call to Action

Integrating Qlik Enterprise Manager with CloudWatch and Grafana Cloud empowers organizations to gain comprehensive visibility into their AWS environments, streamline log management, and unlock valuable insights for optimized performance, security, and compliance.

To learn more about implementing Amazon MSK with Qlik Replicate, visit Qlik documentation on using Amazon MSK as a target.

Ready to get started?