Publish and subscribe messaging is an asynchronous service-to-service form of communication for serverless and micro-services architectures. In the publish and subscribe messaging model, distributed applications in a modern cloud architecture can receive instant event notifications, enabling event-driven architecture and real-time data streaming.

Publish and subscribe messaging is at the heart of technologies like Apache Kafka, Azure Event Hub, Google Cloud Pub/Sub and other major streaming services. Organizations can use technology built on publish and subscribe messaging to aggregate and analyze real- time data streams from hundreds or thousands of sources – IoT sensors, financial transactions, e-commerce purchases, social networks and many others.

But while publish and subscribe messaging systems offer tremendous value, ingesting data from these platforms also presents significant challenges. Production on source systems can suffer, hindering productivity and frustrating users. And integrating data from disparate sources can require a large amount of custom development which can drain IT resources – especially when scaling to include hundreds or thousands of sources.

Qlik Replicate®: simplifying ingestion from publish and subscribe messaging systems

Qlik provides an answer for the challenge of ingesting data quickly from publish and subscribe messaging platforms. Qlik Replicate is a universal data ingestion and replication solution that allows organizations to accelerate replication, ingestion and streaming across a wide range of heterogeneous sources, including databases, data warehouses and Big Data platforms. By making it easier to move data securely and efficiently with minimal impact on operations, Qlik Replicate enables IT teams to maximize the value of publish and subscribe messaging platforms.

Qlik Replicate makes it easy to publish events to all major streaming services. When looking at the benefits of Google Pub Sub vs. Kinesis or Kafka vs. Confluent, IT teams know that whatever publish and subscribe messaging system they choose will be supported by Qlik Replicate. Data administrators can support multi-topic and multi-partition streaming as well as flexible JSON and AVRO file formats, and in platforms like Kafka they can separate data and metadata by Kafka topic to integrate metadata more seamlessly with different schema registries.

Benefits of supporting publish and subscribe messaging platforms with Qlik Replicate

When using Qlik Replicate with publish and subscribe messaging systems, organizations can:

• Eliminate manual coding – Qlik Replicate provides an intuitive and configurable graphical user interface that lets administrators quickly and easily set up data feeds with no need for custom development.
• Minimize impact on source systems – Qlik Replicate’s log-based Change Data Capture (CDC) technology reduces the impact on production sources while zero-footprint architecture eliminates the need to install agents on source database systems.
• Scale easily – Qlik Replicate enables organizations to scale easily to include publish and subscribe messaging data from hundreds and thousands of databases.