Streaming Data Into Snowflake – The Enterprise Way

Today we announced that Qlik has officially joined the Snowflake Partner Connect Program. Woot! But, you’re probably thinking: “What does that mean to me?” Simply stated, Snowflake users, whether bona fide subscribers or trial participants, now have an incredibly easy path to load data into their cloud data warehouses. We’re offering a 14-day Qlik Replicate test drive that has everything you need to quickly ingest data into your data warehouse. The Qlik Replicate test drive runs in the cloud, includes an example source database with sample data and supports Snowflake on AWS or Azure. Just navigate over to Snowflake’s Partner Connect portal from within the Snowflake UI.

Once there, press the Qlik tile to begin.

What Can I Do With the Qlik Replicate Test Drive for Snowflake?

We understand that you might be unfamiliar with Qlik Replicate, so we’ve packaged a well-documented tutorial with sample data so you can quickly get up to speed. Bite-sized exercises walk you through configuring a data source, choosing data subsets for ingesting into Snowflake, initiating the data loading process and monitoring the result.

However, the tutorial doesn’t stop there. We all know that accurate data warehouse reporting relies upon freshness of data it contains and so the Qlik Replicate test drive does two unique things.

  1. A background process continually changes the values in the MySQL data source.
  2. Qlik Replicate continuously updates the Snowflake data warehouse for those changes. You can see the result in the animation below.

No other “data loader” can do that!

Why Is Qlik Replicate Different From Other “Data Loaders”?

Qlik Replicate isn’t just another data loader that is based on batch ETL (Extract, Transform, Load) processes. Qlik Replicate uses log-based change data capture (CDC) to ensure that data in the Snowflake warehouse never lags the current state and is always up-to date.

In addition, Qlik Replicate is also easier to deploy and manage, as it doesn’t require the use of software agents like many ETL services to deliver data from on premise systems. In fact, Qlik Replicate’s agent-less CDC can stream data from over 70 sources including SAP, mainframes, relational databases and data lakes in real-time. We’ve got a great eBook called “Streaming Change Data Capture” that serves as a practical guide for enterprise architects, data managers and CIOs alike. Click here to download your copy today.

Moving Beyond Streaming Data Ingestion

Qlik’s support for Snowflake doesn’t stop at real-time data ingestion. We’ve got a full range of functionality in our Qlik Data Integration platform (QDI) that grows as you adopt Snowflake and roll out bigger footprints into production. In fact, we’ve seen this pattern so many times that it’s worth describing below:

  1. Adding MORE sources of data – The first requirement that appears after folks have piloted data loading is to scan their enterprise and wonder if they can import new types of data to their warehouse. Of course, our answer is yes! QDI can quickly adapt to help you ingest virtually any data into Snowflake.
  2. Data Catalog – The irony is that, once multiple streams of data have been loaded into the warehouse, then the next requirement is that Snowflake users want an easy way to search and share data sets with others. QDI’s smart data catalog helps you to easily expose Snowflake data to more apps and users.
  3. Data warehouse automation – Many Snowflake implementations do not conform to any particular data warehouse design philosophy and can become unwieldy to manage over time. QDI’s visual data warehouse automation helps you operationalize the entire Snowflake data warehouse lifecycle to improve efficiency and productivity.
  4. Discover bolder business insights – Of course, the reason why many people deploy a Snowflake data warehouse is for analytics initiatives. You can maximize the value of your Snowflake data with Qlik Sense. With it’s one-of-a-kind associative analytics engine, sophisticated AI and support for the full range of analytics use cases, Qlik Sense is the perfect analytics platform to complement your Snowflake implementation.
  5. Cost control – Paradoxically, the last item most folks consider is: “How much am I using Snowflake?” Yet, again, we’ve got an answer for that question, too! Our partner engineering team has created a handy set of operational dashboards to help you explore and understand your Snowflake usage. You can download the dashboards for free here.

To sum up, you may not have all these requirements at once, but it’s highly likely you’ll encounter each need at some stage. For more information visit out our dedicated Snowflake partner page.

Next Steps

I started this post with the exciting news that Qlik had joined the Snowflake Partner Connect program and ended it by explaining the many ways in which Qlik supports a Snowflake implementation – from continually ingesting data sources, to visualizing Snowflake usage, and everything else in between. But, let’s remember: a data warehouse implementation is a journey. And, we know that every journey of a thousand miles begins with a single step. So, take the first step and click our tile in the Snowflake Partner Connect portal today.

@Qlik and @SnowflakeDB announced an expanded partnership - Qlik real-time #dataintegration is now available for seamless trial to accelerate #datawarehouse automation for Snowflake customers

 

In this article:

Keep up with the latest insights to drive the most value from your data.

Get ready to transform your entire business with data.

Follow Qlik