Democratizing Machine Learning Capabilities With Qlik Sense and Amazon SageMaker

By John Park, Principal Solution Architect, Qlik; and Dylan Tong, Global Segment Lead Architect, AI Augmented Analytics, Partners, AWS

The ability to discover insights from past events, transactions and interactions is how many customers currently utilize Qlik. Qlik’s unique approach to Business Intelligence (BI) using an in-memory engine and intuitive interface has democratized BI for typical business users, who usually have little to no technical savvy. But, for many years, organizations have only been able to analyze metrics or KPIs of “what has happened” (i.e., descriptive analytics). Analysts have reported that analytics maturity in most organizations is limited to hindsight. In fact, Gartner reported that “87% of organizations have low BI and analytics maturity.” So, even data-driven organizations are limited. Decisions are made by analyzing historical trends and are prone to our biases and the limits of how much data we can consume.

In the past decade, a discipline of new analytics or predictive analytics leveraging Machine Learning (ML) methods has become more accessible due to cloud and its limitless computing capability. ML allows us to create predictive models. These models are trained on potentially large data sets and learn from patterns that are often indiscernible by humans. These models can in turn provide us with business foresight.

Organizations that leverage ML capabilities have deeper visibility into their business and thus possess a competitive advantage. At Qlik, we are on a journey to provide customers the ability to use ML capabilities on our platform, so that business users can perform analytics on various business critical functions. Examples of this include:

  • optimizing supply chain;
  • improving inventory management;
  • avoiding lost opportunities;
  • detecting and preventing customer churn; and,
  • managing employee attrition.

Unfortunately, most organizations are not equipped for ML. Analyst reports often quote skill gap as a top challenge to AI/ML adoption. The notion is supported by reports from Gartner, Deloitte, McKinsey & Company and others. ML typically requires data scientists, data engineers and like roles that are scarce and expensive. Cross-team collaboration is necessary and requires experience to be done effectively. Another problem we see often is an inability to serve ML to hundreds or even thousands of business users, so that the power of predictions and ML can help drive insights and action for business outcomes.

Democratizing AI for BI

Fortunately, the current renaissance in ML extends to ML tooling. Rapid developments in areas like AutoML are alleviating skill gap challenges.

AWS and Qlik have partnered on this opportunity with the joint vision of democratizing predictive analytics for our customers. Qlik is a leader in BI and is an AWS Data and Analytics Competency Partner.

Qlik has taken the first step to alleviate this skill gap challenge by creating the Qlik-SageMaker Connector. It integrates with Amazon SageMaker, a managed service design to drive and scale the entire ML lifecycle, by bridging the Qlik Sense Server and SageMaker Hosting Services.

Qlik's SageMaker Connector

Through the connector, Qlik Sense users can source real-time predictions from an Amazon SageMaker hosted model through Qlik’s in-memory engine. The connector reduces the complexity and effort typically required to integrate ML models into a predictive analytics solution. It removes the need to build and manage scoring pipelines. It also enables better support for use cases that benefit from fresher data. The integration, currently offered as an open-source connector, is available for customers today as an “early access” technology.

Road Ahead

The connector is the first step in our joint vision to democratize AI on AWS by leveraging AutoML tools like Amazon SageMaker Autopilot. AutoML automates the ML experimentation process and creates an opportunity for non-ML experts to use ML. Coupled with services like Amazon SageMaker, the ML lifecycle can be automated at scale to accelerate time-to-value.

For instance, Amazon SageMaker Autopilot (Autopilot) can ingest tabular data and produce optimized regression and classification models. It automates data scientist and data engineering tasks, such as data analysis, feature engineering, algorithm exploration and hyperparameter tuning. The job of the Autopilot user is to identify a tabular dataset with predictive potential and map it to an ML problem.

Autopilot reduces the required ML skillset to that possessed by many core BI analysts.

Qlik SageMaker Solution

Below we’ve provided a high-level overview of the Qlik SageMaker predictive solution. Figure 1 illustrates the core components.


Integrating SageMaker Models With Qlik

Starting with a Qlik Sense Server and at least one SageMaker model endpoint, all that’s needed is to deploy the Qlik-SageMaker Connector, which consists of the following components:

  • Qlik Rapid API Server: This serves as the bridge between the Qlik Engine and the SageMaker model endpoint. It contains the know-how of interfacing with the Qlik Engine and operating with its in-memory data format.
  • Amazon API Gateway REST API: The Qlik Engine communicates with the model endpoint through a REST API managed by Amazon API Gateway. The API provides a standardized interface for SageMaker models. ML models could have a wide variety of interfaces depending on how the model is built and served.
  • Amazon Lambda Format Transformer: A Lambda function is needed to perform data transformations. The Qlik API Server sends requests in a standard format that needs to be translated into a format compatible with the target model. We recommend using Autopilot whenever possible because it creates and packages transformation logic automatically. Thus, it spares you from writing custom code for each unique model.
  • Session Store: The Qlik Server requires stateful communication. Amazon DynamoDB is used to store session information to facilitate stateful bi-directional communication.

The connector provides various configurations documented here. Configurations include micro-batching and WebSocket API support to facilitate performance tuning.

Predictive Analytics Solutions

Once the connector is deployed and configured, you’ll be able to access predictions from your dashboard. The figure below shows this in the context of a Qlik Sense application.


The below diagram shows Qlik Sense with a real-time interactive dashboard working with Amazon SageMaker.

In this example, a business analyst can trigger SageMaker Endpoint in real-time by making modifications to variables, such as radius and area.


This Qlik Sense application was derived from this SageMaker sample repository.

A real-time demo of this Qlik Sense application is available on the Partner Engineering Qlik Server. Access the Qlik SageMaker Breast Cancer Application demonstration.

Additional Qlik Sense examples showing customer churn and term deposit subscriptions can be found here.

Conclusion

The Qlik-SageMaker Connector is the first step in Qlik’s journey toward democratizing ML capabilities with Qlik Sense. We look forward to seeing the solutions you create with these tools, and the impact it will have on your organizations!

Get started with the Qlik SageMaker Connector:

  • Go for a test-drive by downloading and trying code: Github Repository.
  • Reach out to the Qlik Community to share your experiences. Join Qlik Community and post in the Technology Partners Ecosystem section: Qlik Community.
  • Want to start a conversation with us on how to leverage SageMaker with Qlik Sense? Fill out this form to participate in a private pilot program.

@Qlik connector integrates w/ @Amazon SageMaker to unleash power of #machinelearning by democratizing predictive #analytics

Get ready to transform your entire business with data.

Follow Qlik