Qlik Cloud and Amazon SageMaker: Expanding the Power of Predictive Analytics in the Cloud Together

In the past decade, a discipline of new analytics or predictive analytics leveraging machine learning (ML) methods has become more accessible due to cloud and its limitless computing capability. ML allows us to create predictive models. These models are trained on potentially large data sets and learn from patterns that are often indiscernible by humans. These models can in turn provide us with business foresight.

Organizations that leverage ML capabilities have deeper visibility into their business and thus possess a competitive advantage. At Qlik, we are on a journey to provide customers the ability to use ML capabilities on our platform, so that business users can perform analytics on various business critical functions. Examples of this include:

• optimizing supply chain
• improving inventory management
• avoiding lost opportunities
• detecting and preventing customer churn
• managing employee attrition

Our new SageMaker integration increases the breadth of advanced analytics capabilities already available in Qlik Cloud, providing seamless integration to Amazon’s advanced machine learning capabilities all via Qlik’s Active Intelligence Platform. Those capabilities include: Insight Advisor, which auto-generates advanced analytics and insights, automates creation and data preparation, and supports natural language interaction; and Qlik AutoML, a no-code automated ML capability aimed at helping business analysts and analytics teams create models and perform predictive analytics in a complementary fashion to data scientists.

Unfortunately, most organizations are not equipped for ML. Analyst reports often quote skill gap as a top challenge to AI/ML adoption. The notion is supported by reports from Gartner, Deloitte, McKinsey & Company and others. ML typically requires data scientists, data engineers and like roles that are scarce and expensive. Cross-team collaboration is necessary and requires experience to be done effectively. Another problem we see often is an inability to serve ML to hundreds or even thousands of business users, so that the power of predictions and ML can help drive insights and action for business outcomes.

Fortunately, the current renaissance in ML extends to ML tooling. Rapid developments in areas like AutoML are alleviating skill gap challenges. AWS and Qlik have collaborated on this opportunity with the joint vision of democratizing predictive analytics for our customers. Qlik is a leader in BI and is an AWS Data and Analytics Competency Partner.

Qlik has taken the first step to alleviate this skill gap challenge by creating Qlik Sense SaaS, which integrates with Amazon SageMaker and Amazon Comprehend, a managed service designed to drive and scale the entire ML lifecycle, by bridging the Qlik Sense Services and SageMaker ML Platform.

Advanced Analytics Integration

With the SaaS version of Qlik and Qlik Sense Cloud newly issued as a service, Qlik can now communicate bidirectionally with new and exciting SaaS services on the Cloud. The connectors securely encrypt, connect and send data from Qlik Sense Cloud to complimentary products, such as SageMaker. Through the connectors, Qlik Sense users can source real-time predictions from an Amazon SageMaker hosted model through Qlik’s in-memory engine. The connector reduces the complexity and effort typically required to integrate ML models into a predictive analytics solution. It removes the need to build and manage scoring pipelines. It also enables better support for use cases that benefit from fresher data.

For a detailed tutorial please follow community article Analyzing customer churn using Qlik Sense & Amazon SageMaker.

Road Ahead

It is our joint vision to democratize AI on AWS by leveraging AutoML tools like Amazon SageMaker Autopilot. AutoML automates the ML experimentation process and creates an opportunity for non-ML experts to use ML. Coupled with services like Amazon SageMaker, the ML lifecycle can be automated at scale to accelerate time-to-value. For instance, Amazon SageMaker Autopilot (SageMaker Autopilot) can ingest tabular data and produce optimized regression and classification models. It automates data scientist and data engineering tasks, such as data analysis, feature engineering, algorithm exploration and hyperparameter tuning. The job of the SageMaker Autopilot user is to identify a tabular dataset with predictive potential and map it to an ML problem. SageMaker Autopilot reduces the required ML skillset to that possessed by many core BI analysts.

Qlik SageMaker Solution

Below we’ve provided a high-level overview of the Qlik SageMaker predictive solution using Qlik SaaS. Figure 1 illustrates the core components.

Integrating SageMaker Models With Qlik Sense SaaS

Starting with a Qlik Sense SaaS and at least one SageMaker model endpoint, all that’s needed is to activate the Qlik-SageMaker Connector, which consists of the following components:

The connector provides various configurations documented here: https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/LoadData/ac-amazon-sagemaker-overview.htm

Predictive Analytics Solutions

Once the connector is deployed and configured, you’ll be able to access predictions from your dashboard. The figure below shows this in the context of a Qlik Sense application.

The below diagram shows Qlik Sense with a real-time interactive dashboard working with Amazon SageMaker.

In this example, a business analyst can trigger SageMaker Endpoint in real-time by making modifications to variables, such as radius and area.

Additional Qlik Sense examples showing customer churn and term deposit subscriptions can be found.

Qlik Application Automation and SageMaker Autopilot

Additionally with acquisition of Blendr.io in late 2020 Qlik Sense SaaS has ability to use Qlik Application Automation to create SageMaker models. This is designed to allow data engineering teams using Qlik Sense SaaS to take data and generate Machine Learning Models.

Following Figure Shows Qlik Application Automation able to create AutoML jobs to automate the model training and generation.

After the AutoML job complete a SageMaker Endpoint will be automatically created and either Qlik Sense SaaS as seen above can access the endpoint or Qlik Application Automation can run a reload task that loads data through the endpoint.For us this is an important capability because it reduces the complexity of creating models and continuously reloading and reevaluating the data against a machine learning model.

The following figures show re-evaluating the data via Invoking SageMaker Endpoint with Qlik Application Automation and reloading the data to spreadsheet.

Finally, Qlik Application Automation can be used to create processes that work with Qlik Sense and SageMaker to continuously update and reevaluate the data. In the figure below Qlik Application Automation will take the data Invoke Endpoint (evaluate against model) and perform a reload of new data.

A detailed community example showing Qlik Application Automation working with SageMaker Autopilot can be found in this Qlik community article.

Conclusion

Qlik is democratizing ML capabilities with Qlik Sense SaaS. We look forward to seeing the solutions you create with these tools, and the impact it will have on your organizations!

Get started with the Qlik SageMaker Connector:

• Sign up for Qlik Sense SaaS and request “Early Access”
• Reach out to the Qlik Community to share your experiences. Join Qlik Community and post in the Technology Partners Ecosystem section: Qlik Community.
• Want to start a conversation with us on how to leverage SageMaker with Qlik Sense? Fill out this form for more information.