Data Products

At a high level, a data product is a domain-specific, consumable entity aimed at transforming data into actionable insights for stakeholders and AI systems, enabling informed decision-making processes. The goal of data products is to make data accessible, consumable, insightful, and actionable for the increasing number of stakeholders and generative AI that rely on data to inform decision making. These products are created for a specific purpose and have defined and agreed-upon shapes, consumption interfaces, and maintenance and refresh cycles.

Essentially, data products are like self-contained containers that directly address business problems internal to an organization or are monetized externally. They leverage data analysis, processing, and visualization techniques to generate meaningful insights or support AI and machine learning, presenting them in a way that is easy to use.

Here are the 6 characteristics of a well-designed data product:

1. Prepared: Cleaned, transformed, high-quality data ready for analysis.

2. Findable and Understandable: Metadata-driven, domain-centric assets built for effective use.

3. Interoperable: Comprised of one or more datasets that work with each other to bring holistic, unbiased data insights.

4. Shareable: Several datasets and data elements packed into a single trusted cohesive unit, making it easy to distribute.

5. Accessible: Accessible to data consumers when needed in a standardized manner.

6. Reusable: Built of composable elements that can be used to build several data products, as well as derivative data products.

Data products act as a bridge between data producers and data consumers. As a result, according to the Harvard Business Review, organizations that adopt the data product approach have the potential to accelerate the implementation process for new use cases by up to 90%.

Data producers work with data sets (which can be in a table, view, an ML model, or streamed) and the published data model, schema, and business-approved metadata. Your data will likely be from multiple data sources and can be raw data or transformed data.

Data consumers interact with data products via self-service but controlled access to data through visualization, APIs, SQL, and other secure options.

Data products are domain centric, modular and reusable, and are built by a designated owner using product management principles. A data products catalog organizes all data products within your organization. Here we describe each aspect in more detail:

Domain Centric. The domain model enhances data understanding and adds business context. It does this by housing the business logic of transformations, analytics calculations, metrics, and machine learning. It acts as a semantic layer, exposing the business-friendly domain data and insights, while abstracting the technical details.

Modular And Reusable. They are built once and then reused multiple times in various use cases, so new ones can be built using existing ones.

Product Management Principles. Each one is owned and managed by a domain-aligned data team that is responsible for its success in delivering value, satisfying and growing data users, and maintaining its lifecycle. And, every data product goes through versions and enhancements based on customer feedback.

Data Products Catalog. As we discuss in more detail below, this type of catalog is a foundational capability that lets both data producers and data consumers speak the same language. It serves as a centralized marketplace, providing detailed information about each data product, including its purpose, data sources, processing methods, and intended audience.

Here are three top use case examples of data products:

Decision Support. GPS navigation applications serve as decision support data products. They provide real-time guidance to users, helping them make informed decisions about routes and directions.

Algorithms. One of the most well-known examples are recommendation engines used by companies like Netflix and Amazon to suggest products or content based on user behavior and preferences.

Automated Decision-Making. A self-driving car exemplifies automated decision-making. It relies on complex algorithms to make real-time decisions without external user intervention.

Before you begin, you should establish a product management team, led by a data product manager, who steers the lifecycle of these products to meet business objectives. The team should also comprise analysts, data engineers, user experience designers, and data architects.

This team collaborates closely with data consumers to understand their requirements and translates those needs into actionable products. Plus, data products require ongoing management and continuous monitoring – just like real world products.

Here we describe the following key steps of a data product lifecycle:

1. Conceptualization: This stage involves identifying domain-specific business needs – understanding the users, their use cases and how they typically consume data.

2. Design: This stage involves shaping the solution, which includes identifying datasets needed to address the identified need, the sources of that data, and defining the characteristics of the data product by functionality, interactivity, presentation approach, etc.

3. Engineering: This stage involves building the technical infrastructure and functionalities of the data product. This includes gathering and integrating the necessary raw datasets from various sources, ensuring data quality, and the transformation processes.

4. Deployment: This is the “making it accessible” stage, which involves ensuring data product documentation is complete, assigning ownership, integrating with existing business systems or consumption workflows for seamless access, and activating the data product essentially making it ready for consumption.

5. Marketing: Activating the data product in the marketplace is just the first step. This stage focuses on creating awareness and promoting usage. This involves consumer training and setting up communication channels to capture consumer feedback.

6. Usage: With a successful launch, consumers interact with the data product in a self-serve fashion to search, find, understand and gain access to the data product. This stage focuses on monitoring usage and gathering feedback. Typically, data is collected on how users interact with the data product, how the data product is performing, and whether a data product is meeting the domain-centric needs of the business.

7. Maintenance: Data products are not static entities; they need to be improved to ensure continued value. Based on the user feedback collected and the evolving needs of the business, new data dimensions, features, and functionalities are proposed to be added into the next version of the data product.

The gap between data producers and data consumers is only growing wider and, ultimately, this increases the time-to-value for your data. A data product catalog provides a comprehensive repository that organizes and documents various data products within your organization. It serves as a federated resource, providing detailed information about each data product, including its purpose, data sources, processing methods, and intended audience.

It facilitates efficient discovery, understanding, and utilization of your data assets, enabling you and other stakeholders to make informed decisions and leverage data effectively for business processes and analytics.

So, both a traditional data catalog and a data product catalog are needed in your enterprise data stack. Let’s compare them side by side:

Sometimes people confuse these two concepts. While they share some similar aspects, they’re fundamentally different. In summary, “data as a product” emphasizes strategic importance within a data mesh, while “data products” refer to tangible outputs created from that data. Both concepts are essential for effective data management and utilization.

Let’s compare them in terms of monetization: