- Basing decisions on data that is incorrect and not trusted
- Basing decisions on incorrectly built and interpreted visualizations and analytics
- Incorrectly communicating decisions
- Making ineffective decisions based either solely on intuition or solely on data
- Failure to make any decisions because of the fear there is not enough data or due to incorrect mental models
These issues cost organizations millions of dollars in lost opportunities and added costs. Below are some of the common symptoms we see in organizations related to this.
A very common scenario when an organization is trying to make a data-informed decision is that the data needed for the analysis is at some level not correct or trusted. Maybe there is no standardized data definitions or standardized calculation used for a measure. As a result, the data is not accurate or trusted. Another common symptom seen is that decision makers within an organization do not even have access to the right data needed to make appropriate decisions.
Incorrectly Built Analytics
Individuals and organizations who are not data literate will be more likely to use less than ideal analytics and visualizations as part of their data-informed decision making process. This could be as simple as analyzing some measure and calculating it using the wrong aggregation method. Or, it could be more complex like excluding information (data or context) that is vital to making the best decision. In some cases, the correct calculation on a given measure is appropriate for the analysis, but maybe the measure itself was not appropriate to use for the current situation. That measure may not be an accurate Key Performance Indicator (KPI) that ties to organization goals. Organizations then end up driving behaviors and performance to a measure that is not positively impacting the organizational goals. In fact, in most cases, it ends up hurting the organization.
One example of this would be a store that is trying to measure sales performance of their various locations. They created a KPI that measures sales performance and compares the performance to other locations. Rather than driving the right behaviors, it caused locations to pursue and aggressively market the same customers, even offering higher discounts. This had an overall impact of lowering sales, and also higher costs (as the products being bought now had to be delivered from the warehouse to locations that are farther away).
In many organizations, different people are responsible for building analytics and visualizations than those who actually need to consume them to make decisions. In these cases, the consumers of analytics can misinterpret the data or analytics, leading to incorrect conclusions and insights.
One famous example of this is when the engineers in charge of a part on the U.S. Space Shuttle Challenger tried to get the launch stopped because they believed the part would get damaged during the takeoff, which could then lead to a massive explosion. Those engineers showed a visualization to NASA that included only a subset of the data they used to gain their insights. That visualization was misinterpreted by NASA and deemed that the launch was not at risk. The launch happened and unfortunately the space shuttle exploded during takeoff. The after-action report highlights multiple lapses in process and also groupthink, so this is just one example from that report.
In some cases, the interpretation of a given analysis could be incorrect due to a wrong deduction or having an incorrect mental model. Maybe a visualization is created that is accurate, but it is void of context which can lead to misinterpretation. Or, maybe an individual interprets something with a correlation and believes that means there is a causation and acts on it. Potentially an individual is looking at descriptive analytics and incorrectly infers a root cause from this (ignoring the exploratory analytics process). Potentially the analytics are correct, but the individual has a cognitive bias that causes them to have a blind spot in their reasoning which leads to a less than ideal decision. Or, the group of people making the decision suffers from groupthink and a lack of a decision making process.
One famous example of groupthink is the decision of the United States to invade Cuba during the 1960s in the Bay of Pigs conflict.
Incorrectly Communicating Decisions
Another common symptom of lack of data literacy and lack of a data-informed decision making process is when decisions struggle to take hold within an organization. This could be due to a lack of understanding how to communicate the decisions to all the stakeholders. One common mistake is to use the same visualizations and analytics used for gaining the insights to also communicate them.Another common mistake is to not communicate the thought process for the decision. In this case, an organization with a culture that does not trust data is likely to not embrace the decision.
One famous example of ineffective communication is the failure of project managers and engineers which then resulted in a deadly collapse of a Hyatt Regency Walkway.
Blindly trusting the data
There are many situations where an organization will make a decision with not enough data, but another common situation is when an organization over relies on data alone. Data alone is not a magic bullet for the best decisions. Even if an organization has good data, decisions are typically made with assumptions. If those assumptions are not correct, you will make a bad decision regardless of the data. This is why it is important to marry data with intuition and other factors. Data cannot replace intuition, but when data is used along with intuition and other factors, flawed decisions are minimized.
How Can Organizations Improve
All of these symptoms occur in organizations all the time, leading to ineffective decisions and ultimately to worse performance of the organization as a whole. How can organizations improve on this? Follow these 4 steps:
- Start with data
literacy education for everyone who touches data and analytics and makes
decisions. Understanding data fundamentals and how to read data, then how to work
with and analyze the data properly, and also how to argue with the data rather
than just blindly following what it say
- Adopt a data and
- Leverage a systemic
and systematic process for making data-informed decisions.
- Evolve to a
culture that embraces data literacy and data-informed decision making.