What You See Is Rarely All There Is

Gaining wisdom and insights from data requires you to look beyond the obvious to really understand what is happening

In last month’s blog, we introduced you to a 12-step systematic process to help guide you through making data-informed decisions. This process is unique because it is not just about understanding analytics. It is also about understanding psychology and business anthropology as well.

I thought it would help to provide an example of the 12-step process in action that is not specific to business. This is a real-life example that is close to my heart, to demonstrate that this process impacts all types of decisions, including personal ones.

Many years ago, when my son was just starting elementary school, he started having a lot of ‘behaviors’ while he was at school. The school staff, as well as my wife and I, were concerned about the escalating behaviors. After an initial meeting to discuss the situation, the decision was made to give my son a one-to-one aide to help keep him on task and provide support when he needed it. A few months went by and we did not see the behaviors decrease. In fact, the behaviors continued to escalate.

Step 1 – Turn the business questions into analytical question(s).

Decision-making should always start with the goal in mind. Going into a meeting and asking a question like “How is my son doing” is not really a good analytical question. Analytical questions need to be clear, specific, scoped, quantifiable and data-oriented. In this example, the goal was trying to decrease behaviors of my son. We had turned that goal into multiple analytic questions such as “Are the behaviors decreasing since the introduction of a 1-1 aide.”

Step 2 – Find and source all relevant data.

At the team meeting we asked to see the data collected. The school had provided logs where they had performed data collection on behaviors.

Step 3 – Ensure the sourced data is trusted.

We then asked the school how the data was collected. They mentioned that the teacher would report on the number of behaviors at the end of each day. At this point we were not completely sure we could trust the data. The teacher may miss behaviors or may have forgotten to document them. Even if the teacher did collect the information accurately, was any of that really useful? Did that, in and of itself, help us understand how to resolve the escalating behaviors?

Step 4 – Create a measurement framework to describe your data with KPIs.

We asked the school to continue capturing the data, but we wanted to revisit the Acquire phase and asked them to collect additional data. Specifically, what time of day the behavior happened, what my son was doing before, what the specific behavior was (using a categorization of 4 types of behaviors), and what his consequence was as a result of his behavior.

Step 5 – Use exploratory analytics to find patterns and trends and relationships that may exist and not be obvious to start to drill into root cause.

Remember descriptive analytics like a dashboard/measurement framework do not answer questions like why. In this way, it’s more like reporting vs analytics as most people think of it. We know now the behaviors are escalating. But do we know why? Can we make a decision on how to course correct if we do not know why? Enter exploratory analytics (also called diagnostic analytics) to help us try to create a hypothesis into what may be happening.

Through exploratory analytics, along with the additional data we asked to be collected, we were able to view the data across multiple dimensions to see that the overwhelming majority of the incidents happened at times of the day when the setting was unstructured and he did not have a one on one aide (like during lunch and recess).

Step 6 – Review and orientate yourself to the information and data so far, and apply your personal experiences to it.

At the next school meeting, this was discussed. Since the behaviors are happening during unstructured times without an aide, the logical decision would be to request the aide to stay with my son during those times. This is what was recommended by the school. However, this is where you need to challenge the data, apply your perspective and experience, and get input from others who may have different perspectives.

Step 7 – Challenge the data and look for information and data to disprove it.

My wife and I did challenge the data and apply our experience. We went back to the data and exploratory analytics and were able to determine that for just about all of the behaviors, the consequence was for our son to either be sent to the Principal’s office or to go back in the classroom with a teacher. My wife and I applied our perspective and knew that our son craved adult attention for stimulation and would rather be with adults than kids. People without this experience would just apply their own mental model that a punishment of being sent to the Principal’s office would be seen as a negative thing.

Step 8 – Review with a cognitively diverse team (or if you are alone, be aware of your bias and play devil’s advocate and reframe).

As a result, we came up with a new hypothesis. Our son was having the behaviors because he wanted adult attention. According to the data, the consequences were such that he was getting that adult attention when he acted up. What was happening was actually enabling the behavior.

Step 9 – If applicable, leverage predictive analytics to run simulations or similar to test potential decisions/solutions.

It was not applicable to run simulations on this, but we did want to go back to the Acquire phase and get more data. In this case we decided to talk to the Principal and get her input on the time our son is in her office. She highlighted that he was always happy and engaged and always asked lots of questions. This was enough evidence to help us feel confident to move forward with our new decision. The decision was when these behaviors happened, our son was not to be given time with an adult. He was to be sent to his desk by himself.

Step 10 – Announce your decision at the right level to ALL stakeholders (direct, indirect, upstream, and downstream) by leveraging the Rule of 3 and Pyramid Principle in your storytelling.

We ensured that at the school every single adult ranging from the bus driver, the cafeteria workers, any teacher, any staff at all were made aware of this decision and provided instructions in the form of a decision tree what to do when certain behaviors came up in the future.

Step 11 – Setup a review mechanism to monitor the impacts of the decision after it is made and acted upon.

When we saw the next round of data one month later, we saw that behaviors during that time period in fact trend downward. There was a spike in the beginning, but then gradually it came down.

With change we know things may get worse before they get better. This is why it is important to fully analyze and understand the data to ensure not only root causes are investigated but to see if the data truly shows a new pattern or just some symptomatic waves from the change being made.

Step 12 – Leverage that review mechanism and fail/fix/learn fast including improvements to data, measurement frameworks, accountability, decisions, and anything else relevant.

This step was actually leveraged multiple times through this example as we would re-assess and then update our decisions based off the insights we had learned. This included going back and updating the measurement framework to include more sources of data, like time of day, what was happening before the behavior and what the consequence was after. This information gives us much more abilities to gain future insights from.

I know this is a personal example, but this framework and the iterative process was critical in us determining what the best decision was. It was not and is not a perfect answer from the get-go. It can take cycles of going through this and learning more each time.

This same concept can be applied to all levels of decisions in business too. The answer is somewhere in your data, but it requires a systemic and systematic framework like this, a focus on what goals are really important, and the combination of data plus the diverse perspectives from a diverse group to get it out. This is data-informed decision making in action.

Decision making is not an event, it is a non-linear process. Our own Kevin Hanegan shares a personal example of this 12-step process in action!


In this article:

Keep up with the latest insights to drive the most value from your data.

Get ready to transform your entire business with data.

Follow Qlik