Explainable AI
What it is, why you need it, and best practices. This guide provides definitions and practical advice to help you understand and establish explainable AI in your organization.
Explainable AI Guide
What Is Explainable AI?
Explainable AI (XAI) refers to a set of techniques and processes that help you understand the rationale behind the output of a machine learning algorithm. With XAI, you can meet regulatory requirements, improve and debug your models, and have more trust in your AI models’ decisions and predictions.
Benefits
Interest in XAI is accelerating rapidly as organizations realize the need for insights into the decision-making process of opaque or “black box” AI models. These insights bring the five main explainable AI benefits below:
1) Better decision-making by understanding how to influence predicted outcomes. In the XAI example below, your predictive model has generated likely outcomes regarding customer churn based on your data.
With XAI, you also get interpretable and transparent explanations for the decisions made by your AI analytics models. In this case, you see prediction-influencer data to explain these outcomes at the record-level. This helps you know how to influence the predicted outcomes. This example shows how the SHAP explainability tool (explained below) tells you that the top 6 features of your product account for 78% of influence on customer churn. You can use this insight to make changes to your product or service and reduce churn.
2) Faster AI optimization by monitoring and evaluating your models. In the XAI example below, you have transparency into which model is the best performing, what the key drivers are, and how accurate the model is.
With black box models, you don’t have this kind of transparency. And when they fail, it can be very difficult for you to identify why the model did not perform as you’d expected.
3) Raise trust and reduce bias in your AI systems by being able to check models for fairness and accuracy. XAI explanations show the patterns your model found in your data. This helps your MLOps team (machine learning operations team) trace any errors and evaluate for bias and data integrity.
4) Increase adoption of AI systems as your organization, customers, and partners gain more understanding of and trust in your ML and AutoML systems. Your AI models can then power your predictive, prescriptive, and augmented analytics systems.
5) Ensure regulatory compliance as the reasoning behind your AI–based decisions can be audited to ensure conformity with the growing slate of laws and regulations. For example, California’s recent Consumer Privacy Act (CCPA) stipulates that your users be able to know conclusions made about them by your AI systems and know what data was used to make those conclusions.
Learn How to Get Started
Download the AutoML guide with 5 factors for machine learning success
Approaches
There’s no single, optimal way to explain the outputs of a machine learning or AI algorithm. There are three main approaches to consider: global vs local, direct vs. post hoc, and data vs. model. Your choice of approach will depend on the requirements of your ML pipeline and who is consuming the explanations (e.g. a data scientist, regulator, or business decision maker).
Global vs local refers to the scope of the explanation:
- Global XAI models provide a high-level understanding of how your AI model is making predictions. They typically summarize relationships between input features and predictions in a high-level, abstract way. For example, a global explanation might highlight the most important features in your model and how they influence the prediction.
- Local models provide specific, instance-level explanations for individual predictions. They show the exact contribution of each feature for a particular prediction. For example, a local explanation for a prediction of a customer's credit risk might show the specific factors, such as income and credit history, that led to the prediction.
Direct vs post hoc refers to the way your model is designed to provide explanations:
- Direct XAI models (“white box”) are designed to produce interpretable predictions from the outset. You choose your model’s architecture, loss functions, and regularization terms with interpretability in mind. For example, decision trees or logistic regressions are types of direct models, because their structure provides a clear and interpretable explanation of the model’s predictions. More complex Gradient Boosted Decision Tree models such as LightGBM, H20, XGBoost, Catboost, and AdaBoost are in-between white box and black box models on the explainability spectrum. Direct XAI models are usually more interpretable and provide more transparent explanations than post hoc.
- Post hoc models (“black box”) were not originally designed to be interpretable. Still, an explanation can be generated after the fact, often using a separate tool or process, rather than being built into the model itself. For example, a neural network is a type of post hoc model, because it can produce predictions that are difficult to interpret. However, techniques such as SHAP (SHapley Additive exPlanations) can be used to generate post hoc explanations of its predictions. Unlike direct models, post hoc explanations can provide additional insights into the workings of more complex models.
Data vs model refers to the type of explanation being provided:
- Data XAI models provide an explanation based on the ML-ready input data and the relationships between the features. This type of explanation focuses on the relationship between the input features and the predictions, and how changes in the features lead to changes in the predictions. For example, a decision tree model is a type of data XAI model, as it provides an explanation based on the splits in the tree and how the input features influence the prediction. Data XAI models typically provide more transparent explanations than model XAI models.
- Model explainable AI models provide an explanation based on the internal workings of your model. They focus on how the model processes the input data and how the internal representations are used to make predictions. For example, neural networks are a type of model XAI model, because they provide explanations based on the activation patterns in the network and how the activations influence the prediction. Model XAI models provide deeper insights into the workings of your model than data XAI models.
Challenges
Having your XAI provide explanations that are both accurate and easy to understand involves many challenges.
XAI models can be:
- Complex and difficult to understand, even for data scientists and machine learning experts.
- Challenging to verify the correctness and completeness of XAI explanations you receive. While first-order insights may be relatively simple, the audit trail becomes harder to follow as your AI engine interpolates and reinterpolates your data.
- Computationally intensive, which can make it hard for you to scale for large AI datasets and real-world applications.
- Unable to provide explanations that generalize well across different situations and contexts.
- Requiring a trade-off between explainability and accuracy as your XAI models may sacrifice some level of accuracy in order to increase transparency and explainability.
- Difficult to integrate with your existing AI systems, requiring significant changes to existing processes and workflows.
Best Practices
Here are key best practices of implementing Explainable AI (XAI) in your organization:
- Set up a cross-functional AI governance committee that includes not only technical experts, but also business, legal, and risk leaders. This committee will guide your AI development teams by defining the organizational framework for XAI and determining the right tools for your needs. This group will also set standards according to different use cases and associated risk levels.
- Invest in the appropriate talent and set of tools to implement XAI in your organization and to stay up to date with this rapidly evolving space. Your choice of using custom, off-the-shelf, or open-source tools will depend on your short- and long-term needs.
- Clearly define your use case or problem and the decision-making context in which your XAI will be used. This helps ensure that you understand the unique set of risks and legal requirements for each model.
- Consider your audience for your XAI system and what level of explanation they will need to understand it.
- Choose appropriate XAI techniques for the problem and use case you’ve defined, such as feature importance, model-agnostic methods, or model-specific methods.
- Evaluate your XAI models using metrics such as accuracy, transparency, and consistency to ensure they are providing accurate and trustworthy explanations. This may require that you weigh trade-offs between explainability and accuracy.
- Test your XAI models for bias to ensure that they are fair and non-discriminatory.
- Continuously monitor and update your XAI models as needed to maintain their accuracy, transparency, and fairness.
Lastly, you should ensure that your XAI models adhere to the four principles of explainable artificial intelligence as defined by the National Institute of Standards (NIST):
- Explanation: Accompanying evidence or reasons are provided for all outputs by the system.
- Understandable: The system's explanations are tailored to be comprehended by each user.
- Accuracy: The output's generation process is accurately reflected in the explanation.
- Knowledge Limitations: The system only functions within its designed parameters or when it attains a confident level in its output.
Techniques
The specific XAI techniques you employ depends on your problem, the type of AI model you use, and your audience for the explanation. Below are the main XAI techniques used to produce explanations that are both accurate and easy to understand.
- Feature importance: This technique highlights the most important input features that contribute to a particular AI decision.
- Model-agnostic methods: These techniques provide explanations that are not specific to any particular AI model, and can be applied to any black-box model. Examples include saliency maps and LIME (Local Interpretable Model-agnostic Explanations).
- Model-specific methods: These techniques provide explanations that are specific to a particular AI model, such as decision trees and rule-based models.
- Counterfactual explanations: This technique provides explanations for AI decisions by showing what would have to change in the input data for a different decision to be made.
- Visualization: Visualization tools such as graphs, heatmaps, and interactive interfaces can be used to provide clear and intuitive explanations for AI decisions.
Learn more about AutoML and XAI
