Similar to hiring decision makers in the organization, it is important to understand how AI makes decisions. Many organizations want to use AI, but don't feel comfortable letting the model or AI make more effective decisions because they don't yet trust the model. The predictions may be accurate, but without a deeper understanding, the path to the outcome is hard to follow. Data scientists are then faced with the problem of hardly being able to extract any insights from the models. Explainable AI helps by providing insights into the decision-making process of models. It provides answers to three questions:
- Which features in the data does the model classify as particularly important?
- How does each characteristic affect the result of a single prediction of the model?
- How does each feature affect the predictions of the model in the holistic view when it is considered over a large number of possible predictions?
With the increasing complexity of artificial intelligence, the relevance of Explainable AI is growing. It provides an insight into the "brain" of the AI in order to understand not only the result, but also the calculation path. This creates trust and reduces inhibitions about using AI in a productive environment. Data scientists thus benefit from the transparency of their models, as management and those responsible want to understand the decisions.
The main goal of Explainable AI is to create more explainable models while maintaining a high level of learning performance and prediction accuracy. It optimizes the use of AI in the production environment through in-depth model and data exploration. The results bring benefits for all industries and companies: increased efficiency and lower costs.
In AI projects, however, there is often a lack of clarity about the reasons for Explainable AI. It makes AI models more transparent, fairer and more reliable. This article describes why this is the case, how it works and what added value it provides.
Benefits of Explainable AI
Interpretable models help to achieve some of the goals of Machine Learning projects:
- Trust: Without checking some basic facts, many people tend to doubt a model's ability to make a decision. Given the frequency of data errors, this is a wise precaution. Confidence in models increases when insights provide a better understanding of the problem.
- Causality: The data basis of a model can be unreliable, disorganized and incorrect. It is therefore a potential source of error for false correlations. Understanding which patterns a model finds in the data helps to detect random correlations.
- Increased effectiveness: With the knowledge of what role individual features play in model prediction, new features can be created by transforming the raw data (also called feature engineering). Sometimes this process is guided by domain knowledge, but with interpretability, features can be added to the model to increase accuracy.
Explainable AI and the trade-off between interpretability and accuracy
As a rule, simpler machine learning methods such as linear regression or decision trees are easier to interpret than more complex methods such as artificial neural networks. This is because the longer and more complicated the mathematical formula on which a method is based, the more difficult it is to follow the calculation path. However, this also allows a method to react more flexibly to different patterns in the data so that, for example, non-linear relationships can be better mapped. In principle, this is therefore referred to as a compromise between interpretability and accuracy.
Levels of interpretability in Explainable AI
When it comes to the interpretability of models, a distinction is made between global and local interpretability. This refers to the levels of interpretability, which can have different relevance depending on the application.
Local interpretability
This level of interpretability is about understanding an individual prediction of a model. For example, if you want to investigate why a particular customer is classified as "bad", the local interpretability of the model is essential.
Global interpretability
This level of interpretability is about understanding how the model makes decisions. This is based on a holistic view of the features and each of the learned components such as weights, other parameters and structures. Global interpretability makes it possible to understand which features a model takes into account for its decisions. In finance or the social sciences, it is important that users understand the model. If, for example, probabilities of loan defaults are calculated, it is not enough to simply classify a customer as "good" or "bad". Credit approval authorities need a clear scorecard to justify the basis for this classification.
Techniques for Explainable AI
The implementation of Explainable AI requires different approaches depending on the model. These can be divided into two categories: Model-specific and model-agnostic. There are countless techniques for both types, not all of which can be shown in detail here. To give an insight into the application of Explainable AI, some of the best known and most commonly used are presented below.
Let us first look at the model-specific techniques: Model-specific techniques are specific to a single model or a group of models. These tools depend heavily on the functionality and capabilities of a particular model, such as tree interpreters. A tree interpreter can be used to track the results of the decision tree. Each node contains the conditions according to which the tree makes a decision and moves to the next node, e.g. leaf size (petal length) <= 2.45cm. If the condition is true, the index jumps to the left node on the next level. If the condition does not apply, the index selects the right-hand node. If the index arrives at an end node, the result contained in the node is output.
Other model-specific interpretation techniques deal with the explanation of artificial neural networks. Since neural networks, and the relationships and data representations they learn, are so complex and abstract that even experts can no longer understand them, they are also referred to as a black box. Interpretation techniques related to Explainable AI shed light on the black box. They help to understand how a neural network arrived at its result. In image recognition, for example, heat maps are used to illustrate the patterns learned by the model. A high level of traceability in this area should not be underestimated, as it also identifies inaccuracies in the recognition process. The model interpretation can then be used to determine whether a model needs to be retrained or the data needs to be labeled differently.
In contrast to the model-specific techniques, model-independent tools can be used for any machine learning model, no matter how complicated it is. These agnostic methods usually work by analyzing input and output pairs of features. Common frameworks are LIME (Local Interpretable Model-agnostic Explanations) for classification models and SHAP (SHapley Additive exPlanation) for regression models. Both frameworks examine how a slight change to individual input variables affects the result. In this way, it is recognized whether individual features have contributed to the result or not. The "importance" of the features can be displayed in diagrams.
Explainable AI refers to the insight into AI models on several levels. Depending on the area of application, it may be more important to understand the path to an individual prediction or the model in its holistic mode of operation and decision features.
Techniques and tools to increase explainability
can be tailored to procedures or function independently of procedures. The former provide a deeper insight into the model and thus also explain approaches that are difficult to interpret, such as neural networks. The latter primarily examine the effect of individual characteristics on the result. They support the selection of models and the selection of decisive features.
What both types of techniques have in common is that they increase confidence in the predictions and close the gap between more accurate but non-transparent approaches and those that are easy to understand. Explainable AI therefore ensures that fewer compromises have to be made when deciding between interpretability and accuracy.
The safe and reliable integration of AI requires human and artificial intelligence to complement each other. Techniques that enable a qualitative assessment of the decision rules used should form the basis for corresponding recommendations for action when adapting the model.
The traceability of models is not just a scientific question. It helps to understand where models fail, how to fix them and how to explain them to project stakeholders so that everyone knows exactly how a model generates value.
