How does Explainable AI work?

In the first part we have shown why Explainable AI (XAI) is the foundation for trustworthy and responsible AI. AI is. We will now go one step further and show how XAI actually works, what methods are available and what ethical challenges are associated with it.

Methods of Explainable AI

Transparent models

Some models are inherently explainable - They are also called "interpretable". These include linear regression models and decision trees. These models have the following properties:

• Simulatability: People can understand the model and its behavior - for example, how individual inputs affect the output.
• Dismantlability: Every component, whether input value or hyperparameter, is comprehensible. Pixel values of an image are difficult here, for example, as they are too abstract. 
• Algorithmic traceability: The underlying algorithm is transparent and easy to understand. This makes it possible to analyze how a linear model weights risk factors, for example. 

Post-hoc declarations

Not every AI model is inherently comprehensible. Many powerful models, such as neural networks or ensemble methods, are considered black box models: Their internal processes are complex and almost impossible for humans to understand. Explainability is particularly important here.

In such cases, post-hoc methods are used.
The term "post-hoc" means that explainability is generated retrospectively, i.e. not in the model itself, but is built up additionally. These methods are crucial when it comes to models with non-linear or very complex structures, such as language models, medical diagnostics or image processing.

Model-specific vs. model-agnostic
Model-specific: The aim here is to specifically analyze the inner workings of a particular model - for example, which layers of a neural network process which information. 

Model-agnostic: Instead of examining the model itself, only the relationship between input and output is considered. By making specific changes to the input data, conclusions can be drawn about the decision logic - regardless of the model type. 

Among the post-hoc technologies, the two approaches SHAP and LIME are particularly promising.

Tools in focus: SHAP, LIME & Co. 

SHAP (Shapley Additive Explanations) is based on game theory concepts and evaluates the influence of individual characteristics on the prediction of a model. The method supports both global explanations - for example in the form of bar charts or so-called Beeswarm plots - as well as local analyses such as force plots or heat maps.

LIME (Local Interpretable Model-Agnostic Explanations) takes a different approach. It changes individual data points locally and then creates a simple, linear model that serves as an explanatory model. LIME is particularly suitable for text data, images or structured data. It makes visible which words, image areas or features have made a decisive contribution to the prediction - and thus offers locally valid, intuitive insights. Here, too, an example from real estate valuation can be found: the distance to the city center is the strongest feature - and LIME can make this meaning comprehensible through a visual representation.

Also image-based procedures contribute to explainability. Heat maps are used in object recognition in particular. They mark those areas of the image that contribute particularly strongly to the prediction - often highlighted in red. This makes it clear what the model is actually "looking at" - for example, when differentiating between vehicle types or detecting medical abnormalities.

Ethical challenges of XAI

With XAI, biases can be made visible - but they do not automatically solve all problems. One of the biggest challenges is dealing with unbalanced training data. Discrimination is particularly common with facial recognition technologies: women and ethnic minorities are misclassified significantly more often than white men. The reason often lies in the lack of representation in the training data.

Generative AI also harbors ethical risks. Language models can unintentionally generate hate speech, misinformation or psychologically harmful content. This content is particularly problematic if it appears to be credible answers - for example in counseling situations.

In addition, historical biases can be reproduced from data sets - as in the case of ADHD diagnoses, which were recognized less frequently in women. Without explainable models, such biases remain invisible and can have serious social consequences.

Practical examples from XA

Explainable AI is already being used in numerous application areas today:

Generative AI: Generative models such as ChatGPT can have biases, spread misinformation or provide misleading answers. XAI helps to make such models more transparent by showing how user input is processed and which factors influence the results. Local explanations, such as color coding for high trustworthiness or source references for medical questions, enable a better assessment of the quality of information - and thus strengthen trust in the models.

Medical diagnostics: Responsible AI is particularly important in healthcare, as decisions directly affect people's lives. Historical biases, such as the underdiagnosis of ADHD in women and ethnic minorities, can be made visible through XAI. Tools such as SHAP or LIME help to show which characteristics diagnoses are based on and enable doctors and supervisory authorities to understand decisions. In this way, fairness, transparency and patient safety can be strengthened. 

Autonomous vehicles: With autonomous vehicles, a mistake can have fatal consequences. XAI makes decisions comprehensible - for example, when a speed limit is detected and the vehicle accelerates as a result. This transparency allows users to understand the vehicle's behavior and intervene if necessary. At the same time, it creates the basis for manufacturers' legal responsibility in the event of an accident. Feedback on explainable decisions also helps to continuously make the systems more robust and fairer. Even data protection issues - such as swarm-based learning - can be addressed responsibly in this way.