Bayreuth Lectures - Explainable AI with Large Language Models

Artificial intelligence systems based on machine learning algorithms (ML models) are often presented as black boxes, which means that it is difficult for users to understand how and why they produce their outcomes. In literature, this is usually referred to as the black box problem. Resolving this problem constitutes the fundamental objective of the recently-born research programme of Explainable Artificial Intelligence (XAI). One major challenge in XAI involves formulating explanations of ML models' reasoning and outcomes that are understandable to users with limited expertise in AI and related fields. This challenge requires the development of XAI tools that can scrutinise the complex low-level and sub-symbolic inferences executed by an ML model and translate them into natural language explanations that non-expert users can easily read and grasp. Next-generation large language models (LLMs), in particular chatbots based on generative pre-trained transformers, e.g., OpenAI ChatGPT, seem to be particularly suitable for this task.

These models can be easily trained to recognize inferences made by black-box AI systems and explain them via natural language texts that are easily comprehensible to users. The quality of these explanations might also be improved through reinforcement learning mechanisms that leverage users’ feedback. However, the use of LLMs as XAI tools introduces unexpected risks and presents us with new challenges. Unlike conventional XAI techniques, LLMs themselves are equally, if not more, opaque than the ML models they should explain. This raises an additional “meta” black-box problem: how can one places trust in an explanation provided by an LLM-based XAI tool if one does not understand how this explanation has been generated?

In his lecture, Alberto Termine will explore more in-depth the challenges and risks raised by the use of LLMs for XAI purposes. He will particularly focus on the phenomenon of hallucination, i.e., the tendency of LLMs to produce explanations that seem reliable but, in reality, include a plethora of false and potentially misleading information. He will contend that hallucination severely undermines the reliability of LLMs as XAI tools, and that addressing this issue constitutes the primary challenge in the development of safe, fair, and trustworthy LLM-based XAI systems.