XAI in LLM Fine-Tuning

Fine-tuning changes model behavior with labeled examples, instruction data, or preference feedback. We can split explanations of a fine-tuned model into two groups. Local explanations explain a specific prediction or a small set of predictions. Global explanations attempt to characterize what the model represents or computes in general.

Local explanations covered here are feature attribution, attention-based explanation, example-based explanation, and natural language explanation. Global explanations covered here are probing-based explanation, neuron activation explanation, and concept-based explanation. Many of these approaches should look familiar — most of them appeared earlier in this unit for other model types.

Local: Feature Attribution

There are four primary feature attribution approaches used in LLMs.

• Perturbation-based explanation. Perturb input examples by removing, masking, or altering input features — embedding vectors, hidden units, words, or tokens — and evaluate how the model output changes. These are the same occlusion-based methods discussed in the Explainable Attention section.
• Gradient-based explanation. Determine the importance of each input feature by analyzing the partial derivatives of the output with respect to each input dimension. The magnitude of the derivative reflects the sensitivity of the output to changes in that input. Integrated Gradients, covered in the Explainable Attention section, is the primary approach here.
• SHAP. Chen et al. and others adapted SHAP to transformer-based language models in 2023 (Zhao et al., 2024, survey). TransSHAP focuses on adapting SHAP to sub-word text input and provides sequential visualization explanations.
• Decomposition-based methods. These techniques break down a relevance score into linear contributions from the input. Layer-wise relevance propagation (LRP), covered in the Explainable Attention section, is a canonical example.

Local: Example-Based Explanation

Example-based explanation covers two families discussed elsewhere in this unit: counterfactual explanations, which reveal what would have happened under certain observed input changes, and influential instances, which characterize the influence of individual training samples by measuring how much they affect the loss on test points.

State-of-the-art adversarial example methods for LLMs leverage masked language models and adopt a "mask-then-infill" procedure for token-level perturbations such as replacement, insertion, and merging (Garg & Ramakrishnan, 2020; Li et al., 2021).

A more recent method called SemAttack (Wang et al., 2022) proposes a more general framework that operates directly on embedding spaces, including knowledge and contextualized semantics. It iteratively optimizes perturbed embeddings to meet an attack goal.

Local: Natural Language Explanation

Natural language explanation explains a model's decision-making on an input sequence with generated text. The approach trains a language model using both original textual data and human-annotated explanations.

This has been done for a variety of datasets and tasks (see the survey by Zhao et al., 2024). In 2019, the Common Sense Explanations (CoS-E) dataset was introduced and used to train language models to automatically generate explanations usable during both training and inference. A more recent 2022 study examined few-shot out-of-domain transfer learning of natural language explanations from a parent task with many natural language explanations to a child task with few.

Generated natural-language explanations are especially tempting because they read well. They are outputs from the same fallible model. Test them by perturbing the cited evidence and checking whether behavior changes as claimed. For data debugging, influential-instance methods can connect a surprising response to fine-tuning examples that shaped it.

Global: Probing-Based Explanation

A probe:

• Freezes the LLM's parameters.
• Generates representations from the LLM.
• Trains a shallow classifier on those representations to predict a linguistic property.

Probing for syntax asks about parts of speech, morphology, and dependencies. Probing for semantics asks about co-references, entities, and relations.

For example, take the sentence "The hungry cat chased the small mouse." Get the LLM's internal representation for each word, then train a simple classifier to predict whether a word is a noun, adjective, or verb based only on those representations, and test it on new sentences.

Global: Neuron Activation Explanation

The process of neuron activation explanation has five steps.

• Identify important neurons in an unsupervised manner, ranking them by metrics like correlation measurements, learned weights, or supervised classification methods.
• Learn relations between linguistic properties and the individual ranked neurons in supervised tasks.
• Verify the importance of the ranked neurons quantitatively via ablation experiments such as masking and erasure.
• With LLMs, generate natural language explanations that summarize the patterns in text that trigger high activation for a given neuron.
• Evaluate the quality of those natural language explanations by testing how well they let a model simulate the real neuron's activation behavior on new text examples.