XAI in LLM Prompting
In-context examples, label words, ordering, and chain-of-thought demonstrations can all change an LLM's answer. Prompt explanation asks which parts of that context drive the behavior.
Useful tests include flipping labels, perturbing demonstrations, replacing semantically meaningful labels, and measuring token saliency.
Behavioral perturbation
A prompt component is influential when a controlled change to that component reliably changes the output distribution. Contrastive demonstrations separate semantic priors from learned input-label mappings.
Perturb an In-Context Prompt, Then Evaluate the Effect
Apply one controlled change to a few-shot sentiment prompt, watch the changed span highlight, and see how the output, confidence, and run-to-run consistency respond.
Prompt sent to the model
Review: "A stunning debut, full of warmth." Sentiment: positive
Review: "Dull, overlong, and poorly acted." Sentiment: negative
Reasoning: The reviewer uses praise words like "stunning" and "warmth," so the sentiment is positive.
Review: "Surprisingly good for a sequel." Sentiment: ?
Model answer
positive
Confidence
0.91
Answer across 5 repeated runs (temperature > 0)
100% run agreement
Semantic prior and demonstrated label mapping agree, so this tells us little on its own — it is the reference point for every other run.
Chain-of-thought text can improve performance and still fail as a faithful explanation. Models can produce correct answers from incorrect reasoning or polished reasoning after the answer is already determined.
Compare standard and chain-of-thought prompting with perturbations, counterfactuals, and consistency tests.
Why is a coherent chain of thought weak evidence of faithfulness?