XAI in Agentic AI Systems
An agent plans, calls tools, reads results, updates state, and acts. The most useful explanation is often a trace of those events rather than a paragraph describing “reasoning.”
For a failed purchase order, we want the query sent to inventory, the records returned, the calculation performed, and the authorization rule applied.
Decision trace
An agent trace records observations, tool calls, tool outputs, state transitions, guardrail decisions, and final actions. A good trace is timestamped, scoped, and linked to the exact run.
Debug an Agent Run
Step through a purchase-order trace. Hide the failed calculator call and see how the same final hold becomes harder to reconstruct.
Final action
approval hold
First call failed: malformed expression.
Raw traces can leak private data and overwhelm users. Build layered views: a concise user explanation, an operator trace, and a detailed audit record.
Log enough to reconstruct the run, including retries and tool errors. A trace that records only successful actions creates a flattering fiction.
Why should an agent explanation include failed tool calls?