Case Study: Unsafe GenUI Tool Call
Incident summary
An internal AI copilot helped support agents resolve account issues. During a pilot, it suggested a refund and triggered a tool call before the agent understood the policy exception. The action was reversible, but it created audit, customer trust, and support-process problems.
The model made a bad suggestion. The architecture allowed the suggestion to become a product action too easily.
Timeline
| Time | Event |
|---|---|
| Week 0 | Copilot pilot launches for support agents. |
| Week 1 | Read-only policy answers perform well in demos. |
| Week 2 | Reversible refund tool enabled for small amounts. |
| Day 2 | Copilot suggests refund for an account with policy exception. |
| Day 2 | Tool executes after generic confirmation, before policy citation is inspected. |
| Day 3 | Audit review cannot connect prompt, model rationale, approval text, and tool execution clearly. |
| Day 4 | Refund tool disabled; tool risk model redesigned. |
Production signals
- tool execution succeeded
- agent correction note said policy exception was missed
- audit log showed action result but not full rationale/approval context
- eval suite had answer quality tests but no unsafe tool fixtures
- approval UI did not show target, impact, policy source, and rollback details clearly
Root cause
The workflow treated tool calls as model decisions rather than product actions requiring risk classification, permission propagation, human approval, and server policy.
Architecture failures
- tool risk classes were missing
- "reversible" was treated as low-risk
- approval UI was generic
- server policy did not re-check enough workflow context
- audit linked tool execution but not model suggestion, citations, approval, and trace
- evals tested answers, not tool safety
- frontend progress copy made the action feel routine
Bad alternatives
| Alternative | Why it was wrong |
|---|---|
| Ban all tools permanently | loses useful copilot capability |
| Rely on better prompt wording | prompts are not policy boundaries |
| Add a generic "are you sure" modal | approval must be specific and informed |
| Log more raw prompt data | may create privacy and audit-data risks |
Corrected architecture
| Control | Practice |
|---|---|
| Tool risk matrix | read, draft, reversible write, external write, destructive/financial |
| Approval panel | actor, target, impact, reason, source citations, rollback notes |
| Backend policy | server-authorized tools with tenant, role, workflow, and side-effect checks |
| Audit | link user intent, model suggestion, citations, approval, policy check, tool result, trace |
| Evals | fixtures prove high-risk actions require approval or refusal |
| UI copy | progress distinguishes suggestion, approval pending, executing, and completed |
Corrected flow
Prevention controls
- every tool has side-effect class and owner
- approval UI is generated from deterministic policy metadata
- high-risk tools require citations and preview
- server policy checks role, tenant, target, amount, workflow, and idempotency
- eval suite includes malicious, ambiguous, stale, unauthorized, and high-risk tool prompts
- audit stores redacted trace references rather than raw sensitive content
- kill switch exists per tool
Review questions
- What side-effect class does this tool belong to?
- What must the human understand before approval?
- What server policy can still deny the action?
- What audit evidence proves consent and policy compliance?
- What eval would have caught this before pilot?
Reusable lesson
The model may propose, but the product system decides. Approval is a UX control on top of deterministic policy, not a replacement for authorization, validation, idempotency, and audit.
Source lens
Use Part XIII GenUI architecture, GenUI evaluation governance, completed GenUI launch review, completed GenUI eval matrix, and the GenUI AI system review packet.