Completed Architecture Review Example
Scenario
This completed review evaluates a proposal to add a GenUI support copilot to an internal operations portal. The copilot can answer policy questions, show cited sources, render generated task UI, and request approval for account actions.
Review summary
| Field | Value |
|---|---|
| Decision | Approved for canary with blocking follow-ups. |
| Risk level | High, because the system combines retrieval, generated UI, and tool execution. |
| Primary reviewers | frontend architect, security engineer, design-system owner, support product lead. |
| Required packet | review-packets/genui-ai-system-review-packet. |
| Related capstone | capstones/capstone-genui-ai-fullstack-copilot. |
Intake answers
| Question | Answer |
|---|---|
| Which parts are deterministic? | App shell, navigation, auth, component renderer, tool execution, approvals, audit log. |
| Which parts are AI-assisted? | Intent interpretation, answer drafting, component selection, retrieval query rewriting. |
| Can the model invent UI? | No. It can request only registered components. |
| Can the model execute tools directly? | No. The backend applies permission, risk, and approval policy. |
| Are citations required? | Yes for policy and customer-account knowledge answers. |
| What is the fallback? | Text-only answer with "needs review" state, or deterministic task link. |
Required artifacts reviewed
| Artifact | Status | Notes |
|---|---|---|
| Component registry contract | Approved | Requires schema compatibility tests before canary. |
| RAG source policy | Needs changes | Stale-source behavior is underspecified. |
| Tool risk matrix | Approved with condition | High-risk writes require human approval. |
| Eval suite | Needs changes | Missing prompt-injection fixtures from retrieved sources. |
| Observability plan | Approved | Add correction reason taxonomy. |
| Accessibility contract | Approved | Approval panel focus behavior must be manually checked. |
Decision matrix
| Concern | Assessment | Required action |
|---|---|---|
| Generated UI safety | Registry and fallback design are sound. | Add invalid-schema fixtures. |
| Retrieval trust | Citations are visible, but stale source behavior is weak. | Add source freshness policy and UI state. |
| Tool safety | Risk classes are appropriate. | Add denial and rollback telemetry. |
| Security/privacy | Tenant isolation is backend-owned. | Add prompt-injection tests before canary. |
| Reliability | Model outage fallback is acceptable. | Add runbook for retrieval outage. |
| Accessibility | Uses design-system primitives. | Manual screen-reader pass for approval flow. |
Reviewer comments
- "The registry boundary is strong. Do not add markdown passthrough as an escape hatch."
- "Policy answers need a stale-source state. A citation is not enough if the source is expired."
- "The approval panel should show action preview, actor, target account, risk, and rollback notes."
- "Tool denial should be a first-class outcome, not an error toast."
- "Canary should be limited to support agents trained on correction behavior."
Blocking follow-ups
| Follow-up | Owner | Due before |
|---|---|---|
| Add stale-source UI and retrieval policy | AI platform | canary |
| Add prompt-injection eval fixtures | security | canary |
| Add invalid schema fixtures | frontend platform | canary |
| Add approval panel manual accessibility notes | design system | canary |
Canary criteria
- 100% pass on critical tool safety evals.
- 100% pass on component schema validation fixtures.
- No high-risk tool executes without approval.
- Retrieval answers with stale sources show review-needed state.
- Observability captures component fallback, user correction, tool denial, and retrieval confidence.
Final decision
Approved for limited canary after blocking follow-ups. Not approved for broad rollout until production correction data and retrieval quality are reviewed for at least one full support cycle.
Follow-up review questions
- Which corrections indicate a missing component versus poor retrieval?
- Are support agents over-trusting medium-confidence answers?
- Are denied tools creating confusing dead ends?
- Are generated components slower or less accessible than deterministic equivalents?
- Which eval fixtures should be added from real production failures?