Rubric Calibration Exercises
Purpose
Rubrics are only useful when reviewers can apply them consistently. These calibration exercises help mentors, interviewers, and learners distinguish senior implementation skill from architect-level decision capability.
How to use
For each scenario:
- Read the candidate answer.
- Score each capability from 1 to 5.
- Identify missing evidence.
- Rewrite one weak part into a level 4 answer.
Scenario 1: Dashboard performance regression
Prompt: "The SaaS dashboard has poor mobile INP after a table redesign. What do you do?"
Weak answer:
I would memoize rows, add virtualization, lazy load the chart, and use React Compiler.
Score:
| Capability | Score | Why |
|---|---|---|
| Performance diagnosis | 2 | jumps to fixes before trace evidence |
| Rendering/data tradeoffs | 2 | does not mention data volume, table interaction, hydration, or route budget |
| Observability | 1 | no field metric segmentation |
| Communication | 2 | no user/business framing |
Level 4 rewrite:
First I would confirm the regression with field INP segmented by route, device class, release, and experiment. Then I would capture a trace for the slow table interaction and separate input delay, processing, rendering, layout, and third-party cost. Candidate fixes might include reducing row render work, virtualizing large result sets, moving non-critical chart work out of the interaction path, and adjusting hydration boundaries. I would add a table interaction budget, a regression fixture, and a release gate before rolling the redesign forward.
Scenario 2: Server action for billing update
Weak answer:
Use a server action because it is easier than creating an API route.
Score:
| Capability | Score | Why |
|---|---|---|
| Security/reliability | 1 | no authorization, validation, CSRF/session, or idempotency |
| Data architecture | 2 | no cache or mutation lifecycle |
| Decision quality | 2 | convenience is the only rationale |
Level 4 rewrite:
A server function is acceptable if the operation is owned by this UI workflow and the boundary is explicit. The action must authenticate and authorize on the server, validate a narrow input schema, reject over-posted fields, use an idempotency key, return typed validation errors, update/revalidate affected account data, and write an audit event. If mobile app, support tooling, or external integrations also need the operation, we should expose a versioned API endpoint instead.
Scenario 3: Third-party script request
Weak answer:
Add the script with async loading so it does not block the page.
Score:
| Capability | Score | Why |
|---|---|---|
| Client-side security/privacy | 1 | ignores data access, consent, CSP, ownership, drift |
| Performance | 2 | mentions async but no budget or measurement |
| Operating model | 1 | no register or kill switch |
Level 4 rewrite:
Before adding it, we need a third-party register entry: business owner, engineering owner, purpose, route allowlist, data accessed, consent category, load strategy, CSP impact, performance budget, kill switch, and review date. It should not run on billing/security routes. We should measure transfer and main-thread cost, verify no sensitive data is collected, and test the kill switch in staging.
Scenario 4: Design-system component
Weak answer:
Put all combobox options into a reusable component so teams can share it.
Score:
| Capability | Score | Why |
|---|---|---|
| Design-system thinking | 2 | reuse is named but API durability is missing |
| Accessibility | 1 | no keyboard/focus/semantics contract |
| Platform operating model | 2 | no adoption or migration plan |
Level 4 rewrite:
Build a headless combobox primitive for behavior and accessibility, then a styled design-system wrapper for visual states. Product-specific wrappers own domain rendering and data fetching. The stable contract should define labels, keyboard behavior, focus, async loading states, error states, option identity, virtualization constraints, and tests. Pilot it in two workflows before deprecating legacy select/search components.
Scenario 5: GenUI copilot
Weak answer:
Use an AI SDK to stream generated components and let users approve actions.
Score:
| Capability | Score | Why |
|---|---|---|
| AI/GenUI governance | 2 | no evals, registry, permissions, or fallback |
| Security/privacy | 2 | approval is treated as the main control |
| Observability | 1 | no traces or quality signals |
Level 4 rewrite:
The copilot needs a workflow boundary, permission-filtered retrieval, component registry, schema validation, tool side-effect classes, approval gates, deterministic fallback, eval suites, and kill switches. Approval is not enough: server policy must prevent unauthorized retrieval and tool calls. Traces must connect user prompt, retrieval, model/prompt version, generated component, validation result, user correction, cost, and terminal outcome.
Reviewer checklist
Look for:
- constraints before tools
- rejected alternatives
- ownership and operation after launch
- pre-release and production verification
- security/privacy/accessibility considered early
- user/business risk explained plainly
- failure modes and rollback
- evidence that other teams can reuse the decision
Exercise
Take one recent architecture proposal and write three versions:
- level 2: tool-led answer
- level 3: reasonable bounded design
- level 4: cross-team architecture decision with verification and ownership