Case Study: Multi-Tenant Cache Leak
Incident summary
A SaaS dashboard introduced edge caching for an authenticated customer summary route. Median response time improved, but one tenant occasionally saw another tenant's summary count after navigating from an email link.
The failure was not a CDN bug. It was a cache variation and data-classification failure.
Timeline
| Time | Event |
|---|---|
| Week 0 | Team proposes edge caching for dashboard summary route. |
| Week 1 | Lab and staging tests show faster TTFB. |
| Week 2 | Change ships to 25% of tenants. |
| Day 2 | Support ticket reports incorrect summary count after email link navigation. |
| Day 3 | Logs show cache hits across requests with different tenant context. |
| Day 3 | Frontend cache key also found to use route path only. |
| Day 4 | Edge caching disabled; BFF headers updated. |
| Week 4 | Cache classification and edge delivery review adopted. |
Production signals
- TTFB improved
- support tickets mention impossible customer counts
- cache hit logs lacked safe tenant/role dimensions
- issue reproduced after tenant switch and email deep link
- frontend server-state cache reused data by route path
Root cause
Tenant-specific data was treated as generic dashboard metadata. Neither edge cache keys nor frontend cache keys encoded the dimensions that made the response private: tenant, role, locale, and experiment assignment.
Architecture failures
- tenant data was not classified before delivery optimization
- response cacheability was not declared by BFF contract
- edge cache variation rules were not documented
- frontend cache keys ignored ownership dimensions
- review optimized latency without privacy review
- observability could not safely explain cache behavior
Bad alternatives
| Alternative | Why it was wrong |
|---|---|
| Add tenant id directly to public logs | creates another privacy problem |
| Disable all caching forever | avoids leak but loses safe optimization opportunities |
| Rely on frontend role checks | browser checks cannot fix shared-cache behavior |
| Patch only CDN config | leaves frontend cache-key bug intact |
Corrected architecture
| Layer | Corrective action |
|---|---|
| Data classification | mark route data as public, tenant-private, user-private, or role-private |
| BFF | response headers declare cacheability and variation dimensions |
| Edge | shared cache only for proven public or safely varied responses |
| Frontend | server-state keys include tenant-safe identity and role dimensions |
| Observability | log route, cache status, release, and safe tenant class |
| Review | edge delivery packet required for cache changes |
Cache decision flow
Prevention controls
- cache review requires data classification
- BFF contract includes cacheability and variation
- edge changes require rollback and purge plan
- frontend query keys include ownership dimensions
- tenant switch and permission change tests exist
- cache diagnostics use safe identifiers/classes
Review questions
- Is this response public, tenant-private, user-private, or role-private?
- What dimensions affect correctness: tenant, user, role, locale, experiment, region?
- Which cache layers store the response?
- How do logout, tenant switch, permission change, and deep links behave?
- How can we debug cache behavior without leaking tenant data?
Reusable lesson
Performance changes can become privacy incidents. Shared caches require explicit data classification and variation rules before optimization.
Source lens
Use Part V edge delivery, Part III cache layering, Part VIII privacy/security, completed route architecture note, and edge delivery review packet.