Skip to main content

SECTION 0 — RELIABILITY IS DESIGNING FOR NORMAL FAILURE

Production is partial failure:

  • timeouts

  • dependency brownouts

  • retries

  • backpressure

Senior fullstack means the UI and backend cooperate to fail safely.

SECTION 1 — TIMEOUTS + RETRIES + BACKOFF (THE TRINITY)

Rules:

  • always set timeouts

  • retry only idempotent operations (or make them idempotent)

  • exponential backoff + jitter

Avoid:

  • synchronized retries (retry storm)

SECTION 2 — CIRCUIT BREAKERS + BULKHEADS

  • circuit breaker stops calling a failing dependency

  • bulkheads isolate resources so one failure doesn’t sink the system

UI equivalent:

  • degrade features instead of blocking the whole app

SECTION 3 — RATE LIMITING + BACKPRESSURE

  • apply limits per user/tenant

  • protect expensive endpoints

  • apply queue depth limits

Signal backpressure clearly:

  • 429 with retry-after

  • typed error response

SECTION 4 — POISON MESSAGES + DLQs

Async systems must assume:

  • malformed payloads

  • unprocessable messages

  • repeated failures

Pattern:

  • limited retries

  • then DLQ

  • then replay tooling

Worker rule:

Consumers must be idempotent because delivery is at-least-once.

SECTION 5 — ROLLOUTS THAT DON’T HURT

  • feature flags

  • canary deployments

  • progressive delivery

  • fast rollback

Senior rule:

Rollback is part of the design, not an emergency improvisation.

SECTION 6 — EXERCISES

  1. Identify endpoints that must be idempotent and how you’ll guarantee it.

  2. Design a retry policy (backoff + jitter) for a flaky dependency.

  3. Define DLQ rules and replay process.

  4. Write a rollout plan for a risky change.

🏁 END — RELIABILITY PATTERNS