Skip to main content

Observability and Experimentation

(Closing the loop between code and user experience)

Senior rule:

Without production signals, frontend decisions are guesses.

Atlas engineers need to know whether users can load reports, apply filters, save views, recover from errors, and complete workflows on real devices.

Conceptual Model​

Frontend observability answers:

  • what failed?
  • who was affected?
  • which route, release, browser, and device?
  • what did the user experience?
  • which decision likely caused it?

Signals​

Track:

  • JavaScript errors
  • unhandled promise rejections
  • failed API calls
  • route transition timing
  • LCP, INP, CLS
  • long tasks
  • feature flag exposure
  • experiment assignment
  • failed form submissions
  • accessibility regressions where detectable

Web Vitals Collection Shape​

type WebVitalName = "LCP" | "INP" | "CLS" | "TTFB" | "FCP";

type WebVitalEvent = {
  name: WebVitalName;
  value: number;
  route: string;
  release: string;
  deviceClass: "mobile" | "desktop";
};

export function reportWebVital(event: WebVitalEvent) {
  navigator.sendBeacon("/analytics/web-vitals", JSON.stringify(event));
}

Use sendBeacon for low-priority unload-safe telemetry, but keep payloads small and privacy-safe.

Error Boundaries and Context​

"use client";

import { Component, ReactNode } from "react";

type Props = { children: ReactNode; route: string };
type State = { hasError: boolean };

export class RouteErrorBoundary extends Component<Props, State> {
  state: State = { hasError: false };

  static getDerivedStateFromError() {
    return { hasError: true };
  }

  componentDidCatch(error: Error) {
    reportClientError({
      message: error.message,
      route: this.props.route,
      release: process.env.NEXT_PUBLIC_RELEASE ?? "unknown",
    });
  }

  render() {
    if (this.state.hasError) return <p>Report could not be displayed.</p>;
    return this.props.children;
  }
}

The fallback should match the failure scope. Do not collapse the whole app for one failed widget.

Feature Flags and Experiments​

type ExperimentExposure = {
  experiment: "new-report-filter";
  variant: "control" | "compact";
  userId: string;
  route: string;
  release: string;
};

export function recordExposure(exposure: ExperimentExposure) {
  navigator.sendBeacon("/analytics/exposure", JSON.stringify(exposure));
}

Experiments need:

  • owner
  • hypothesis
  • exposure event
  • success metric
  • guardrail metrics
  • cleanup date
  • accessibility and performance review

A/B Testing Guardrails​

Atlas tests a compact report filter. Success metric: more saved reports. Guardrails:

  • no INP regression
  • no increase in validation errors
  • no lower keyboard completion rate in manual review
  • no support-ticket spike

Good experiments measure harm as well as success.

Mini Case Study: Conversion Improved, Performance Regressed​

Atlas launched a new onboarding checklist. Activation improved, but INP worsened because the checklist script loaded on every dashboard route.

Fix:

  • limit experiment script to onboarding routes
  • lazy-load non-critical checklist logic
  • segment metrics by route
  • add performance guardrail to experiment review

The experiment was locally successful and systemically harmful until observability connected both truths.

Common Failure Modes​

  • Logging errors without release, route, or user segment.
  • Measuring averages only.
  • Running experiments without cleanup.
  • Collecting PII in analytics.
  • Ignoring denied consent.
  • Using lab data as proof of real-user experience.

Review Checklist​

  • Are errors tied to route and release?
  • Are Web Vitals collected from real users?
  • Are failed recoveries visible?
  • Are analytics consent and PII rules clear?
  • Do experiments have guardrails and cleanup dates?
  • Can telemetry answer whether users are better off?

Senior Deep Dive: Observability As Feedback Design​

Observability is how senior engineers learn whether their assumptions survived production. It is not a pile of dashboards. It is a feedback system connecting user intent, release changes, runtime behavior, and business outcomes.

Signal taxonomy​

SignalAnswers
Web Vitalsdid users experience load, interaction, or stability problems?
JavaScript errorsdid client code fail by route, browser, device, or release?
API errorsdid backend or network behavior break the UI contract?
Custom eventsdid users complete the intended task?
Feature flag exposurewho saw the new behavior?
Experiment assignmentwhich variant changed outcomes?
Trace/request idscan client symptoms connect to server logs?
Support/session linkscan qualitative reports be tied to technical evidence?

The weakness this corrects: adding analytics events without a debugging model. A senior event answers a future question.

Telemetry event design​

A useful event has stable name, product meaning, route/surface, release/version, feature flag or experiment context, success/failure classification, duration where relevant, safe error category rather than sensitive raw data, and correlation id when crossing the network.

Avoid events that record everything and explain nothing. Also avoid events that leak personal or sensitive content. Telemetry is part of privacy architecture.

Alert design​

Do not alert on every metric. Alert when action is required.

AlertGood trigger
frontend error spikeuser-impacting error rate by route and release
INP regressionp75/p95 degradation for meaningful cohort
conversion droptask completion drop with experiment/release correlation
API contract failureschema/error shape mismatch affecting critical flow
feature flag incidentvariant-specific error or guardrail breach

Every alert should have an owner, dashboard, runbook, and rollback/degrade option. Otherwise it is noise.

Experiment guardrail model​

Experiments should not optimize one metric while damaging the system. Define primary success metric, guardrail metrics, exposure rules, sample limitations, stopping conditions, rollback path, and post-experiment cleanup. Example: a conversion variant that increases signups but worsens INP and form error recovery may not be a product win. Senior engineers insist on guardrails because users experience the whole system.

Debugging from production​

When a report arrives, the team should answer: route, release, browser, device, region, active feature flags, active experiments, Web Vitals trend, JavaScript/API error trend, trace id, and what changed recently. If these questions cannot be answered, observability is incomplete for senior ownership.

Exercises​

Exercise 1 - Signal Map​

For one Atlas-like flow, define success, failure, performance, and recovery signals.

Exercise 2 - Error Context Review​

Take one logged error and ask whether it includes enough context to debug.

Exercise 3 - Experiment Review​

Design an A/B test with one success metric and three guardrail metrics.

Further Reading​