AI & Technical question

Design a monitoring and alerting system.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests systems design for observability: balancing signal quality against alert fatigue.

Clarify scope: infrastructure metrics versus business metrics like payment failure rate, since thresholds differ.
Design the pipeline: services emit metrics to a time-series store, with a rules engine evaluating thresholds.
Choose alerting logic: static thresholds for well-understood metrics, anomaly detection for naturally variable ones.
Reduce alert fatigue: require sustained breaches over a time window before alerting, and route by severity.
Address on-call ergonomics: alerts include enough context, like recent deploys, to act quickly.
Define success as mean time to detect and acknowledge real incidents, plus a low false-positive rate.

Explicitly designs against alert fatigue, the most common real failure of alerting systems.
Chooses different alerting strategies matched to each metric's actual behavior, not one blanket approach.
Picks both detection speed and false-positive rate as success metrics, the real trade-off in alerting.

Alerting on every threshold breach with no sustained-window logic, causing ignored pages.
No context included in alerts, slowing down on-call response.

How would you tune thresholds to avoid false positives after a legitimate traffic spike?
How would you handle an alert storm from one root cause cascading across services?
How would you measure if the alerting system itself is working well?