Metrics question

Your new feature boosts Amazon Search by 10%, adds 2s to load time. What do you do?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests how you resolve a metrics tradeoff between a short-term conversion win and a long-term retention risk from latency.

Confirm the numbers: 10% lift in search conversion or relevance against a 2 second increase in page load, and ask over what test duration.
State the known tradeoff: Amazon's own data has tied every 100ms of latency to measurable sales loss, so 2s is a real cost.
Check whether the 10% is a short-term novelty effect by holding out a longer-running cohort before deciding.
Segment the impact: does the 2s hit mobile users or slow-network regions harder than desktop, since that changes the net effect.
Recommend optimizing the feature to cut load time before full rollout, or gating it behind lazy-loading, rather than shipping either metric blind.

Quantifies the latency cost using known industry data (Amazon found 100ms of latency costs about 1% in sales) to argue 2s is not trivial.
Proposes a staged rollout: ship to a segment, measure conversion and 7 or 30-day retention together, not just the immediate search metric.
Suggests a technical mitigation, like progressive rendering so the perceived latency drops even if total load time doesn't, as a way to keep both wins.
Recommends deciding by net revenue impact (conversion lift minus estimated churn from slowness) rather than by feature-owner instinct.

Accepting the 10% lift at face value without questioning whether it holds up long-term.
Ignoring that latency compounds; a single feature adding 2s can be part of a much larger latency budget being eaten away.

How would you set up the A/B test to isolate a long-term retention effect from a short-term novelty effect?
What latency budget would you set for future features on this page?