Metrics question

Amazon is testing a new voice-shopping feature. How would you design an A/B test to validate whether it increases purchase frequency?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Experiment design for a new feature's impact on a downstream behavioral metric, not just feature usage.

Define the test: randomize users into a group with voice shopping enabled versus a control without it, at the account or device level.
Choose the primary metric: purchase frequency over a defined window, not just voice shopping usage itself.
Add guardrail metrics: return rate and average order value, since voice shopping with less product detail visible could increase mis purchases.
Run long enough to capture a full habit formation cycle, since a new feature often has a novelty spike that fades.
Define the decision rule upfront: the minimum detectable lift in purchase frequency that would justify a full rollout.

Correctly picks purchase frequency, the actual business question, as primary, not feature engagement as a vanity proxy.
Adds return rate as a guardrail, recognizing reduced visual context could increase wrong or regretted purchases.
Sets the decision threshold before running the test, avoiding a post hoc it looks good call.

Measuring only how often people use voice shopping, not whether it increases actual purchase frequency.
No guardrail against increased returns from lower visibility purchases.

How would you account for a novelty effect fading after the first few weeks?
What would you do if voice shopping increased frequency but also increased returns significantly?