Strategy question

You ran an A/B test and saw that it drops engagement. What would you do about it?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests disciplined response to a negative experiment result, distinguishing real product harm from an isolated or short-term metric dip.

Confirm statistical validity first: check sample size, test duration and whether the result is statistically significant, not just numerically lower.
Segment the drop by user type and platform to see if it's broad or concentrated in one segment, which changes the interpretation.
Check for a novelty or adjustment effect by looking at whether the engagement drop persists or recovers over a longer test window.
Weigh the tradeoff: if the feature serves a real strategic goal, like reduced misinformation or improved trust, some engagement cost may be acceptable.
Decide based on the full picture: kill the change if the drop is real, broad and unjustified by any offsetting goal, iterate if it's isolated to a fixable UX issue, or accept it if it serves a higher-priority goal.

Checks statistical significance and test duration first, avoiding a premature reaction to noise.
Explicitly allows that engagement isn't the only thing that matters, some strategic goals justify an engagement cost.
Separates a temporary novelty effect from a persistent real harm using time-based analysis.

Reacting immediately to a negative headline number without checking statistical significance or test length.
Assuming engagement is always the only metric that matters, even when the feature serves another real goal.

Chapters of the AI PM course that teach what this question tests.