Metrics question

A user satisfaction survey was conducted for two groups of facebook users (each with a 50k sample size). Group 1: Enabled certain login security features (Experiment). Group 2: Did not enable these security features. It was found that user satisfaction with Group 1 was 30% lower than with Group 2. Why do you think are the reasons? Comment on how the survey was conducted.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests critical evaluation of survey methodology alongside product hypothesis generation, since a flawed experiment design can itself explain the gap.

Restate the setup: Group 1 with login security features on reported 30% lower satisfaction than Group 2, each with a 50k sample.
Question the survey design first: check if it was administered right after the friction of the security step, biasing sentiment toward that friction rather than overall satisfaction.
Check randomization: confirm the groups were truly randomly assigned and comparable in tenure and device, not self-selected security-conscious users.
Generate product hypotheses assuming the gap is real: added login friction directly reduces immediate satisfaction, even if the feature helps account safety long-term.
Distinguish short-term friction from long-term trust: propose a follow-up survey weeks later to see if satisfaction recovers as users adapt.
Recommend next steps: segment the gap by new versus existing users, and pair the survey with behavioral data like login success rate.

Interrogates the survey methodology first, timing bias and randomization, rather than accepting the gap as a pure product signal.
Explicitly separates immediate friction sentiment from durable satisfaction, a nuanced distinction many candidates miss.
Proposes a concrete follow-up test, a delayed re-survey, to check whether the drop is temporary adaptation friction.
Recommends pairing self-reported satisfaction with behavioral data for a fuller picture, not survey results alone.

Accepting the 30% gap at face value as proof the security feature is bad, without questioning survey design.
Not addressing the explicit prompt to comment on how the survey was conducted.

How would you redesign the survey to reduce timing bias?
Would you still ship the security feature if satisfaction stayed lower long term? Why?