Metrics question

Midjourney is testing a new creative tool and early feedback is polarized: some users say the concept is valuable but hard to use, while others say usability is fine but the outputs are not good enough. How would you isolate whether the main issue is user value, UX, or model performance, and what evidence would you need to recommend iterate, pivot, or stop?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests isolating whether polarized feedback on a new tool stems from user value, UX, or model performance, and deciding iterate, pivot, or stop.

Segment the feedback explicitly by theme, value-but-hard-to-use versus usable-but-output-not-good-enough, rather than treating it as one mixed signal.
For the value-but-hard-to-use group, check task completion and time-to-first-output data to confirm whether UX friction is measurably blocking usage.
For the usable-but-output-not-good-enough group, check output quality against a defined bar, comparing user-rated output quality to existing tools they compare it against.
Cross-reference both groups against retention: if both segments retain poorly despite different complaints, that signals a deeper value problem beneath the surface feedback.
Recommend iterate if the issue is isolated and fixable, UX rework for one group, model tuning for the other, with distinct owners for each fix.
Recommend pivot or stop only if neither group retains even after the fix hypothesis is tested, or if the underlying value proposition itself is unclear to both groups.

Segments the polarized feedback by theme first, avoiding a single blended read of what is really two different problems.
Uses retention data across both segments as the real signal for whether this is a value problem beneath the surface complaints.
Sets a clear decision rule, iterate if isolated and fixable, pivot or stop only if value itself is unproven, avoiding a premature kill decision.

Treating polarized feedback as one undifferentiated signal instead of two distinct groups.
Recommending pivot or stop without first testing whether a targeted fix resolves each group's complaint.

How would you decide the fix isn't working and it's time to pivot?
What would you do if the two groups turn out to be the same users on different days?