Product design question
A prospective customer wants an agentic or RL data solution but can only describe the desired outcome, not the tasks, feedback signals, or delivery constraints. How would you run discovery, separate must-have from nice-to-have requirements, and turn the conversation into a concrete plan for product, operations, and next customer validation?
- Scale AI
- Product design
- Medium
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests discovery skill for translating a vague outcome into a concrete delivery plan across product, operations, and validation.
How to approach it
- Ask the customer to describe the outcome in terms of what success looks like downstream, for example a model behavior they want improved.
- Work backward from the outcome to infer candidate tasks and feedback signals, then validate those inferences directly with the customer.
- Separate must haves, the minimum data or feedback loop needed to move the metric, from nice to haves, like edge case coverage.
- Propose a small scoped pilot task set to validate the approach before committing to full delivery scale.
- Translate the validated scope into an operations plan, staffing, quality review process, and a product plan, tooling needed to support annotators.
- Set a checkpoint with the customer after the pilot to confirm the data is actually producing the outcome they described.
What a strong answer includes
- Works backward from the described outcome to infer concrete tasks rather than asking the customer to specify something they clearly cannot articulate.
- Proposes a small validation pilot before full commitment, protecting both sides from committing to the wrong task design at scale.
- Separates must have and nice to have requirements explicitly, which keeps the plan buildable instead of trying to spec everything upfront.
Common mistakes
- Asking the customer to specify tasks and feedback signals directly when they have already said they cannot describe them.
- Committing to full scale delivery without a validation pilot to confirm the inferred task design actually works.
Likely follow-up questions
- How would you handle it if the pilot data does not move the outcome the customer described?
- What would you do if the customer changes the desired outcome after the pilot has started?
More product design questions
- You're asked to deliver an enterprise GenAI application on Scale’s platform in 10 weeks for a customer with ambiguous requirements, strict security/compliance review, and multiple stakeholder groups. How would you scope the v1, convert discovery into clear requirements, run testing and pilot rollout, and decide what to cut versus what must ship for launch?Scale AI · Product design · Hard
- A ministry outside the U.S. asks Scale to build a bespoke GenAI application on top of its proprietary data, but end users cannot clearly explain where the workflow is breaking today. How would you run the first client workshops to uncover the real job-to-be-done, select the highest-value use case, define an MVP, and align the client with Scale’s engineering, MLE, and ops teams on scope?Scale AI · Product design · Hard
- Scale forward deploys to understand real workflows before building. If future end-users in a government agency have different needs from the senior sponsor who is funding the project, how would you gather the right feedback, separate core pain points from feature requests, and turn that into a prioritized roadmap for the first release?Scale AI · Product design · Medium
- A ministry outside the U.S. has several candidate workflows for a bespoke AI solution, but its leadership team is not aligned on which problem is highest priority. How would you run discovery and design workshops to identify the best workflow to target, define a measurable success outcome, and decide whether Scale should build an AI application on top of existing models or invest in a custom LLM?Scale AI · Product design · Hard
- You need to ship the first agentic workflow product for defense analysts on a controlled network where internet access, model updates, and human review are tightly constrained. What is the MVP, which user/job would you target first, and what tradeoffs would you make among agent autonomy, user experience speed, and security/risk controls?Scale AI · Product design · Hard
- A defense customer asks for "an AI assistant for analysts" but cannot clearly describe the day-to-day workflow or failure modes. How would you work with engineers and ML teammates to turn that vague request into a concrete v1 product, including the user task you would target, the human-in-the-loop design, and what you would explicitly leave out?Scale AI · Product design · Hard
More questions from Scale AI
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 4: Discovery and strategy for AI products
- Chapter 7: AI UX and human oversight: design for a system that is wrong sometimes
- Chapter 14: Get the job: the AI PM interview loop