Product design question
A ministry outside the U.S. has several candidate workflows for a bespoke AI solution, but its leadership team is not aligned on which problem is highest priority. How would you run discovery and design workshops to identify the best workflow to target, define a measurable success outcome, and decide whether Scale should build an AI application on top of existing models or invest in a custom LLM?
- Scale AI
- Product design
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can run structured discovery with a misaligned leadership team, land on one measurable target workflow, and make a build-versus-custom-model call grounded in evidence rather than assumption.
How to approach it
- Run individual stakeholder interviews before a joint workshop, since leadership misalignment often hides real disagreement that a group session would paper over.
- In the joint workshop, surface each candidate workflow's underlying pain with a concrete scenario walkthrough, then have the room rank them together against shared criteria like frequency, cost, and citizen or user impact.
- Define one measurable success outcome for the top-ranked workflow before leaving the workshop, for example reducing processing time for a specific request type by a stated amount.
- Decide build-versus-custom-model based on data readiness and domain specificity: if strong existing models plus the ministry's own documents can hit the target via retrieval and prompting, build on existing models; only invest in a custom LLM if the domain language or data is specialized enough that general models consistently underperform.
- Close the workshop with a documented decision and named owner on both sides, since leadership misalignment tends to resurface later without a written commitment.
What a strong answer includes
- Runs individual interviews before the group workshop specifically to surface hidden misalignment, rather than assuming a joint session alone will produce consensus.
- Defines the success outcome as measurable and specific, not just a directional goal like improving efficiency.
- Defaults to building on existing models and only justifies a custom LLM with a specific, named technical reason.
Common mistakes
- Jumps straight to a joint workshop without individual interviews, missing hidden disagreement.
- Defaults to a custom LLM without evidence that existing models plus retrieval cannot meet the bar.
Likely follow-up questions
- What would you do if the workshop ends without full leadership agreement.
- How would you validate the success outcome is actually measurable with the ministry's available data.
More product design questions
- You're asked to deliver an enterprise GenAI application on Scale’s platform in 10 weeks for a customer with ambiguous requirements, strict security/compliance review, and multiple stakeholder groups. How would you scope the v1, convert discovery into clear requirements, run testing and pilot rollout, and decide what to cut versus what must ship for launch?Scale AI · Product design · Hard
- A ministry outside the U.S. asks Scale to build a bespoke GenAI application on top of its proprietary data, but end users cannot clearly explain where the workflow is breaking today. How would you run the first client workshops to uncover the real job-to-be-done, select the highest-value use case, define an MVP, and align the client with Scale’s engineering, MLE, and ops teams on scope?Scale AI · Product design · Hard
- Scale forward deploys to understand real workflows before building. If future end-users in a government agency have different needs from the senior sponsor who is funding the project, how would you gather the right feedback, separate core pain points from feature requests, and turn that into a prioritized roadmap for the first release?Scale AI · Product design · Medium
- A prospective customer wants an agentic or RL data solution but can only describe the desired outcome, not the tasks, feedback signals, or delivery constraints. How would you run discovery, separate must-have from nice-to-have requirements, and turn the conversation into a concrete plan for product, operations, and next customer validation?Scale AI · Product design · Medium
- You need to ship the first agentic workflow product for defense analysts on a controlled network where internet access, model updates, and human review are tightly constrained. What is the MVP, which user/job would you target first, and what tradeoffs would you make among agent autonomy, user experience speed, and security/risk controls?Scale AI · Product design · Hard
- A defense customer asks for "an AI assistant for analysts" but cannot clearly describe the day-to-day workflow or failure modes. How would you work with engineers and ML teammates to turn that vague request into a concrete v1 product, including the user task you would target, the human-in-the-loop design, and what you would explicitly leave out?Scale AI · Product design · Hard
More questions from Scale AI
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 4: Discovery and strategy for AI products
- Chapter 7: AI UX and human oversight: design for a system that is wrong sometimes
- Chapter 14: Get the job: the AI PM interview loop