Metrics question
What metrics prove Devin is delivering ROI to an enterprise like Goldman Sachs?
- Cognition
- Metrics
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Enterprise metrics: proving ROI for an autonomous coding agent to a risk-averse, high-stakes financial services buyer.
How to approach it
- Clarify the buyer's priorities: Goldman Sachs will care about engineering throughput, cost, and risk, especially code quality and security in a regulated environment.
- Define the primary ROI metric: engineering hours saved per week, converted to a dollar figure using average loaded engineer cost.
- Add throughput metrics: number of tickets (bug fixes, migrations, test coverage) completed by Devin versus a human baseline in comparable time.
- Add quality guardrails: defect rate in Devin-authored code post-merge, and time engineers spend reviewing versus rewriting.
- Add compliance-specific proof points: audit trail completeness and adherence to internal coding and security standards.
- Present a before and after pilot comparison in one team before extrapolating org-wide savings, since a large bank will demand a controlled proof point.
What a strong answer includes
- Converts time saved into a concrete dollar ROI figure using a stated, reasonable loaded-cost assumption.
- Names quality and compliance metrics specifically relevant to a bank, not just generic 'speed' metrics.
- Proposes a controlled pilot design, comparing a Devin-assisted team against a baseline team, which is what a rigorous enterprise buyer expects.
- Distinguishes throughput (volume of tickets) from efficiency (cost per ticket) as separate proof points.
- Flags audit trail and security review pass rate as metrics unique to regulated enterprises.
Common mistakes
- Using generic SaaS metrics like DAUs instead of ROI-focused, finance-relevant metrics.
- Ignoring compliance and audit requirements specific to a bank.
Likely follow-up questions
- How would you structure the pilot comparison?
- What would you do if defect rates were higher in Devin's code?
- How would you present this ROI story to a non-technical executive?
More metrics questions
- How would you measure the performance and health of a Netflix Original?Microsoft · Metrics · Hard
- Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?Meta · Metrics · Hard
- You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?Google · Metrics · Hard
- Facebook Ads dropped by 20%. Mark Zuckerberg calls you and asks you to fix it. What do you do?Meta · Metrics · Hard
- Imagine you're the product manager for Facebook Marketplace. Since many sellers don't mark items as sold, what existing functionality and metrics could you use to determine whether an item has likely sold?Meta · Metrics · Hard
- Uber has seen a dip in the ride confirmation rate. As a PM at Uber, you are responsible to investigate and fix the issue. How will you go about it?Uber · Metrics · Hard
More questions from Cognition
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 9: Prove it paid off: outcomes, economics, and pricing
- Chapter 2: Data fluency: SQL, logs, and reading the truth yourself
- Chapter 14: Get the job: the AI PM interview loop