AI & Technical question
Ramp wants to launch an LLM-powered tax feature that extracts tax-relevant data from bills and proposes filing-ready outputs. What offline evals, online guardrail metrics, and launch thresholds would you require before general availability? How would you decide which error classes can auto-resolve, which must escalate to manual review, and which should block launch entirely?
- Ramp
- AI & Technical
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Whether you can set launch-gating evals and error-handling policy for an LLM feature whose mistakes have direct financial and compliance consequences.
How to approach it
- Build offline evals against labeled bills across document quality and complexity, scoring extraction accuracy per field (amount, tax category, jurisdiction) since consequences differ by field.
- Define online guardrail metrics: rate of low-confidence extractions, rate of outputs later corrected by a human reviewer, and rate of downstream filing errors traced back to this feature.
- Set launch thresholds per field, for example higher accuracy required on tax category classification than on vendor name, since misclassification has direct compliance risk.
- Classify error types into three tiers: auto-resolve (high-confidence, low-risk fields), escalate to manual review (moderate confidence or higher-risk fields), and block launch entirely (categories with insufficient data or unacceptable risk, such as unusual cross-border VAT cases).
- Require every auto-resolved output to still carry an audit trail and be sampled for periodic human QA, even after launch, since tax rules and edge cases evolve.
- Set the GA gate at a level where the combined precision on auto-resolve categories meets an assumed compliance-grade bar, for example above 98%, with everything below routed to review.
What a strong answer includes
- Uses field-specific accuracy bars rather than one blended score, since a wrong tax category is more consequential than a wrong description string.
- Defines the three-tier error classification (auto-resolve, escalate, block) explicitly, directly answering the question's core ask.
- Requires ongoing sampling and audit even for auto-resolved outputs, recognizing that tax rules and edge cases shift over time.
- Names a concrete compliance-grade threshold as an illustrative assumption, showing what 'good enough' means in this domain.
Common mistakes
- Proposing one blended accuracy score instead of field-level and risk-tiered thresholds.
- No plan for continued sampling and audit after launch, treating the eval as a one-time gate.
- No clear rule for which error classes are blocked from launch entirely.
Likely follow-up questions
- How would you decide which error classes are severe enough to block launch entirely?
- What would you do if accuracy is strong overall but weak on one specific jurisdiction?
- How would you handle a customer dispute over an auto-resolved output?
More ai & technical questions
- How would you measure whether Ramp's AI agents make better decisions than humans?Ramp · AI & Technical · Hard
- Ramp is considering an API-based partnership that would expand the product surface area. From first conversation to launch decision, how would you structure the evaluation? Be specific about the product and engineering inputs you would need, such as API coverage, auth model, data flows, SLAs, implementation effort, and ongoing partner dependencies, and how those technical facts would affect whether Ramp should build the integration.Ramp · AI & Technical · Hard
- Pick one concrete international-payments workflow where an LLM could create real leverage at Ramp, for example, KYB document review, payment exception handling, or explaining why a cross-border transfer was delayed. How would you design the human-in-the-loop experience, identify failure modes such as hallucinated compliance conclusions or incorrect document extraction, and decide whether the system is safe and useful enough to ship?Ramp · AI & Technical · Hard
- You own AI-powered contract analysis that extracts pricing terms, renewal dates, and risk flags from vendor agreements. Design the pre-launch evaluation framework: what labeled datasets and test slices would you use, how would you score extraction accuracy and explanation quality, what failure modes must trigger fallback or human review, and what launch gate would you set for enterprise readiness?Ramp · AI & Technical · Hard
- Ramp needs to support changing tax rules across US entities and international VAT/GST regimes. How would you design the product and underlying rule system so that most tax logic can be updated through configuration or data rather than bespoke code? Be specific about the abstractions, versioning, and auditability you would need.Ramp · AI & Technical · Hard
- Design a simple load balancer for Google.com. What data structures would you use?Google · AI & Technical · Hard
More questions from Ramp
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Foundations: the model and the decisions it forces on you
- Chapter 8: Evals: define good and make the number defensible
- Chapter 6: Agents and agentic architecture