Amplitude · AI product · Mid · 45 to 60 minutes

How far should AI automate analytics?

Amplitude is a digital analytics platform that helps companies understand user behavior and product usage. Answer this product management case and get it graded line by line against a weighted rubric.

product strategy
AI scoping
prioritization
risk management
experimentation

The situation

The director leading AI work at Amplitude asks you to recommend what kind of AI experience the company should ship next in its analytics product area. Large customers are asking for more natural language help and more automation, while leadership wants clear AI progress in the next planning cycle. At the same time, Amplitude serves enterprise teams whose event naming, instrumentation quality, and data models vary a lot, and many analytics questions are open ended rather than single correct tasks. If the product sounds too autonomous and returns wrong answers, trust in the core analytics product could fall. If it stays too narrow, customers may see it as a gimmick and competitors may look faster. Assume you have one cross functional team for the next two quarters and enough engineering support to ship either a narrowly scoped assistant, a more agentic workflow, or enabling pieces that make later automation safer. Your job is to define what problem to tackle first, how much autonomy to allow, and how you would know it is working before a broad release.

What to produce

A 1 to 2 page product strategy memo.

Your answer must

Define the primary user and the top one or two jobs to be done for the first AI release.
Recommend a specific first scope and autonomy level, and name at least one attractive idea that should be explicitly out of scope for now.
Explain how the product will verify or validate AI outputs before users act on them, including where human confirmation is required.
Describe how differences in customer data readiness should affect eligibility, onboarding, or rollout for the first release.
Propose an evaluation and launch plan that includes success metrics, failure metrics, and a clear decision gate for expansion, iteration, or rollback.

Constraints

You cannot assume customer data is clean, complete, or consistently instrumented across enterprise accounts.
The first release must protect trust in Amplitude's core analytics product for enterprise customers.
You have one product area team for two quarters, so approaches that require a major platform rebuild are not feasible.
The solution should work as a product capability across many customers, not as a bespoke services workflow.

Format

Use short section headers and make the trade offs explicit.

How Amplitude actually played it

Amplitude's director of AI analytics reflects on shipping three AI products and over twenty AI features to thousands of enterprise customers, and on what broke. The core point is that agents work best where outputs are easy to verify, which makes analytics a hard fit because data is messy and questions are open-ended. A candid read for PMs deciding how much autonomy to promise.

Match the agent to tasks whose outputs can be verified, or invest in verification first.
Customer data readiness limits what agents can do in production.
Evals can act as the requirements document for AI behaviour.
Expect to learn more from what broke than from what worked.

Built from What We Learned Building AI Products in 2025 by Nirmal Utwani, Amplitude.