PostHog · AI product · Senior · 45 to 60 minutes

Scope PostHog's first in product AI agent

PostHog is a platform with product analytics, feature flags, session replay, experiments, and related tools used by product and engineering teams. Answer this product management case and get it graded line by line against a weighted rubric.

AI product strategy
scoping and prioritization
technical judgment
trust and UX
experimentation and metrics

The situation

After a hackathon, PostHog has a rough internal prototype of an in product AI assistant that can answer simple questions by calling one tool. The CEO and AI lead ask you, a senior PM, whether this should become a real product bet inside PostHog and what the first release should look like. Teams are excited, but there is disagreement on core choices. Some want narrow predefined workflows that feel safer and easier to measure. Others want a more open ended agent that can choose tools, plan work, and handle a wider set of requests. Engineering warns that model capabilities are improving fast, so heavy orchestration, brittle prompts, or many hand built workarounds may age badly. Design worries that users will not trust answers that appear as magic, and research points out that strong offline evals may still miss real product failure modes. Assume you have one PM, four engineers, one designer, and six weeks to propose a launchable v1. Recommend what to build first, how transparent the experience should be, and how you will learn fast without locking the team into a complex architecture too early.

What to produce

A 3 page product strategy memo.

Your answer must

Identify the primary target user for v1 and the top two jobs to be done, and explain why this is the right wedge for PostHog now.
Recommend the v1 product shape and scope, including whether to start with predefined workflows, a more open ended agent, or a hybrid, and state what you would explicitly not build yet.
Propose the core interaction and system design principles for v1, including planning, tool use, whether to use one controller loop or multiple specialized components, and how much action history to show users.
Define success metrics and a validation plan that combines offline evals with real world usage signals, and explain how you will review failures and user confusion.
Outline rollout and the next two release decisions, including triggers to simplify, expand, or change the approach as models improve.

Constraints

Must work as an in product experience inside PostHog, not as a separate assistant product.
Cannot rely on manual human review or support intervention in the normal user path.
Must fit the team and six week timeline in the scenario.
Do not assume today's model limits or prompt workarounds will still matter next quarter.

Format

Use short sections with bullets. A simple option tradeoff table is helpful.

How PostHog actually played it

PostHog shares eight lessons from a year building its in-product agent, including that agents beat workflows, a single loop beats subagents, to-do lists help, and evals are not enough. It starts from a one-tool prototype built in a hackathon. Valuable for PMs deciding how to scope an AI agent feature.

Model improvements can make last quarter's workaround obsolete
Showing every step builds trust in agent output
Evals alone do not catch product problems, so watch real usage

Built from 8 learnings from 1 year of agents (PostHog AI) by Michael Matloka.