PostHog · AI product · Mid · 45 to 60 minutes

Should PostHog automate fixes from product signals?

PostHog is a product analytics and developer tools platform that helps teams understand, test, and improve their products. Answer this product management case and get it graded line by line against a weighted rubric.

AI product strategy
Prioritization
Risk management
Metrics
Systems thinking

The situation

PostHog sees a steady stream of product signals that suggest something is broken, confusing, or slow, including errors, failed queries, and direct user feedback. Leadership asks you, a PM working on product experience, to recommend whether PostHog should invest in an AI powered workflow that turns those signals into prioritized engineering work, and if so, how far the first version should go. The opportunity is speed: customers want issues found and fixed faster, and internal teams spend significant time triaging noisy reports. The risk is trust: many signals are ambiguous, some come from public or user controlled inputs, and a bad recommendation could waste engineering time or create security or quality problems. There is also a practical concern that running agents across large volumes of signals could become expensive quickly. You need to propose a next quarter plan that balances automation with human judgment, defines what the system should attempt first, and explains how the team will know whether it is safe, useful, and affordable enough to expand.

What to produce

Write a 1 to 2 page decision memo.

Your answer must

Define the product problem, the primary internal user or team, and the jobs this workflow should improve.
Recommend the scope of the first version, including which signal types to include or exclude at launch and why.
Describe the end to end workflow from incoming signal to engineering action, including where human review is required.
Specify how the system will decide whether an issue is actionable, including clear gating criteria.
Propose a trust, safety, and abuse handling approach for public or user controlled inputs, including prompt injection risk.
Define success metrics and an evaluation plan that covers quality, speed, and cost per run.

Constraints

You cannot assume the system is allowed to ship code directly to production without human approval.
Some input signals are public or user controlled, so they must be treated as untrusted.
Engineering time is limited, so the first version must be narrow enough to build and evaluate within one quarter.
LLM usage has a real budget impact, so your plan must include cost control.

Format

Use sections with headings such as goal, users and signals, proposed workflow, actionability gate, safety and trust, metrics and evals, and rollout recommendation.

How PostHog actually played it

PostHog describes a pipeline that takes product signals such as errors and failed queries, groups them into problems, researches them with agents, decides what is actionable, and opens pull requests. The authors share what was hard, including prompt injection from public signals, evals and token cost. A PM can use it as a model for turning feedback and analytics into a closed loop with human review.

A closed loop needs a gate that decides what is actionable, not only a way to generate fixes.
Public inputs can be attacked, so classify signals before acting on them.
Evals matter more than prompts when you want the loop to be trusted.
Cost per run is a product constraint, so scope what the agent attempts.

Built from What if your product built itself? by Josh Snyder and Cleo Lant, PostHog.