PostHog · AI product · Mid · 45 to 60 minutes

Where should PostHog focus AI for error backlogs?

PostHog is a product and developer tools platform that helps teams understand, test, and improve software. Answer this product management case and get it graded line by line against a weighted rubric.

product strategy
AI scoping
prioritization
metrics design
risk management

The situation

At PostHog, the team working on AI features for developer workflows has early evidence that users want help with error backlogs, but the requests point in different directions. Some teammates want a broad agent that can investigate and handle errors end to end. Others think the bigger opportunity is in repetitive backlog work such as sorting, grouping, and deciding what needs attention first. The team has product telemetry from how people interact with errors today, including what they search for before taking action, but it is not yet clear which jobs are common enough, valuable enough, and safe enough for an agent to own. The head of product asks you to recommend what PostHog should build next in this area. Why now: interest in AI is high, engineering time is limited, and a poor decision could reduce trust if the system closes or downgrades important errors. Assume you can fund one meaningful agent investment next quarter. You need to choose a direction, explain what evidence you would use to support it, and show how you would de-risk the rollout.

What to produce

A 1 to 2 page product recommendation memo.

Your answer must

Name the primary user and the specific error backlog job you would target first.
Compare 2 to 3 realistic product directions for agent support in error workflows, then recommend one.
Explain what existing evidence you would analyze first, including search or query behavior, to decide where the agent can add the most value.
Define success metrics and guardrails, including at least one quality or trust metric related to incorrect handling of errors.
Outline a rollout or experiment plan that shows how you would validate value before expanding scope.

Constraints

Keep the scope to error backlog management, not general purpose coding or repo level auto fixing.
You can rely only on current product usage data and telemetry, not a large new labeling effort or external partnership.
Engineering capacity supports one major workflow investment next quarter.
The solution cannot depend on heavy manual review for every action the agent takes.

Format

Use short headings and bullets. A simple options table is encouraged.

How PostHog actually played it

PostHog analyzes how its agents handled 4,063 errors, finding they do three jobs in a backlog and that smaller, sharper tools win. The post lists the kinds of errors agents handle best and what search behavior revealed. A data-based look at an AI product's real usage.

Study what agents actually do before expanding them
Bulk triage is where agents add the most value
Smaller, sharper tools beat broad ones

Built from 4,063 errors closed without a human opening PostHog: here's what we learned by Sara Miteva.