Honeycomb · Execution · Senior · 45 to 60 minutes

How to Ship Natural Language Querying

Honeycomb is an observability platform that helps engineers explore and understand production systems using telemetry data. Answer this product management case and get it graded line by line against a weighted rubric.

Product strategy
Execution scoping
Technical judgment
Risk management
Metrics

The situation

You are the PM responsible for a new Honeycomb capability that lets users describe a query in natural language and get back a Honeycomb query they can refine in the existing UI. The VP of Product wants a recommendation this week on whether and how to launch a first version soon, because customers are asking about AI and competitors are making flashy claims. Engineering has a rough prototype, but it is unstable. Some customer schemas are extremely large, model responses can take several seconds, and the team is seeing inconsistent output quality for vague requests. Security and legal are also raising concerns about prompt injection, privacy, enterprise terms, and what customer data could be sent to an external model provider. Design is pushing back on adding a chat experience if it complicates the core workflow. Assume you have one month to define an initial release plan with limited engineering capacity. Your job is to recommend the MVP scope, the launch approach, and the safeguards that make this a real product feature rather than a demo that breaks under real customer use.

What to produce

Write a 1 to 2 page product recommendation memo.

Your answer must

State a clear recommendation on whether to launch now, delay, or launch with a constrained MVP, and define the target user job to be done.
Define the MVP experience, including where it fits in the existing product and what the system should and should not do in the first release.
Analyze the key execution tradeoffs around context limits, large customer schemas, output quality for broad inputs, and latency, and explain how those tradeoffs shape scope.
Identify the main security, privacy, legal, and compliance risks, and propose launch gates or mitigations for each.
Provide a rollout and learning plan that includes success metrics, guardrails, customer controls, and what you would test or instrument after launch.

Constraints

The solution must rely on currently available third-party commercial LLM APIs, not a custom trained model.
Some customers have schemas with thousands of fields, which may exceed model context limits.
The experience must feel interactive enough for product use, so long multi-step LLM chains are risky.
The feature cannot create destructive actions or require direct LLM access to Honeycomb production systems or customer data stores.

Format

Use a short executive recommendation first, then sections with bullets for scope, tradeoffs, risks, and rollout.

How Honeycomb actually played it

A product lead explains the obstacles in building a natural language query feature, from context limits to latency and legal review. Early access alone will not reveal them.

An LLM is an engine for features, not a product
Latency can break chained designs
Budget time for legal and prompt injection work

Built from All the Hard Stuff Nobody Talks About when Building Products with LLMs by Phillip Carter.