Product design question

How would you improve LangSmith's observability for production agents?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests product design for debugging tooling serving developers running complex agents in live production environments.

Define the user need precisely: a developer needs to quickly understand why a production agent produced a wrong or unexpected output, often across many chained steps and tool calls.
Identify the core pain today: agent behavior involves many steps, and without clear tracing, developers must guess which step caused a downstream failure.
Prioritize full trace visibility: a clear, step by step view of every tool call, model response, and state change in a single agent run, not just the final output.
Add comparison tooling: let developers compare a failing run against a similar successful run to spot exactly where the behavior diverged.
Add proactive alerting: flag anomalous agent behavior, like unusually long runs or repeated tool call failures, before a developer even goes looking for a problem.
Confirm with the interviewer whether the priority is debugging individual failures after the fact, or proactively catching production issues before they affect many users.

Prioritizes full step by step trace visibility as the core capability, correctly identifying that multi step agent failures are hard to diagnose without seeing every intermediate step.
Proposes comparing a failing run against a similar successful run, a powerful debugging technique that isolates exactly where behavior diverged rather than requiring manual guesswork.
Adds proactive anomaly alerting, shifting from purely reactive debugging to catching issues before they compound across many production runs.
Frames improvements around reducing time to root cause, a concrete, measurable outcome for the developer using the observability tool.

Providing only the final output and error message without full step by step trace visibility into the agent's intermediate reasoning and tool calls.
No comparison tooling, forcing developers to manually reconstruct why a failing run differs from a working one.
Purely reactive design, requiring a developer to notice a problem before any debugging tooling helps them investigate it.

How would you help a developer debug an issue that only reproduces intermittently?
What would you alert on to catch problems proactively without causing alert fatigue?
How would you scale trace storage and search as agent run volume grows very large?