Context
Petra Wille and Teresa Torres discuss Teresa's recent writing on AI evals, but the conversation quickly moves past tooling and into a more durable product question: what does 'good' actually mean? Teresa's argument is that evaluating non-deterministic AI outputs is new, but defining success is not. For PMs, that is the useful reframing. The episode shows that the real work is the same work product teams have always had to do: define customer value, define business value, and measure whether the product is creating both.
The Big Idea
AI evals only become useful when a team defines what good looks like in its own product context. The hard part is not scoring model output, it is being explicit about the customer value and business value the output is supposed to create.
Teresa's point is that PMs should treat evals like any other product measurement problem. Generic checks like 'helpful' or 'concise' may be a starting point, but they are not a substitute for knowing which user you serve, what job they are trying to do, and what outcome would prove the product helped.
Key Insights
AI evals reuse old work
Teresa frames AI evals as part of the same family of work as writing a PRD, setting outcomes, or instrumenting a feature. The new part is that AI outputs are non-deterministic, so teams need new ways to measure them. The old part, and still the hardest part, is deciding what success actually is. For PMs, that means the core skill is not model scoring. It is defining the target clearly enough that a team can tell whether the feature is working.
Technical quality is insufficient
- Early product analytics often focused on easy technical measures like uptime, speed, and performance.
- Teresa's point is that those metrics matter, but they do not tell you whether a customer reached a value creation moment.
- She sees the same pattern repeating with AI evals. Teams measure the system because that is easiest, while skipping the harder question of whether the output created value for the user.
Metrics mature with context
Teresa uses engagement metrics as an analogy for how evals should evolve:
- Teams often start with broad, simple metrics like monthly active users.
- Then they refine those measures, for example with daily active users or a stickiness ratio like daily active users divided by monthly active users.
- The strongest teams go one step further and measure the specific behavior that signals value in their own product.
The lesson for AI products is that off-the-shelf evals can help teams get started, but mature teams eventually define quality in a way that is tied to their own product's value creation.
Concise depends on user context
Teresa's example is simple and important. A student learning astrophysics for the first time may need a long, step-by-step explanation. An astrophysics professor may want a short, direct answer. That means even a common eval trait like 'concise' has no stable meaning without context.
For PMs, this is the warning against universal quality labels. A trait that looks obviously good at first can flip meaning based on user expertise, use case, or moment in the journey.
Cross functional thinking matters here
Teresa does not blame engineers for starting with technical performance. She says that is a natural first lens, especially because AI evals arrived through a more engineering-heavy path. The problem is not that technical metrics are wrong. The problem is that they are incomplete.
This is why she argues for a cross-functional view. A strong eval suite should reflect:
- technical performance,
- customer value,
- business value.
Without that broader lens, a team can optimize the system and still miss the product.
AI amplifies weak processes
Petra pushes the conversation beyond evals into a broader operating point. If a product team lacks clear direction, weakly tracks outcomes, or has not agreed on which users it serves, AI makes those weaknesses more painful. It helps the team move faster, but possibly in the wrong direction.
Teresa connects this to a bigger pattern: teams already do work with unclear impact. If that work gets automated before anyone asks whether it matters, the team has not become more effective. It has just become more efficient at producing output that may not change anything.
Mental Models & Frameworks
Outcome first eval design
Think of an eval suite as product instrumentation for an AI feature.
- Start with the customer value you want to create.
- Then define the business value that should follow if the customer value is real.
- Only then decide which output checks and technical signals help you monitor the system.
Use this model when a team is tempted to jump straight into scoring answers before it has defined what a good answer actually helps a user accomplish.
Metric maturity ladder
This framework explains how teams usually get better at measurement over time.
- Level 1: broad, universal metrics that almost any team can adopt.
- Level 2: slightly richer measures that say more about frequency or depth.
- Level 3: product-specific behaviors that best represent value creation.
Teresa's DocuSign example makes the ladder concrete. A user signing a received document shows some engagement. A user sending their own document for signature shows deeper value, because it signals the user has adopted the product for their own workflow. Use this ladder when a team is stuck between vendor-default evals and the harder work of defining product-specific success.
Expectation inspection loop
Teresa reduces a lot of product work to one habit: say what you expect to happen, then check whether it happened. She applies that habit to outcomes, feature success, role clarity, and even day-to-day work.
Use this loop when a team is unclear about success criteria:
- Write the expected outcome in plain language.
- Name the evidence that would show the expectation was met.
- Review where the expectation was wrong, vague, or incomplete.
That is the same thinking behind good metrics, good evals, and good product judgment.
Decision Principles
Principle: Define value before evals
When: a team is deciding how to evaluate an AI feature. Why: a score for 'helpful' or 'concise' is not meaningful until the team knows which user, which task, and which value creation moment the answer is supposed to support.
Principle: Measure value not fluency
When: an AI feature produces polished answers and scores well on surface-quality checks. Why: fluent output can still miss the user's need. Teresa's standard is whether the output helps create customer value and eventually business value, not whether it simply sounds good.
Principle: Fix process before automation
When: a team wants to apply AI to a workflow that already feels noisy, slow, or low leverage. Why: AI accelerates the current system. If the underlying work has weak goals or weak signal, automation scales the confusion instead of solving it.
Trade-offs & Nuance
Simple metrics still have value
Teresa is not arguing that speed, reliability, or other technical measures should be ignored. She is arguing that they are necessary but not sufficient. Early, simple metrics are often the right place to start, especially for teams that are new to AI. The mistake is treating that first layer as the whole definition of success.
Context reduces comparability
Off-the-shelf evals are appealing because they are fast to adopt and easy to standardize across teams. The downside is that they often flatten the real context of the user and task. Product-specific evals are harder to design and harder to compare across companies, but they are more likely to measure what actually matters. The trade-off is convenience versus truth.
Automation raises clarity demands
AI can produce output before a team has produced clarity. That is useful when the product strategy, user segment, and success metrics are already clear. It becomes risky when those foundations are weak, because the team can now scale the wrong work faster than before.
Common Mistakes
Mistake: Measuring only technical output
- This happens because technical performance is easier to observe first.
- It is harmful because a product can look healthy while the user never reaches a value creation moment.
- The better approach is to pair technical evals with measures of customer behavior and business impact.
Mistake: Outsourcing your value judgment
Generic eval vendors often offer scores for traits like 'concise', 'helpful', or 'friendly'. Teresa's point is that those labels quietly assume a user, a task, and a context. If the team accepts those defaults, it may inherit the wrong definition of quality.
The better approach is to write a product-specific definition of what a good answer does for a specific user in a specific situation.
Mistake: Automating work without purpose
Teresa references a broader risk: teams automate tasks that did not matter much even before AI. That creates more output, but not necessarily more impact. A team can feel productive while still failing to change customer behavior or business results.
The better approach is to ask what decision, outcome, or user value a workflow is supposed to improve before using AI to speed it up.
Practical Application
Define good in writing
Before choosing eval metrics, write one short paragraph that completes this sentence: a good AI response for a specific user in a specific situation helps that user accomplish a specific outcome. Then add the evidence that would show customer value and the evidence that would show business value. This forces hidden assumptions into the open.
Map the value moment
For each AI feature, identify the customer behavior that proves value was created, not just the model behavior that looks polished.
- Teresa's DocuSign example shows the difference well.
- Any signing activity can look like engagement.
- Sending your own document for signature is a stronger signal because it shows deeper adoption.
Use the same logic to define the activation moment for an AI feature.
Layer your eval metrics
Build an eval suite with three layers:
- Technical: system health measures like reliability, latency, or cost.
- Customer: evidence that the output helped the user complete the job they came to do.
- Business: evidence that the feature improved retention, adoption, or another strategic outcome.
This structure helps a PM avoid the trap of treating polished output as proof of product value.
Test adjectives by persona
Take any vague quality word in current evals, such as 'helpful' or 'concise', and test it across very different user types. Teresa's student versus professor example is the model. If the word changes meaning sharply by user expertise or use case, the metric is too generic and needs to be rewritten.
Prune work before automating
Pick one recurring team workflow, such as weekly summaries, backlog triage, or research synthesis. Ask what decision or outcome that workflow is supposed to improve. If nobody can answer clearly, redesign or stop the workflow before adding AI. Faster waste is still waste.
Questions to Consider
- Which quality judgments in current AI evals are really vague adjectives, such as 'helpful' or 'concise', with no user-specific definition behind them?
- What customer value creation moment would prove an AI feature helped a user, instead of simply generating a fluent answer?
- Where is the product team still measuring only technical performance, such as speed or uptime, instead of whether customer behavior improved?
- Which recurring workflows would get faster with AI but still would not matter enough to justify doing them at all?
- Where does the product team need a product-specific definition of success, instead of a vendor-default eval or a generic benchmark?
Bottom Line
Teresa Torres's core message is reassuring and demanding at the same time. AI evals introduce new tools, but the hardest part is still classic product work: define what good looks like for a real user in a real context, then measure whether the product created customer and business value.
