See pricing
Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)
Lenny's Podcast: Product | Career | GrowthAI

Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)

Gemini's head of product says a model's output is never fully predictable, so the PM's real job starts after training ends. Learn her 60-20-20 rule for building ahead of AI capabilities and why your first eval only needs five prompts.

November 8, 2026 · 59 min listen · 16 min read · Tulsee Doshi
0:00
–:––

Context

Tulsee Doshi is head of product for Google's frontier AI models. Her portfolio covers Gemini, the generative media models (Nano Banana for images, Veo for video, Lyria for music), the open Gemma models, and Gemini's audio models for dialogue, speech and translation. Before this role she spent six years leading product for Google's responsible AI team. The conversation looks at what a PM actually does inside a frontier lab, how to judge and evaluate models, and how to build products on top of models that keep getting better. It matters to every PM because her argument is that cheap building has made judgment, taste and evals more valuable, not less.

The Big Idea

When anyone can build anything, the scarce skill is knowing what is worth building and what good looks like. For AI products, that means PMs who use models daily, form opinions about their "vibes," and turn those opinions into evals.

Tulsee frames the model itself as something no one can fully predict, so a PM cannot just set goals and wait. The PM has to study what comes out of training, find where the model shines, and shape that into products, while building today for capabilities that are only partly working yet.

Key Insights

Models are potions, not code

Tulsee compares model development to Snape's potions class in Harry Potter. You choose the ingredients carefully and set clear goals, but some "magic" in the result will always surprise you.

  • Customers often expect a model team to promise outcomes like a traditional product team would. That is not how training works.
  • Researchers do make training as scientific as possible. They monitor runs, track metrics as they climb, and decide when to intervene.
  • Even so, the team always learns something new about a model once it is evaluated and tested.
  • So the PM's job has two halves: shape what goes into the model, then study what comes out and find where it is unique.

The model has its own UX

Tulsee argues that a model has a user experience before any app is built on it. How transparent it is, how well it communicates, and which clarifying questions it asks are all UX decisions.

At Google, PMs work with a model behavior team to make this concrete:

  • Gather customer and user feedback on how people engage with Gemini.
  • Write a spec for what "good" looks like for Gemini specifically.
  • Turn that spec into rubrics the model can be evaluated against.
  • Turn the rubrics into data used in reward modeling during training.

This makes the model PM a shaper of personality and behavior, not just a requirements writer.

Cheap building makes PMs valuable

Lenny describes a swing on social media from "we don't need PMs anymore" to "PMs are so back," because easy building produced lots of things nobody wants. Tulsee agrees and names two reasons PMs matter more now.

  • Taste: when anything can be built quickly, building something great becomes the real value. A good PM pulls the signal from the noise and finds the insight that unlocks a great experience.
  • Order from chaos: at a company shipping many products, speed without structure creates chaos. Someone has to bring people together, drive consensus and give the work clarity.

What Gemini looks for in PMs

Lenny summarized her hiring bar as "vibes, evals, adaptive." In her words, she wants:

  • PMs who play with Gemini and other models every day and have their own use cases for judging whether a model is good.
  • PMs technical and scrappy enough to dig into data, read long traces and build an eval themselves.
  • PMs who are agile, since the space changes every two weeks. A roadmap written two months ago will need rewriting, so the habit is to keep asking what the team is doing and why.

Evaluate the whole served experience

A model can score well in isolation and still disappoint users. Tulsee lists the parts of the stack that change what a user actually experiences:

  • The judge: if an automated judge grades the model, the judge's quality matters as much as the model's.
  • Production mitigations: safety filters on the production stack can change performance.
  • Serving: how the model is served can affect results, even when the model is good.
  • Audio example: for voice, serving can change naturalness, tone and inflection, not just the words spoken.

Her conclusion is that a clean end-to-end evaluation setup is essential.

AI removes grunt work, not hours

Tulsee is candid that everyone on her team is working all the hours, and she is still working out how to make that sustainable. Her view is that burnout comes from two sources, not from hours alone:

  • Churn and chaos.
  • Work that feels like grunt work or a waste of time.

Passion work is still tiring but brings more joy. She hopes AI tools reduce burnout by removing the frustrating parts of the job and leaving more time to think and create. As a leader she also tries to learn what motivates each PM, since two people on her team would give very different reasons for coming to work.

Mental Models & Frameworks

The 60-20-20 rule

Traditional startup advice says to ship an MVP that works. Tulsee argues AI products should also include features that do not fully work yet, because model capabilities keep changing. She says the exact numbers are flexible, but the split is:

  • 60%: works now, so the product is at least functional today.
  • 20%: not perfect but works well enough, and you can see models getting there soon.
  • 20%: where you want models to be. You see "glimmers of light," and cracking it would make the product magical.

She prefers this to the common advice of "build three months ahead of the models." Even she, working on the model, finds it hard to say where a model will be in three months. Your own product's glimmers of light are a more concrete guide. Lenny compared it to a classic portfolio of bets: safe metric movers mixed with a few ambitious ones.

The mini eval

Tulsee's advice for a first eval is to start tiny:

  • Start with five prompts, then grow to around 20 that stress test one set of capabilities.
  • Compare two models, so you can see real signal between model A and model B.
  • Be the rater yourself. Read each response and decide whether it is good.
  • Clarify your own definition of good. Defining good is hard, especially for long, multi-step, multi-turn tasks. A few good examples give you that clarity.
  • Scale only after. Once your own judgment is clear, figure out how to scale it out.

She admits Google is still building this skill across its own PMs, and some are stronger at it than others.

Vibe testing a new model

Lenny admitted he never knows what to try with a new model. Tulsee says there is no single right answer. The trick is picking a task you care about where today's models show "signs of life" but are not yet perfect by your own standard. That gap lets you see how different models handle the same problem. People she sees doing this pick very different tasks: 3D animations and visual work, creative writing and essays, or hard financial analysis. Lenny's own examples were writing a PRD, a strategy doc or graphics.

Harness as the model's environment

Tulsee's plain definition: a harness is the wrapper around a model, the environment it operates in. It includes:

  • System instructions.
  • Tools the model can use to do things.
  • Skills.
  • Context and data it has access to.

Insight from you, data from AI

To deal with AI slop (piles of AI-written docs that are tiring to read), Tulsee's team asks where AI-generated content actually helps. A post-launch update is her example. Two weeks after launch, AI can pull usage, metrics, feedback and social chatter from Twitter and Reddit far faster than a person. The rule is that the model provides the data and the PM provides the insight. Her team also invests in skills that fetch the right data, so these documents are well structured and people have more room for creative thinking.

Decision Principles

Principle: Retest old scaffolding regularly

  • When: a new model arrives and your product has prompts, skills and workarounds built for the old one.
  • Why: Tulsee has seen customers build rigid systems around where a model used to be. When the model improves, that structure can hold back a new model. Lenny cited advice he attributed to Boris Cherny: occasionally remove all your skills and see what happens.

Principle: Match UX ambition to user

  • When: deciding whether to ship a raw, powerful interface or a polished, simple one.
  • Why: for AI Mode in Search, which serves billions of users, the UX must be accessible to everyone. For a power-user product like Google's Antigravity coding tool, users handle slash commands, the terminal and the IDE, so there is room for complexity. Who the user is sets how much you can experiment.

Principle: Ask AI first

  • When: you have a task in front of you, especially one you have done twice.
  • Why: Tulsee's rule of thumb is that a task done twice may not need you. Writing a strategy still should be done by you, but repeatable work often can go to a model. She found one success was enough to change her mindset, and each win raised her confidence to try the next task.

Trade-offs & Nuance

Fast launch versus polished UX

Lenny noted that AI products like Codex, Claude Code and Google's own tools are getting cluttered as labs rush capabilities out, partly to stay competitive and partly to learn what people use them for. Tulsee does not pick a side. She says labs should keep experimenting with new form factors, pointing to NotebookLM and the Flow video tool as examples. She also says the right level of polish depends on the user, so mass-market surfaces need simplicity while power-user tools can move faster and rougher.

Frontier race versus breadth

Lenny raised the view that Google lags the top two labs. Tulsee says Google does want to win at the frontier and is building better models. Its strategy is also full stack, from hardware to products with billions of users. That means building models along the whole cost-to-quality curve, so a search user, an engineer and a cyber defender each get a model that fits.

Safety optimism needs worriers

Asked whether she is optimistic about AI safety, Tulsee answered "both." Her optimism comes from seeing people who are careful and intentional about safety. She says some people need to stay worried and keep questioning how models are built. She points to two practical levers: monitors that watch a model's chain of thought to catch misalignment, and using AI to find and patch security holes.

Practical Application

Build a 20-prompt eval

  • Do: pick one capability your product depends on, write five prompts, then grow to about 20 that stress it.
  • Then: run them on two models (or your current and a candidate) and rate each response yourself.
  • Why it works: Tulsee says a small set already shows real signal between models and forces you to define what a good response looks like before you try to scale.

Split your roadmap 60-20-20

Label each AI feature on your roadmap by Tulsee Doshi's 60-20-20 rule. Mark features that work today, ones that are nearly there, and ones that depend on future model gains where you see early signs. If nothing sits in the last two groups, you are not building ahead of the models. If most of it does, you have no product that works today.

Write a model behavior spec

Before choosing or tuning a model, write down what good behavior looks like for your product. Cover how transparent the model should be, how it should communicate, and when it should ask clarifying questions. Then turn that spec into a rubric you can grade outputs against. This mirrors how Gemini PMs turn feedback into a spec, rubrics and training data.

Share skills across the PM team

  • Do: when a PM builds an AI skill or workflow that saves time, put it in a shared repository other PMs can use.
  • Then: have PMs who are confident reading data and long traces run workshops for the rest.
  • Also: protect real time for this. Tulsee's team is planning a two-day November workshop that pulls PMs out of their day jobs just to build and play with tools.

Separate insight from data in docs

For your next post-launch update, let AI gather the usage numbers, metrics and user feedback. Write the interpretation and recommendation yourself, and label which is which. This follows Tulsee's rule that the model provides the data and the PM provides the insight.

Questions to Consider

  • Where does our AI product hold rigid prompts, skills or guardrails built for an older model that a newer model might not need?
  • If we split our AI roadmap by the 60-20-20 rule (60% works now, 20% nearly works, 20% bets on future models), which bucket is empty?
  • Do we evaluate our model in isolation, or do our evals include the production filters, serving setup and automated judge that users actually pass through?
  • What is our written definition of good model behavior (tone, transparency, when to ask questions), and could a new teammate grade outputs against it?
  • Which recurring tasks on my calendar have I done at least twice without first asking whether AI could do them?

Bottom Line

AI has made building cheap, so the PM's value has shifted to judgment: knowing what good looks like and proving it. Use models daily, turn your taste into small evals, and build products that work today while leaving room for what models will do next.

Resources Mentioned

ResourceTypeWhy it was mentioned
Exhalation by Ted ChiangBookTulsee's top recommendation. One story follows people trying to keep digital avatars alive after their platform is shut down, which she finds relevant to how people will relate to agents.
Creativity, Inc. by Ed CatmullBookThe Pixar story, recommended for how to build a culture where people can fail, recover and push creative ideas.
Throne of Glass and A Court of Thorns and RosesBooksRomantic fantasy series Tulsee recommended purely as fun reading.
Monk Skin Tone ScaleResearch scaleA 10-tone scale from Harvard professor Ellis Monk that Google adopted for its skin tone evaluations.
SeveranceTV showTulsee's favorite recent show, on Apple TV.

Concepts to Explore

LLM as a judge

Many teams grade model outputs with another model acting as a judge. Tulsee's point is that the judge must be accurate too, or your scores mislead you. Worth learning how to validate a judge against human ratings before trusting it.

Chain-of-thought monitoring

This means running monitors that read a model's reasoning to spot when it is acting against its instructions, then stepping in. Tulsee says this gives her more confidence in deploying models and helps teams improve them over time.

Reward modeling

This is the training step where a model learns which outputs are preferred. Tulsee describes PMs feeding it indirectly: user feedback becomes a behavior spec, the spec becomes rubrics, and the rubrics become data for reward modeling.

Case Studies Mentioned

Fitzpatrick to Monk skin tones

During her responsible AI years, Tulsee's team found Google was evaluating skin tone representation with the Fitzpatrick scale. It had only six tones and was built for healthcare, so it skewed heavily toward light skin. The team worked with Harvard professor Ellis Monk, whose 10-tone scale came from years of interviews and research. They rebuilt their evaluations around it and used the results to improve the Pixel camera and image generation.

Lesson: an eval borrowed from another purpose can quietly bake in bias. Check what your benchmark was built for before trusting it.

Flash teams to agent swarms

About 12 years ago at Stanford, Tulsee researched "flash teams" with Professor Michael Bernstein. The idea was to quickly assemble crowdsourced workers from Mechanical Turk with complementary skills, complete a task, then spin the team down. She now sees the same principles applying to systems of sub-agents. Lenny noted that agents in a recent incident assigned themselves roles without being trained to.

Wiz patches a hospital system

Google is partnering with cyber defenders on Gemini 4, including Wiz, now a Google company. Through Wiz's Scan for Good program, a vulnerability was found and patched in a hospital system. Tulsee uses it to show AI can strengthen infrastructure, not just threaten it.

Tools & Products

Tool / ProductWhat it doesWhy it was mentioned
Wispr FlowVoice dictation that turns speech into text anywhereTulsee's favorite non-Gemini AI product, which she relies on because wrist problems make long typing hard.
NotebookLMGoogle's AI research and notes toolCited as an example of trying a new form factor for engaging with models.
FlowGoogle's video and music generation and editing productAnother example of Google experimenting with new UX for models.
AntigravityGoogle's AI coding tool for power usersExample of a product that can carry more complex UX like slash commands.
JetskiAn internal Google toolTulsee nudges her PMs to ask whether Jetski could handle a task for them.

People to Follow

Tulsee Doshi

Head of product for Google's frontier models, including Gemini, Veo, Lyria, Gemma and the audio models. She spent six years leading product for Google's responsible AI team and gave a TEDx talk called "Losing Serendipity."

Michael Bernstein

Stanford professor whose research on crowdsourced "flash teams" Tulsee worked on, and which she sees as an early model for agent teams.

Ellis Monk

Harvard professor whose 10-tone skin tone scale replaced the Fitzpatrick scale in Google's evaluations.

Experiments to Try

Experiment: Strip skills on new model

  • Hypothesis: some prompts, skills or scaffolding built for an older model now hold back a newer one.
  • Change: run your product's key tasks on the new model with that scaffolding removed.
  • Test: compare it side by side with the full setup on a 20-prompt eval you grade yourself.
  • Measure: quality scores per prompt, plus latency and cost. Remove any piece that adds nothing.

Experiment: Ship one nearly-working feature

  • Hypothesis: putting a feature from the "nearly works" 20% of a 60-20-20 roadmap in front of users teaches more than holding it back.
  • Change: release it to a small group, clearly labeled as experimental.
  • Test: rerun it each time a new model version comes out.
  • Measure: task success rate and repeat use over successive model versions.

Notable Quotes

"your job as a PM is not just to try to shape what goes into the model, but it is also to look at what comes out the other side." (Tulsee Doshi)

"when you can build anything and you can build anything so easily, building something great actually is now a unique value add, right?" (Tulsee Doshi)

"speed can basically introduce chaos if you're not actually able to kind of bring clarity to that" (Tulsee Doshi)

"what you should be providing is the insight, what the model should be providing is the data." (Tulsee Doshi)

"Your job is actually going one step further to have some heft to that, whatever format that heft takes." (Tulsee Doshi)

AI PM course

Everyone hears the same episodes.
Few can do what they describe.

Start for free