AI & Technical question
Several enterprise customers report that the ServiceNow and GitHub connectors are ingesting data successfully, but search results and assistant answers still feel incomplete or low quality. How would you diagnose where the breakdown is across crawl freshness, permissions sync, metadata extraction, entity mapping, ranking, and UX, and how would you decide which fixes to ship first?
- Glean
- AI & Technical
- Hard
Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.
Start a mock interview on this question · Mock interview from a job description
What this question tests
Tests systematically diagnosing a search and answer quality problem across the full pipeline, from crawl freshness to UX, and sequencing fixes by evidence rather than guesswork.
How to approach it
- Check crawl freshness first: are ServiceNow and GitHub documents actually up to date, or is stale content the root cause of incomplete-feeling answers.
- Check permissions sync: are permission changes propagating correctly, since overly restrictive sync could silently exclude relevant, accessible documents from results.
- Check metadata extraction and entity mapping: are ticket fields, code references, and identifiers being parsed correctly, since garbled metadata degrades both ranking and answer synthesis.
- Check ranking: even with clean data, is the ranking model weighting these newer connector types appropriately against more established sources.
- Check UX: are users phrasing queries in a way the system doesn't map well to ServiceNow or GitHub-specific terminology, versus a genuine retrieval failure.
- Instrument each stage with a metric, freshness lag, permission sync error rate, entity extraction accuracy, so the sequence of checks becomes repeatable for future connector issues, and fix the earliest broken stage first since downstream stages can't fix upstream data problems.
What a strong answer includes
- Orders the diagnosis from data pipeline (crawl, permissions, metadata) to model (ranking) to UX, since fixing UX first would be wasted if the underlying data is broken.
- Names a specific measurable check for each stage, freshness lag and entity extraction accuracy, instead of a vague audit.
- Recognizes that permissions sync errors can cause silent under-retrieval, a subtler cause than obviously stale or missing data.
- Proposes fixing the earliest broken stage first, since downstream stages inherit upstream problems.
Common mistakes
- Jumping to a ranking or UX fix without first confirming the underlying data pipeline is healthy.
- Treating incomplete or low-quality as one problem instead of testing each pipeline stage separately.
- Skipping instrumentation, making the same diagnosis process unrepeatable for the next connector issue.
Likely follow-up questions
- How would you prioritize if both crawl freshness and permissions sync are broken?
- What would you check first if this pattern showed up on a third connector?
More ai & technical questions
- How would you measure whether Glean's AI answers are accurate and well-cited?Glean · AI & Technical · Medium
- How would you design a governance model that behaves consistently across Glean's UI, public APIs, and heterogeneous enterprise connectors, for example, a shared policy layer for permissions, action scopes, audit logging, and redaction, even when source systems expose different authentication models and data granularity?Glean · AI & Technical · Hard
- A Fortune 500 customer asks for an audit trail for every AI-agent answer: which data sources were accessed, which permissions were applied, why the response was generated, and whether sensitive content was filtered or redacted. Design the admin and end-user experience, and the core APIs/data model needed to support this without overwhelming either audience.Glean · AI & Technical · Hard
- Design the next version of Glean’s Indexing API and custom connector platform for large enterprises that need to bring proprietary data sources online quickly. What core capabilities would you prioritize, such as schema/modeling, auth and permissions, incremental sync, observability, and governance, and what principles would you use to keep the platform flexible without creating an inconsistent product experience?Glean · AI & Technical · Hard
- Glean Model Hub lets enterprise customers choose among LLMs for search, assistant, and agent workflows. How would you build a model-provider evaluation framework to decide which new providers to add, including the gating criteria, offline/online evals, and the tradeoffs you would make across answer quality, latency, security, cost, and enterprise-specific requirements?Glean · AI & Technical · Hard
- Suppose Glean wants to launch a new answer-generation experience in core search. How would you define the MVP and rollout plan given enterprise constraints around permissions, citations, latency, model choice, and admin controls? What would need to be true before you scaled it broadly?Glean · AI & Technical · Hard
More questions from Glean
Learn the skill behind it
Chapters of the AI PM course that teach what this question tests.
- Chapter 1: Foundations: the model and the decisions it forces on you
- Chapter 8: Evals: define good and make the number defensible
- Chapter 6: Agents and agentic architecture