AI & Technical question

Glean Model Hub may support multiple LLMs for search, assistant, and agent workflows. Design an evaluation framework you would use to decide whether a new model should be added to the portfolio, including how you would assess answer quality, latency, cost, safety, and enterprise-specific requirements such as permissions, grounding, and consistency.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Tests ability to design a rigorous, enterprise aware model evaluation framework covering quality, cost, safety, and Glean specific requirements.

Define what the evaluation decides: whether a candidate model earns a slot in Model Hub for search, assistant, or agent workflows.
Build an offline eval suite with a golden set of real enterprise queries, scored on accuracy, groundedness to cited sources, and latency under load.
Add enterprise checks: does the model respect document level permissions, does it stay consistent in tone and format across repeated queries.
Layer in safety testing, for example red teaming for prompt injection through retrieved documents and cross workspace data leakage.
Run a limited live pilot with a design partner workspace comparing the candidate against the current default on real usage, not just offline scores.

Treats permission compliance as a hard gate, for example the model fails outright if it ever surfaces content a tester lacks access to.
Separates model quality from system quality, since a great base model can still fail if Glean's retrieval or grounding is weak.
Proposes concrete thresholds, for example groundedness within two points of the current default and latency under a set ceiling at the ninety fifth percentile.
Includes a live pilot step, since offline evals on curated queries often overstate real world enterprise search quality.

Evaluating raw model quality on public benchmarks without testing it against Glean's actual retrieval and permission system.
Skipping adversarial or safety testing because the model scores well on accuracy alone.

How would you weight cost against quality if the best model is also the most expensive?
What would you do if the model passes offline evals but underperforms in the live pilot?