Strategy question

Design an evaluation framework for ads ranking on Meta.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Strategic and metrics design: building a rigorous evaluation framework for a core, high-stakes ranking system.

Clarify the goal: ads ranking should balance advertiser value, user relevance, and Meta's revenue.
Define offline evaluation metrics: predicted click-through rate accuracy and predicted conversion rate accuracy against held-out data.
Define online evaluation metrics: actual click-through rate, conversion rate, and revenue per impression from live A/B tests.
Add guardrail metrics: user-reported ad relevance or annoyance scores, and advertiser return on ad spend.
Define the decision framework: a ranking change ships only if it improves revenue without degrading user experience or advertiser value guardrails.

Balances three stakeholders explicitly (users, advertisers, Meta) rather than optimizing for revenue alone.
Distinguishes offline model evaluation from online live-test evaluation, showing technical rigor.
Names a concrete guardrail, like advertiser ROAS, ensuring ranking changes don't quietly hurt advertiser outcomes.
Proposes a clear ship or no-ship decision rule tying metrics to guardrails, not just directional improvement.

How would you weigh a revenue gain against a small drop in user-reported ad relevance?
How would you detect if a ranking change is quietly hurting a specific advertiser segment?
How often would you re-evaluate this framework as the ad ecosystem changes?