AI & Technical question

Design to Code quality is strong on Figma-owned demos but inconsistent across real customer repositories. How would you design the evaluation system for this product: define the task taxonomy, build the annotation pipeline, choose offline and human-in-the-loop quality measures, and set clear ship/no-ship criteria before expanding availability?

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Whether you can design a rigorous evaluation system for a generative AI feature whose quality is inconsistent across real-world inputs.

Define a task taxonomy: component fidelity, layout and spacing accuracy, responsive behavior, and code-style conformance to the target framework and design system.
Build an annotation pipeline using real customer repositories, not just Figma-owned demos, with human reviewers scoring each taxonomy dimension against a rubric.
Choose offline measures: pixel or structural diff against the source design, code compile success, and lint or style-guide conformance.
Add human-in-the-loop measures: designer and engineer satisfaction ratings on a sample of real outputs, since some quality dimensions (does this look right) are hard to automate.
Set ship or no-ship criteria per taxonomy category, for example component fidelity must clear a higher bar than exact spacing, since fidelity errors are more visible and costly to fix.
Expand availability only after the offline and human scores are both stable across a rotating sample of real customer repos, not just the original demo set.

Insists the eval set include real customer repositories, since demo performance is known to be misleadingly strong.
Breaks the single 'quality' question into named dimensions (fidelity, layout, responsiveness, code style) with separate bars, since averaging them hides real weak spots.
Uses a mix of automated diff metrics and human review, since visual and code-quality judgment cannot be fully automated yet.
Names a concrete gate, for example do not expand to a new framework or repo type until component fidelity clears an assumed 90% on that segment.

Evaluating only on Figma-owned demos, which is exactly the gap the question flags.
Reporting one overall quality score instead of dimension-level bars.
No plan for how the eval set stays representative as customer repos change over time.

How would you source a representative sample of real customer repositories for the eval set?
What would you do if fidelity is strong but code style consistently fails lint checks?
How often would you refresh the eval set, and why?