Estimation question

Estimate the gross margin on serverless inference vs. renting GPU clusters.

Practice this question out loud. An AI interviewer asks it, follows up like a real interviewer would, and scores your answer. Type or speak.

What this question tests

Structured cost and margin estimation comparing two different infrastructure business models.

Define both models: serverless inference bills per token with shared, multi-tenant GPU utilization, while dedicated clusters bill a flat reserved rate regardless of utilization.
Estimate serverless margin: assume GPU cost of roughly 2 dollars per hour, serving many customers' requests on shared capacity at high utilization, say 70 percent, yielding a gross margin in the range of 60 to 70 percent.
Estimate dedicated cluster margin: assume the same GPU cost but a customer often runs it at lower average utilization, say 40 percent, since they pay for guaranteed capacity whether or not they use it fully, giving Together a similar or better margin on the same hardware.
Compare: dedicated clusters can have higher or comparable margin per GPU-hour sold because the customer, not Together, absorbs the underutilization risk, while serverless margin depends heavily on achieving high shared utilization.
Conclude with the key driver: utilization rate is the single biggest swing factor for serverless margin, while committed contract length is the biggest factor for dedicated margin.

Identifies utilization as the central variable separating the two models, since serverless margin lives or dies on packing many customers onto shared GPUs efficiently.
Notes correctly that dedicated clusters shift underutilization risk to the customer, which can make them safer margin even if the headline price per hour is lower.
Gives concrete assumed percentages for margin in both models, clearly marked as illustrative estimates rather than real reported figures.
Names the practical lever, improving serverless batching and scheduling to raise utilization, as the way to close any margin gap.

Assuming dedicated clusters are automatically higher margin without accounting for who bears the utilization risk.
Skipping the utilization assumption entirely and comparing only sticker prices per GPU-hour.

What utilization rate would make serverless more profitable than dedicated clusters?
How would a long-term GPU price decline affect this margin comparison?