Anyone can call an LLM API. Designing one that stays fast and affordable at scale is the real engineering, and a favorite interview topic. Once your feature has users, three forces start fighting: latency (how fast each response feels), throughput (how many requests you serve), and cost (what you pay per request).
Latency: make it feel instant
Stream tokens so output appears immediately; time-to-first-token matters more than total time for perceived speed. Right-size the model, since a smaller or distilled one often hits the same quality bar far faster. Cache aggressively with exact-match and semantic caching, reuse the KV cache, and consider speculative decoding. A reasonable budget: retrieval under 100ms, first token under 1 second, full answer in a few seconds.
Throughput: keep the GPUs busy
Continuous batching (vLLM, TGI) lets requests join and leave a running batch each decoding step, keeping expensive GPUs saturated, the single biggest win over naive serving. Quantization (INT8/INT4, AWQ/GPTQ) shrinks memory and speeds inference with minor quality loss; validate on your evals first. Use tensor and pipeline parallelism for models too big for one GPU.
Cost: spend where it counts
Run a model cascade: route easy or short queries to a cheap small model and escalate only the hard ones to a large model. Cache repeated prompts, which can cut token cost 30 to 90 percent. Trim context and cap max tokens, since you pay for every token in and out. Batch offline jobs instead of paying for real-time when you don’t need it.
The mental model
Latency, throughput, and cost form a triangle, push one and you usually trade against another. The art is choosing which to optimize for the use case at hand: a chatbot prioritizes latency; a nightly summarization job prioritizes cost and throughput.
How you’d be graded
Strong answers name the bottleneck (usually GPU memory bounded by the KV cache and context length), reach for continuous batching and caching before throwing hardware at the problem, and tie every choice back to an explicit SLO and budget.
This is exactly the kind of trade-off question interviewers love. Refer 3 friends to unlock the full Interview Question Bank, or refer just 1 to grab the Cheat Sheet. Your share link is below.