System Design Interviews for AI and LLM Systems
A candidate I was coaching earlier this year got a question he had not prepared for at all. Not a hard one, exactly. "Design a system that answers customer support questions from our internal documentation." He had done a lot of classic system design preparation. He drew a load balancer, application servers, a database, a cache. Then he drew a box labeled LLM, connected it with an arrow, and stopped, because he could feel that the interviewer was waiting for something and he did not know what.
What the interviewer was waiting for is the subject of this article.
These questions are showing up in more loops, at more companies, and at more levels than they were a year ago. That is not surprising. Companies are solving similar problems, so they interview for it. What is surprising is how many otherwise strong system design candidates freeze in exactly the way my client did, because the classic preparation material does not cover the parts that are actually different.
Why This Round Exists Now
Two things happened at once.
Companies started shipping features with a model in the path, which means they now need engineers who have thought carefully about what that entails. And system design as a round gained weight generally, because when the coding round is partially tool-assisted, interviewers lean harder on the round that resists it. I have watched loops rebalance in this direction repeatedly over the past year.
Put those together and you get a round that is both more common and more consequential than it was.
The good news, and I want to be clear about this before going further: you do not need to be an AI specialist to do well here. Most of what you need is engineering judgment you already have, applied to a component with unfamiliar properties. The candidates who struggle are almost never the ones who lack machine learning depth. They are the ones who never sat down and thought about how a model behaves differently from a database.
Five Things That Are Genuinely Different
Everything you know about scoping, trade-offs, and failure handling still applies. But a model in the path introduces properties that no component in the classic problems has, and your design has to account for them explicitly.
The output is not deterministic. Call it twice with the same input and you may get two different answers. This breaks assumptions that are so ingrained they are invisible. Your caching strategy assumed determinism. Your tests assumed determinism. Your idempotency story assumed determinism. Say out loud that you have noticed this, because a lot of candidates design straight past it.
It can be confidently wrong. A database that cannot answer returns an error. A model that cannot answer often produces something fluent and incorrect instead. That means correctness is not a property you can check at the boundary the way you check a schema. It has to be designed for, usually by constraining what the model is allowed to draw on and by verifying the output before it reaches a user.
Cost scales with usage, in real money, per request. This is the constraint that most distinguishes people who have shipped one of these from people who have only read about them. Classic system design optimizes for machines you already pay for. Here, a design that works beautifully can be commercially impossible at a hundred times the traffic. Bring up cost per request unprompted and you will separate yourself immediately.
Latency is measured in seconds. A model call is orders of magnitude slower than a cache hit, sometimes slower than the entire request budget you would normally allow. That pushes streaming, asynchronous patterns, and honest conversations about what the user sees while waiting into the core of the design rather than the polish at the end.
Quality can regress without anything failing. This is the deepest one. Your error rate is flat, no alert fires, latency is fine, and the answers have quietly gotten worse because a prompt changed or the underlying model was updated. Every monitoring instinct you have is tuned to detect crashes. None of it detects this.
Questions to Ask Before You Design Anything
The classic advice to scope before solving applies here with even more force, because these problems are usually stated vaguely. Alongside your normal requirements questions, I would ask:
What is the cost of a wrong answer? A wrong answer in an internal search tool is an annoyance. A wrong answer in a medical or financial context is a different system entirely, and probably one with a human in the loop.
Does the answer need to be grounded in specific sources, or is general knowledge acceptable? This single question determines whether you are building a retrieval system or not.
Is a human reviewing the output before it reaches anyone? If yes, a great deal of your risk budget just moved, and you should design differently.
What is the acceptable latency, and can we stream? Streaming changes what "acceptable" means, because perceived latency and actual latency stop being the same number.
What volume are we expecting, and is there a cost ceiling? Ask for both. Then actually do the arithmetic in front of the interviewer.
How will we know if quality drops? Ask this one even if the interviewer did not raise it. It is the question that most reliably signals you have operated one of these systems rather than only designed one on a whiteboard.
Working Through the Grounded Assistant
Take my client's question, since it is the most common shape you will get. Answer support questions from internal documentation.
The core insight is that the model is not the interesting part of this design. Retrieval is. The model is a formatting and reasoning layer over context you supply. Almost everything that determines whether the system is good happens before the model is called, which is exactly backwards from where most candidates spend their time.
So the design has two halves that deserve unequal attention. There is an offline path where documents are ingested, split into chunks, embedded, and indexed, with the messy details being chunk boundaries, what to do about documents that update, and how you handle permissions so the index does not become a way to read things you should not. And there is an online path where a question arrives, gets embedded, retrieves candidate chunks, and those chunks plus the question go to the model.
Where strong candidates go deep is retrieval quality, because that is where the system actually succeeds or fails. What happens when retrieval returns nothing relevant? The right answer is that the system says it does not know, and you should say so, because the tempting alternative is letting the model answer from general knowledge, which is precisely how a grounded assistant starts inventing company policy. What if the answer spans several documents? What if two documents contradict each other because one is out of date? What if the user asks something the documentation genuinely does not cover?
Then handle permissions properly. Retrieval must filter by the asking user's access before anything reaches the model, not after. I have seen candidates put the authorization check on the response, which is a security hole.
The Evaluation Question, Which Most Candidates Miss
If you take one thing from this article, take this.
At some point a good interviewer will ask how you know the system is working. Most candidates answer with uptime, latency, and error rates. Those are necessary and they are not what is being asked. The question is how you know the answers are good, and how you would find out that they had stopped being good.
A credible answer has a few parts. You need a fixed set of representative questions with known good answers that you can run against any change, which is your regression suite and which you should treat with the same seriousness you treat unit tests. You need a way to score output that does not require a human every time, whether that is exact matching where possible, checking that cited sources actually support the claim, or using a model to grade against a rubric with all the caveats that carries. You need production signal, because offline evaluation always drifts from reality, so capture thumbs up and down, escalations to a human agent, repeated rephrasing of the same question. And you need to treat prompt and model changes as deployments, with the same rollout discipline you would apply to a schema migration, because a prompt edit is a production change with no compiler to catch it.
Very few candidates get to the fourth point. If you do, you will be remembered.
Cost and Latency, Said Out Loud
Do the arithmetic in the room. If you estimate a hundred thousand requests a day, and each one sends a few thousand tokens of retrieved context, that is a number you can reason about and it will drive your design.
The levers worth naming: cache aggressively at the semantic level rather than on exact string matches, since similar questions are extremely common in support workloads. Route by difficulty, using a smaller model for the majority of straightforward cases and reserving the expensive one for what needs it. Keep retrieved context tight, because more context is not free and past a point it is not even better. Stream the response so that time to first token, not total generation time, is what the user experiences.
Each of those is a real trade-off with a real downside, which makes them good material for the trade-off conversation that these interviews are ultimately about. Semantic caching risks serving a stale or subtly mismatched answer. Model routing means your quality now varies by request in ways that are hard to reason about.
If the Question Involves Agents
Some loops now push further and ask you to design something where a model takes actions rather than only producing text. Processing a queue of tickets, making tool calls, updating records.
The framing that helps most is to treat the agent as an unreliable distributed worker that you cannot fully instruct in advance. Everything you know about unreliable workers applies. You need bounded retries, because an agent that loops is an agent generating cost with no upper limit. You need idempotency, because it may repeat an action. You need a blast radius, meaning explicit limits on what it can touch and a hard boundary around anything destructive or irreversible. You need to persist the reasoning trace, because when something goes wrong the trace is the only artifact that tells you why. And you need a human approval step for high-consequence actions, positioned deliberately rather than added later.
If you say "I would not let it delete anything without a human confirming," you have said something more valuable than any architectural diagram you could draw.
What the Interviewer Is Actually Listening For
Not machine learning depth. I want to be emphatic about this, because it is the assumption that makes people avoid preparing for these questions at all.
They are listening for whether you treat the model as a component with properties, or as magic. Whether you noticed that non-determinism and per-request cost break assumptions you did not know you were making. Whether you thought about being wrong, not just about being slow. Whether you have any answer at all for how quality gets measured. And whether you are honest about the limits of your experience instead of reaching for vocabulary.
That last one deserves emphasis. If you have not built one of these, say so early and then reason from what you do know. Rate limiting, cost control, caching, graceful degradation, and measuring whether a system works are all things you have done elsewhere. Transferring that judgment openly reads as senior. Bluffing reads as junior, and it contaminates everything else you said in the interview.
How to Prepare Without Becoming a Specialist
Build one small thing end to end. A retrieval system over a few hundred documents, running locally, is a weekend. You will hit chunking, retrieval quality, and cost within the first hour, and having hit them yourself is worth more in an interview than anything you could read.
Then practice three or four problems in this shape out loud, using the same structure you use for classic problems. A grounded assistant over internal documents. A content moderation pipeline. A system that summarizes long documents at scale. An agent that triages incoming tickets.
Force yourself to raise cost, non-determinism, and evaluation in every one, until it becomes automatic. Those three are the difference between a competent answer and a memorable one.
The underlying structure has not changed, so build on System Design Interviews: A Practical Guide, and if you want feedback on these under real interview conditions, that is what Staff Engineer Interview Coaching is for.
About Me
Nimesh Patel is an engineering leader and career coach with over 20 years of experience building cloud-native enterprise and consumer software systems in Big Tech (including Google) and high-growth AI startups. He has led globally distributed engineering organizations of 60+ engineers and leaders, conducted 650+ interviews across engineering, management, and executive roles, made 50+ hires, and coached and promoted 30+ engineers and leaders. He provides interview and career coaching through ScaleYourCareer. Follow him on LinkedIn.
Ready to accelerate your interview preparation or grow into your next role? Explore the coaching programs to find the right fit.