AI engineering interview questions: a step-by-step guide

AI engineering interview questions: a step-by-step guide

When you're interviewing for a role building LLM features, you'll face questions that dig into the real engineering decisions your team makes every day. These aren't theoretical puzzles—they're the problems you solve in production, from retrieval accuracy to cost management. This guide walks you through the most common AI engineering interview questions and how to approach them with the confidence of someone who's shipped these systems.

How does RAG work?

Retrieval-Augmented Generation (RAG) is the bread and butter of practical LLM systems. The core idea is straightforward: don't let the model hallucinate answers. Instead, retrieve relevant context from your document store and pass it as grounding material to the LLM.

Here's the flow:

1. User asks a question.
2. Convert the question into an embedding using a model like OpenAI's text-embedding-3-small or Cohere's embed-v3.
3. Search your vector database (Pinecone, Weaviate, Milvus, or even PostgreSQL with pgvector) for the nearest neighbors.
4. Retrieve the top K chunks (typically 3–10).
5. Pass those chunks as context into your LLM prompt.
6. The LLM generates an answer grounded in real data.

In an interview, interviewers want to see that you understand the trade-offs. For example:

• Chunk size: Smaller chunks (512 tokens) are precise but may miss context. Larger chunks (2048 tokens) preserve narrative but introduce noise. You pick based on your retrieval quality metrics.
• Embedding model choice: Smaller, faster models like text-embedding-3-small work well for most e-commerce and SaaS use cases. You might use text-embedding-3-large only if precision is critical and latency isn't.
• Retrieval strategy: Simple cosine similarity works, but hybrid search (combining dense vector search with BM25 keyword matching) often outperforms pure embedding-based retrieval.

A practical answer in an interview might sound like this: "We chunk our documents at 512 tokens with 10% overlap, embed them with text-embedding-3-small, and store them in Pinecone. For each query, we retrieve the top 5 chunks by cosine similarity, then pass them into a GPT-4 prompt with a system instruction to cite sources. We've found this gives us 92% accuracy on our test set while keeping latency under 800ms."

How do you evaluate an LLM feature?

This is where many engineers stumble. Evaluating an LLM system is fundamentally different from evaluating a classifier. You can't just compute F1 scores and call it done.

Test sets. Start with a curated test set of 100–500 examples that represent real user behavior. For a customer support bot, this might be actual support tickets. For a RAG system, it's real questions paired with ground truth answers.

Rubrics. Define what "good" means. Create a rubric with clear criteria:

• Is the answer factually correct?
• Does it cite sources?
• Is it relevant to the question?
• Does it handle edge cases (refusals, out-of-scope questions)?

Score each answer on a scale (e.g., 1–5), and track consistency across raters.

Model-graded evaluation. Have a stronger LLM (like GPT-4) grade responses from your production model (like GPT-3.5) against your rubric. This scales evaluation beyond manual review and lets you test hundreds of examples cheaply. You might use a prompt like:

evaluation_prompt = """
You are an evaluator. Grade this response on:
1. Factual accuracy (1-5)
2. Relevance (1-5)
3. Citation quality (1-5)

Context: {context}
Question: {question}
Response: {response}

Respond with JSON: {"accuracy": X, "relevance": Y, "citations": Z}
"""

Human review. Always reserve a subset (20–30%) for human review. Your rubric might be imperfect, or edge cases might reveal flaws in both your model and your grading logic. Have engineers or domain experts spot-check results weekly.

In an interview setting, explain that you'd start with 200 test examples, create a rubric, grade them manually, then use GPT-4 to grade new batches while doing spot-checks. Track scores over time—a 5% drop after a deployment is a signal to investigate.

How do you reduce hallucinations?

Hallucinations are the enemy of production LLM systems. When your customer support bot invents a return policy, that's a business problem. Four strategies reduce hallucinations:

Grounding through retrieval. This is RAG. The LLM can only generate answers supported by your retrieved context. If the answer isn't in your knowledge base, the model struggles to hallucinate.

Citations. Force the model to cite sources. Modify your prompt to require citations for every factual claim. For example:

prompt = """
You are a helpful assistant. Answer the question based on the context below.
For every factual claim, include a citation like [Source 1].

Context:
{context}

Question: {question}

Answer: (cite every fact)
"""

Then validate citations programmatically by checking that each cited chunk actually contains the claimed information.

Constrained output. Use structured output formats. Instead of free-form text, request JSON or a specific template:

structured_prompt = """
Respond with this JSON structure:
{
  "answer": "your answer here",
  "sources": ["chunk_id_1", "chunk_id_2"],
  "confidence": 0.95,
  "is_grounded": true
}
"""

Many model providers (OpenAI, Anthropic) support JSON mode, which ensures responses match your schema.

Refusal paths. Teach the model to refuse gracefully. If retrieved context doesn't support an answer, the model should say "I don't have that information" rather than guess. Include examples in your prompt:

refusal_example = """
Question: What's your CEO's personal phone number?
Answer: I don't have access to personal contact information.

Question: Can you process refunds instantly?
Answer: I don't have refund policy details in my knowledge base. 
Please contact support@company.com.
"""

In interviews, demonstrate that you understand the hierarchy: first prevent hallucinations through grounding, then add citations to catch them, then use structured output to enforce consistency, and finally teach refusals as a safety net.

How do you control cost?

LLM inference costs add up fast, especially at scale. A ride-hailing platform processing 100K requests daily can easily hit five figures monthly without optimization.

Prompt caching. Both OpenAI and Anthropic offer prompt caching, which deduplicates the processing cost for repeated context. If you're answering 1000 questions about the same 50-page document, caching reduces cost by 90%:

import anthropic

client = anthropic.Anthropic()

system_context = """
You are a customer support bot. Use only the knowledge base below.

[50-page knowledge base here...]
"""

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": system_context,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": "How do I reset my password?"}]
)

Smaller models. Not every task needs GPT-4. Use smaller, cheaper models (GPT-3.5, Llama 2) for classification, routing, and simple retrieval. Reserve large models for complex reasoning. We've found that a pipeline using GPT-3.5 for routing plus Claude 3 Haiku for grading cuts costs by 40% with minimal quality loss.

Trimmed context. Every token you send costs money. If your retrieved chunks total 3000 tokens but only the first 800 are relevant, trim the rest. Use extractive summarization or a cheap model to pre-filter context:

def trim_context(chunks, query, max_tokens=1000):
    combined = " ".join(chunks)
    if len(combined.split()) > max_tokens:
        # Use GPT-3.5 to extract relevant excerpts
        extraction_prompt = f"""
Extract only the sentences relevant to: {query}
Keep it under {max_tokens} words.
Text: {combined}
"""
        # Call cheaper model for extraction
    return trimmed_context

In an interview, walk through your cost optimization strategy: "We route simple queries to GPT-3.5, complex ones to Claude. We cache the knowledge base across requests. We retrieve 5 chunks but trim to 1000 tokens before calling the LLM. This keeps our per-request cost under $0.01 while maintaining 94% accuracy."

Common mistakes

• Assuming RAG is a silver bullet. RAG helps, but it doesn't eliminate hallucinations entirely. You still need citations, refusal paths, and human review.
• Over-optimizing for one metric. If you optimize only for latency, you'll sacrifice accuracy. Define multiple metrics (accuracy, latency, cost, hallucination rate) and track trade-offs.
• Forgetting about drift. Your test set from three months ago might not reflect today's user behavior. Continuously log production queries, evaluate against them, and retune.
• Using manual evaluation at scale. Human review is expensive and inconsistent. Build model-graded evaluation early, use it to detect regressions, then spot-check manually.
• Picking the wrong chunk size. Default to 512 tokens with 10% overlap. Test smaller and larger sizes on your test set. What works for customer support might not work for technical docs.

Key takeaways

• RAG works by embedding queries, retrieving relevant chunks, and passing them as context. Interviewers want to hear about chunk size, embedding model choice, and hybrid retrieval trade-offs.
• Evaluate LLM features with test sets, rubrics, model-graded evaluation, and human spot-checks. A 200-example test set and GPT-4 grading is a practical starting point.
• Reduce hallucinations through grounding (RAG), citations, structured output, and refusal paths. Implement all four, not just one.
• Control cost by caching prompts, using smaller models for routing and classification, and trimming context to only relevant tokens. Even small optimizations compound across thousands of requests.
• Have a concrete story ready: "We built a RAG system that retrieves 5 chunks, trims to 1000 tokens, uses GPT-3.5 for classification and Claude for complex reasoning, caches the knowledge base, and achieves 92% accuracy at $0.008 per request."