Building an AI chat feature: what developers should know

Building an AI chat feature: what developers should know

Shipping an AI chat feature feels straightforward until you hit production. You've got the LLM API working in a notebook, responses flow naturally in your prototype, but then real users start conversations that span hours, ask about private data, and the system needs to remember context while staying within cost and latency budgets. Building an AI chat feature: what developers should know starts with understanding that the gap between demo and production isn't about the model—it's about the system around it.

In this guide, we'll walk through four critical areas that separate toy chatbots from systems people actually use: conversation state management, context limits, grounding answers in your data, and guardrails that keep the feature reliable and safe.

Conversation state: storing messages per conversation in your database

Every conversation needs a home. Whether you're building a support chatbot, customer onboarding flow, or internal knowledge assistant, you must persist every message exchange—both user input and model output—in a structured way.

A straightforward schema looks like this:

CREATE TABLE conversations (
  id UUID PRIMARY KEY,
  user_id UUID NOT NULL,
  created_at TIMESTAMP DEFAULT NOW(),
  updated_at TIMESTAMP DEFAULT NOW(),
  metadata JSONB  -- for title, tags, or feature flags
);

CREATE TABLE messages (
  id UUID PRIMARY KEY,
  conversation_id UUID NOT NULL REFERENCES conversations(id),
  role TEXT NOT NULL,  -- 'user' or 'assistant'
  content TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW(),
  tokens_used INT,  -- for cost tracking
  model_version TEXT  -- for debugging model changes
);

CREATE INDEX ON messages(conversation_id, created_at);

The role field matters because you're not just storing the assistant's responses—you need the full thread. When you fetch a conversation to send to the LLM, you reconstruct the message history in order. The tokens_used column is subtle but important: log it during inference so you can track spend per user, per conversation, and per feature.

In a typical flow, after you get a response from your LLM provider (OpenAI, Anthropic, etc.), you save both the user message and the assistant response in a single database transaction. This ensures you never lose a turn, even if your backend crashes mid-response.

Always insert the user message before calling the LLM API. If the API fails, you've still recorded what the user asked. If you insert after, you risk losing the user's intent if there's a network hiccup.

You'll also want to index conversations by user and creation time so you can show users their conversation history quickly. In production, pagination is your friend—don't fetch all messages at once for a 6-month-old conversation; load them in batches.

Context limits: summarizing and trimming old messages to fit the context window

LLMs have finite context windows. GPT-4 Turbo accepts 128K tokens; Claude 3 Opus accepts 200K. But token limits aren't just about fitting messages—they're about cost and latency. Every token you send costs money, and every token increases response time. A multi-month conversation with hundreds of messages will quickly blow your budget and slow your inference.

You have three strategies: truncate, summarize, or hybrid.

Truncate: Keep only the most recent N messages. Simple, fast, and stateless. For a customer support chatbot over a short session (1–2 hours), this often works fine. You lose historical context, but the user can scroll and re-read if needed.

Summarize: Every 10–20 messages, call the LLM again with a prompt like "Summarize this conversation in 2–3 sentences" and store that summary. Replace the old messages with a single summary record. This preserves context but costs extra API calls and introduces latency delays.

Hybrid: Keep the last 10 messages verbatim, summarize everything older, and concatenate both when you call the LLM. This balances fresh context with historical awareness.

Here's a practical example:

async def get_conversation_context(conversation_id: str, max_recent: int = 10):
    """
    Fetch conversation context for LLM inference.
    Returns recent messages in full, plus a summary of older messages.
    """
    messages = await db.query("""
        SELECT role, content, created_at FROM messages
        WHERE conversation_id = %s
        ORDER BY created_at DESC
        LIMIT %s
    """, (conversation_id, max_recent))
    
    recent_messages = list(reversed(messages))  # chronological order
    
    # Fetch summary if conversation is long
    summary = await db.query_one("""
        SELECT content FROM conversation_summaries
        WHERE conversation_id = %s
        ORDER BY created_at DESC LIMIT 1
    """, (conversation_id,))
    
    context = []
    if summary:
        context.append({
            "role": "system",
            "content": f"Earlier in this conversation: {summary['content']}"
        })
    
    context.extend([
        {"role": m['role'], "content": m['content']}
        for m in recent_messages
    ])
    
    return context

Decide your truncation strategy based on your use case. Support chats: truncate aggressively. Research or onboarding conversations where context matters deeply: summarize. Monitor token usage in production—if users hit context limits often, your strategy needs tuning.

Ground answers: using RAG so answers come from your data

An ungrounded chatbot hallucinates. It confidently invents pricing, product features, or policy details that don't exist in your system. Retrieval-Augmented Generation (RAG) solves this: before you call the LLM, you fetch relevant documents or data from your own database, then pass those as context in the prompt.

The pattern is:

1. User asks a question.
2. Convert the question into a vector embedding.
3. Search your vector database (Pinecone, Weaviate, pgvector in PostgreSQL) for semantically similar documents.
4. Retrieve the top 3–5 results.
5. Construct a prompt that includes these documents plus the user's question.
6. Call the LLM, which now grounds its answer in your data.

A minimal example using PostgreSQL with pgvector:

from openai import AsyncOpenAI
import asyncpg

async def answer_with_rag(user_question: str, conversation_id: str, pool: asyncpg.Pool):
    """
    Answer a user question using RAG.
    """
    client = AsyncOpenAI()
    
    # Step 1: Embed the user question
    embedding_response = await client.embeddings.create(
        model="text-embedding-3-small",
        input=user_question
    )
    question_embedding = embedding_response.data[0].embedding
    
    # Step 2: Retrieve relevant documents from your vector store
    async with pool.acquire() as conn:
        documents = await conn.fetch("""
            SELECT content, source FROM documents
            ORDER BY embedding <-> %s
            LIMIT 5
        """, question_embedding)
    
    # Step 3: Build context from retrieved documents
    context = "\n".join([
        f"[{doc['source']}]\n{doc['content']}"
        for doc in documents
    ])
    
    # Step 4: Fetch conversation history
    async with pool.acquire() as conn:
        messages_raw = await conn.fetch("""
            SELECT role, content FROM messages
            WHERE conversation_id = %s
            ORDER BY created_at ASC
            LIMIT 20
        """, conversation_id)
    
    # Step 5: Call LLM with grounded context
    system_prompt = f"""You are a helpful assistant. 
Answer based on the following information:
{context}

If the information does not contain the answer, say so."""
    
    messages = [
        {"role": "system", "content": system_prompt}
    ]
    messages.extend([
        {"role": m['role'], "content": m['content']}
        for m in messages_raw
    ])
    messages.append({"role": "user", "content": user_question})
    
    response = await client.chat.completions.create(
        model="gpt-4",
        messages=messages,
        temperature=0.7
    )
    
    return response.choices[0].message.content

The key insight: RAG is not magic. The quality of your retrieved documents directly impacts answer quality. If your vector database has stale, incomplete, or irrelevant documents, the LLM's grounded answer will still be poor. Invest in keeping your document corpus clean and up-to-date. Also log which documents were retrieved for each answer—you'll need this for debugging when a user says "that's not what your documentation says."

Guardrails: moderating inputs, restricting topics, and handling failures gracefully

Production chatbots encounter edge cases and adversarial inputs constantly. A user asks the bot to help them break the law. Another user pastes 50MB of raw logs. A third asks for a refund for an order that doesn't exist. Without guardrails, your chatbot becomes a liability and a cost sink.

Input validation: Check message length, rate-limit per user, and optionally run content moderation (OpenAI's moderation API is a low-friction option).

async def validate_and_moderate(user_id: str, message: str, redis_client):
    """
    Validate input and check for policy violations.
    """
    # Rate limiting: max 30 messages per hour per user
    key = f"chat:rate:{user_id}"
    count = await redis_client.incr(key)
    if count == 1:
        await redis_client.expire(key, 3600)
    if count > 30:
        raise RateLimitExceeded()
    
    # Length check
    if len(message) > 5000:
        raise MessageTooLong()
    
    # Content moderation
    moderation = await openai_client.moderations.create(input=message)
    if moderation.results[0].flagged:
        raise ContentPolicyViolation()
    
    return True

Topic restriction: Some questions are out of scope. A support chatbot shouldn't offer medical or legal advice. Use a classification step: before answering, ask the LLM "Is this question related to [our product]? Answer yes or no only."

Graceful failure: If the LLM API times out, your vector search returns nothing, or you hit rate limits, respond honestly to the user instead of hallucinating:

try:
    answer = await answer_with_rag(question, conversation_id, pool)
except asyncio.TimeoutError:
    answer = "I'm taking longer than usual. Could you please try again in a moment?"
except NoDocumentsFound:
    answer = "I don't have information about that in my knowledge base. Please contact support at support@example.com."
except OpenAIRateLimitError:
    answer = "I'm processing a lot of requests right now. Please try again shortly."

Also add observability: log every inference with the user input, retrieved documents, LLM response, latency, and token count. Use these logs to catch patterns—if 20% of users ask about refunds and the chatbot consistently gives wrong answers, you've found a training priority.

Common mistakes

• Storing conversation state in memory only. Your app restarts, the entire conversation history vanishes. Store in the database from day one, even if you cache in Redis for performance.
• Sending full conversation history to the LLM every time. Token costs explode exponentially. Truncate or summarize from the start. Monitor tokens per conversation.
• Assuming RAG solves hallucination completely. It reduces hallucination if your documents are accurate and complete. Stale or incomplete data just grounds the hallucination in your business context, which is worse.
• Shipping without input validation. Malicious users will find creative ways to exploit unfiltered input. Validate length, rate-limit, and run moderation.
• Treating LLM failures as exceptions, not normal. APIs fail, rate limits hit, models drift. Handle failures in your feature, not with a 500 error.

Key takeaways

• Persist every message in your database with a clear schema; index by conversation and user for fast retrieval.
• Implement context truncation or summarization early; monitor token usage and cost per conversation.
• Use RAG to ground answers in your own data; log retrieved documents for debugging.
• Add input validation, rate limiting, and content moderation before launch; handle LLM failures with user-friendly fallbacks.
• Build observability into your chat feature from the start; logs of inference, latency, and tokens are invaluable for tuning in production.