Fine-tuning vs RAG vs prompting: lessons from production

After shipping AI features across e-commerce platforms, recommendation engines, and customer support systems, we've learned one thing: the choice between fine-tuning vs RAG vs prompting isn't about which is technically superior—it's about which solves your problem at the lowest cost without sacrificing quality. We've watched teams spend months fine-tuning models when a well-engineered prompt would have worked, and we've seen others try to RAG their way around poor prompt design. This article distills what actually works in production.

Why this decision matters

When you're deciding how to make an LLM do what you need, you're really choosing between three trade-offs: speed of implementation, cost per inference, and the quality of outputs for your specific use case. Each approach has a different failure mode in production.

Prompting is fast but fragile. RAG is flexible but operationally heavy. Fine-tuning is powerful but expensive and slow to iterate. You need to know which failure mode you can tolerate, and more importantly, which one will keep your feature running cheaply.

Prompting first: fast and flexible

Start here. Seriously. A well-crafted prompt with good examples (few-shot learning) will solve most of your problems. We've seen teams ship customer service classification, FAQ matching, and even basic content moderation using nothing but prompt engineering and GPT-4.

The advantage is obvious: zero training time, zero infrastructure. You change a prompt, deploy in seconds. You iterate based on real user feedback without retraining anything. For a ride-hailing platform we worked with, we built driver feedback categorization entirely through prompting—the model learned to bucket complaints into safety, payment, vehicle, and communication issues with just four labeled examples in the prompt.

Here's what that looked like:

from openai import OpenAI

client = OpenAI()

def categorize_feedback(feedback_text: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {
                "role": "system",
                "content": """You are a feedback categorizer for a ride-hailing platform.
Categorize driver feedback into exactly one of: SAFETY, PAYMENT, VEHICLE, COMMUNICATION, OTHER.
Return only the category name, no explanation."""
            },
            {
                "role": "user",
                "content": f"Categorize this feedback: {feedback_text}"
            }
        ],
        temperature=0.1,
        max_tokens=10
    )
    return {
        "category": response.choices[0].message.content.strip(),
        "model": "gpt-4",
        "cost_usd": 0.00001  # rough estimate for this call
    }

# Usage
result = categorize_feedback("Driver charged me twice for the same ride")
print(result)  # {'category': 'PAYMENT', 'model': 'gpt-4', 'cost_usd': 0.00001}

The weakness of pure prompting shows up when your domain gets specific or your quality bar gets high. A prompt works great for classification, but if you need the model to respond like your brand—with your tone, your phrasing, your specific product knowledge—prompts become brittle. You end up writing longer and longer system prompts trying to encode style, and it still drifts.

Start with prompting. Measure accuracy against a holdout test set. Only move to something more complex if you're hitting a wall.

RAG for knowledge that changes

Retrieval-augmented generation is your answer when you have a corpus of documents, policies, or data that changes frequently and the model needs to cite or reference specific information. We used this heavily for a travel SaaS platform that needed to answer questions about pricing policies, cancellation rules, and partner benefits that updated weekly.

RAG works by retrieving relevant documents from your knowledge base and injecting them into the prompt, so the model answers with current information instead of relying on its training data. The key insight: you're not training the model at all. You're changing what it has access to.

Here's a practical example:

from openai import OpenAI
import json

client = OpenAI()

# Simulated vector store. In production, use Pinecone, Weaviate, or PostgreSQL pgvector
KNOWLEDGE_BASE = {
    "policy_001": "Customers can cancel bookings up to 24 hours before departure for full refund.",
    "policy_002": "Group bookings (5+ people) receive 15% discount on total price.",
    "policy_003": "Premium members get free cancellation up to 12 hours before departure.",
}

def retrieve_relevant_docs(query: str, top_k: int = 2) -> list[str]:
    # In production, use semantic search (embeddings). 
    # For demo, naive keyword matching.
    query_lower = query.lower()
    scores = []
    for doc_id, doc_text in KNOWLEDGE_BASE.items():
        score = sum(1 for word in query_lower.split() if word in doc_text.lower())
        scores.append((doc_id, doc_text, score))
    
    scores.sort(key=lambda x: x[2], reverse=True)
    return [text for _, text, _ in scores[:top_k]]

def answer_with_rag(user_question: str) -> dict:
    docs = retrieve_relevant_docs(user_question)
    context = "\n".join([f"- {doc}" for doc in docs])
    
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {
                "role": "system",
                "content": f"""You are a helpful support agent for a travel platform.
Answer questions using ONLY the following policies. Do not make up information.

Policies:
{context}"""
            },
            {
                "role": "user",
                "content": user_question
            }
        ],
        temperature=0.7,
        max_tokens=150
    )
    
    return {
        "answer": response.choices[0].message.content,
        "retrieved_docs": docs,
        "approach": "RAG"
    }

# Usage
result = answer_with_rag("Can I cancel my booking 20 hours before departure?")
print(result)

RAG's strength: your knowledge base can be updated without retraining or redeploying the model. Add a new policy, update a price, fix an error—it's live immediately. This is huge for platforms with dynamic content.

RAG's cost is operational: you need a vector database (or semantic search infrastructure), you need to embed documents, and retrieval quality depends heavily on your embedding model and chunking strategy. A bad retrieval step tanks your answer quality downstream. We've seen production RAG systems fail silently because they retrieved the wrong documents, giving confidently wrong answers to users.

Use RAG when your knowledge base updates frequently and you need source attribution. Skip it if your data is stable or small enough to fit in a prompt.

Fine-tuning for style and narrow tasks

Fine-tuning is expensive and slow, but it's the right choice when you need consistent behavior on a narrow, well-defined task where prompt engineering has hit its limits. We fine-tuned a model for a CRM platform that needed to generate customer outreach emails matching the company's exact tone and brand voice. Prompting alone couldn't capture the nuance—every email felt slightly off-brand until we showed the model hundreds of examples of what "on-brand" looked like.

Fine-tuning updates the model's weights based on your examples. After training, the model doesn't need those examples in the prompt anymore; the behavior is baked in. This also reduces inference cost in some cases because you can use a smaller, fine-tuned model instead of a large one.

Here's what the workflow looks like:

import jsonl
import json
from openai import OpenAI

client = OpenAI()

# Step 1: Prepare training data in JSONL format
training_examples = [
    {
        "messages": [
            {"role": "user", "content": "Customer bought three items but returned one due to damage."},
            {"role": "assistant", "content": "Hi there, thanks for reaching out about your return. We're sorry the item arrived damaged—that's not the experience we want for you. We'll process a full refund right away and even throw in a 20% discount on your next purchase as our apology."}
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Customer hasn't opened their order email yet."},
            {"role": "assistant", "content": "Hey! Just wanted to check in—your order shipped yesterday and you should see it soon. Let me know if you have any questions about tracking or delivery."}
        ]
    },
    # ... 50-100 more examples of your brand voice
]

# Save to JSONL file
with open("training_data.jsonl", "w") as f:
    for example in training_examples:
        f.write(json.dumps(example) + "\n")

# Step 2: Upload training data
with open("training_data.jsonl", "rb") as f:
    response = client.files.create(file=f, purpose="fine-tune")
    training_file_id = response.id

# Step 3: Start fine-tuning job
finetune_job = client.fine_tuning.jobs.create(
    training_file=training_file_id,
    model="gpt-3.5-turbo",  # cheaper base model
    hyperparameters={"n_epochs": 3}
)

print(f"Fine-tune job started: {finetune_job.id}")
# Wait for job to complete (typically 30 mins to several hours)

# Step 4: Use the fine-tuned model
def generate_email_with_finetuned_model(customer_scenario: str) -> str:
    response = client.chat.completions.create(
        model=finetune_job.fine_tuned_model,  # Use your fine-tuned model
        messages=[
            {"role": "user", "content": customer_scenario}
        ],
        temperature=0.7,
        max_tokens=200
    )
    return response.choices[0].message.content

# Usage
email = generate_email_with_finetuned_model("Customer needs help with password reset")
print(email)

Fine-tuning costs: training time (30 mins to hours), API costs for training (usually $0.08–$0.30 per 1K tokens), and higher inference costs for larger models. But once trained, the model is consistent. No more prompt engineering arms race. No more "the model changed its tone." This matters in customer-facing features.

The trade-off: you're locked into that model version until you retrain. If you need to change behavior, you retrain. This is fine for stable tasks (customer email tone, product description style) but terrible for dynamic content (answering FAQ questions from an updated knowledge base).

Fine-tune when: style matters, you have 100+ labeled examples, and the task is narrow and stable. Don't fine-tune for general knowledge or when your requirements change weekly.

Combine them when needed

Production systems rarely use just one approach. We've built systems that do all three:

• Prompting for quick classification and routing
• RAG for retrieving up-to-date policies and documents
• Fine-tuning for generating responses in a specific brand voice

A concrete example: an e-commerce support system that receives customer messages. The flow was:

1. Prompt-based classification: route the message to the right category (order status, returns, technical issue, etc.)
2. RAG: retrieve relevant FAQs and policies for that category
3. Fine-tuned model: generate a response in the company's customer service voice using the retrieved information

Each layer did what it was best at. The cost was moderate because we used a smaller fine-tuned model, and the quality was high because the retrieved information was current and the tone was consistent.

Here's the orchestration pattern:

def process_customer_message(message: str) -> dict:
    # Step 1: Classify with prompt
    classification_response = client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[{
            "role": "system",
            "content": "Classify this message into: order_status, returns, technical, billing, or other.",
        }, {
            "role": "user",
            "content": message
        }],
        temperature=0.1,
        max_tokens=10
    )
    category = classification_response.choices[0].message.content.strip()
    
    # Step 2: Retrieve relevant docs for this category with RAG
    docs = retrieve_relevant_docs(message, category=category, top_k=3)
    
    # Step 3: Generate response with fine-tuned model
    response = client.chat.completions.create(
        model="your-finetuned-model-id",
        messages=[{
            "role": "system",
            "content": f"""You are a helpful support agent. Use the following information to answer the customer:
{chr(10).join(docs)}"""
        }, {
            "role": "user",
            "content": message
        }],
        temperature=0.7,
        max_tokens=300
    )
    
    return {
        "category": category,
        "response": response.choices[0].message.content,
        "pipeline": "prompt → RAG → fine-tuned"
    }

Common mistakes

Over-fine-tuning: Teams fine-tune for problems that prompting or RAG could solve. We've seen this burn budget and slow down iteration. Default to prompting, move to RAG if knowledge changes, fine-tune only when nothing else works.

Under-testing RAG quality: RAG breaks silently. The model will confidently give wrong answers if retrieval fails. Always measure retrieval accuracy separately from generation quality. We've caught bugs where the system retrieved documents about the wrong product category 15% of the time—users got correct-sounding but totally wrong advice.

Ignoring cost in production: A prompt with a large model at scale gets expensive fast. We worked with a team that didn't optimize prompt length or model choice early, and their API bill tripled when they went from 100 requests per day to 100,000. Fine-tuning a smaller model would have been cheaper after the first few days. Measure cost per inference from day one.

Treating prompting as "done" once: Prompts degrade as the world changes. What worked in January might not work in March. Log model outputs, measure accuracy, and iterate on prompts monthly. This is operational maintenance, not a one-time engineering task.

Key takeaways

• Start with prompting. It's fast, cheap, and you iterate in seconds.
• Move to RAG when your knowledge base changes and you need up-to-date answers. Invest in retrieval quality; it's where RAG fails.
• Fine-tune when you need consistent style or narrowly-scoped behavior on stable tasks and you have 100+ labeled examples.
• Combine all three in production systems. Each solves a different problem.
• Optimize for cost. Prompting is usually cheapest, but measure actual spend per inference in your environment.
• Monitor production quality continuously. Classification accuracy drifts, retrieval fails, fine-tuned models become outdated. Build observability into your pipeline.
• Default to iteration velocity over perfect quality. Shipped and evolving beats perfect in the lab.