100 days of practical AI and ML: the recap — a deep dive

We've spent the last hundred days building, testing, and refining AI and machine learning systems in production. We've learned what actually works when the code goes live, when real users depend on it, and when cost matters as much as accuracy. This 100 days of practical AI and ML: the recap — a deep dive captures the lessons that separate projects that ship from projects that stall.

Data first — Clean, relevant data beats clever models

The hardest truth we've learned is simple: no architecture, no fine-tuning, no ensemble method saves you from bad data. We've seen teams spend weeks optimizing model hyperparameters only to gain 0.3% accuracy. We've also seen teams spend two days cleaning their training set and jump 8% in production performance.

When we started a fraud detection system for a payments platform, our initial dataset had duplicates, mislabeled transactions, and missing features that should have been signals. The model looked good on validation. It failed catastrophically in production because the patterns it learned didn't reflect real-world fraud. We had to rebuild the data pipeline from scratch.

The practical lesson: invest early in data exploration and cleaning. Write scripts that surface anomalies. Set up automated checks that run before training. We use a simple pattern:

import pandas as pd
import numpy as np

def audit_dataset(df, label_col):
    report = {
        "total_rows": len(df),
        "duplicates": df.duplicated().sum(),
        "null_counts": df.isnull().sum().to_dict(),
        "label_distribution": df[label_col].value_counts().to_dict(),
        "class_balance": (df[label_col].value_counts() / len(df)).to_dict(),
    }
    
    # Flag severe imbalance
    ratios = list(report["class_balance"].values())
    if max(ratios) / min(ratios) > 100:
        report["warning"] = "Severe class imbalance detected"
    
    return report

Also, validate that your features are actually relevant. We've watched teams include features that correlate with the target but for the wrong reasons—proxies for data bias rather than real signal. Use domain knowledge to question every column.

Measure honestly — Proper splits, the right metrics and no leakage

A model that reports 96% accuracy and fails at 60% in production isn't a production bug. It's a measurement bug. We've seen this pattern often enough to make it a rule: if your test set doesn't match your production data distribution, your metrics are fiction.

Data leakage is the most common culprit. We worked with a churn prediction model that looked incredible on the test set. The feature engineering pipeline included a column derived from customer behavior in the month after the prediction date—information the model wouldn't have at prediction time. The model had learned to cheat. Once we removed it, accuracy dropped from 91% to 74%, but the model actually worked in production.

Proper train-test splits matter more than you might think. Time-based splits beat random splits for any problem where sequence matters. If you're predicting next month's revenue, train on months 1-10, validate on month 11, test on month 12. Don't shuffle them randomly.

def time_based_split(df, date_col, train_ratio=0.6, val_ratio=0.2):
    df_sorted = df.sort_values(date_col)
    n = len(df_sorted)
    
    train_end = int(n * train_ratio)
    val_end = int(n * (train_ratio + val_ratio))
    
    train = df_sorted.iloc[:train_end]
    val = df_sorted.iloc[train_end:val_end]
    test = df_sorted.iloc[val_end:]
    
    return train, val, test

Pick the right metric for your business problem, not the metric that looks best. Accuracy is often useless. For fraud detection, recall might matter more than precision (catch fraud, accept false positives). For spam filtering, precision dominates. For e-commerce recommendations, lift and engagement beat raw accuracy. We define success metrics before building the model, and we measure them on the test set only—never on data used for training or tuning.

LLMs are tools — RAG, structured output and evaluation turn them into features

LLMs aren't replacement machine learning models. They're interfaces to unstructured knowledge and reasoning. The teams that win use them tactically—generating features, augmenting training data, extracting structure—rather than replacing the entire pipeline.

Retrieval-Augmented Generation (RAG) is the pattern we use most often. Instead of fine-tuning or hoping the model remembers facts, we give it relevant context at inference time. We built a support assistant that answers product questions. Without RAG, the LLM hallucinated feature details. With RAG, it retrieves the actual product documentation and answers accurately.

The real production problem with LLMs: they're unreliable. They hallucinate. They're expensive to run. They're slow. So we use them where their weaknesses don't matter and their strengths do. We ask them to:

• Extract structured data from unstructured text (customer feedback, support tickets).
• Generate training examples to augment small datasets.
• Score or rank candidates based on criteria we define.
• Rewrite or simplify content for different audiences.

We always add a validation layer. If the LLM generates structured output, we parse it and validate the schema. If the stakes are high, we pair the LLM with a smaller, deterministic model to double-check. We log everything—the prompt, the response, whether it passed validation—so we can measure quality and cost.

import json
import anthropic

def extract_with_validation(text, schema):
    client = anthropic.Anthropic()
    
    prompt = f"""Extract structured data from the text below.
Return valid JSON matching this schema: {json.dumps(schema)}

Text: {text}"""
    
    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    
    try:
        result = json.loads(message.content[0].text)
        # Validate against schema
        for key in schema:
            if key not in result:
                return None
        return result
    except (json.JSONDecodeError, KeyError):
        return None

Evaluation is non-negotiable. We run regular checks on LLM outputs: do they match expected format, do they contain the right information, do they avoid hallucinations on known facts. We keep a growing test set of examples and measure how often the LLM passes.

Production is the job — Versioning, monitoring and cost control make AI useful

A model that works in a notebook doesn't work in production. Production means versioning every artifact, monitoring every prediction, and controlling costs that can spiral without guardrails.

We version models, not just code. When we deploy, we tag the model artifact with the exact training data hash, the feature engineering code, the hyperparameters, everything needed to reproduce it. We use a simple registry:

import hashlib
import json
from datetime import datetime

class ModelRegistry:
    def register(self, model_path, config, data_hash):
        version = {
            "timestamp": datetime.utcnow().isoformat(),
            "model_path": model_path,
            "config": config,
            "data_hash": data_hash,
            "model_hash": self._hash_file(model_path),
        }
        version_id = f"v{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}"
        self.store(version_id, version)
        return version_id
    
    def _hash_file(self, path):
        with open(path, 'rb') as f:
            return hashlib.sha256(f.read()).hexdigest()

Monitoring is where most teams fall short. We log predictions in real time: what features went in, what the model predicted, what actually happened. We surface data drift—when the incoming data no longer matches what the model trained on. We track model performance on actual outcomes, not just inference metrics.

Cost control is often forgotten until it's too late. We worked with a team running an LLM-powered feature that cost $15K per month at scale. They hadn't measured cost per prediction or set any budget guardrails. We added simple checks: cache repeated queries, batch requests where possible, fall back to a cheaper model when accuracy requirements are lower, and hard-cap API calls per day.

import time
from functools import wraps

class CostController:
    def __init__(self, daily_budget_usd, cost_per_request):
        self.daily_budget = daily_budget_usd
        self.cost_per_request = cost_per_request
        self.today_cost = 0
        self.last_reset = time.time()
    
    def should_allow(self):
        now = time.time()
        if now - self.last_reset > 86400:  # 24 hours
            self.today_cost = 0
            self.last_reset = now
        
        if self.today_cost + self.cost_per_request > self.daily_budget:
            return False
        return True
    
    def record_usage(self):
        self.today_cost += self.cost_per_request

Common mistakes

• Training on the full dataset: We've seen models that look perfect because they memorize training data. Always hold out a test set before you even look at it.
• Ignoring class imbalance: A 99% accurate model on a 99:1 imbalanced dataset is just predicting the majority class. Use stratified splits and appropriate metrics like F1 or AUC-ROC.
• Deploying without monitoring: The model that works in staging will drift in production. Log predictions from day one.
• Feature engineering without validation: Don't assume a feature is useful because it correlates. Measure its contribution to model performance on holdout data.
• Running LLMs without cost controls: A model that seems cheap at 10 requests per day becomes expensive at 100K requests per day. Set budgets upfront.

Key takeaways

• Data quality and cleanliness drive results more than model sophistication. Invest in data pipelines early.
• Proper evaluation requires time-based splits, no data leakage, and metrics aligned with business goals—not raw accuracy.
• Use LLMs tactically: RAG for knowledge, extraction for structure, evaluation for reliability.
• Production AI is engineering: version everything, monitor continuously, control costs, and measure real outcomes.
• Test sets are sacred. Once you look at them, they're no longer test sets. Hold them back from tuning and exploration.