Running models locally: AI & ML best practices explained

Running models locally: AI & ML best practices explained

When you run machine learning models on your own hardware instead of calling a cloud API, you gain control over latency, data privacy, and cost. Running models locally: AI & ML best practices explained is about understanding the trade-offs and techniques that make this practical for production systems. In this article, we'll walk through the tools, quantization strategies, and real-world scenarios where local inference becomes not just optional but essential.

Why local inference matters in production

Cloud APIs are convenient, but they come with drawbacks. Every inference call adds network latency—typically 100–500ms round-trip. For a user-facing feature that runs five inferences per action, that's half a second of overhead. Beyond latency, there's data sensitivity: medical records, financial data, or user PII shouldn't leave your infrastructure if you can avoid it. And cost scales with volume; at scale, per-token pricing gets expensive fast.

Local inference solves these problems. A quantized 7B parameter model runs inference in 50–200ms on a modern laptop or modest server. Your data never leaves your network. And after the initial model download, marginal inference cost is near-zero. The trade-off is that you manage hardware, model updates, and quality degradation from quantization. For many teams—especially those handling sensitive data or operating at scale—that trade-off is worth it.

Ollama for quick local LLMs

Ollama is the fastest way to get a working local LLM. It handles model downloading, quantization, and serving via a simple CLI and HTTP API. You don't need to understand GGML formats or quantization parameters; Ollama abstracts that away.

Getting started:

1. Install Ollama from ollama.ai (or ollama.com, depending on current URLs).
2. Run ollama pull llama2 to download a quantized Llama 2 model (~4GB for the 7B variant).
3. Start the server with ollama serve.
4. Make requests to http://localhost:11434/api/generate.

Here's a practical example. Imagine you're building a document summarization feature for a SaaS platform. You want fast, private summaries without sending documents to OpenAI:

import requests
import json

def summarize_document(text: str) -> str:
    """Summarize using local Ollama instance."""
    payload = {
        "model": "llama2",
        "prompt": f"Summarize this in 2-3 sentences:\n\n{text}",
        "stream": False
    }
    response = requests.post(
        "http://localhost:11434/api/generate",
        json=payload,
        timeout=60
    )
    result = response.json()
    return result["response"].strip()

# Usage
doc = "Your 5000-word document here..."
summary = summarize_document(doc)
print(summary)

Ollama supports multiple backends and models. Llama 2, Mistral, Neural Chat, and Orca are popular choices. Each comes pre-quantized at various sizes (7B, 13B, 70B parameters). For most use cases, 7B or 13B models deliver reasonable quality while fitting on a single GPU or even CPU with patience.

Production considerations: Ollama listens on localhost by default. For production, you'll want to proxy it behind an authentication layer (nginx + OAuth, for instance) or run it on a private network. Consider containerizing it with Docker so model downloads and dependencies are reproducible. And monitor GPU memory; a shared GPU across multiple services can cause OOM errors if you're not careful.

Quantized models fit on laptops

Quantization is the key technique that makes local inference practical. Full-precision (FP32 or FP16) large language models are enormous. A 7B parameter Llama model in FP16 is roughly 14GB. That's too large for most laptops and wastes inference time because GPUs must load all those bytes from memory.

Quantization reduces precision—typically from FP16 to INT8 or INT4—which shrinks the model 2–8x while keeping quality surprisingly good. A 4-bit quantized 7B Llama model is around 3.5GB and fits comfortably on a laptop with a modest GPU or even CPU.

How it works: During quantization, the model's weights are converted to lower-bit integers. The conversion is calibrated on a small dataset to minimize accuracy loss. At inference time, weights are dequantized on-the-fly (a cheap operation), and computations happen in higher precision. Modern quantization schemes—like GPTQ or bitsandbytes—are highly optimized.

Real scenario: You're building a mobile companion app that should work offline. You embed a 4-bit quantized Mistral 7B model (3.2GB) on the device. Users get instant responses, even in airplane mode, with no API calls. Battery usage is lower than cloud API calls (no network radio overhead). For on-device ML at scale, this is standard practice in travel, ride-hailing, and fitness apps.

You can quantize models yourself using libraries like bitsandbytes or GPTQ, but Ollama and Hugging Face already host quantized versions. For most teams, grabbing a pre-quantized model is faster than rolling your own.

Good for privacy and offline use

Privacy is non-negotiable in healthcare, finance, and legal tech. Running models locally means your data—patient records, financial statements, case files—never touches a third-party API. This is crucial for HIPAA compliance, GDPR, and regulatory audits. Some jurisdictions explicitly require data to stay on-premises.

Practical example: A hospital's radiology department uses a local AI model to assist in tumor detection. Images are processed on-premises, never sent to the cloud. The model is validated against the hospital's historical data, tuned for their equipment and patient population. This is more defensible legally and ethically than relying on a generic cloud service.

Offline capability is equally valuable. A field service app for technicians working in remote areas can run inference locally without needing connectivity. A recommendation engine for an e-commerce app can provide suggestions even during network hiccups. These features degrade gracefully; if the local model fails or the inference is slow, you fall back to precomputed results or simpler logic.

Implementation pattern:

import ollama
import logging

def infer_with_fallback(prompt: str, fallback_response: str) -> str:
    """Run local inference with fallback to precomputed response."""
    try:
        response = ollama.generate(
            model="mistral",
            prompt=prompt,
            stream=False,
            timeout=5  # Fail fast
        )
        return response["response"]
    except Exception as e:
        logging.warning(f"Local inference failed: {e}, using fallback")
        return fallback_response

# Usage in a recommendation engine
user_history = "User viewed: shoes, hats, bags"
fallback = "Try our trending collection"
recommendation = infer_with_fallback(
    f"Suggest next item: {user_history}",
    fallback
)

The offline benefit extends to batch processing too. Process logs, emails, or documents overnight when network load is low, storing results in a local cache. Then serve cached results during business hours with no API overhead.

Smaller models trade quality for cost

The popular narrative is "bigger models are always better." In production, that's often wrong. A 7B model is 60x smaller than GPT-3 (175B) but handles most tasks—classification, summarization, Q&A, extraction—at 80–95% of GPT-3 quality. A 3B model (like Phi or MobileLLM) runs 2–3x faster than 7B with maybe 5–10% quality drop on simpler tasks.

The trade-off is task-dependent. For sentiment analysis or classification, a 3B model is often sufficient and noticeably faster. For creative writing or complex reasoning, you'll want 13B or larger. The sweet spot for most production systems is 7B—it's fast enough, accurate enough, and fits most hardware.

Benchmarking approach: Before committing to a model size, run a small evaluation on your actual use case. Generate 100 test examples, run them through multiple model sizes (3B, 7B, 13B), and compare outputs against a ground truth or expert judgment. Measure latency and cost. Often you'll find that 7B is optimal; sometimes a 3B model surprises you with acceptable quality.

import time
import ollama

models = ["phi", "mistral", "neural-chat"]
test_prompt = "Classify sentiment: I love this product!"

for model in models:
    start = time.time()
    response = ollama.generate(model=model, prompt=test_prompt, stream=False)
    elapsed = time.time() - start
    print(f"{model}: {elapsed:.2f}s - {response['response'][:50]}")

Smaller models also enable on-device deployment. A 3B quantized model fits on a mobile phone's storage and runs inference in a few seconds. This unlocks features for users in low-bandwidth regions or those who value privacy by default.

Cost equation: Local inference cost is dominated by hardware (amortized GPU or CPU cost). Once you've bought a $500 GPU, running 1 million inferences costs a few dollars in electricity. Compare that to cloud APIs: 1 million inferences at $0.001 per token (typical pricing) and 100 tokens per inference = $100. At volume, local inference wins decisively.

Common mistakes

• Ignoring latency SLAs: Local models are fast, but not instant. A 7B model might take 200ms per inference. If your frontend timeout is 300ms, you're walking a knife edge. Always test end-to-end latency, including model loading, tokenization, and deserialization.
• Not monitoring model staleness: If you download a model once and never update it, you miss bug fixes and improvements. Set up a process to periodically check for new quantized versions and run validation tests before deploying.
• Underestimating memory needs: A quantized model file is 3GB, but in memory, you need space for the model, tokenizer, intermediate activations, and batching. A safe estimate is 2x the model size in RAM. Running two concurrent requests might require 12GB for a 3GB model.
• Using generic models without fine-tuning: A base Llama 2 model works reasonably well for many tasks, but fine-tuning it on your domain data (customer support queries, technical documentation, etc.) typically yields 10–20% quality improvement at minimal extra cost.
• Forgetting about inference optimization: Even with quantization, there are wins to be had. Use smaller tokenizers, batch requests when possible, and leverage libraries like vLLM that optimize the inference pipeline. These can 2–3x throughput.

Key takeaways

• Local inference eliminates latency, protects privacy, and reduces cost at scale—use Ollama to get started in minutes.
• Quantization shrinks models 2–8x with minimal quality loss; a 4-bit 7B model is practical for laptops and mobile devices.
• For sensitive data or offline-required features, local inference is often mandatory, not optional.
• Smaller models (3B–7B) trade quality for speed and cost; benchmark your use case to find the optimal size.
• Local inference cost is hardware amortization plus electricity; at high volume, it beats cloud APIs by 10–100x.
• Monitor model memory usage, update models regularly, and optimize the inference pipeline for production reliability.