Embeddings explained in Practical AI & Machine Learning

Day 37: Embeddings explained in Practical AI & Machine Learning

Embeddings are vectors—lists of numbers—that capture the semantic meaning of text, images, or other data in a form a machine can understand and compare. Instead of treating words or documents as isolated tokens, embeddings let you represent meaning mathematically, so similar content produces similar vectors. Once you master embeddings, you unlock semantic search, recommendations, deduplication, and clustering in your applications.

What embeddings are: numbers that capture meaning

An embedding is a fixed-length array of floating-point numbers. Each position in that array represents some learned dimension of meaning. When an embedding model processes the text "the user loved this product," it doesn't output a category or score—it outputs a vector, say 384 or 1536 dimensions long, where each number encodes some aspect of the phrase's semantics.

The magic is that semantically similar text produces similar vectors. "The customer adored this item" will be close in vector space to "the user loved this product." Meanwhile, "system error occurred" will be far away. This property—that meaning becomes distance—is what makes embeddings powerful for production systems.

Think of embeddings as coordinates in a high-dimensional semantic space. Just as you can measure physical distance in meters, you measure semantic distance using mathematical formulas (like cosine similarity, which we'll cover). Two product reviews that say almost the same thing will cluster near each other in this space; a spam message will float off alone.

Create embeddings: API or local models

You have two main paths to generate embeddings in production: call a managed API, or run a local model on your infrastructure.

API approach: Services like OpenAI's Embeddings API, Cohere, Anthropic, and others expose embedding endpoints. You send text and get a vector back. This is the simplest integration path: one HTTPS request, no model hosting burden, and you get production-grade embeddings tuned by domain experts.

import openai

client = openai.OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="the user loved this product"
)

embedding = response.data[0].embedding
print(f"Embedding dimension: {len(embedding)}")
print(f"First 5 values: {embedding[:5]}")

Local model approach: Libraries like Sentence Transformers (built on Hugging Face transformers) let you download and run embedding models on your own hardware. Models like all-MiniLM-L6-v2 (384 dims) or all-mpnet-base-v2 (768 dims) are lightweight enough for a single server or container.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

texts = [
    "the user loved this product",
    "the customer adored this item",
    "system error occurred"
]

embeddings = model.encode(texts)
print(f"Shape: {embeddings.shape}")  # (3, 384)

Choose based on your constraints: APIs are fast to integrate and scale elastically, but add latency and per-call cost. Local models are free after deployment, latency-sensitive-friendly, and work offline, but require infrastructure and care (GPU, memory, updates).

Compare embeddings: cosine similarity

Once you have two embeddings, you need a metric to measure how close they are. Cosine similarity is the standard choice in production. It measures the angle between two vectors—ranging from -1 (opposite) to 1 (identical)—without caring about their magnitude.

Why cosine and not Euclidean distance? Because in high-dimensional spaces, all points are far apart. Cosine similarity ignores magnitude and focuses purely on direction, which aligns better with how neural networks learn meaning.

import numpy as np

def cosine_similarity(a, b):
    """
    Compute cosine similarity between two embedding vectors.
    Returns a score from -1 to 1.
    """
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Example
embedding_1 = np.array([0.1, 0.2, 0.3, 0.4])
embedding_2 = np.array([0.11, 0.19, 0.31, 0.39])
embedding_3 = np.array([0.9, 0.8, 0.7, 0.6])

sim_1_2 = cosine_similarity(embedding_1, embedding_2)
sim_1_3 = cosine_similarity(embedding_1, embedding_3)

print(f"Similarity 1–2: {sim_1_2:.4f}")  # High, ~0.99
print(f"Similarity 1–3: {sim_1_3:.4f}")  # Low, ~-0.13

In practice, you don't compute this by hand. Vector databases (Pinecone, Weaviate, Milvus) and search libraries (Elasticsearch, PostgreSQL pgvector) handle similarity search efficiently at scale, indexing millions of embeddings so lookups are fast.

Use cases: semantic search, recommendations, clustering, deduplication

Semantic search

A user types "is this product good for beginners?" You embed that query and compare it (via cosine similarity) to embeddings of all your product reviews. The highest-scoring reviews bubble to the top—not because they contain the exact words "beginner," but because the vector space learned that "good for beginners" and "easy to use for new users" live nearby.

Recommendations

Embed all items in your catalog (products, articles, videos). When a user views one item, find the K-nearest neighbors in embedding space. Those neighbors are semantically similar, making them strong recommendations. A user reading a machine learning article gets recommended related posts without explicit tagging.

Clustering and deduplication

Embeddings let you group similar items automatically. Feed embeddings from customer support tickets into a clustering algorithm (K-means, DBSCAN). Tickets cluster by topic—payments, shipping, returns—without you manually labeling them. For deduplication, compute embeddings of incoming user-generated content and check if it's close (say, cosine > 0.95) to existing content; if so, it's likely a duplicate.

Example: a simple recommendation pipeline

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarities
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

# Your catalog
items = [
    "How to learn Python for data science",
    "Beginner's guide to machine learning",
    "Advanced NumPy optimization techniques",
    "Building APIs with FastAPI",
    "SQL query performance tuning"
]

# Embed all items
item_embeddings = model.encode(items)

# User is reading item 0
user_item_idx = 0
user_embedding = item_embeddings[user_item_idx].reshape(1, -1)

# Compute similarity to all others
similarities = cosine_similarities(user_embedding, item_embeddings)[0]

# Get top 2 recommendations (excluding the current item)
top_indices = np.argsort(similarities)[::-1][1:3]

print(f"User is reading: {items[user_item_idx]}")
print("Recommendations:")
for idx in top_indices:
    print(f"  - {items[idx]} (similarity: {similarities[idx]:.3f})")

Common mistakes

• Comparing embeddings from different models. Each model defines its own semantic space. An embedding from OpenAI's model is not directly comparable to one from Sentence Transformers. Always use the same model for all embeddings in a pipeline.
• Forgetting to normalize queries and documents. If your corpus embeddings come from a batch process but you embed a live query differently (e.g., with different preprocessing), the semantic space drifts. Ensure consistent encoding.
• Using raw Euclidean distance. High-dimensional spaces make all distances meaningless. Stick to cosine similarity or dot product for embeddings.
• Neglecting the cost of embedding at scale. If you embed millions of documents repeatedly, you'll rack up API costs. Cache embeddings in a database or vector store so you only compute them once.
• Picking a model that's too large. A 1536-dimensional OpenAI embedding is rich but slow to search and expensive to store. For many use cases, a 384-dim local model is faster and cheaper.

Key takeaways

• Embeddings are vectors of numbers that represent meaning; similar text produces similar vectors.
• Generate embeddings via API (OpenAI, Cohere) for simplicity or local models (Sentence Transformers) for control and cost savings.
• Use cosine similarity to measure closeness between embeddings—it's robust and fast in high dimensions.
• Apply embeddings to semantic search, recommendations, clustering, and deduplication in production systems.
• Cache embeddings in a vector database to avoid recomputation and enable fast nearest-neighbor queries at scale.
• Keep the same embedding model throughout your pipeline; never mix embeddings from different sources.