Text classification with transformers: a practical guide

Support teams drown in unstructured text every day—tickets, emails, reviews, and chat messages that need to be routed, prioritized, or analyzed. Text classification with transformers: a practical guide walks you through the two paths that actually work in production: fine-tuning a small pretrained model for high-accuracy custom tasks, or prompting a large language model for quick prototypes. We'll show you how to pick the right approach, prepare your data, and ship something that handles real-world messy input.

Why transformers for text classification

Transformers like BERT, DistilBERT, and RoBERTa have made text classification dramatically simpler than it was five years ago. They capture semantic meaning and context in ways bag-of-words models never could. A pretrained transformer already "understands" language from billions of tokens, so you don't start from scratch.

The practical upside: you can fine-tune to your specific problem—support ticket urgency, review sentiment, user intent—with a few hundred examples and get 85–95% accuracy in most cases. No PhD in NLP required.

Pick a base model — small is often enough

Your first instinct might be to reach for a massive model. Resist it. For text classification, a lightweight pretrained transformer does the job and costs far less to host and run.

Popular choices:

• DistilBERT: 40% smaller and 60% faster than BERT, drops only 3–5% in accuracy. Start here for most projects.
• RoBERTa-base: Slight edge over BERT on most benchmarks, widely available via Hugging Face.
• MobileBERT: If you need edge deployment or mobile, this compresses further.
• Llama 2 7B or Mistral 7B: If you want a general-purpose model you can also use for other NLP tasks, these are lighter than larger LLMs but more capable than task-specific classifiers.

A real example: one team we worked with classified support tickets (urgent, routine, billing, technical) using DistilBERT. On a GPU, inference latency was under 50ms per ticket. That's fast enough for real-time dashboards and automated routing.

Model size matters for deployment cost. DistilBERT is roughly 268M parameters; BERT-base is 110M. Smaller models fit in memory better, batch faster, and run cheaper on AWS, GCP, or your on-prem hardware.

Prepare labels — a few hundred clean examples per class

The biggest mistake we see: teams spend weeks engineering features but only hours labeling data. Your model is only as good as your labels.

Good labeling practice:

1. Start with 50–100 examples per class. Deliberately pick edge cases and ambiguous ones. Label them yourself or with a colleague so you understand the boundaries.
2. Once you have a sense of the task, use a labeling tool (Prodigy, Label Studio, or even a shared spreadsheet) and involve 1–2 domain experts. For support tickets, that's your support lead. For reviews, someone from the product team.
3. Aim for 300–500 examples per class if your classes are balanced. Imbalanced data (80% routine, 10% urgent, 10% billing) needs more careful handling—stratified sampling, class weights, or oversampling.
4. Hold back 10–20% as a test set. Don't touch it until you're done tuning.

Real-world size: we've shipped production classifiers on 400–800 total examples across 3–5 classes. More data always helps, but it has diminishing returns after a few hundred.

Format your data as CSV or JSONL:

# CSV format
text,label
"My account is locked I can't log in",urgent
"How do I update my billing address?",billing
"Feature request: dark mode",routine

# JSONL format (one per line)
{"text": "My account is locked I can't log in", "label": "urgent"}
{"text": "How do I update my billing address?", "label": "billing"}

Fine-tune with Hugging Face Trainer

Once you have 300+ labeled examples, fine-tuning a transformer on your specific task beats zero-shot classification every time. Hugging Face Trainer abstracts away the training loop, mixed precision, gradient accumulation, and checkpointing.

Here's a complete example:

from datasets import load_dataset, DatasetDict, Dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
import pandas as pd
from sklearn.model_selection import train_test_split

# 1. Load and split your data
df = pd.read_csv("support_tickets.csv")
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42, stratify=df["label"])

# Convert to Hugging Face Dataset
train_dataset = Dataset.from_pandas(train_df)
test_dataset = Dataset.from_pandas(test_df)

# 2. Tokenize
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=128)

train_tokens = train_dataset.map(tokenize_function, batched=True)
test_tokens = test_dataset.map(tokenize_function, batched=True)

# 3. Map labels to IDs
label2id = {"routine": 0, "billing": 1, "technical": 2, "urgent": 3}
id2label = {v: k for k, v in label2id.items()}

train_tokens = train_tokens.map(
    lambda x: {"label": label2id[x["label"]]}, remove_columns=["__index_level_0__"] if "__index_level_0__" in train_tokens.column_names else []
)
test_tokens = test_tokens.map(
    lambda x: {"label": label2id[x["label"]]}, remove_columns=["__index_level_0__"] if "__index_level_0__" in test_tokens.column_names else []
)

# 4. Load model and configure trainer
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=len(label2id),
    id2label=id2label,
    label2id=label2id
)

training_args = TrainingArguments(
    output_dir="./support_classifier",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    warmup_steps=100,
    weight_decay=0.01,
    logging_dir="./logs",
    logging_steps=50,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_tokens,
    eval_dataset=test_tokens,
)

# 5. Train
trainer.train()

# 6. Evaluate and save
results = trainer.evaluate()
print(results)
trainer.save_model("./support_classifier_final")
tokenizer.save_pretrained("./support_classifier_final")

Training typically takes 10–30 minutes on a GPU for a few hundred examples. After training, you'll see validation accuracy, F1 scores, and loss curves in the logs directory.

Then use your classifier in production:

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="./support_classifier_final",
    device=0  # GPU index, or -1 for CPU
)

ticket_text = "I can't reset my password"
result = classifier(ticket_text)
print(result)
# Output: [{'label': 'urgent', 'score': 0.94}]

Or prompt an LLM — zero-shot for prototypes and low volume

Not every classification task needs fine-tuning. If you have a prototype, low volume, or labels that are hard to define, zero-shot classification with an LLM is faster to ship and requires no labeled data.

With OpenAI or similar APIs:

import openai

client = openai.Client(api_key="your-key")

ticket = "My account is locked I can't log in"
categories = ["urgent", "billing", "technical", "routine"]

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": "You are a support ticket classifier. Classify the ticket into ONE of these categories: urgent, billing, technical, routine. Respond with only the category."
        },
        {
            "role": "user",
            "content": ticket
        }
    ],
    temperature=0,
    max_tokens=10
)

label = response.choices[0].message.content.strip().lower()
print(label)  # Output: "urgent"

Or use Hugging Face's zero-shot pipeline locally (no API calls, no keys):

from transformers import pipeline

zero_shot_classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli",
    device=0
)

ticket = "My account is locked I can't log in"
labels = ["urgent", "billing", "technical", "routine"]

result = zero_shot_classifier(ticket, labels)
print(result)
# Output:
# {
#   'sequence': 'My account is locked I can't log in',
#   'labels': ['urgent', 'technical', 'routine', 'billing'],
#   'scores': [0.89, 0.08, 0.02, 0.01]
# }

Trade-offs:

• Fine-tuned model: Faster inference, cheaper at scale, works offline, high accuracy on your specific domain. Needs labeled data upfront.
• Zero-shot LLM: No labeling, ships immediately, handles edge cases and nuance well. Slower, more expensive per call, needs internet or a local LLM.

Practical hybrid: start with zero-shot for your first week. Collect predictions and feedback. Once you have 300–500 labeled examples, switch to fine-tuning. You'll save money and get faster response times.

Common mistakes

• Skipping the test set. Always hold back 10–20% of your data and evaluate on it only at the end. Otherwise you'll overfit and your reported accuracy means nothing.
• Imbalanced labels without handling. If 80% of tickets are "routine" and 5% are "urgent," your model will be biased. Use class weights or stratified sampling.
• Using the wrong max_length. Setting max_length too high wastes compute; too low and you truncate important context. 128 tokens works for most support tickets. Review a few samples to find the 95th percentile.
• Not monitoring drift. Six months after deployment, your ticket distribution shifts or language changes. Collect predictions and retrain quarterly.
• Choosing the wrong base model. Reaching for GPT-4 when DistilBERT will do is expensive and unnecessary. Start small.

Key takeaways

• For most text classification tasks, a small pretrained transformer (DistilBERT, RoBERTa-base) fine-tuned on 300–500 examples beats custom heuristics or bag-of-words models.
• Label your data carefully. A few hundred clean examples with good agreement outweigh thousands of noisy ones.
• Use Hugging Face Trainer to handle the training loop, saving time and reducing bugs.
• Zero-shot classification with an LLM is perfect for prototypes and low-volume tasks, but fine-tuning is faster and cheaper at scale.
• Always hold back a test set, monitor for label drift in production, and retrain as language and distribution shift.