Designing a notification system
By day 80 of this series, you've built event pipelines and real-time communication layers. Now comes the hard part: connecting them reliably to users across multiple channels. Designing a notification system means orchestrating events, user preferences, and delivery guarantees across in-app, push, and email channels without overwhelming your infrastructure—or your users.
I've shipped notifications for e-commerce platforms (order updates, wishlists), SaaS dashboards (alerts, invitations), and ride-hailing apps (driver assignment, surge pricing). Every one taught me that a "one size fits all" notification service breaks under real constraints. This article walks you through a production-grade architecture that scales.
Events in — Services publish events; the notification service decides what to send
Your notification pipeline starts where all event-driven systems do: with events. A user places an order, a payment fails, a team member is invited to a workspace. Each event is published to a Kafka topic that your notification service consumes.
The key design decision here is who decides what notifications are sent. Option one: let each domain service publish pre-formatted notifications directly to users. This breaks quickly. You end up with dozens of notification types hardcoded across services, no consistency, and no central place to change behavior. Option two: publish domain events, and let a dedicated notification service decide.
Here's a simplified event contract for an order service:
interface OrderEvent {
eventType: 'order.created' | 'order.shipped' | 'order.delivered' | 'order.cancelled';
orderId: string;
userId: string;
timestamp: number;
data: {
orderId: string;
amount: number;
estimatedDelivery?: string;
reason?: string;
};
}
Your notification service consumes this event and decides: does this user want an in-app notification? A push notification? An email? That logic lives in one place.
// Notification service consumer
kafkaConsumer.subscribe(['orders'], async (event: OrderEvent) => {
const preferences = await getNotificationPreferences(event.userId);
if (preferences.channels.includes('in-app')) {
await queue.push('in-app', {
userId: event.userId,
title: 'Order Shipped',
body: `Your order #${event.data.orderId} is on the way`,
actionUrl: `/orders/${event.data.orderId}`,
});
}
if (preferences.channels.includes('push')) {
await queue.push('push', {
userId: event.userId,
title: 'Order Shipped',
body: `Your order #${event.data.orderId} is on the way`,
});
}
if (preferences.channels.includes('email') && preferences.emailFrequency !== 'never') {
await queue.push('email', {
userId: event.userId,
templateId: 'order-shipped',
data: event.data,
});
}
});
This decouples your domain services from notification logic. A payment service publishes a payment.failed event; the notification service decides the best way to reach the user about it. If you later change your notification strategy, you don't touch the payment service.
Preferences — Users choose channels and frequency
Not all notifications are equal to all users. Some want every order update via push; others tolerate email summaries once a day. Your notification system must respect user choice, both for user experience and to stay compliant with regulations like CAN-SPAM and GDPR.
I've seen systems that store preferences in Redis for speed but lose them on a cache flush. Better approach: store in PostgreSQL with a Redis cache layer. Preferences change infrequently enough that a cache miss is acceptable, but fast reads are critical during event processing.
interface NotificationPreference {
userId: string;
channels: ('in-app' | 'push' | 'email')[];
emailFrequency: 'instant' | 'daily-digest' | 'weekly-digest' | 'never';
pushEnabled: boolean;
mutedCategories: string[]; // e.g., 'marketing', 'promotions'
doNotDisturbStart?: string; // HH:mm format
doNotDisturbEnd?: string;
}
When a user updates preferences (via your settings UI), publish an event to Kafka and invalidate the cache:
app.patch('/api/notification-preferences', async (req, res) => {
const { userId } = req.user;
const updates = req.body;
// Update in database
const prefs = await db.notificationPreferences.update(userId, updates);
// Invalidate cache
await redis.del(`prefs:${userId}`);
// Publish event for audit/compliance
await kafka.publish('notification-preferences', {
eventType: 'preferences.updated',
userId,
changes: updates,
timestamp: Date.now(),
});
res.json(prefs);
});
When the notification service retrieves preferences, it checks the cache first, then the database:
async function getNotificationPreferences(userId: string): Promise {
const cached = await redis.get(`prefs:${userId}`);
if (cached) return JSON.parse(cached);
const prefs = await db.notificationPreferences.findOne(userId);
if (!prefs) {
// Return sensible defaults if user never set preferences
return getDefaultPreferences(userId);
}
// Cache for 1 hour
await redis.set(`prefs:${userId}`, JSON.stringify(prefs), 'EX', 3600);
return prefs;
}
Also enforce frequency limits. If a user has emailFrequency: 'daily-digest', batch their email notifications and send once a day, not with every event. This requires a scheduler (like Bull or node-schedule) to run daily email digests.
Channels — In-app via WebSocket, push via FCM or APNs, email via a provider
Each channel has different delivery mechanics, latency, and reliability characteristics. In-app notifications via WebSocket are near-instant but only reach users currently online. Push notifications via Firebase Cloud Messaging (FCM) or Apple Push Notification (APNs) reach users offline but have queuing delays and can be noisy. Email is reliable and permanent but slow—minutes to hours—and can land in spam folders.
I recommend a channel-agnostic queue abstraction. Treat each channel as a worker pool:
interface NotificationJob {
id: string;
channel: 'in-app' | 'push' | 'email';
userId: string;
title: string;
body: string;
data?: Record;
templateId?: string;
createdAt: number;
status: 'pending' | 'sent' | 'failed' | 'delivered' | 'read';
retries: number;
nextRetryAt?: number;
}
In-app via WebSocket: Simplest to implement. Push a message to the user's WebSocket connection if they're connected; if not, store a "new notification" badge in the database. When the user reconnects, their client fetches unread notifications.
// When a notification job arrives for 'in-app'
async function sendInApp(job: NotificationJob) {
const connection = wsServer.getConnection(job.userId);
if (connection && connection.isOpen) {
connection.send(JSON.stringify({
type: 'notification',
id: job.id,
title: job.title,
body: job.body,
data: job.data,
timestamp: Date.now(),
}));
await db.notifications.update(job.id, { status: 'sent' });
} else {
// Store as unread notification in database
await db.notifications.create({
id: job.id,
userId: job.userId,
title: job.title,
body: job.body,
status: 'pending',
readAt: null,
});
}
}
Push via FCM/APNs: Requires device tokens (stored per user, per device). Use a third-party service like Firebase to handle the complexity. Implement exponential backoff for retries; if a device token is invalid, mark it as stale and don't retry indefinitely.
async function sendPush(job: NotificationJob) {
const tokens = await db.deviceTokens.findAll(job.userId);
if (!tokens || tokens.length === 0) {
// No device tokens; mark as undeliverable
await db.notifications.update(job.id, { status: 'failed' });
return;
}
for (const token of tokens) {
try {
const response = await admin.messaging().sendMulticast({
tokens: [token.value],
notification: {
title: job.title,
body: job.body,
},
data: job.data,
});
if (response.failureCount > 0) {
response.responses.forEach((resp, idx) => {
if (!resp.success && resp.error.code === 'messaging/invalid-registration-token') {
// Remove stale token
db.deviceTokens.delete(token.id);
}
});
}
await db.notifications.update(job.id, { status: 'sent' });
} catch (err) {
// Retry with exponential backoff
const retries = (job.retries || 0) + 1;
if (retries < 5) {
const delayMs = Math.pow(2, retries) * 1000;
job.nextRetryAt = Date.now() + delayMs;
job.retries = retries;
await queue.push('push', job);
} else {
await db.notifications.update(job.id, { status: 'failed' });
}
}
}
}
Email via a provider: Use SendGrid, AWS SES, or Postmark. These services handle deliverability, bounces, and compliance. Template emails client-side or use the provider's templating. Store the provider's message ID to track delivery.
async function sendEmail(job: NotificationJob) {
const user = await db.users.findOne(job.userId);
if (!user || !user.email) {
await db.notifications.update(job.id, { status: 'failed' });
return;
}
try {
const result = await sendgrid.send({
to: user.email,
from: 'notifications@myapp.com',
templateId: job.templateId,
dynamicTemplateData: job.data,
trackingSettings: {
clickTracking: { enable: true },
openTracking: { enable: true },
},
});
await db.notifications.update(job.id, {
status: 'sent',
providerId: result[0].headers['x-message-id'],
});
} catch (err) {
// Retry with backoff (max 3 retries for email)
if ((job.retries || 0) < 3) {
job.retries = (job.retries || 0) + 1;
job.nextRetryAt = Date.now() + 60000; // Retry after 1 minute
await queue.push('email', job);
} else {
await db.notifications.update(job.id, { status: 'failed' });
}
}
}
Delivery tracking — Sent, delivered, read — with retries per channel
A notification's lifecycle spans from creation to read. Tracking each state is crucial for debugging, compliance audits, and resend logic. Store this in a dedicated table with indexed queries:
interface NotificationRecord {
id: string;
userId: string;
channel: 'in-app' | 'push' | 'email';
status: 'pending' | 'sent' | 'delivered' | 'read' | 'failed' | 'bounced';
title: string;
body: string;
createdAt: number;
sentAt?: number;
deliveredAt?: number;
readAt?: number;
failureReason?: string;
retries: number;
nextRetryAt?: number;
providerId?: string; // For email, the SendGrid message ID
}
Set up webhooks from your providers to update delivery status. FCM, APNs, and SendGrid all support webhooks or status callbacks.
// Webhook from SendGrid
app.post('/webhooks/sendgrid', express.json(), async (req, res) => {
const events = req.body;
for (const event of events) {
if (event.event === 'delivered') {
await db.notifications.update({ providerId: event['message-id'] }, {
status: 'delivered',
deliveredAt: Date.now(),
});
} else if (event.event === 'open') {
await db.notifications.update({ providerId: event['message-id'] }, {
status: 'read',
readAt: Date.now(),
});
} else if (event.event === 'bounce') {
await db.notifications.update({ providerId: event['message-id'] }, {
status: 'bounced',
failureReason: event['bounce_reason'],
});
}
}
res.status(200).send('OK');
});
For in-app notifications, track reads client-side. When a user dismisses or opens a notification, send an event to your backend:
// Client-side
const markNotificationAsRead = async (notificationId: string) => {
await fetch(`/api/notifications/${notificationId}/read`, { method: 'PATCH' });
};
// Backend
app.patch('/api/notifications/:notificationId/read', async (req, res) => {
const { notificationId } = req.params;
const { userId } = req.user;
await db.notifications.update(notificationId, {
status: 'read',
readAt: Date.now(),
});
res.json({ ok: true });
});
Implement retry logic per channel with exponential backoff and maximum retry counts. Failed in-app notifications can retry more aggressively (they're free), while email retries should be conservative (you pay per email).
Common mistakes
- Storing preferences in Redis only: Lose them on a cache flush or pod restart. Always use a persistent database; Redis is a layer on top.
- Ignoring user preferences in production: Shipping notifications that users have explicitly disabled erodes trust and invites unsubscribes. Respect preferences religiously.
- No tracking of notification status: If a user says "I never got that notification," you can't debug it. Track every state: pending, sent, delivered, read, failed.
- Infinite retries on failed sends: A bad email address or revoked FCM token will fail every time. Set a max retry count and give up gracefully.
- Same payload for all channels: A 200-character email subject looks fine; a 200-character push notification is a wall of text. Tailor content per channel.
- No rate limiting: A bug in your event pipeline can spam users with hundreds of notifications. Implement per-user, per-channel rate limits at the queue level.
Key takeaways
- Separate domain events from notification decisions. Publish events; let a dedicated notification service decide what to send based on user preferences.
- Store user preferences in a database with a Redis cache layer for fast lookups during event processing.
- Treat each channel (in-app, push, email) as a separate worker, respecting their unique constraints (latency, cost, reliability).
- Track notification status from creation to read, and store delivery metadata (provider IDs, bounce reasons) for debugging and compliance.
- Implement exponential backoff retries per channel, with conservative max-retry counts for expensive channels like email.
- Set up webhooks from providers to track delivery and read events in real time.
- Respect user preferences ruthlessly—unsolicited notifications are the fastest way to lose users.
Comments
Loading comments…