Lessons from dispatch notifications

I learned early in my ride-hailing platform work that real-time dispatch systems fail silently when we treat WebSocket connections as reliable. A driver misses a high-value ride offer not because the backend never sent it, but because they switched apps, their network dropped, or their browser tab wasn't focused. Lessons from dispatch notifications matter because they're the difference between a smooth user experience and frustrated drivers watching competitors pick up their rides.

Over the past 49 days building Kafka and WebSocket systems, I've encountered nearly every failure mode in real-time notifications. Today I'm sharing the four patterns that actually keep drivers engaged: assume disconnection, fall back to push notifications, acknowledge explicitly, and expire stale offers. These aren't theoretical—they're production lessons from platforms handling thousands of concurrent drivers.

Assume the socket is disconnected

The first mental shift is hardest: stop assuming your WebSocket connection will deliver your message. I've seen teams spend weeks debugging "why the notification never arrived" only to discover the driver's phone went into airplane mode for 30 seconds. By then, the offer had expired on the backend and the client never knew it existed.

In a ride-hailing system, when a ride request comes through Kafka, you have maybe 30–60 seconds to notify drivers before the offer gets auto-cancelled. If you only push to their WebSocket and they're disconnected, they miss the entire window. The driver never sees the notification, the backend never knows why they didn't accept, and you lose revenue.

Here's how I handle it in production:

typescript


interface DispatchEvent {
  rideRequestId: string;
  driverId: string;
  fare: number;
  pickupLocation: {
    lat: number;
    lng: number;
  };
  expiresAt: number; // Unix timestamp in milliseconds
  notificationId: string;
}

// In your Kafka consumer for new ride requests
async function handleRideRequest(event: DispatchEvent) {
  // Always assume WebSocket might not deliver
  // Queue the notification for multiple delivery attempts
  
  const notificationRecord = {
    id: event.notificationId,
    driverId: event.driverId,
    rideRequestId: event.rideRequestId,
    type: 'RIDE_OFFER',
    payload: event,
    createdAt: Date.now(),
    expiresAt: event.expiresAt,
    attempts: 0,
    maxAttempts: 5,
    lastAttemptAt: null,
    delivered: false,
    acknowledged: false,
  };

  // Persist notification to database for retry logic
  await notificationRepository.create(notificationRecord);

  // Try WebSocket immediately (best case: instant delivery)
  const driver = activeConnections.get(event.driverId);
  if (driver && driver.isConnected()) {
    driver.socket.emit('ride:offer', event);
    console.log(`WebSocket delivered to driver ${event.driverId}`);
  } else {
    console.log(`Driver ${event.driverId} not connected, will retry`);
  }

  // Queue for push notification (fallback)
  await pushNotificationQueue.enqueue({
    driverId: event.driverId,
    title: 'New Ride Available',
    body: `₹${event.fare} • ${event.pickupLocation}`,
    data: {
      notificationId: event.notificationId,
      rideRequestId: event.rideRequestId,
    },
    delay: 2000, // Wait 2 seconds—maybe they'll reconnect
    priority: 'high',
  });
}

This approach gives you three chances: immediate WebSocket delivery, a delayed push notification, and a background retry system. The driver experiences the offer through whichever channel works first.

Fall back to push notifications

Push notifications aren't glamorous compared to WebSockets, but they're your safety net. A driver with your app installed will almost certainly get a push notification, even if they're in another app or their WebSocket is stale. The latency is higher (usually 1–5 seconds), but it beats missing the offer entirely.

The tricky part is coordinating between channels so you don't send duplicate alerts. I've watched drivers get a WebSocket notification, then 3 seconds later get an identical push notification because the WebSocket delivery wasn't acknowledged quickly enough.

typescript


// Push notification service
async function sendPushNotification(
  driverId: string,
  notification: DispatchEvent
): Promise {
  const deviceTokens = await deviceTokenRepository.find(driverId);

  if (deviceTokens.length === 0) {
    console.warn(`No device tokens for driver ${driverId}`);
    return;
  }

  const message = {
    notification: {
      title: `₹${notification.fare} ride`,
      body: `Pickup at ${formatLocation(notification.pickupLocation)}`,
    },
    data: {
      notificationId: notification.notificationId,
      rideRequestId: notification.rideRequestId,
      type: 'RIDE_OFFER',
      expiresAt: String(notification.expiresAt),
    },
    android: {
      priority: 'high',
      ttl: Math.min(notification.expiresAt - Date.now(), 60000), // Expires with offer
    },
    apns: {
      headers: {
        'apns-priority': '10',
        'apns-expiration': String(Math.floor(notification.expiresAt / 1000)),
      },
    },
  };

  const results = await Promise.allSettled(
    deviceTokens.map(token => firebaseAdmin.messaging().send({
      ...message,
      token: token.fcmToken,
    }))
  );

  // Log failures for debugging
  results.forEach((result, index) => {
    if (result.status === 'rejected') {
      console.error(`Push failed for token ${deviceTokens[index].fcmToken}:`, result.reason);
      // Mark token for removal if it's permanently invalid
      if (result.reason.code === 'messaging/invalid-registration-token') {
        deviceTokenRepository.delete(deviceTokens[index].id);
      }
    }
  });
}

// Enqueue with exponential backoff
async function enqueuePushWithRetry(
  driverId: string,
  notification: DispatchEvent,
  attempt: number = 0
): Promise {
  const maxAttempts = 3;
  const delay = Math.min(1000 * Math.pow(2, attempt), 5000);

  try {
    await sendPushNotification(driverId, notification);
  } catch (error) {
    if (attempt < maxAttempts) {
      console.log(`Push retry ${attempt + 1}/${maxAttempts} for driver ${driverId}`);
      await scheduleJob(
        () => enqueuePushWithRetry(driverId, notification, attempt + 1),
        delay
      );
    } else {
      console.error(`Push notification exhausted retries for driver ${driverId}`, error);
    }
  }
}

By the time your push notification arrives, the driver might already be reconnected via WebSocket. That's fine—the acknowledgment mechanism (next section) prevents double-processing.

Acknowledge receipt explicitly

This is where most systems break down. A notification arrives through WebSocket, the driver sees it, but the backend has no idea. If the WebSocket disconnects immediately after, the server thinks the driver never got the message and tries to re-deliver it through every channel. The driver gets spammed, trust erodes, and engagement tanks.

Your app must send an explicit acknowledgment the moment it receives the notification, not when the driver acts on it. This tells the backend: "I got it, stop retrying."

typescript


// Client side (React Native / React)
useEffect(() => {
  socket.on('ride:offer', (offer: DispatchEvent) => {
    console.log('Received offer:', offer.rideRequestId);
    
    // Display the offer UI immediately
    dispatch(setCurrentOffer(offer));
    playNotificationSound();
    
    // Send acknowledgment right away
    socket.emit('ride:offer:ack', {
      notificationId: offer.notificationId,
      rideRequestId: offer.rideRequestId,
      receivedAt: Date.now(),
    });
    
    // Mark it as no longer needed in push notifications
    if (global.fcm) {
      global.fcm.removeNotification(offer.notificationId);
    }
  });

  return () => socket.off('ride:offer');
}, [socket]);

// Server side
socket.on('ride:offer:ack', async (ack: {
  notificationId: string;
  rideRequestId: string;
  receivedAt: number;
}) => {
  const driverId = socket.data.driverId;

  // Mark notification as acknowledged
  await notificationRepository.update(ack.notificationId, {
    acknowledged: true,
    acknowledgedAt: ack.receivedAt,
    acknowledgedBy: 'websocket',
  });

  // Cancel any pending push notifications
  await pushNotificationQueue.cancel(ack.notificationId);

  // Log latency for monitoring
  const latency = ack.receivedAt - notificationRepository.getCreatedAt(ack.notificationId);
  metrics.dispatch.ackLatency.observe(latency);

  console.log(`Offer acknowledged by driver ${driverId} in ${latency}ms`);
});

Without explicit acknowledgment, your retry logic has no signal to stop. With it, you optimize delivery: the moment a driver receives the notification through any channel, all other delivery attempts cancel.

Expire stale offers automatically

A ride offer is only valuable for a limited window. After 45 seconds, you want to auto-cancel the offer, clean up pending notifications, and remove it from the client's view. If you don't handle expiry correctly, drivers accept rides that already went to another driver, push notifications arrive for offers that no longer exist, and your backend accumulates garbage data.

typescript


interface RideOfferExpiry {
  rideRequestId: string;
  offerId: string;
  driverId: string;
  expiresAt: number;
}

// When creating the offer, schedule its expiry
async function createAndDispatchRideOffer(
  rideRequest: RideRequest,
  targetDriver: Driver
): Promise {
  const offerId = generateUUID();
  const expiresAt = Date.now() + 45000; // 45 seconds

  const offer = {
    id: offerId,
    rideRequestId: rideRequest.id,
    driverId: targetDriver.id,
    fare: rideRequest.fare,
    pickupLocation: rideRequest.pickupLocation,
    expiresAt,
  };

  await rideOfferRepository.create(offer);

  // Dispatch via WebSocket and queue for push
  await handleRideRequest(offer);

  // Schedule automatic expiry
  await scheduleJob(
    () => expireRideOffer(offerId, targetDriver.id, rideRequest.id),
    expiresAt - Date.now()
  );

  return offerId;
}

async function expireRideOffer(
  offerId: string,
  driverId: string,
  rideRequestId: string
): Promise {
  const offer = await rideOfferRepository.findById(offerId);

  if (!offer || offer.status !== 'pending') {
    console.log(`Offer ${offerId} already handled`);
    return;
  }

  // Mark as expired
  await rideOfferRepository.update(offerId, {
    status: 'expired',
    expiredAt: Date.now(),
  });

  // Cancel any pending push notifications
  const notifications = await notificationRepository.findByOfferId(offerId);
  for (const notif of notifications) {
    if (!notif.delivered) {
      await pushNotificationQueue.cancel(notif.id);
    }
  }

  // Notify driver if connected (so they see the offer disappear)
  const socket = activeConnections.get(driverId);
  if (socket && socket.isConnected()) {
    socket.emit('ride:offer:expired', {
      offerId,
      rideRequestId,
    });
  }

  // Log expiry for analytics
  metrics.dispatch.offerExpired.inc({
    reason: 'timeout',
  });

  console.log(`Offer ${offerId} expired`);
}

Expiry is also your chance to clean up. Notifications that were never acknowledged get marked as expired. Push tokens that never received the notification are marked for investigation. This data helps you debug delivery failures and improve your notification system over time.

Common mistakes

Relying only on WebSockets: Teams often deploy systems where WebSocket is the sole notification channel. The first network hiccup causes offer misses. Always have a fallback.
No acknowledgment strategy: Servers retry notifications indefinitely because they don't know if the client received them. Implement explicit acks from day one.
Ignoring expiry: Offers that never expire clutter the database and can be accepted long after they're invalid. Set and enforce expiry on every offer-like entity.
Silent push notification failures: FCM tokens fail and expire. If you don't log and handle these failures, drivers gradually stop receiving notifications and never tell you why.
Acknowledgment after user action: Waiting for the driver to accept or reject the offer before acknowledging is too late. Acknowledge on receipt, not on action.

Key takeaways

Design dispatch notifications assuming the WebSocket will disconnect at the worst moment.
Push notifications are not a luxury—they're your primary reliability mechanism for time-sensitive alerts.
Explicit acknowledgment from the client tells your server to stop retrying and cancel fallback deliveries.
Automatic expiry prevents stale offers from lingering and protects your database from unbounded growth.
Multi-channel delivery with deduplication (WebSocket + push + retry) is the standard in production ride-hailing and dispatch systems.
Monitor latency, retry exhaustion, and token failures—these metrics reveal gaps in your notification reliability before drivers complain.

Lessons from dispatch notifications in Kafka & WebSockets