Design a news feed: a practical Engineering Craft guide

Design a news feed: a practical Engineering Craft guide

A news feed looks simple from the user's perspective—just a chronological list of posts from people you follow. Behind that interface lies one of the hardest problems in distributed systems: how do you efficiently aggregate posts from thousands of sources, rank them, and serve them in milliseconds at scale? This article walks through the architectural decisions that shape every feed system, from early startups to platforms handling billions of daily active users.

Why the feed problem matters

Feed design forces you to make explicit trade-offs between write latency, read latency, storage, and computation. Every choice has cost implications. A social network or content platform typically faces one of two extreme scenarios: either users create posts rarely but read feeds constantly (high read volume), or users post frequently and followers number in the millions (high write volume). Most real systems sit somewhere in between, which is why the architectural decisions you make early determine whether you scale gracefully or hit a wall.

We've built and scaled feed systems across e-commerce recommendations, ride-hailing driver matching, and social platforms. The lessons apply broadly: understand your write-to-read ratio, know your hotspots, and choose the approach that optimizes for your bottleneck.

Fan-out on write: Push posts into followers' feeds when created

Fan-out on write means that when a user publishes a post, you immediately push it to all their followers' feed lists. Think of it as a write-time fanout: the post gets duplicated across many feed buckets, one per follower.

How it works: A user creates a post. Your service enqueues a job (via Kafka, Bull, or similar) that iterates through all followers and writes the post ID to their feed list (typically a sorted set in Redis or a table row in your database). By the time the follower refreshes their feed, the post is already there.

Real-world scenario: A Twitter-like platform with 10 million daily active users. Most users have fewer than 1,000 followers. When a celebrity posts, you fire off a job that writes to millions of follower feeds in the background. The follower who refreshes their feed moments later gets an instant read.

Advantages:

• Feed reads are blazingly fast—you just fetch pre-computed feed lists from cache or a database index.
• No complex ranking or sorting at read time; you already have the feed ordered.
• Simple to implement and debug: each user has a deterministic feed location.

Disadvantages:

• High write amplification. A user with 100,000 followers triggers 100,000 writes on every post.
• Celebrity accounts (millions of followers) create write storms that can overload your queue and storage.
• Deletes and edits must propagate to all follower feeds, which is expensive.

Implementation sketch:

// When a post is created
async function createPost(userId: string, content: string) {
  const post = await db.posts.insert({
    userId,
    content,
    createdAt: new Date(),
  });

  // Fan out to all followers
  const followers = await db.followers.find({ followingId: userId });
  
  for (const follower of followers) {
    await feedQueue.enqueue({
      job: 'add-to-feed',
      postId: post.id,
      userId: follower.followerId,
      timestamp: post.createdAt.getTime(),
    });
  }

  return post;
}

// Worker processes the queue
feedQueue.process(async (job) => {
  const { postId, userId, timestamp } = job.data;
  // Add post to user's feed (Redis sorted set, keyed by user ID)
  await redis.zadd(`feed:${userId}`, timestamp, postId);
});

Fan-out on read: Build the feed when requested

Fan-out on read is the inverse: when a user requests their feed, you query all posts from all accounts they follow and assemble the list on demand. No pre-computation, just aggregation at read time.

How it works: The user opens their feed. Your service looks up all accounts they follow, fetches the latest N posts from each, and merges them into a single sorted list. The response is sent back to the client.

Real-world scenario: A small team collaboration tool where users follow dozens of team channels and teammates. Write volume is high (posts every few minutes), but read patterns are clustered—the same person might request their feed once every hour. You don't want to maintain millions of pre-built feed copies; instead, you build it on demand.

Advantages:

• Trivial write path. A post is inserted once into the database; no fanout queue needed.
• Deletes and edits are immediate; you just update the post record.
• Follows and unfollows take effect immediately without needing to backfill or cleanup feeds.
• No storage overhead for maintaining per-user feed copies.

Disadvantages:

• Read latency can be high. If a user follows 5,000 accounts, you're potentially querying 5,000 tables or indexes.
• Expensive at scale. Each feed read is a fan-out query that hits your database hard.
• Inconsistent performance. A user following popular accounts with millions of posts will have slow feed loads.

Implementation sketch:

// When feed is requested
async function getFeed(userId: string, limit: number = 50) {
  // Get all accounts this user follows
  const following = await db.follows.find({ followerId: userId });
  const followedIds = following.map(f => f.followingId);

  // Fetch recent posts from each
  const postsByAuthor = await Promise.all(
    followedIds.map(authorId =>
      db.posts.find({ userId: authorId }).sort({ createdAt: -1 }).limit(limit)
    )
  );

  // Merge and sort
  const allPosts = postsByAuthor.flat().sort((a, b) => 
    b.createdAt.getTime() - a.createdAt.getTime()
  );

  return allPosts.slice(0, limit);
}

The hybrid: Push for normal users, pull for accounts with millions of followers

The winning architecture in production combines both approaches. You push feeds for normal users but pull for high-follower accounts.

How it works: When a post is created, check the author's follower count. If they have fewer than, say, 100,000 followers, use fan-out on write. If they're a celebrity, treat their posts differently: store the post in a dedicated "celebrity feed" table and merge it at read time using a secondary query.

Real-world scenario: A ride-hailing platform where we need to surface driver promotions to users. Most drivers have zero followers (they post for their own benefit). A few platform accounts have hundreds of thousands of followers. We fan-out on write for the 99.9% of drivers, but pull celebrity driver updates at read time.

Implementation sketch:

const FANOUT_THRESHOLD = 100000;

async function createPost(userId: string, content: string) {
  const post = await db.posts.insert({
    userId,
    content,
    createdAt: new Date(),
  });

  const followerCount = await db.followers.count({ followingId: userId });

  if (followerCount < FANOUT_THRESHOLD) {
    // Fan out on write for normal users
    const followers = await db.followers.find({ followingId: userId });
    for (const follower of followers) {
      await feedQueue.enqueue({
        job: 'add-to-feed',
        postId: post.id,
        userId: follower.followerId,
        timestamp: post.createdAt.getTime(),
      });
    }
  } else {
    // For celebrities, just mark the post as "celebrity"
    await db.posts.update(post.id, { isCelebrity: true });
  }

  return post;
}

async function getFeed(userId: string, limit: number = 50) {
  // Get pre-computed feed for normal users
  const feedItems = await redis.zrevrange(
    `feed:${userId}`,
    0,
    limit * 2, // Fetch extra to account for filtered items
    'WITHSCORES'
  );

  // Also pull in recent posts from celebrity accounts this user follows
  const celebFollows = await db.follows.find({
    followerId: userId,
    isCelebrity: true,
  });
  const celebPosts = await db.posts.find({
    userId: { $in: celebFollows.map(f => f.followingId) },
    isCelebrity: true,
  }).sort({ createdAt: -1 }).limit(20);

  // Merge and re-rank
  const allPosts = [
    ...feedItems.map(id => fetchPostById(id)),
    ...celebPosts,
  ];

  return allPosts.sort((a, b) =>
    b.createdAt.getTime() - a.createdAt.getTime()
  ).slice(0, limit);
}

Ranking and caching: Cache the first page, rank with simple signals before machine learning

Once you've solved the throughput problem, users want to see the most relevant posts first, not just the most recent. Ranking is a separate concern from feed architecture, but the two interact tightly.

Caching strategy: The first page (typically 20–50 posts) of a user's feed should be cached aggressively. Most users only scroll through the first page. Cache it for 1–5 minutes depending on your traffic. Use Redis or similar with a cache key like feed:page1:{userId}. On each new post from a followed account, invalidate that key.

Ranking signals (before machine learning):

• Recency: Newer posts rank higher. Use a decay function so posts older than 24 hours drop off.
• Engagement: Posts with likes, comments, or shares from mutual friends rank higher. This requires denormalization—store engagement counts on the post record or in a secondary index.
• Author relevance: Posts from accounts the user interacts with frequently rank higher. Track interaction frequency (likes, comments, shares) per user pair.
• Content type: Some users prefer video, others text. Store a user profile of content preferences and boost matching types.

Implementation: Rather than full machine learning at read time (expensive), compute a simple ranking score at write time or during a background job. Use a formula like:

// Simplified ranking score
function rankingScore(post: Post, userProfile: UserProfile): number {
  const recencyDecay = Math.exp(-0.0001 * (Date.now() - post.createdAt.getTime()));
  const engagementScore = Math.log(post.likes + post.comments + 1);
  const authorRelevance = userProfile.authorAffinity[post.userId] || 0.1;
  const contentTypeBoost = userProfile.contentTypePrefs[post.type] || 0.5;

  return (
    recencyDecay * 0.4 +
    engagementScore * 0.3 +
    authorRelevance * 0.2 +
    contentTypeBoost * 0.1
  );
}

Cache the top-ranked posts per user and update them every few minutes via a background job. This balances freshness and performance.

Common mistakes

• Choosing one approach and never revisiting it: Your platform's user and follower distribution will change. A system that works for 100,000 users may fail at 10 million. Monitor your write and read latencies; if reads start degrading, it's time to reconsider the hybrid approach.
• Ignoring the celebrity problem: A single user with 10 million followers can bring down a pure fan-out-on-write system. Always have a threshold and a fallback strategy.
• Over-ranking too early: Don't build a machine learning ranking pipeline before you have basic caching and simple signals working. Profile your bottleneck first. Often, recency and basic engagement are enough.
• Forgetting about deletes and edits: If you push posts to followers, deletes require pulling them from all follower feeds. Plan for this cost upfront.
• Not measuring cache hit rates: Cache invalidation is hard. Monitor your Redis cache hit rate for the feed namespace. If it's below 70%, your TTL is too aggressive or your invalidation logic is buggy.

Key takeaways

• Fan-out on write optimizes for read latency; fan-out on read optimizes for write latency. Choose based on your traffic pattern.
• The hybrid approach (push for normal users, pull for celebrities) scales to billions of users by isolating high-follower accounts.
• Cache the first page of every user's feed aggressively. Most users never scroll past it.
• Rank with simple signals (recency, engagement, author affinity) before investing in machine learning. Measure the improvement.
• Monitor your system continuously. As your user base and follower distribution evolve, your feed architecture may need to shift.
• Build observability into your feed pipeline. Track queue depth, cache hit rates, p99 read latency, and post fanout duration separately.