Idempotent consumers in Java: what developers should know

Introduction

Distributed systems never guarantee exactly-once delivery—they guarantee at-least-once. That means your consumers will see duplicate messages, and the difference between a robust system and a production nightmare often comes down to whether you've planned for them. Understanding idempotent consumers in Java is essential for anyone building event-driven systems with Kafka, RabbitMQ, or similar brokers, and the patterns we'll cover today can save you from data corruption, duplicate charges, and frustrated customers at 3 AM.

At-least-once delivery semantics are the industry standard because they're simpler and more reliable than alternatives. Your broker will retry failed deliveries, network partitions will cause re-delivery, and consumer restarts will cause offset resets. When you consume the same message twice, your application must produce the same observable outcome both times—or your system isn't truly idempotent.

Store processed message IDs with a unique constraint

The most straightforward way to prevent duplicate processing is to record that you've already handled a message. When a duplicate arrives, you can detect and skip it immediately. We do this by storing the message ID (or a derived key) in a database table with a unique constraint.

Here's why this matters: if you process a message, crash, and restart, you need to know whether that message was already committed to your business logic. A simple check against a deduplicated store answers that question in milliseconds.

@Entity
@Table(name = "processed_messages")
public class ProcessedMessage {
    
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    
    @Column(nullable = false, unique = true)
    private String messageId;
    
    @Column(nullable = false)
    private String consumerGroup;
    
    @Column(nullable = false)
    private LocalDateTime processedAt;
    
    // getters, setters
}

@Repository
public interface ProcessedMessageRepository extends JpaRepository<ProcessedMessage, Long> {
    Optional<ProcessedMessage> findByMessageIdAndConsumerGroup(String messageId, String group);
}

@Service
@Transactional
public class OrderService {
    
    @Autowired
    private ProcessedMessageRepository deduplicationRepo;
    
    @Autowired
    private OrderRepository orderRepo;
    
    public void handleOrderEvent(String messageId, OrderEvent event) {
        
        // Check if already processed
        if (deduplicationRepo.findByMessageIdAndConsumerGroup(messageId, "order-consumer").isPresent()) {
            log.info("Message {} already processed, skipping", messageId);
            return;
        }
        
        // Process the event
        Order order = new Order();
        order.setCustomerId(event.getCustomerId());
        order.setAmount(event.getAmount());
        orderRepo.save(order);
        
        // Record that we've processed this message
        ProcessedMessage processed = new ProcessedMessage();
        processed.setMessageId(messageId);
        processed.setConsumerGroup("order-consumer");
        processed.setProcessedAt(LocalDateTime.now());
        deduplicationRepo.save(processed);
    }
}

The unique constraint on messageId is critical. If two threads attempt to process the same message simultaneously, the database will reject the duplicate insert, preventing double-processing even under concurrent load.

Process and record in the same transaction

Storing the message ID is only effective if you also guarantee that your business logic succeeds when the deduplication record is created. This is where transaction boundaries matter.

Consider what happens if you process an order, then crash before recording the message ID. When you restart, you'll see the message again and process it a second time. Conversely, if you record the ID but fail to process the order, the message is lost. The only safe pattern is to wrap both operations in a single database transaction.

@Service
public class KafkaOrderConsumer {
    
    @Autowired
    private OrderService orderService;
    
    @KafkaListener(topics = "orders", groupId = "order-consumer")
    @Transactional  // Single transaction for both operations
    public void consume(ConsumerRecord<String, OrderEvent> record) {
        String messageId = record.key(); // Use Kafka key as idempotency key
        OrderEvent event = record.value();
        
        orderService.handleOrderEvent(messageId, event);
        // Both the order creation and the deduplication record 
        // commit together or roll back together
    }
}

Spring's @Transactional annotation ensures that the database transaction wraps both the order processing logic and the deduplication record insert. If either operation fails, the entire transaction rolls back, and the message will be redelivered.

This approach works well when your business logic is entirely database-driven. For systems that call external APIs or publish to other brokers, you'll need additional safeguards—but the core principle remains: make your critical operations atomic.

Use natural idempotency where possible (upserts)

Not every operation needs an explicit deduplication table. Many events are naturally idempotent if you design them correctly.

Consider a user profile update event. If the same "user profile changed" message arrives twice, you want the same end state both times. An upsert (update if exists, insert if not) is naturally idempotent—running it once or ten times produces the same result.

@Service
public class UserProfileService {
    
    @Autowired
    private UserRepository userRepo;
    
    @Transactional
    public void handleProfileUpdate(String userId, ProfileUpdateEvent event) {
        // Upsert: naturally idempotent
        User user = userRepo.findById(userId)
            .orElseGet(() -> new User(userId));
        
        user.setEmail(event.getEmail());
        user.setName(event.getName());
        user.setUpdatedAt(LocalDateTime.now());
        
        userRepo.save(user);
        // Running this twice with the same event produces the same state
    }
}

@Service
public class InventoryService {
    
    @Autowired
    private InventoryRepository inventoryRepo;
    
    @Transactional
    public void handleInventoryAdjustment(String productId, InventoryEvent event) {
        // NOT naturally idempotent—this is additive
        Product product = inventoryRepo.findByIdForUpdate(productId);
        product.setStock(product.getStock() + event.getDelta());
        inventoryRepo.save(product);
        // If called twice, stock increases twice—BAD
    }
    
    @Transactional
    public void handleInventorySet(String productId, InventoryEvent event) {
        // Naturally idempotent—sets to a specific value
        Product product = inventoryRepo.findByIdForUpdate(productId)
            .orElseGet(() -> new Product(productId));
        product.setStock(event.getNewStock());
        inventoryRepo.save(product);
        // If called twice, stock is still the same—GOOD
    }
}

The key distinction: increments are not idempotent, but set operations are. When designing events and consumers, prefer set-based semantics wherever possible. This dramatically simplifies your architecture and reduces the need for explicit deduplication.

Expire IDs after the retention window

Storing message IDs indefinitely wastes storage and slows down lookups. In practice, you only need to retain them for as long as the broker might re-deliver the message.

For Kafka, this is typically the log retention period (often 7 days in production). After that window, if a message hasn't been delivered, it won't be. You can safely delete its deduplication record.

@Configuration
public class DeduplicationCleanupConfig {
    
    @Autowired
    private ProcessedMessageRepository deduplicationRepo;
    
    @Scheduled(cron = "0 0 2 * * *")  // Run daily at 2 AM
    @Transactional
    public void cleanupExpiredDeduplicationRecords() {
        LocalDateTime cutoff = LocalDateTime.now().minusDays(8);
        
        int deleted = deduplicationRepo.deleteByProcessedAtBefore(cutoff);
        log.info("Cleaned up {} deduplication records older than {}", deleted, cutoff);
    }
}

@Repository
public interface ProcessedMessageRepository extends JpaRepository<ProcessedMessage, Long> {
    Optional<ProcessedMessage> findByMessageIdAndConsumerGroup(String messageId, String group);
    
    @Query("DELETE FROM ProcessedMessage p WHERE p.processedAt < ?1")
    int deleteByProcessedAtBefore(LocalDateTime cutoff);
}

This cleanup task is crucial for systems handling high message volume. Without it, your deduplication table grows unbounded, and queries slow down. With it, you maintain a fixed-size window of deduplication state.

Set the cleanup cutoff slightly beyond your broker's retention period. If Kafka retains 7 days, delete records after 8 days. This gives you a safety margin without wasting storage.

Common mistakes

Forgetting the unique constraint. If your deduplication table has no constraint, concurrent processing of the same message can succeed twice. Always use a unique index on the message ID column.

Checking idempotency outside the transaction. If you check whether a message was processed outside a transaction, then process it inside a new transaction, a race condition can occur between the check and the processing. Check and process within the same transaction.

Mixing deduplication logic with business logic. Store deduplication records in a separate table with a separate repository. This keeps your domain models clean and makes the intent clear.

Never cleaning up old records. If your deduplication table grows to millions of rows, queries slow down and storage becomes expensive. Schedule regular cleanup based on your broker's retention window.

Assuming your business logic is idempotent when it isn't. Before relying on upserts, verify that re-running your operation with the same input produces the same output. Increments, decrements, and time-based calculations are rarely idempotent.

Key takeaways

• At-least-once delivery is the standard; design your consumers to handle duplicates as a normal case, not an edge case.
• Store processed message IDs in a database table with a unique constraint to detect duplicates instantly.
• Wrap both business logic and deduplication recording in a single transaction to ensure atomicity.
• When possible, design operations to be naturally idempotent (upserts, set operations) rather than relying on explicit deduplication.
• Clean up old deduplication records regularly, keeping only as much state as your broker might re-deliver.
• Use Kafka message keys or correlation IDs as your idempotency keys for consistency across your system.
• Test your consumer logic by manually replaying messages and verifying that the outcome doesn't change.