S3 event notifications: a practical guide for developers

Introduction

S3 buckets sit at the heart of most AWS architectures, but they're often treated as passive storage. The moment you need to react to uploads—whether that's generating thumbnails, scanning for malware, or triggering downstream workflows—you need to know about S3 event notifications. This guide covers S3 event notifications: a practical guide for developers who want to automate file processing without polling and keep their systems responsive at scale.

We've handled real production scenarios where a single misplaced filter rule caused millions of unnecessary Lambda invocations, and where missing idempotency checks led to duplicate processing. Let's build systems that actually work.

Understanding S3 event notifications

S3 can emit events when objects are created, deleted, or restored from Glacier. These events contain the bucket name, object key, event time, and metadata. The key advantage over polling is immediate reaction: your handlers fire within seconds of the upload completing.

S3 supports three destination types: Lambda, SQS, and SNS. EventBridge integration is also available as a filter-rich alternative. For most teams, you'll reach for Lambda for lightweight processing or SQS when you need reliable buffering and retry logic.

Event notifications are best-effort delivery. AWS guarantees that at least one notification will be sent for each event, but in rare cases you may receive duplicates. This is why idempotency matters, which we'll cover in depth later.

Triggering Lambda, SQS, or EventBridge

Lambda: Direct and immediate

Lambda is the natural choice for synchronous processing where latency isn't a concern and you have predictable volume. The event payload arrives directly in your handler, and you can process it immediately.

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "eu-west-1",
      "eventTime": "2025-01-15T10:30:45.123Z",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "bucket": {
          "name": "my-uploads",
          "arn": "arn:aws:s3:::my-uploads"
        },
        "object": {
          "key": "documents/invoice-2025-01-15.pdf",
          "size": 256000,
          "eTag": "\"abc123def456\""
        }
      }
    }
  ]
}

A basic Lambda handler in Node.js might look like this:

exports.handler = async (event) => {
  for (const record of event.Records) {
    const bucket = record.s3.bucket.name;
    const key = decodeURIComponent(record.s3.object.key);
    console.log(`Processing ${key} from ${bucket}`);
    
    // Your processing logic here
    // Download from S3, transform, store result
  }
  return { statusCode: 200 };
};

Set up the S3 notification in the AWS console or via Infrastructure as Code. We recommend Terraform or CDK to avoid manual drift.

SQS: Reliable buffering for high volume

When upload traffic spikes or your processor needs retry logic, SQS provides a buffer. S3 sends event notifications to an SQS queue, and your workers consume messages at their own pace. This decoupling prevents Lambda throttling and gives you built-in retry mechanics.

The SQS message body contains the S3 event payload. Your consumer can process it, and if processing fails, the message returns to the queue for retry. After the visibility timeout expires (default 30 seconds), the message reappears for another attempt.

EventBridge: Filtering and routing at scale

EventBridge sits between S3 and your targets, letting you filter events before they trigger expensive Lambda invocations. This is particularly valuable when you only care about specific file types or prefixes.

An EventBridge rule might route only PDFs from a certain prefix to one Lambda, images to a thumbnail generator, and everything else to a DLQ for manual review. The filtering happens before invocation, saving both cost and latency.

Generate thumbnails and scan files

Real-world scenario: Image uploads

Consider an e-commerce platform where users upload product images. When a JPG or PNG lands in S3, you want to generate thumbnails (small, medium, large) and extract metadata like dimensions and format.

Your Lambda receives the S3 event, downloads the original image, uses a library like Sharp to resize it, and uploads the three thumbnail variants back to S3. All of this can happen within the Lambda timeout (we'd allocate 30 seconds and 512 MB memory for this).

const AWS = require('aws-sdk');
const sharp = require('sharp');

const s3 = new AWS.S3();

exports.handler = async (event) => {
  const bucket = event.Records[0].s3.bucket.name;
  const key = decodeURIComponent(event.Records[0].s3.object.key);

  // Download original
  const obj = await s3.getObject({ Bucket: bucket, Key: key }).promise();
  const buffer = obj.Body;

  // Generate thumbnails
  const sizes = [
    { name: 'small', width: 100, height: 100 },
    { name: 'medium', width: 300, height: 300 },
    { name: 'large', width: 800, height: 800 }
  ];

  for (const size of sizes) {
    const thumb = await sharp(buffer)
      .resize(size.width, size.height, { fit: 'contain', background: '#fff' })
      .toBuffer();

    const thumbKey = key.replace(/\.[^.]+$/, `-${size.name}.jpg`);
    await s3.putObject({
      Bucket: bucket,
      Key: thumbKey,
      Body: thumb,
      ContentType: 'image/jpeg'
    }).promise();
  }

  console.log(`Generated thumbnails for ${key}`);
};

File scanning with third-party APIs

For security-sensitive workloads, you'll scan uploads for malware or sensitive data. Services like ClamAV or Rapid7 can be invoked from Lambda. The pattern is the same: receive the event, fetch the file, scan it, and store the result (clean/infected) in a metadata table.

Always add error handling. If scanning fails, you might want to quarantine the file and alert your security team rather than silently ignoring the error.

Filter by prefix and suffix

Avoiding unnecessary invocations

Not every object in your bucket needs processing. User profile pictures and system logs might live in the same bucket but require different handling.

S3 event notification filters let you specify key prefixes and suffixes. Set up one filter for uploads/images/ to trigger the thumbnail generator, and another for uploads/documents/ with suffix .pdf to trigger a document processor.

Filtering at the S3 layer is cheaper than filtering in your Lambda code. If you configure the prefix correctly, you'll avoid invoking Lambda thousands of times per day for objects you don't care about.

Multi-destination routing with EventBridge

If your filtering logic gets complex—e.g., process images only if they're under 5 MB and not in a quarantine folder—use EventBridge rules. You can match on object key patterns and route to different targets.

{
  "Name": "route-s3-events",
  "EventBusName": "default",
  "EventPattern": {
    "source": ["aws.s3"],
    "detail-type": ["Object Created"],
    "detail": {
      "bucket": {
        "name": ["my-uploads"]
      },
      "object": {
        "key": [{
          "prefix": "products/",
          "suffix": ".jpg"
        }]
      }
    }
  },
  "Targets": [
    {
      "Arn": "arn:aws:lambda:eu-west-1:123456789:function:thumbnail-generator",
      "RoleArn": "arn:aws:iam::123456789:role/EventBridgeRole"
    }
  ]
}

Make handlers idempotent

Why idempotency is non-negotiable

S3 event notifications guarantee at-least-once delivery. If a network glitch occurs after your Lambda processes an upload but before S3 marks the notification as sent, you'll receive a duplicate event. Without idempotency, you'll generate thumbnails twice, scan the file twice, or insert duplicate records into your database.

An idempotent handler produces the same result regardless of how many times it runs for the same input. For file processing, this means using the object's ETag or a content hash as a deduplication key.

Implementation pattern

Store a record of processed events in DynamoDB with the bucket name, object key, and ETag as the composite key. Before processing, check if this combination already exists. If it does, return early.

const AWS = require('aws-sdk');
const dynamodb = new AWS.DynamoDB.DocumentClient();

const TABLE = 'S3ProcessedEvents';

exports.handler = async (event) => {
  const bucket = event.Records[0].s3.bucket.name;
  const key = decodeURIComponent(event.Records[0].s3.object.key);
  const eTag = event.Records[0].s3.object.eTag;

  const id = `${bucket}#${key}#${eTag}`;

  // Check if already processed
  const existing = await dynamodb.get({
    TableName: TABLE,
    Key: { id }
  }).promise();

  if (existing.Item) {
    console.log(`Duplicate event for ${key}, skipping`);
    return { statusCode: 200, message: 'skipped' };
  }

  // Process file
  console.log(`Processing ${key}`);
  // ... thumbnail generation, scanning, etc ...

  // Mark as processed
  await dynamodb.put({
    TableName: TABLE,
    Item: {
      id,
      processedAt: new Date().toISOString(),
      ttl: Math.floor(Date.now() / 1000) + (24 * 60 * 60) // 24h TTL
    }
  }).promise();

  return { statusCode: 200, message: 'processed' };
};

The TTL attribute tells DynamoDB to automatically delete old records, keeping storage costs low. A 24-hour TTL covers most duplicate scenarios while being safe to extend if you need longer deduplication windows.

Stateless idempotency with output naming

For some workloads, you can achieve idempotency without external state. When generating thumbnails, always write the output to the same key. If the same file is processed twice, the second write simply overwrites the first with identical data. This only works if your processing is deterministic (same input, identical output).

This approach fails if you're performing state mutations like incrementing a counter or sending a unique notification. Stick with explicit deduplication for those cases.

Common mistakes

• Forgetting to configure IAM permissions: S3 needs permission to invoke your Lambda or write to your SQS queue. The resource policy on the destination must explicitly allow the principal s3.amazonaws.com.
• Using overly broad filters: A prefix filter of / means every object in the bucket triggers an event. This can lead to massive Lambda bills. Be specific about which prefixes and suffixes you actually need to process.
• Ignoring duplicates in production: Assuming you'll never receive duplicate events is a common path to production incidents. Implement idempotency from the start, not as an afterthought.
• Not setting Lambda timeout appropriately: If your handler downloads large files, 3 seconds isn't enough. Calculate realistic timeout values based on file size and network conditions. We typically allocate 30–60 seconds for file processing workloads.
• Treating event notifications as transactional: They're not. If you need guaranteed, ordered processing, use SQS FIFO queues instead, though this comes with throughput tradeoffs.
• Logging sensitive data from the event: S3 events can contain file paths or metadata that might be confidential. Be selective about what you log to CloudWatch.

Key takeaways

• S3 event notifications trigger Lambda, SQS, or EventBridge instantly when objects are created, deleted, or restored—no polling required.
• Choose Lambda for lightweight processing, SQS for high volume or when you need retry logic, and EventBridge for complex filtering.
• Use prefix and suffix filters to avoid invoking handlers for objects you don't care about; filtering at the S3 layer is cheaper than filtering in code.
• Make your handlers idempotent by tracking processed events in DynamoDB or by using deterministic output naming; assume duplicates will occur in production.
• Real-world use cases like thumbnail generation and file scanning follow the same pattern: fetch the object, transform it, store the result, and handle errors gracefully.
• Common pitfalls include missing IAM permissions, overly permissive filters, and ignoring duplicate events—address all three in any production setup.