Storing files in MongoDB with GridFS in MongoDB & NoSQL

Storing files in MongoDB with GridFS in MongoDB & NoSQL

Most of us reach for object storage like S3 when files enter the picture, but MongoDB offers GridFS as an alternative for cases where your binary data genuinely belongs in the same database as your application state. The question isn't whether GridFS works—it does—but whether it's the right choice for your problem. We'll walk through when to use it, how it actually works under the hood, and why S3 remains the default for most production systems.

GridFS splits files into chunks

MongoDB documents have a size limit of 16 MB. That's large enough for most individual records, but it becomes a problem if you want to store a 500 MB video file or a multi-gigabyte dataset directly alongside your application data. GridFS solves this by splitting files into smaller chunks—typically 255 KB each by default—and storing them as separate documents in a collection.

Here's the conceptual model: when you upload a file through GridFS, the driver creates two collections. The first, usually named fs.files, holds metadata—the filename, upload date, content type, total file size, and a unique ID. The second collection, fs.chunks, holds the actual binary content broken into sequential pieces, each with a reference back to the parent file ID and a chunk index.

In Node.js with the MongoDB driver, uploading a file looks straightforward:

const { MongoClient } = require('mongodb');
const fs = require('fs');

async function uploadFile() {
  const client = new MongoClient('mongodb://localhost:27017');
  const db = client.db('myapp');
  
  const bucket = new MongoClient.GridFSBucket(db);
  const readStream = fs.createReadStream('./large-video.mp4');
  const uploadStream = bucket.openUploadStream('large-video.mp4', {
    contentType: 'video/mp4'
  });
  
  readStream.pipe(uploadStream);
  
  uploadStream.on('finish', () => {
    console.log('File uploaded successfully');
    client.close();
  });
}

uploadFile();

Notice we're streaming the file rather than loading it entirely into memory. The driver chunks it automatically, writes each chunk as a separate document, and updates the metadata record. This streaming approach is critical—it means you can upload terabyte-scale files without running out of RAM.

Retrieving the file follows the same pattern:

async function downloadFile(fileId, outputPath) {
  const client = new MongoClient('mongodb://localhost:27017');
  const db = client.db('myapp');
  
  const bucket = new MongoClient.GridFSBucket(db);
  const downloadStream = bucket.openDownloadStream(fileId);
  const writeStream = fs.createWriteStream(outputPath);
  
  downloadStream.pipe(writeStream);
  
  writeStream.on('finish', () => {
    console.log('File downloaded');
    client.close();
  });
}

downloadFile(objectId, './output.mp4');

GridFS handles all the chunking logic for you. The MongoDB driver queries fs.chunks, fetches chunks in order, reassembles them, and streams the result to your application. From the client's perspective, you're just piping data—the distribution across chunks is transparent.

Useful when files must live with data

GridFS makes sense in a narrower set of scenarios than many assume. The primary use case is when your file's lifecycle is tightly coupled to your application data and you need transactional guarantees across both.

Consider a contract management system. When a user creates a contract record, they upload a PDF. In a traditional setup, you'd store the metadata in MongoDB and the PDF in S3. But what if the user decides to delete the contract? Now you need to:

1. Remove the document from MongoDB
2. Delete the file from S3
3. Handle the case where step 2 fails but step 1 succeeded

With GridFS, both the metadata and the file live in the same MongoDB database. A single transaction can atomically delete the contract record and the associated file chunks. If the transaction rolls back, everything is intact. This eliminates orphaned files and ensures consistency.

Another practical case is temporary data that lives for a short duration. Imagine a data pipeline that processes uploaded CSV files. The file might exist for just a few hours—long enough for workers to process it, then it's discarded. Spinning up S3 for ephemeral data introduces operational overhead. Storing it in GridFS keeps everything in one place during the file's short life, and cleanup is automatic when the document expires (using TTL indexes).

User-generated content in collaborative features also fits this pattern. A real-time document editing platform might store draft documents and their associated media (embedded images, videos) together in MongoDB. Until the document is published, everything is temporary. Publishing moves it to permanent storage (S3) or archives it. Keeping drafts and media together in GridFS simplifies the draft lifecycle.

Geographic distribution is another valid reason. If your MongoDB cluster is already distributed across regions (with multi-region replication), your files automatically replicate alongside your data. S3, by contrast, would require separate replication setup. For scenarios where you're already paying for a geographically distributed MongoDB cluster and file replication is valuable, GridFS leverages that investment.

S3 is usually cheaper and simpler

Despite GridFS's merits, object storage like S3 remains the default for most file workloads, and for good reasons rooted in both economics and operational complexity.

S3 pricing is aggressive. You're paying roughly $0.023 per GB stored per month (US regions), with cheaper rates for older data. A 1 TB file stored for a month costs about $23. Compare that to MongoDB Atlas: the same 1 TB on a cluster sized to handle other application data often costs $300–$500 monthly just for the baseline compute and storage allocation. Even if your cluster has excess capacity, GridFS files consume disk space that directly inflates your monthly bill because storage scales with your instance tier.

Bandwidth differs too. S3's CloudFront integration gives you CDN delivery at scale. Serving large files to a global audience through S3 + CloudFront is cheaper and faster than routing everything through your MongoDB cluster. If your use case involves video streaming or downloadable assets served worldwide, S3 is both more cost-effective and provides better user experience.

Operational simplicity favors S3 as well. MongoDB backups and snapshots now include your file data, making backups larger and slower. Restore times extend because the database has to rebuild indices not just for application data but for file chunks too. S3 has separate backup policies, lifecycle rules for archival, and clear separation of concerns. Your database backup strategy doesn't become entangled with file retention policies.

Scaling behavior differs. MongoDB replication means every file chunk is written to every replica. A 100 MB file creates roughly 400 chunk documents; on a 3-node replica set, that's 1,200 writes. S3 handles durability and distribution internally—you write once, AWS replicates. For high-volume file uploads, GridFS amplifies the write load on your database cluster, whereas S3's architecture is purpose-built for this.

There's also the matter of query integration. While you can query file metadata in MongoDB (finding contracts by upload date, for example), the actual file content isn't queryable. GridFS files are opaque blobs, just like S3 objects. Once you've separated file storage, you might as well use the tool optimized for it.

Store only metadata in MongoDB

The pragmatic middle ground most teams land on is storing file metadata in MongoDB and the actual file in S3. This gives you the best of both worlds: efficient queries and transactional consistency for metadata, cost-effective and scalable file storage for the actual bytes.

Here's a realistic schema for a document management app:

db.documents.insertOne({
  _id: ObjectId(),
  title: 'Q4 2024 Financials',
  userId: ObjectId('...'),
  createdAt: new Date(),
  file: {
    s3Key: 'documents/user-123/q4-2024.pdf',
    bucket: 'myapp-docs',
    size: 2458624,
    contentType: 'application/pdf',
    etag: '"abc123def456"',
    uploadedAt: new Date(),
    url: 'https://d1234.cloudfront.net/documents/user-123/q4-2024.pdf'
  },
  tags: ['financial', 'q4'],
  isPublished: true
});

The metadata—what the file is, who owns it, when it was uploaded, its size—lives in MongoDB. This lets you query documents by publication status, find recent uploads, tag them, and enforce access controls. The actual PDF is in S3, identified by the s3Key.

When a user downloads the document, you check permissions against the MongoDB record, then serve the url from CloudFront. When the user deletes the document, you:

1. Delete the MongoDB document (includes the metadata)
2. Asynchronously delete the S3 object using the stored s3Key

If step 2 fails, you have the key stored and can retry. If the user somehow re-queries and finds no record in MongoDB, they can't access the file anyway because you've deleted the metadata. Consistency is maintained at the application level without needing distributed transactions.

For temporary files or backups that should auto-delete, add a TTL index to the metadata document. When the document expires, it's automatically removed from MongoDB. Separately, configure an S3 lifecycle policy to delete objects older than X days. The two cleanup operations are independent but complementary—if one fails, the other still works.

db.documents.createIndex({ createdAt: 1 }, { expireAfterSeconds: 86400 });

This approach also simplifies permission checking. Want to restrict access to a document? Check MongoDB. Want to share a temporary download link? Generate a signed S3 URL (valid for 1 hour) from the metadata. Want to audit who downloaded what? Log the MongoDB access, not S3 downloads. Your application logic stays in one database.

Common mistakes

Storing large binaries in regular MongoDB documents: Some teams try to base64-encode files and store them directly as document fields. This bypasses GridFS's chunking, keeping the entire file in memory during upload/download and hitting the 16 MB document limit quickly. If you're committed to MongoDB-only storage, always use GridFS.

Forgetting to implement cleanup after deletion: Deleting a GridFS file record doesn't automatically delete its chunks. You must explicitly call bucket.delete(fileId) or manage cleanup through cleanup jobs. Similarly, if you use S3, deleting the MongoDB metadata doesn't delete the S3 object. Always implement explicit cleanup logic.

Not streaming: Loading entire files into memory defeats the purpose of GridFS's chunking. Always use streams for upload and download. Loading a 500 MB file into a Node.js string or buffer will crash the process.

Ignoring access control: GridFS doesn't provide built-in authorization. You must check permissions in your application before opening a download stream. Similarly, with S3, signed URLs are temporary but still need proper scoping. Don't assume storing data somewhere makes it secure.

Underestimating replication costs: On a 3-node replica set, every GridFS file is written three times. Monitor your disk usage and write throughput. For high-volume file uploads, this overhead can become significant.

Key takeaways

• GridFS chunks files into 255 KB segments and stores them alongside metadata in MongoDB collections, making files larger than 16 MB manageable.
• Use GridFS when files are short-lived, tightly coupled to application data, or when you already have a multi-region MongoDB cluster and need file replication built-in.
• S3 is cheaper, simpler operationally, and better for high-volume file serving; it's the default choice for most production systems.
• The practical middle ground is storing file metadata in MongoDB and actual files in S3, giving you queryable metadata and cost-effective, scalable storage.
• Always stream large files; never load them entirely into memory or as base64-encoded strings in documents.
• Implement explicit cleanup logic for deleted files, whether using GridFS or S3—deletion in one system doesn't automatically cascade to the other.
• Remember to enforce access control at the application level; storing files in MongoDB or S3 doesn't automatically secure them.