File uploads in Go: a practical guide for developers

File uploads in Go: a practical guide for developers

File uploads are a deceptively complex feature in production systems. Get them wrong, and you'll face security vulnerabilities, memory exhaustion, or corrupted data in your storage layer. This guide walks through the practical patterns we've used across e-commerce platforms, CRM systems, and SaaS applications to handle multipart forms safely and efficiently.

Why file uploads matter in production

Most developers treat file uploads as a simple feature: accept a form, save a file, move on. But in real-world systems, uploads touch security, performance, and reliability. A malicious actor could send gigabytes of data to exhaust your server's memory. An untrusted MIME type could be exploited. Concurrent uploads could overwhelm your local disk or cloud storage rate limits. We'll cover the patterns that actually work.

Limiting upload size with http.MaxBytesReader

The first line of defense is enforcing size limits before the request body is fully read. Go's http.MaxBytesReader stops reading after a specified number of bytes, returning an error if the client sends more. This prevents memory exhaustion and runaway bandwidth.

Here's the pattern we use:

func handleUpload(w http.ResponseWriter, r *http.Request) {
    // Limit the entire request body to 50MB
    r.Body = http.MaxBytesReader(w, r.Body, 50*1024*1024)
    
    // ParseMultipartForm will respect the MaxBytesReader limit
    err := r.ParseMultipartForm(10 * 1024 * 1024) // 10MB in-memory threshold
    if err != nil {
        http.Error(w, "Request too large", http.StatusRequestEntityTooLarge)
        return
    }
    defer r.MultipartForm.RemoveAll()
    
    file, handler, err := r.FormFile("document")
    if err != nil {
        http.Error(w, "No file uploaded", http.StatusBadRequest)
        return
    }
    defer file.Close()
    
    // Process file...
}

Two parameters matter here. The 50MB limit on the entire request body is your global safety valve—this is what http.MaxBytesReader enforces. The 10MB parameter to ParseMultipartForm is different: it controls how much form data stays in memory before spilling to disk. Files smaller than 10MB go into memory; larger ones are stored temporarily on disk. This is a reasonable middle ground.

When a client exceeds the limit, they get an HTTP 413 Payload Too Large response. No crash, no memory spike. The connection is closed cleanly.

Detecting content type from bytes, not headers

Never trust the Content-Type header sent by the client. In a travel platform, we once had a user upload an executable file with a fake image MIME type. The header can be spoofed in seconds. Real validation comes from inspecting the file's magic bytes.

Go's net/http package includes DetectContentType, which reads the first 512 bytes and identifies common formats:

func validateContentType(file multipart.File) (string, error) {
    // Read the first 512 bytes to detect type
    buffer := make([]byte, 512)
    n, err := file.Read(buffer)
    if err != nil && err != io.EOF {
        return "", err
    }
    
    // Reset file pointer for later processing
    file.Seek(0, 0)
    
    // Detect actual type from bytes
    detectedType := http.DetectContentType(buffer[:n])
    
    // Whitelist allowed types
    allowedTypes := map[string]bool{
        "image/jpeg": true,
        "image/png":  true,
        "image/gif":  true,
        "application/pdf": true,
    }
    
    if !allowedTypes[detectedType] {
        return "", fmt.Errorf("unsupported file type: %s", detectedType)
    }
    
    return detectedType, nil
}

The DetectContentType function covers common formats, but for stricter validation in high-security contexts (think financial platforms), consider using a library like github.com/h2non/filetype. However, for most systems, the built-in detection is sufficient and has no external dependencies.

Always reset the file pointer to position 0 after reading the magic bytes, otherwise downstream processing will start mid-file.

Stream to S3 instead of memory

The most critical pattern for production systems: stream files directly to cloud storage instead of buffering them in memory or on disk. This keeps your process lean and lets S3 handle durability and scalability.

Here's how we structure it with the AWS SDK for Go:

import (
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func uploadToS3(ctx context.Context, client *s3.Client, 
    file multipart.File, filename string, contentType string) error {
    
    uploader := manager.NewUploader(client)
    
    // Stream directly from the multipart reader to S3
    result, err := uploader.Upload(ctx, &s3.PutObjectInput{
        Bucket:      aws.String("my-bucket"),
        Key:         aws.String(filename),
        Body:        file,
        ContentType: aws.String(contentType),
    })
    
    if err != nil {
        return fmt.Errorf("s3 upload failed: %w", err)
    }
    
    log.Printf("Uploaded to %s", result.Location)
    return nil
}

The uploader automatically handles multipart uploads for large files, retries, and buffering. The Body parameter accepts any io.Reader—in this case, the multipart file stream. Nothing hits your local disk or memory.

In a CRM system handling customer documents, this pattern let us process uploads from multiple API regions without any single server becoming a bottleneck. Each request streams directly to S3, and your application layer remains stateless.

Generating random file names

Never save files with the name the client provided. Client-supplied filenames can contain path traversal sequences (../../../etc/passwd) or conflicts. We always generate random names on the server side.

Here's the approach:

import (
    "crypto/rand"
    "encoding/hex"
    "path/filepath"
)

func generateSecureFilename(originalFilename string) string {
    // Extract the extension from the original name
    ext := filepath.Ext(originalFilename)
    
    // Generate 16 random bytes
    randomBytes := make([]byte, 16)
    if _, err := rand.Read(randomBytes); err != nil {
        // Fallback if random fails (shouldn't happen in practice)
        return "upload-" + time.Now().Format("20060102150405")
    }
    
    // Convert to hex string and append extension
    randomName := hex.EncodeToString(randomBytes)
    return randomName + ext
}

This gives you a name like a3f2b8c1e4d6f9a2.pdf—cryptographically random and safe. Store the original filename in your database if you need to show it to users later, but never use it for storage paths.

In an e-commerce platform, this pattern prevented issues with duplicate names and ensured concurrent uploads never collided, even when multiple users uploaded files with the same original filename.

Putting it together: a complete example

Here's how these patterns combine in a real handler:

func handleDocumentUpload(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
    defer cancel()
    
    // Enforce global size limit
    r.Body = http.MaxBytesReader(w, r.Body, 50*1024*1024)
    
    // Parse the multipart form
    err := r.ParseMultipartForm(10 * 1024 * 1024)
    if err != nil {
        http.Error(w, "Request too large", http.StatusRequestEntityTooLarge)
        return
    }
    defer r.MultipartForm.RemoveAll()
    
    // Get the file
    file, handler, err := r.FormFile("document")
    if err != nil {
        http.Error(w, "No file uploaded", http.StatusBadRequest)
        return
    }
    defer file.Close()
    
    // Validate content type from bytes
    contentType, err := validateContentType(file)
    if err != nil {
        http.Error(w, err.Error(), http.StatusUnsupportedMediaType)
        return
    }
    
    // Generate secure filename
    secureFilename := generateSecureFilename(handler.Filename)
    s3Key := fmt.Sprintf("uploads/%s", secureFilename)
    
    // Stream to S3
    s3Client := getS3Client() // Your initialization logic
    err = uploadToS3(ctx, s3Client, file, s3Key, contentType)
    if err != nil {
        http.Error(w, "Upload failed", http.StatusInternalServerError)
        return
    }
    
    // Return success with S3 path
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(map[string]string{
        "path": s3Key,
        "type": contentType,
    })
}

Common mistakes

• Trusting the Content-Type header. We've seen production incidents where this assumption led to security vulnerabilities. Always validate using magic bytes.
• Not resetting the file pointer after reading magic bytes. This causes silent data corruption when the file is written. Always call Seek(0, 0) after reading for detection.
• Buffering large uploads in memory. A 100MB file upload will exhaust memory on a shared resource. Stream to cloud storage instead.
• Forgetting to call RemoveAll() on MultipartForm. This leaves temporary files on disk. Defer the cleanup immediately after ParseMultipartForm.
• No timeout on upload handlers. A slow or stalled upload can tie up a goroutine indefinitely. Use context with a reasonable deadline.
• Assuming the client's filename is safe. Path traversal and unicode tricks can hide in filenames. Generate your own names server-side.

Key takeaways

• Use http.MaxBytesReader to enforce a global size limit and prevent memory exhaustion before the request is fully parsed.
• Validate file type by reading magic bytes with http.DetectContentType, never by trusting the Content-Type header.
• Stream uploads directly to S3 using the manager's uploader to keep your application stateless and lean.
• Generate cryptographically random filenames server-side and never use client-supplied names for storage paths.
• Always defer cleanup of MultipartForm and reset file pointers after reading for validation.
• Add timeouts to upload handlers and handle errors gracefully with appropriate HTTP status codes.