Envelope encryption

If you've ever wondered how AWS handles encryption at scale without grinding your infrastructure to a halt, envelope encryption is the answer you've been looking for. It's a pattern that lets you leverage the security of AWS Key Management Service (KMS) while keeping your application performant enough to handle thousands of requests per second. In this article, I'll walk you through how envelope encryption works, why it matters for production systems, and how to implement it correctly in your own services.

Why we need envelope encryption

Let's start with a problem I've faced in almost every production system I've built. You need to encrypt sensitive data—customer PII, payment details, API tokens—but sending every encryption and decryption request to KMS creates a bottleneck. KMS has rate limits (though high ones), adds network latency, and costs money per request. In my experience at scale, these costs compound quickly.

Envelope encryption solves this by splitting the work: KMS handles the really sensitive cryptographic material, and your application handles the actual data encryption locally. It's the same principle you'd use if you had to manually encrypt files—you'd use a strong key to encrypt a smaller key, then use that smaller key to encrypt the actual content.

KMS generates a data key

The first step in envelope encryption is asking KMS to generate what's called a data key. This is a unique, randomly generated key that KMS creates specifically for encrypting your data. You never store the plaintext version—that's the critical security property here.

When you call the GenerateDataKey API, KMS returns two things: the plaintext data key (which you'll use immediately) and the encrypted data key (which you'll store). Think of it like KMS saying, "Here's your key in plaintext so you can use it right now, and here's the same key encrypted under my master key so you can store it safely."

Here's how you'd do this in a .NET application:

csharp

using Amazon.KeyManagementService;
using Amazon.KeyManagementService.Model;

var kmsClient = new AmazonKeyManagementServiceClient();

var request = new GenerateDataKeyRequest
{
    KeyId = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012",
    KeySpec = DataKeySpec.AES_256
};

var response = await kmsClient.GenerateDataKeyAsync(request);

byte[] plaintextDataKey = response.Plaintext.ToArray();
byte[] encryptedDataKey = response.CiphertextBlob.ToArray();

I typically use AES_256 for the data key spec. It's strong enough for virtually any use case, and AWS hardware accelerates AES operations, so performance is never an issue.

Encrypt data locally with the data key

Once you have the plaintext data key from KMS, you use it to encrypt your actual data. This happens entirely in your application, with no additional KMS calls. This is where the performance win comes from—you're doing symmetric encryption locally using a standard library, which is blazingly fast.

In my trading platform, we were encrypting millions of transaction records daily. Moving this encryption step locally cut our KMS API call volume by over 90% and shaved 200ms off our average write latency.

Here's a practical example using AES-GCM, which I recommend for authenticated encryption:

csharp

using System.Security.Cryptography;
using System.Text;

public class DataEncryptor
{
    public static (byte[] ciphertext, byte[] iv, byte[] tag) EncryptData(
        byte[] plaintextDataKey, 
        string dataToEncrypt)
    {
        using (var aes = new AesGcm(plaintextDataKey))
        {
            byte[] plaintext = Encoding.UTF8.GetBytes(dataToEncrypt);
            byte[] iv = new byte[12]; // Standard IV size for GCM
            byte[] ciphertext = new byte[plaintext.Length];
            byte[] tag = new byte[16]; // GCM tag
            
            using (var rng = RandomNumberGenerator.Create())
            {
                rng.GetBytes(iv);
            }
            
            aes.Encrypt(iv, plaintext, ciphertext, tag);
            
            return (ciphertext, iv, tag);
        }
    }
}

// Usage
var (ciphertext, iv, tag) = DataEncryptor.EncryptData(plaintextDataKey, "sensitive-data");

Notice that I'm generating a random IV for each encryption. This is crucial—using the same IV with the same key twice completely breaks GCM encryption. I've seen this mistake cause security vulnerabilities in production before.

Store the encrypted data key alongside

After encrypting your data locally, you need to store both the ciphertext and the encrypted data key together. Typically, this means writing them to your database as part of the same record.

The structure I use in production looks something like this:

csharp

public class EncryptedRecord
{
    public string Id { get; set; }
    public byte[] Ciphertext { get; set; }
    public byte[] EncryptedDataKey { get; set; }
    public byte[] Iv { get; set; }
    public byte[] AuthTag { get; set; }
    public DateTime CreatedAt { get; set; }
}

In my e-commerce platform, I store these as binary columns in PostgreSQL. The encrypted data key is small (256 bytes for AES-256), so there's minimal storage overhead. The critical point is that the encrypted data key is useless without KMS—it's not a key to your data, it's just an encrypted blob that only KMS can decrypt.

You can also include metadata like which KMS key version encrypted this data key, which becomes important when you're rotating master keys. I add this to the database record:

csharp

public class EncryptedRecord
{
    public string Id { get; set; }
    public byte[] Ciphertext { get; set; }
    public byte[] EncryptedDataKey { get; set; }
    public byte[] Iv { get; set; }
    public byte[] AuthTag { get; set; }
    public string KmsKeyVersion { get; set; }
    public DateTime CreatedAt { get; set; }
}

Decrypt the data key with KMS when needed

When you need to read the encrypted data, you reverse the process. You take the stored encrypted data key and call KMS to decrypt it. KMS returns the plaintext data key, which you then use to decrypt your actual data locally.

This is the only step that requires a KMS API call. In my ride-hailing platform, we're decrypting thousands of customer records per second, but because we batch these decryption calls and implement caching (I'll explain that next), the actual KMS load stays manageable.

Here's the decryption flow:

csharp

public async Task DecryptRecord(EncryptedRecord record)
{
    // Step 1: Decrypt the data key with KMS
    var decryptRequest = new DecryptRequest
    {
        CiphertextBlob = new MemoryStream(record.EncryptedDataKey)
    };
    
    var decryptResponse = await kmsClient.DecryptAsync(decryptRequest);
    byte[] plaintextDataKey = decryptResponse.Plaintext.ToArray();
    
    // Step 2: Decrypt the data locally
    using (var aes = new AesGcm(plaintextDataKey))
    {
        byte[] plaintext = new byte[record.Ciphertext.Length];
        
        aes.Decrypt(
            record.Iv,
            record.Ciphertext,
            record.AuthTag,
            plaintext
        );
        
        return Encoding.UTF8.GetString(plaintext);
    }
}

One optimization I always implement: cache the plaintext data key in memory for a short period (typically 5-15 minutes depending on sensitivity). This means you only hit KMS once per data key, not once per record encrypted with that key. Just make sure to clear these keys from memory after the TTL expires.

Common mistakes

Reusing IVs: I mentioned this earlier, but it's worth repeating because it's catastrophic. Generate a new random IV for every single encryption operation. No exceptions.

Storing the plaintext data key: Never, ever write the plaintext data key to disk or logs. After you've encrypted your data, overwrite it in memory and let it be garbage collected. In C#, consider using SecureString or clearing arrays immediately after use.

Not validating the authentication tag: If you're using GCM mode (which you should), always validate the tag during decryption. The AesGcm.Decrypt method will throw an exception if the tag doesn't match, which means someone modified the ciphertext. Don't catch and ignore this exception.

Assuming one data key per encryption request: You don't need a new data key for every single record. One data key can encrypt multiple records, which is actually more efficient. Just generate a new IV for each record.

Forgetting about key rotation: KMS keys should be rotated regularly (AWS recommends annually for customer-managed keys). When you rotate your master key, new data keys will be encrypted under the new version automatically, but old data will still decrypt correctly as long as you don't delete the old key version.

Key takeaways

Envelope encryption separates the security of your master key from the performance of your encryption operations
Request data keys from KMS once, use them to encrypt multiple records locally, and cache them briefly to reduce API calls
Always use authenticated encryption (AES-GCM), generate fresh IVs for each operation, and validate authentication tags during decryption
Store the encrypted data key alongside your ciphertext, IV, and authentication tag as a complete encrypted record
This pattern scales to millions of operations per second because the only KMS bottleneck is decrypting data keys, which happens far less frequently than encrypting individual records

Envelope encryption: what every AWS developer should know