Rotate database credentials: AWS lessons from production

Hardcoded database passwords are one of the easiest attack vectors to exploit, yet I still see production systems with credentials baked into configuration files or environment variables that rarely change. Rotate database credentials using AWS Secrets Manager and RDS, and you eliminate the manual overhead while dramatically reducing the blast radius if a secret ever leaks.

Why Static Passwords Are a Liability

Every day that a database password remains unchanged increases the window of exposure. If someone gains access to your codebase, a CI/CD log, or a team member's laptop, that credential is live indefinitely—unless you rotate it manually, which most teams forget to do or skip altogether. I've seen engineering teams go years without rotating a single RDS password.

The real cost isn't just the risk of unauthorized access; it's the operational burden. When you do discover a leaked credential, you have to:

• Identify every service and application using that credential
• Coordinate a simultaneous password change across all consumers
• Handle the inevitable downtime and failed connections
• Audit logs to see if the compromised credential was actually used

Automated rotation eliminates this problem by changing your database password on a schedule—typically every 30 days—and ensuring your applications always fetch the current secret before connecting. No coordination needed, no manual steps, no excuses.

How Secrets Manager Rotates RDS Credentials

AWS Secrets Manager handles the heavy lifting. You enable automatic rotation on your secret, and Secrets Manager orchestrates the process:

1. Create a new temporary password in Secrets Manager
2. Update the RDS database user with the new password
3. Test the connection using the new credential
4. Store the new secret version in Secrets Manager
5. Mark the old version as deprecated

The best part: this happens without taking down your database or interrupting existing connections. RDS allows multiple valid passwords for the same user during the rotation window, so in-flight queries keep working while new connections use the fresh credential.

To set this up, you need a Lambda function that Secrets Manager invokes to perform the actual rotation. AWS provides a template, but here's what the Lambda typically does:

import json
import boto3
import psycopg2
import os

secrets_client = boto3.client('secretsmanager')
rds_client = boto3.client('rds')

def lambda_handler(event, context):
    secret_id = event['ClientRequestToken']
    service_secret = event['SecretId']
    step = event['ClientRequestToken']
    
    # Retrieve current secret
    secret_response = secrets_client.get_secret_value(SecretId=service_secret)
    secret = json.loads(secret_response['SecretString'])
    
    # Step 1: Create new secret version
    if step == 'createSecret':
        new_password = secrets_client.get_random_password(PasswordLength=32)
        new_secret = secret.copy()
        new_secret['password'] = new_password['RandomPassword']
        
        secrets_client.put_secret_value(
            SecretId=service_secret,
            ClientRequestToken=secret_id,
            Secret=json.dumps(new_secret),
            VersionStages=['AWSPENDING']
        )
    
    # Step 2: Set secret in database
    elif step == 'setSecret':
        conn = psycopg2.connect(
            host=secret['host'],
            user=secret['username'],
            password=secret['password'],
            database=secret['dbname']
        )
        cursor = conn.cursor()
        cursor.execute(
            f"ALTER USER {secret['username']} WITH PASSWORD %s",
            (new_secret['password'],)
        )
        conn.commit()
        conn.close()
    
    # Step 3: Test secret
    elif step == 'testSecret':
        conn = psycopg2.connect(
            host=new_secret['host'],
            user=new_secret['username'],
            password=new_secret['password'],
            database=new_secret['dbname']
        )
        conn.close()
    
    # Step 4: Finish secret
    elif step == 'finishSecret':
        secrets_client.update_secret_version_stage(
            SecretId=service_secret,
            VersionStage='AWSCURRENT',
            MoveToVersionId=secret_id
        )
    
    return {'statusCode': 200}

In practice, you can use the AWS Secrets Manager rotation templates for RDS databases—they handle the boilerplate. The key is that you attach this Lambda to your secret with a rotation rule (e.g., rotate every 30 days), and Secrets Manager handles the orchestration.

Apps Fetch the Current Secret at Startup

Your application never hardcodes or caches a password long-term. Instead, it retrieves the secret from Secrets Manager when it starts up, creates a connection pool, and uses that pool for the lifetime of the app process. When the pod or instance restarts (which should happen regularly with your deployment pipeline), it fetches the fresh credential.

Here's a practical Node.js example using a connection pool:

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
import { Pool } from 'pg';

const secretsManager = new SecretsManagerClient({ region: 'us-east-1' });

async function getDbSecret(secretName) {
    const command = new GetSecretValueCommand({ SecretId: secretName });
    const response = await secretsManager.send(command);
    return JSON.parse(response.SecretString);
}

let dbPool;

export async function initializeDatabase() {
    const secret = await getDbSecret('prod/myapp/postgres');
    
    dbPool = new Pool({
        host: secret.host,
        port: secret.port,
        user: secret.username,
        password: secret.password,
        database: secret.dbname,
        max: 20,
        idleTimeoutMillis: 30000,
        connectionTimeoutMillis: 2000,
    });
    
    console.log('Database pool initialized with rotated credentials');
    return dbPool;
}

export function getDb() {
    if (!dbPool) {
        throw new Error('Database not initialized');
    }
    return dbPool;
}

In your application startup (in main.ts or your entry point), call initializeDatabase() before your server binds to a port. This ensures you have a valid credential before accepting traffic. If the secret fetch fails, the pod fails to start, and your orchestration layer (Kubernetes, ECS) restarts it—the same way it would handle a startup crash.

This approach is clean because your app logic never touches credential management; the pool handles it transparently. A query just calls getDb().query(sql, params) like normal.

Use Caching Clients to Reduce Secrets Manager Calls

Calling Secrets Manager on every request is wasteful and slow. Each call incurs network latency and API throttling. Instead, cache the secret value in memory and refresh it periodically or on error.

Here's a production-grade caching wrapper:

class CachedSecretsManager {
    constructor(region = 'us-east-1', cacheTtlSeconds = 3600) {
        this.client = new SecretsManagerClient({ region });
        this.cache = new Map();
        this.cacheTtlSeconds = cacheTtlSeconds;
    }
    
    async getSecret(secretName) {
        const cached = this.cache.get(secretName);
        
        if (cached && Date.now() - cached.timestamp < this.cacheTtlSeconds * 1000) {
            return cached.value;
        }
        
        try {
            const command = new GetSecretValueCommand({ SecretId: secretName });
            const response = await this.client.send(command);
            const secret = JSON.parse(response.SecretString);
            
            this.cache.set(secretName, {
                value: secret,
                timestamp: Date.now(),
            });
            
            return secret;
        } catch (error) {
            // If fetch fails, fall back to stale cache if available
            if (cached) {
                console.warn(`Failed to fetch secret ${secretName}, using stale cache`, error);
                return cached.value;
            }
            throw error;
        }
    }
    
    invalidate(secretName) {
        this.cache.delete(secretName);
    }
}

export const secretsClient = new CachedSecretsManager('us-east-1', 3600);

With this caching layer, the first call to initializeDatabase() fetches from Secrets Manager, but subsequent app restarts within an hour hit memory. More importantly, if Secrets Manager is temporarily unavailable, your app doesn't fail immediately—it falls back to the cached secret, giving you resilience during AWS service degradation.

The TTL of 3600 seconds (one hour) balances freshness and availability. If you rotate credentials every 30 days, a one-hour cache misses exactly zero rotations in practice. You could even push it to 24 hours safely.

Test Rotation in Staging First

Before enabling rotation in production, test it end-to-end in a staging environment that mimics your prod setup. Deploy the same RDS instance type, same application code, same Lambda function, and trigger a manual rotation to verify the entire flow.

What to check during a test rotation:

• Lambda execution: Does the rotation Lambda invoke successfully? Check CloudWatch logs for errors.
• Database connectivity: Can the app connect before rotation? After rotation?
• In-flight queries: If you have long-running queries, do they complete without interruption?
• Cache behavior: Does your caching client properly invalidate and refresh after rotation?
• Monitoring and alerts: Do your connection pool metrics stay healthy? Are there any spike in error logs?

To trigger a manual rotation in the AWS Console:

Secrets Manager → Select your secret → Rotation → Rotate secret immediately

Or via the CLI:

aws secretsmanager rotate-secret \
    --secret-id prod/myapp/postgres \
    --rotation-rules AutomaticallyAfterDays=30

Watch the rotation step through (createSecret → setSecret → testSecret → finishSecret) in the Secrets Manager console. Once you confirm the rotation completes and your app continues to function, you can enable automatic rotation on a schedule.

After rotating, restart one of your app instances manually and verify it successfully fetches the new secret and connects to the database. This proves your startup logic is sound.

Common Mistakes

• Fetching secrets on every request: This tanks performance and hits API quotas. Fetch at startup and cache.
• Not testing rotation before enabling in production: Rotation logic bugs surface immediately in production with downtime. Test in staging.
• Forgetting to grant the rotation Lambda the right IAM permissions: The Lambda needs permission to call RDS, read/write secrets, and modify database users. Missing permissions cause silent rotation failures.
• Using a short TTL cache: A five-minute cache defeats the purpose and causes unnecessary Secrets Manager calls. Use at least one hour unless you have a specific compliance requirement.
• Not monitoring rotation failures: Enable CloudWatch alarms on Lambda errors or RotationFailed events. A failed rotation leaves you with a stale secret.
• Rotating credentials on a running instance without a connection pool: Direct connections will fail mid-rotation. Always use connection pooling so old connections drain gracefully while new ones pick up the new credential.

Key Takeaways

• Static database passwords are a liability; automate rotation every 30 days using Secrets Manager and RDS.
• Your app fetches the secret at startup and creates a connection pool; it never hardcodes or caches credentials long-term.
• Implement a caching wrapper around the Secrets Manager client to reduce API calls and add resilience to transient failures.
• Test rotation end-to-end in staging before enabling automatic rotation in production.
• Monitor rotation Lambda execution and set up alerts for rotation failures so you catch issues early.
• Use connection pooling to ensure old connections drain gracefully during credential rotation without dropping traffic.