Turn on CloudTrail everywhere: a AWS guide for developers

I've been burned by surprise AWS bills, mysterious permission changes, and security incidents that took weeks to trace back to their root cause. The fix? Turn on CloudTrail everywhere. It's your immutable audit log for every API call across every account, and I wish I'd configured it properly on day one of my infrastructure journey.

CloudTrail isn't glamorous. It won't make your application faster or your deployments smoother. But when someone asks "who deleted that database backup?" or "when did this IAM policy change?" at 2 AM, you'll be grateful it's been running quietly in the background, recording everything.

Organization trail for all accounts

If you're managing multiple AWS accounts—whether you have 3 or 30—you need a single, centralized CloudTrail setup that spans your entire organization. This is where AWS Organizations trails come in. Instead of enabling CloudTrail manually in each account and fumbling with cross-account access, you create one organization trail that automatically logs API activity across all member accounts.

Here's why this matters: last year I inherited a multi-account setup where each account had its own CloudTrail bucket. When we needed to investigate a security incident, we had to jump between seven different S3 buckets, seven different log formats, and seven different retention policies. An organization trail would have given us a single source of truth.

To set this up, you'll need AWS Organizations enabled and an account designated as your delegated administrator (or the management account itself can own it). The process looks like this:

1. Create an S3 bucket in your logging account (we'll talk about protecting it next)
2. Create an organization trail from the CloudTrail console, selecting the bucket
3. Enable it for all AWS regions (not optional if you care about compliance)
4. Verify that member accounts can write to the bucket

The CloudTrail console will show you a simple toggle: "Enable for all accounts in my organization." Flip it. Every new account you create will automatically log to this trail without any additional setup.

aws cloudtrail create-organization-trail \
  --name prod-org-trail \
  --s3-bucket-name my-org-cloudtrail-logs \
  --is-multi-region-trail \
  --enable-log-file-validation \
  --organization-trail

One practical note: if you're using Terraform or CloudFormation, define your organization trail in your infrastructure-as-code layer. I've seen teams set it up manually, forget it exists, and then accidentally delete it during a cleanup pass. Code-driven infrastructure forces intentionality.

Store logs in a protected bucket

CloudTrail logs are valuable—they're proof of who did what and when. So of course they need to be protected from deletion, modification, or accidental exposure. This means your CloudTrail S3 bucket should be locked down tighter than your production database.

Here's the minimum configuration:

• Block public access: Enable all four "Block Public Access" settings at the bucket level. This prevents any accidental policy that exposes logs to the internet.
• Enable versioning: Enables S3 Object Lock, which prevents overwriting or deleting objects within a retention period.
• Enable MFA Delete: Requires MFA to permanently delete any object. I've never needed to delete a CloudTrail log, but if I did, I'd want it to require hardware confirmation.
• Restrict IAM permissions: Only CloudTrail's service principal and your logging administrators should be able to write or read from this bucket.
• Enable encryption: Use KMS-managed keys, and restrict key access to your logging account only.

Here's a bucket policy that enforces this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailAclCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::my-org-cloudtrail-logs"
    },
    {
      "Sid": "AWSCloudTrailWrite",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-org-cloudtrail-logs/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

I also recommend setting a lifecycle policy to move logs older than 90 days to Glacier and delete them after 7 years. CloudTrail logs are verbose—a production account can generate tens of GB per day. Glacier keeps costs reasonable while maintaining your audit trail.

Enable log file validation

CloudTrail can digitally sign every log file it writes using SHA-256 hashing. This means if someone (including AWS staff) modifies a log file after it's written, the validation will detect it. For compliance audits, this feature is often mandatory. For peace of mind, it's essential.

Enabling it is one checkbox in the CloudTrail console, or one boolean in Terraform:

aws cloudtrail create-organization-trail \
  --name prod-org-trail \
  --s3-bucket-name my-org-cloudtrail-logs \
  --is-multi-region-trail \
  --enable-log-file-validation

Once enabled, CloudTrail generates a digest file every hour containing hashes of all logs written in that period. These digest files are also hashed and chained, so tampering with one digest file would invalidate all subsequent ones. It's like a blockchain for audit logs.

To verify a log file hasn't been tampered with, you can use the CloudTrail console's validation tool, or run this:

aws cloudtrail validate-logs \
  --trail-name prod-org-trail \
  --start-time 2024-01-15T00:00:00Z \
  --end-time 2024-01-16T00:00:00Z

I've never had to validate logs in an active incident, but I've run this check during security audits, and it passes every time. It's the kind of feature you enable and forget about until you actually need it.

Alert on sensitive actions like root logins

Having logs is one thing. Acting on them is another. CloudTrail alone doesn't alert you—it just records. You need EventBridge and SNS to turn those logs into actionable notifications.

A root account login is a red flag. Root should never be used for day-to-day operations; if you see a root login, something is wrong. Same for things like disabling CloudTrail, deleting S3 buckets, or modifying IAM policies for your logging role.

Here's how to set up an alert for root logins:

1. CloudTrail logs the event to CloudWatch Logs
2. EventBridge detects the event based on a pattern
3. EventBridge triggers an SNS notification to your team

First, enable CloudTrail to log to CloudWatch Logs:

aws cloudtrail put-event-selectors \
  --trail-name prod-org-trail \
  --cloudwatch-logs-log-group-arn arn:aws:logs:us-east-1:ACCOUNT:log-group:/aws/cloudtrail:* \
  --cloudwatch-logs-role-arn arn:aws:iam::ACCOUNT:role/CloudTrailCloudWatchLogsRole

Then create an EventBridge rule to catch root logins:

aws events put-rule \
  --name detect-root-login \
  --event-pattern '{
    "detail": {
      "userIdentity": {
        "type": ["Root"],
        "principalId": ["AIDAI*"]
      },
      "eventName": ["ConsoleLogin"],
      "eventSource": ["signin.amazonaws.com"]
    }
  }' \
  --state ENABLED

Wire that rule to an SNS topic so your team gets a text message or Slack notification immediately. I'd also add rules for:

• CloudTrail being disabled or modified
• S3 bucket policies being changed on your logging bucket
• IAM policies for your logging role being altered
• KMS key permissions being modified

These aren't just security best practices—they're sanity checks. If someone changes your logging infrastructure, you need to know within minutes, not days.

Common mistakes

• Enabling CloudTrail only in your home region: API calls happen everywhere. An attacker could spin up resources in a region you're not monitoring. Always enable multi-region trails.
• Forgetting about log retention: Logs pile up fast. Set a lifecycle policy to move old logs to Glacier and delete them after your compliance period. Otherwise your S3 bill will creep up.
• Not protecting the logging bucket: Your CloudTrail bucket is a prize target. If someone compromises it, they can erase the evidence of their actions. Treat it like Fort Knox.
• Creating the trail but not starting it: I've actually seen this. The trail was created but never enabled. For months, zero logs. Check the "Logging" column in the CloudTrail console.
• Alerting on everything: If you alert your team on every API call, they'll ignore the alerts. Focus on high-risk actions: root logins, policy changes, deletion events.

Key takeaways

• Set up an organization trail to log all API activity across all accounts in one place, not scattered buckets.
• Protect your CloudTrail bucket with versioning, MFA Delete, block public access, and encryption. It's your audit log—treat it like evidence.
• Enable log file validation so you can prove logs haven't been tampered with during security audits or incident investigations.
• Use EventBridge and SNS to alert on sensitive actions like root logins, CloudTrail disablement, and policy changes. Logs are useless if you don't act on them.
• Test your setup: verify logs are being written, validation is working, and alerts are firing. Don't discover CloudTrail is broken during an incident.