CloudWatch logs, metrics and alarms in AWS for Developers

After 60 days of building on AWS, you've probably deployed Lambda functions, pushed data through SQS, or scaled EC2 instances. But if you can't see what's happening inside those services when things go wrong, you're flying blind. CloudWatch logs, metrics and alarms are the observability backbone of AWS—they let you centralize logs with retention policies, collect built-in and custom metrics, and notify your team when thresholds breach or anomalies emerge.

Logs — Centralize application logs with retention policies

Every application generates logs. In a production environment, those logs are gold when debugging fires at 2 AM. CloudWatch Logs is where they live on AWS, and it integrates seamlessly with almost every service: Lambda automatically sends logs there, ECS containers can stream to CloudWatch, and your EC2 instances can forward logs via the CloudWatch agent.

I learned early that log management isn't just about collection—it's about retention and cost. By default, logs persist indefinitely, and CloudWatch charges per gigabyte ingested and stored. For a high-volume system, that bill can surprise you.

Let's say you have a NestJS API running in ECS. Here's how I'd set up log groups with a sensible retention policy:

aws logs create-log-group --log-group-name /ecs/api-service

aws logs put-retention-policy \
  --log-group-name /ecs/api-service \
  --retention-in-days 7

That retention policy keeps logs for 7 days, then purges them automatically. For compliance-heavy systems, you might set 30 or 90 days. The point is: be intentional. And if you need archival, use log group subscriptions to ship older logs to S3 before CloudWatch deletes them.

Here's a practical NestJS logger configuration that sends structured logs to CloudWatch:

import { Logger } from '@nestjs/common';
import * as winston from 'winston';
import WinstonCloudWatch from 'winston-cloudwatch';

const logger = winston.createLogger({
  transports: [
    new WinstonCloudWatch({
      logGroupName: '/ecs/api-service',
      logStreamName: `api-${process.env.NODE_ENV}`,
      awsRegion: 'us-east-1',
      messageFormatter: ({ level, message, meta }) =>
        `[${level}] ${message} ${JSON.stringify(meta)}`,
    }),
  ],
});

logger.info('Service started', { port: 3000, env: process.env.NODE_ENV });

Structured logging—JSON with consistent fields—matters more as your system grows. Instead of parsing "User 123 logged in," you log {userId: 123, action: 'login', timestamp}. That's searchable, filterable, and indexable in CloudWatch Insights.

Metrics — Built-in service metrics plus your custom metrics

Metrics are time-series data: CPU usage, request count, database latency. AWS services publish metrics automatically to CloudWatch. An EC2 instance gives you CPU utilization and network throughput out of the box. A Lambda invocation publishes duration and error count. An RDS database streams query latency.

But those built-in metrics only tell part of the story. You need custom metrics for business logic: how many orders were processed, how long the payment gateway took to respond, how many users are currently active.

I'll publish a custom metric from a Lambda function processing orders. Here's a real example:

import { CloudWatchClient, PutMetricDataCommand } from '@aws-sdk/client-cloudwatch';

const cloudwatch = new CloudWatchClient({ region: 'us-east-1' });

export async function handleOrder(event) {
  const startTime = Date.now();
  
  try {
    // Process order
    const orderId = await processPayment(event.amount);
    
    const duration = Date.now() - startTime;
    
    // Publish custom metric
    await cloudwatch.send(new PutMetricDataCommand({
      Namespace: 'MyEcommerce',
      MetricData: [
        {
          MetricName: 'OrderProcessingDuration',
          Value: duration,
          Unit: 'Milliseconds',
          Timestamp: new Date(),
          Dimensions: [
            { Name: 'PaymentMethod', Value: event.paymentMethod },
            { Name: 'Region', Value: process.env.AWS_REGION },
          ],
        },
      ],
    }));
    
    return { orderId, success: true };
  } catch (error) {
    await cloudwatch.send(new PutMetricDataCommand({
      Namespace: 'MyEcommerce',
      MetricData: [
        {
          MetricName: 'OrderProcessingErrors',
          Value: 1,
          Unit: 'Count',
          Timestamp: new Date(),
        },
      ],
    }));
    throw error;
  }
}

Notice I'm using dimensions: PaymentMethod and Region. Dimensions let you slice metrics later. You can ask, "What's the average order processing time for credit card payments versus PayPal?" That's powerful for root-cause analysis.

Metrics are aggregated: a single Lambda function might execute 1,000 times a minute, but CloudWatch rolls that into one data point per minute (average, max, sum, count). That's why it's cheap—you're not storing 1,000 individual invocations, just the aggregates. For detailed analysis, use X-Ray or structured logs.

Alarms — Notify on thresholds and anomalies

Metrics without alerting are just history. An alarm watches a metric and triggers an action when thresholds breach. That action might be sending an SNS notification to your Slack channel, invoking a Lambda to auto-scale resources, or paging an engineer.

Let's set up an alarm for the order processing duration metric we just created. If orders are taking longer than 5 seconds on average, we want to know:

aws cloudwatch put-metric-alarm \
  --alarm-name OrderProcessingLatencyHigh \
  --alarm-description "Alert when order processing > 5s" \
  --metric-name OrderProcessingDuration \
  --namespace MyEcommerce \
  --statistic Average \
  --period 300 \
  --threshold 5000 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:alerts

This alarm evaluates every 5 minutes (period 300 seconds). If the average duration exceeds 5 seconds for 2 consecutive periods, it fires. The alarm-actions sends an SNS message—you can subscribe a Slack bot, an email, or a PagerDuty integration to that SNS topic.

I also use anomaly detection. Instead of hardcoding a threshold, let CloudWatch learn normal behavior and alert when it deviates:

aws cloudwatch put-metric-alarm \
  --alarm-name OrderProcessingAnomaly \
  --alarm-description "Alert on unusual order processing patterns" \
  --comparison-operator LessThanLowerOrGreaterThanUpperThreshold \
  --evaluation-periods 2 \
  --metrics '[
    {
      "Id": "m1",
      "ReturnData": true,
      "MetricStat": {
        "Metric": {
          "Namespace": "MyEcommerce",
          "MetricName": "OrderProcessingDuration"
        },
        "Period": 300,
        "Stat": "Average"
      }
    },
    {
      "Id": "ad1",
      "Expression": "ANOMALY_DETECTION_BAND(m1, 2)",
      "ReturnData": true
    }
  ]' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:alerts

CloudWatch learns from historical data and detects when current behavior falls outside the expected band. This is valuable for catching subtle degradations that a static threshold would miss.

Dashboards — One view for latency, errors and saturation

Latency, errors, and saturation are the RED metrics (Request rate, Error rate, Duration). A dashboard should surface these at a glance so you don't need to jump between CloudWatch, your APM, and your logs.

I build dashboards as code using CloudFormation or Terraform. Here's a CloudFormation snippet for a simple e-commerce API dashboard:

Resources:
  APIServiceDashboard:
    Type: AWS::CloudWatch::Dashboard
    Properties:
      DashboardName: api-service-overview
      DashboardBody: !Sub |
        {
          "widgets": [
            {
              "type": "metric",
              "properties": {
                "metrics": [
                  [ "AWS/Lambda", "Duration", { "stat": "Average" } ],
                  [ ".", "Errors", { "stat": "Sum" } ],
                  [ "MyEcommerce", "OrderProcessingDuration", { "stat": "Average" } ]
                ],
                "period": 300,
                "stat": "Average",
                "region": "us-east-1",
                "title": "API Performance"
              }
            },
            {
              "type": "log",
              "properties": {
                "query": "fields @timestamp, @message, @duration | filter @message like /ERROR/ | stats count() by @message",
                "region": "us-east-1",
                "title": "Top Errors (Last Hour)"
              }
            }
          ]
        }

The first widget shows latency (Duration) and errors across Lambda and your custom metrics. The second widget runs a CloudWatch Insights query to surface top error messages. In production, I refresh this every 30 seconds and check it during deployments and incidents.

Common mistakes

• Logging everything without structure. Unstructured logs are hard to search and expensive. Use JSON with consistent field names, and only log events that matter—not every database query.
• Setting retention policies too high. Logs cost money. Unless compliance mandates it, 7-14 days is reasonable for operational logs. Archive old logs to S3 if needed.
• Using alarms without runbooks. An alarm firing at 3 AM is useless if your team doesn't know what to do. Document response steps in a runbook and link it from the alarm description.
• Forgetting about metric costs. Custom metrics cost $0.30 per metric per month. If you publish 100 custom metrics, that's $30/month per region. Be selective.
• Building dashboards nobody watches. Dashboards are great, but they're passive. Use alarms to push notifications actively. A dashboard gathering dust isn't observability.
• Not correlating logs, metrics, and traces. When debugging, you need context. Use correlation IDs in logs and traces so you can jump from a high-latency metric to the specific requests that caused it.

Key takeaways

• CloudWatch Logs centralize application output from all services; set retention policies to control costs.
• Built-in metrics are free, but custom metrics let you monitor business logic and service-specific KPIs.
• Alarms watch metrics and trigger actions; use thresholds for known baselines and anomaly detection for learning normal behavior.
• Dashboards aggregate RED metrics (latency, errors, saturation) in one view, but they're most useful when paired with active alarms.
• Structured logging and thoughtful metric design compound over time—invest early in naming conventions and dimensions.
• Observability is a habit: check dashboards during deploys, document alarm runbooks, and use correlation IDs to connect logs, metrics, and traces.