Monitoring serverless functions: lessons from production

Serverless architectures promise simplified operations, but observability remains as critical as ever—sometimes more so. When you lose direct access to infrastructure logs and metrics scatter across managed services, understanding what's happening inside Lambda functions and edge compute becomes a puzzle that costs real money to solve. In this article, we'll share lessons from monitoring serverless functions in production environments, focusing on the strategies that caught real issues before they became outages.

Why serverless monitoring differs from traditional systems

Serverless functions operate under different constraints than containerized or bare-metal workloads. You don't control provisioning, patching, or even some aspects of networking. Cold starts introduce unpredictable latency spikes. Concurrency limits are invisible until you hit them. Billing is consumption-based, which means a monitoring blind spot can directly impact your AWS bill. Traditional application metrics alone aren't enough; you need visibility into the execution environment itself.

CloudWatch is the default sink, but it's designed for breadth rather than depth. To build a comprehensive monitoring picture for serverless, most teams export CloudWatch data to Prometheus and Grafana, where you can correlate metrics, set precise alerts, and maintain a single pane of glass across your infrastructure.

CloudWatch metrics via exporters

AWS publishes Lambda metrics to CloudWatch: invocations, duration, errors, throttles, and concurrent executions. However, pulling these metrics into Prometheus requires an exporter. The most common approach is to use a Prometheus CloudWatch exporter running on an EC2 instance or within a container, or to leverage AWS Lambda Insights combined with custom metric publishing.

A practical setup involves running the prometheus-cloudwatch-exporter as a sidecar or scheduled job. It periodically queries CloudWatch APIs and exposes metrics in Prometheus format. Here's a typical configuration:

region: us-east-1
metrics:
  - aws_namespace: AWS/Lambda
    aws_metric_name: Invocations
    aws_dimensions:
      FunctionName:
        - my-api-handler
        - my-async-processor
    period_seconds: 60
    range_seconds: 300
    set_timestamp: false
    metrics:
      - name: lambda_invocations_total
        help: Total Lambda invocations
        type: counter
  - aws_namespace: AWS/Lambda
    aws_metric_name: Duration
    aws_dimensions:
      FunctionName:
        - my-api-handler
        - my-async-processor
    period_seconds: 60
    range_seconds: 300
    statistics:
      - Average
      - Maximum
    metrics:
      - name: lambda_duration_milliseconds
        help: Lambda execution duration in milliseconds
        type: gauge

Once metrics flow into Prometheus, you can build dashboards that correlate function performance with downstream services, database queries, and user-facing latency. The exporter typically runs every 60 seconds, introducing a small lag—acceptable for most use cases but worth noting when investigating recent incidents.

For functions deployed at edge locations (CloudFront, Lambda@Edge), metrics flow through different CloudWatch namespaces. You'll need separate exporter configurations for AWS/CloudFront and regional Lambda metrics. Ensure your exporter scrape intervals and CloudWatch query windows don't create gaps in your data.

Cold start rates and their impact

A cold start occurs when AWS provisions a new execution environment for your function. The latency penalty ranges from 100ms to several seconds depending on runtime, code size, and memory allocation. In user-facing APIs, cold starts are visible to end users. In asynchronous workloads, they're less visible but still waste money if they happen frequently.

CloudWatch doesn't directly report cold starts, but you can infer them by tracking initialization time within your function code. The standard approach is to log and emit a metric whenever the Lambda handler runs for the first time in an execution context:

import json
import time
from datetime import datetime

start_init = time.time()

def lambda_handler(event, context):
    end_init = time.time()
    init_duration = (end_init - start_init) * 1000
    
    if init_duration > 50:  # threshold indicating cold start
        print(json.dumps({
            "event": "cold_start",
            "duration_ms": init_duration,
            "timestamp": datetime.utcnow().isoformat()
        }))
    
    # Your business logic here
    return {"statusCode": 200, "body": "OK"}

Parse these logs with CloudWatch Insights or a log shipper (Fluent Bit, Logstash) and emit them as Prometheus metrics. Track cold start rate as a percentage of total invocations, broken down by function name and memory allocation. Higher memory allocations reduce cold start duration but increase cost—the tradeoff is worth monitoring.

On the dashboard, cold start percentage should be a primary metric. A sudden spike often indicates a deployment, autoscaling event, or traffic spike that overwhelmed available execution environments. Setting an alert threshold (typically 5–10% for user-facing functions) helps catch performance regressions early.

Duration and errors: the reliability picture

Function duration tells you how long your code actually executes. Errors tell you how many requests failed. Together, they form the backbone of serverless reliability monitoring. CloudWatch already publishes both; the challenge is making them actionable.

For duration, track percentiles—not just averages. A function with average duration of 100ms but p99 of 5 seconds is experiencing tail latency that will impact your users. Configure your exporter to pull Duration metrics with statistics like Average, Maximum, and consider using custom percentile metrics via Lambda Insights.

Here's an example Prometheus query to alert on duration:

histogram_quantile(0.99, 
  rate(lambda_duration_milliseconds_bucket{function_name="my-api-handler"}[5m])
) > 2000

For errors, CloudWatch publishes an Errors metric—count of invocations that failed or threw an exception. Monitor error rate as a percentage of total invocations. A 1% error rate on a function called millions of times is a major incident; the same rate on a rarely-called administrative function may be acceptable. Use function-level filtering and alerting rules.

Correlate errors with other signals: did they coincide with a deployment, a downstream service outage, or a traffic spike? Tagging functions with deployment version, team, and service metadata helps. Many teams inject a git_commit or deployment_id as an environment variable, then include it in structured logs or custom metrics for post-incident correlation.

Throttles and concurrency limits

AWS Lambda enforces account-level and function-level concurrency limits. By default, you have 1,000 concurrent executions per region per account. If you exceed this, Lambda throttles incoming requests—they fail immediately with a 429 error. This is a hard ceiling that surprises many teams in production.

CloudWatch publishes a Throttles metric. Export it to Prometheus and set a critical alert at any non-zero value. A single throttle event indicates you've hit concurrency limits; at that moment, you're dropping traffic. Here's a sample alert:

- alert: LambdaThrottling
  expr: rate(lambda_throttles_total{function_name="my-api-handler"}[1m]) > 0
  for: 1m
  annotations:
    summary: "Lambda function {{ $labels.function_name }} is being throttled"
    description: "Throttles detected in the last minute. Concurrent executions may have exceeded limits."

Beyond throttles, track concurrent executions in real time. CloudWatch publishes ConcurrentExecutions as a gauge. If it's consistently near your limit, you're at risk. Consider provisioning concurrent capacity (reserved concurrency) for critical functions, or implement backpressure mechanisms (queues, rate limiting) upstream.

For functions that serve multiple tenants or internal teams, segment concurrency metrics by tenant or team. A single tenant's spike shouldn't starve others. Some teams use Lambda provisioned concurrency or layer functions behind SQS to smooth spikes and prevent throttles altogether.

Edge functions (Lambda@Edge) have separate, lower concurrency limits. Monitor them separately. A misconfigured CloudFront rule that triggers thousands of edge function invocations can quickly exhaust capacity and degrade your site's performance globally.

Building a comprehensive serverless dashboard

Combine all of these signals into a single Grafana dashboard:

• Top section: Invocation count, error rate, throttle count, cold start rate—the executive summary.
• Middle section: Duration percentiles (p50, p99, p999) and concurrent executions over time—the performance detail.
• Bottom section: Errors and throttles broken down by function, region, or tenant—the drill-down view.

Common mistakes

Relying solely on CloudWatch dashboards: CloudWatch is slow to refresh and lacks powerful query capabilities. Export to Prometheus and Grafana for real-time, queryable metrics.

Ignoring cold starts in production tuning: Cold starts are real, measurable, and fixable. If you see 20% cold start rate, increasing memory allocation or using provisioned concurrency will improve user experience.

Not alerting on throttles: A single throttle is a production incident. Set the threshold to zero and investigate every trigger.

Aggregating across too many functions: Tempting to graph all functions on one metric, but it hides outliers. Always segment by function name at minimum.

Forgetting cost correlation: Serverless billing is based on invocations, duration, and memory. A spike in duration or concurrency directly increases cost. Include cost forecasts in your dashboards.

Key takeaways

• Export CloudWatch metrics to Prometheus using a CloudWatch exporter; CloudWatch alone lacks the query power and performance needed for production observability.
• Measure cold starts via custom instrumentation in your function code, then aggregate the data as metrics; cold start rate is a leading indicator of performance issues.
• Track duration percentiles and error rates at the function level; averages hide tail latency and severity differences.
• Alert immediately on any throttle; it means you're dropping traffic and need to scale up reserved capacity or implement backpressure.
• Correlate metrics with deployments, code changes, and downstream service health to understand root causes quickly.
• Build segmented dashboards that show overall health and allow drill-down into specific functions, regions, or tenants.