A post-release monitoring checklist in Engineering Craft

A post-release monitoring checklist

You've just shipped a feature that took weeks to build. Your team is celebrating in Slack. But the first hour after deployment is when most production issues surface, and you won't catch them by watching deployment logs. A post-release monitoring checklist matters most in those crucial first moments, when you can still roll back or hotfix before significant customer impact accumulates.

I learned this the hard way during a trading platform release where we skipped the latency check. Within 30 minutes, our p95 response time doubled because of a subtle N+1 query introduced in the portfolio calculation service. We didn't notice until customer support started getting complaints about "slow order placement." Had we followed a structured post-release monitoring checklist, we'd have caught it in the first five minutes and either rolled back or deployed a fix immediately.

This checklist isn't about guilt-driven heroics or obsessive monitoring dashboards. It's about being systematic during the window when you're most alert and your team is most available.

Error rate compared with before the release

The first metric you should look at is error rate. Open your APM dashboard or log aggregator and compare the error rate in the last 10 minutes against the same metric from the hour before deployment. A meaningful increase—anything beyond normal variance—is a signal to investigate immediately.

I define "meaningful increase" as a spike that exceeds your baseline by more than 2–3 standard deviations or breaks a threshold you've set. On an e-commerce platform, if your baseline error rate is 0.05%, and you see 0.3% after deployment, that's worth pausing and investigating. On a ride-hailing service where you handle constant traffic, even a 0.1% increase can mean hundreds of users experiencing failures.

When comparing error rates, segment by error type and endpoint. A spike in 5xx errors on the checkout API is more urgent than a gradual increase in validation errors. Use your error tracking tool—whether that's Sentry, DataDog, New Relic, or custom logs—to group exceptions by class and frequency.

// Example: Error rate comparison logic
interface ErrorMetrics {
  timestamp: Date;
  totalRequests: number;
  errorCount: number;
  errorRate: number;
}

function shouldInvestigate(before: ErrorMetrics, after: ErrorMetrics): boolean {
  const rateIncrease = after.errorRate - before.errorRate;
  const percentageIncrease = (rateIncrease / before.errorRate) * 100;
  
  // Alert if error rate increases by more than 50% or exceeds 0.2%
  return percentageIncrease > 50 || after.errorRate > 0.002;
}

Pay special attention to new error types you haven't seen before. If your logs suddenly show OutOfMemoryError or DeadlockException, that's a red flag even if the overall error count is small. Filter your errors to show only those that appeared in the last 15 minutes to catch new patterns early.

Latency percentiles on key endpoints

Error rate alone isn't enough. A service can have zero errors but still serve users slowly, and slow responses drive abandonment just as much as failures do. Check latency percentiles—p50, p95, and p99—on your critical user paths.

For an e-commerce platform, critical paths include product search, add-to-cart, checkout initiation, and payment processing. For a CRM, it's contact lookup, deal creation, and email send. For a SaaS dashboard, it's login, data fetch on page load, and export operations. You should have these endpoints already identified in your monitoring setup.

Compare the p95 latency before and after deployment. A 20% increase on p95 is often a sign of something wrong—maybe a new database query, a missing index, or inefficient caching logic. The p99 is even more sensitive to performance regressions because it captures the tail latency that affects your worst-case users.

When you spot a latency increase, drill into the specific service layer. Did the database query time increase? Is the external API call taking longer? Did network I/O get worse? Your APM tool should show you a flame graph or call tree that breaks down where time is spent.

// Example: Latency percentile calculation and alerting
interface LatencyMetric {
  endpoint: string;
  p50: number;
  p95: number;
  p99: number;
  sampleSize: number;
}

function checkLatencyRegression(
  baseline: LatencyMetric,
  current: LatencyMetric,
  threshold: number = 0.25 // 25% increase
): { hasRegression: boolean; severity: 'low' | 'medium' | 'high' } {
  const p95Increase = (current.p95 - baseline.p95) / baseline.p95;
  const p99Increase = (current.p99 - baseline.p99) / baseline.p99;
  
  if (p99Increase > threshold) {
    return { hasRegression: true, severity: 'high' };
  }
  
  if (p95Increase > threshold) {
    return { hasRegression: true, severity: 'medium' };
  }
  
  return { hasRegression: false, severity: 'low' };
}

Don't wait for customers to report slowness. The first hour is when you have the luxury of testing in production with relatively low traffic. Act on latency data immediately.

Business metrics like orders and signups

Technical metrics tell you how your system is performing, but business metrics tell you how your users are behaving. These are your most honest signals that something is wrong.

For an e-commerce platform, track orders created per minute. For a SaaS app, track signups and successful trial activations. For a travel booking service, track searches completed and bookings made. These metrics should be directly tied to revenue or growth.

Pull a 30-minute window after deployment and compare it to the same window from yesterday at the same time. If order volume drops by more than 10%, that's a major red flag. You might have a subtle bug that doesn't trigger errors but breaks the happy path—a missing form field, a broken redirect, or a payment gateway timeout that silently fails instead of retrying.

I once deployed a change that refactored how we calculated shipping costs. Technically, the calculation was correct. No errors. Latency was fine. But customer signups dropped 35% because the new shipping cost estimate was showing much higher than before, even though the final price was the same. The business metric caught it instantly; the technical metrics didn't.

Set up simple queries against your analytics database or event tracking system to pull these numbers in real time. Some platforms use Kafka topics to emit business events that you can aggregate in a monitoring dashboard.

// Example: Business metric tracking
interface BusinessMetrics {
  ordersLastMinute: number;
  signupsLastMinute: number;
  checkoutsStartedLastMinute: number;
  checkoutsCompletedLastMinute: number;
  checkoutAbandonmentRate: number;
}

async function compareBusinessMetrics(
  beforeWindow: BusinessMetrics,
  afterWindow: BusinessMetrics
): Promise {
  const alerts: string[] = [];
  
  const orderDropPercent = (
    (beforeWindow.ordersLastMinute - afterWindow.ordersLastMinute) /
    beforeWindow.ordersLastMinute
  ) * 100;
  
  if (orderDropPercent > 15) {
    alerts.push(`Orders dropped ${orderDropPercent.toFixed(1)}% after deployment`);
  }
  
  if (afterWindow.checkoutAbandonmentRate > beforeWindow.checkoutAbandonmentRate * 1.2) {
    alerts.push('Checkout abandonment rate increased significantly');
  }
  
  return alerts;
}

Business metrics are often overlooked in post-release checklists because they require slightly more infrastructure to track. Invest in that infrastructure early. It will save you from shipping silent failures that bleed revenue.

Logs for new error types

Beyond aggregated error rates, read through your logs manually—yes, manually—for the first 15 minutes after deployment. Set a filter for ERROR, WARN, and FATAL levels, sorted by time descending. Look for patterns you didn't expect.

New error types often appear with low frequency, so they won't trigger an alert based on error rate alone. A single NullPointerException in a rarely-used code path might not move your error rate needle, but it signals a bug that's waiting to bite you at scale.

Use your log aggregation tool to filter by deployment version or tag. This lets you isolate logs from your new deployment and ignore older noise. Group by error class and message to spot clusters.

I look for specific red flags in logs:

• Database connection pool exhaustion: Any log lines mentioning connection timeouts, pool depletion, or "waiting for available connection" suggest you've introduced a connection leak or increased connection demand.
• Memory issues: Lines mentioning heap size, garbage collection pauses longer than expected, or out-of-memory errors are catastrophic signals.
• External service timeouts: If you're now timing out on calls to payment gateways, notification services, or third-party APIs, check whether you changed retry logic or timeout values.
• Deadlocks and lock timeouts: Database deadlock messages in logs mean your schema or transaction logic has introduced contention you didn't expect.
• Authentication or permission errors: Sudden spikes in 403 Forbidden or InvalidToken errors suggest you changed authorization logic or secrets incorrectly.

Create a saved log query or dashboard that shows you these conditions at a glance. Make it part of your post-release ritual to click that dashboard and scan for color changes or anomalies.

// Example: Log parsing for critical error patterns
interface LogAlert {
  pattern: string;
  severity: 'critical' | 'high' | 'medium';
  regex: RegExp;
}

const criticalLogPatterns: LogAlert[] = [
  {
    pattern: 'Connection pool exhausted',
    severity: 'critical',
    regex: /connection pool.*exhausted|no available.*connection/i,
  },
  {
    pattern: 'Out of memory',
    severity: 'critical',
    regex: /OutOfMemory|heap space|GC overhead/i,
  },
  {
    pattern: 'Database deadlock',
    severity: 'high',
    regex: /Deadlock|deadlock detected|Lock wait timeout/i,
  },
  {
    pattern: 'External API timeout',
    severity: 'high',
    regex: /timeout.*gateway|gateway.*timeout|service unavailable/i,
  },
];

The logs are your system telling you what's actually happening, unfiltered by aggregation or sampling. Listen to them.

Common mistakes

Relying only on automated alerts. Alerts are great for catching spike patterns at scale, but they're too blunt for the first hour. A single spike in error rate might be noise, but if you read the actual logs, you'll see it's all the same root cause. Automated alerts often trigger 5–10 minutes after you would have noticed by looking at the dashboard.

Checking metrics too early. If you look at dashboards 10 seconds after deployment, they're still warming up. Request volume might not be high enough to show meaningful signals yet. Wait 2–3 minutes for traffic to normalize and metrics to stabilize before comparing.

Comparing to the wrong baseline. Don't compare to the error rate from last week if your traffic pattern has changed. Compare to the same time window yesterday or 30 minutes before deployment. Seasonal traffic patterns matter.

Ignoring p99 latency. Teams often focus on p95 or average latency because the numbers look better. But p99 is where customer frustration lives. A deployment that makes p99 latency 3x worse is a problem even if p50 is unchanged.

Forgetting about feature flags. If you deployed with feature flags off, your monitoring won't catch issues in the new code path until the flag is turned on. Plan to re-run your post-release checklist again after enabling the flag for users.

Key takeaways

• The first hour after deployment is your window to catch issues before they affect scale. Make post-release monitoring intentional, not reactive.
• Start with error rate compared to your pre-deployment baseline, but dig into error types—new errors matter more than a few extra percentage points.
• Check latency percentiles (especially p95 and p99) on your critical endpoints. Latency increases often signal database or external service problems.
• Track business metrics like orders, signups, and checkout completion. A silent failure that breaks the user journey won't always show up in technical metrics.
• Read your logs manually for the first 15 minutes. Filter by new error types and watch for connection pool issues, memory problems, deadlocks, timeouts, and auth errors.
• Have a runbook ready. If you find a problem, know whether you'll roll back, apply a hotfix, or disable the feature flag. Decide this before you deploy, not after.