Production releases as a tech lead: a step-by-step guide

Production releases as a tech lead

I've watched releases go sideways at 2 AM on a Sunday because someone skipped a single checklist item. I've also coordinated deployments across three services with zero customer impact because we treated the release process like a checklist, not a gamble. The difference between those two outcomes comes down to discipline: treating production releases not as a heroic sprint, but as a boring, repeatable, almost mechanical process.

As a tech lead, you own the release. Not just the code—the entire ceremony around getting it live safely. That means migrations, feature flags, dashboards, rollback plans, and timing. It means saying no to Friday afternoon deploys even when business pushes back. It means watching metrics like your life depends on it. In this post, I'll walk through how I keep deployments boring—and why that's the highest compliment you can give a release.

Release checklist: Your contract with production

Every production release I coordinate starts with the same question: what can go wrong? The checklist is the artifact of that paranoia.

I keep a release checklist in our team wiki that gets updated per release. It's not a one-time document; it evolves as we learn from incidents. Here's what mine covers:

• Database migrations: Have they been reviewed by at least one other person? Do they have rollback scripts? Will they block reads or writes during peak traffic?
• Feature flags: Are all new features behind flags? Are we rolling them out incrementally, or flipping them all at once?
• Secrets and config: Have all environment variables been set in staging? Did we rotate any keys or tokens?
• Dependencies and contracts: Do we know which other services this touches? Have we alerted their teams?
• Data validation: If we're changing schemas, have we tested with production-like data volumes?
• Rollback plan: Can we rollback code? Can we rollback migrations? What data might be orphaned?
• Smoke tests: Have we run integration tests in staging that mirror the production deploy?

Let me walk through a concrete example. Last month we deployed a trading platform feature that changed how we store order metadata. The migration needed to:

1. Add the new column (non-blocking)
2. Backfill existing orders in batches (tested to take 45 seconds per 10k rows)
3. Add a uniqueness constraint
4. Remove the old column

On the checklist, we explicitly noted: "Backfill must run during low-traffic window. If it takes longer than estimated, we have a 2-minute window to rollback before the constraint adds lock contention." We also wrote the rollback script upfront:

-- Rollback: order_metadata migration
-- Restores schema to pre-migration state
ALTER TABLE orders DROP CONSTRAINT orders_metadata_unique;
ALTER TABLE orders DROP COLUMN order_metadata_new;

-- Verify count matches backup
SELECT COUNT(*) FROM orders;
-- Expected: 1247834

The rollback script wasn't written during the incident—it was written, tested, and committed alongside the migration. That's the checklist in action: every scenario pre-gamed, every escape hatch mapped.

Deploy at good times

Choosing when to release is one of the most underrated decisions a tech lead makes. I have a simple rule: never release on Friday after 3 PM, never release during peak traffic, and never release alone.

This isn't superstition. It's probability. If something breaks at 11 PM on a Tuesday, we have the whole morning to fix it. If it breaks at 4:30 PM on a Friday, you're coordinating a response across people who want to leave for the weekend, and your on-call rotation starts scrambling.

I build a deploy calendar. For our e-commerce platform, peak traffic runs 7–10 PM and 12–1 PM. For our SaaS CRM, it's 9–11 AM. I know this from dashboards, not guessing. I schedule releases during troughs: 2–3 AM or 2–4 PM on mid-week days when the team is fresh and paying attention.

There's a business pressure to release faster. I acknowledge it and I push back: "We release when it's safe, not when it's urgent. If this is truly urgent, that's a signal our release process is broken." Most of the time, urgency is theater. Occasionally it's real—and in those cases, we still follow the checklist, we just do it during a controlled window rather than ad hoc.

I also never deploy solo. At minimum, there's me and a senior engineer watching together. For major releases (new service, significant data changes), we have three people: one deploying, one watching metrics, one ready to rollback. If I can't find a second person, the release waits.

Watch the dashboards for the first hour

The release isn't done when the code lands. It's done when you've stared at dashboards for 60 minutes and nothing broke.

I watch four categories of metrics during the first hour post-deploy:

1. Error rates (by service, by endpoint, by error type)
2. Latency (p50, p95, p99—not just average)
3. Business metrics (transactions processed, orders placed, API calls, whatever matters for your domain)
4. Infrastructure (CPU, memory, disk I/O, database connections)

Dashboards lie by omission. A service can look healthy overall while a specific endpoint tanks. I set up focused views:

When I deploy, I don't multitask. I sit with the dashboards open. I watch them like a trader watches a position. If I see a metric twitch in the wrong direction, I ask myself: could this be the new code? If the answer is yes, I move to the rollback decision tree immediately.

I've seen teams miss incidents because they deployed and then moved on. A gradual increase in error rate that's not alarming at minute 3 becomes catastrophic by minute 15. I stay present.

Roll back fast: When in doubt, roll back first

The single best decision I make as a tech lead is rolling back quickly. Not hesitantly. Not after 30 minutes of investigation. Fast.

My rollback threshold is simple: if I'm unsure whether the new code caused a metric change, and the impact could be user-facing, I roll back. I investigate after. Here's why: rolling back is cheap. Investigation is expensive. A customer losing an order is catastrophic.

I've rolled back code that wasn't actually broken because the correlation seemed plausible. Each time, investigation later showed the issue was unrelated (a third-party API slowdown, a database lock from a scheduled job). Inconvenient? Yes. Better than shipping the blame-game between teams.

A fast rollback means having the plan ready before you deploy. For code, that's as simple as redeploying the previous Docker image or reverting the feature flag. For databases, it's your rollback migration script, tested and ready to run.

I also have a clear chain of command. If I'm deploying and a metric looks bad, I make the rollback call. I don't wait for consensus. I don't ask the product team. I don't run it up to engineering leadership. I say, "Rolling back," and I do it. Then we all sync up 10 minutes later to talk about what happened.

// Example: Automated rollback detection
// Deployed new order processing service at 2:30 PM
// At 2:35 PM, error rate on /orders spikes from 0.1% to 2.3%
// Decision tree:
// 1. Is the spike within the deployment window? Yes.
// 2. Did we touch order processing? Yes.
// 3. Am I certain the new code is safe? No.
// Action: Rollback immediately. Time to rollback: 90 seconds.
// Investigation: Post-incident review shows a race condition in the new code.
// Outcome: Code is fixed, tested, and re-deployed 2 hours later during low traffic.

Common mistakes

Skipping the checklist on "small" releases. Small releases cause big incidents. I've seen a one-line config change break production. Every release, every time, the checklist gets run.

Assuming the team knows the deploy is happening. I send a Slack message 30 minutes before, tag the relevant teams, and confirm who's watching. "Deploying in 30 minutes" isn't enough. "Deploying at 2:30 PM. On-call is Alice. Fallback is Bob. We're watching the trading service latency dashboard" is clear.

Watching the wrong metrics. I've seen teams obsess over CPU while the database connection pool was maxed out. Know your domain. For a trading platform, latency on the order endpoint is critical. For a CRM, data consistency is critical. Instrument what matters.

Waiting too long to rollback. The sunk cost fallacy kills production. "We've investigated for 10 minutes and it's not the code, so let's wait." Wrong. Roll back. Investigate. Deploy the fix once you're confident.

Not testing the rollback plan. Your rollback script might not work. I've tested every rollback at least once in staging before the release. A database migration rollback that takes 20 minutes and locks tables isn't a rollback—it's a second incident.

Key takeaways

• A release checklist prevents the 2 AM incidents. Run it every time, for every release, no exceptions.
• Deploy during low-traffic windows with at least one other person watching. Friday afternoons are for code review, not production deploys.
• Sit with the dashboards for the first hour. Don't multitask. Watch for correlation between the deploy and any metric changes.
• Roll back fast. If you're unsure, roll back. Investigate after. Speed beats perfection in a production incident.
• Boring releases are successful releases. Treat it as a mechanical process, not an art form.