100 days of AWS: lessons — practical tips and trade-offs

We've spent 100 days building real systems on AWS, learning what works and what costs you money at 3 AM. This series has taken us from VPCs and IAM to Kubernetes, serverless, and multi-region deployments. Today we're consolidating what actually matters: the patterns, trade-offs, and hard-won lessons that separate production-ready architecture from code that looks good in a demo.

IAM is the foundation — Most security issues trace back to permissions

If there's one thing you take from 100 days on AWS, let it be this: IAM misconfiguration is the root cause of nearly every security incident we've responded to. Not weak passwords. Not unpatched servers. Overly permissive roles, forgotten inline policies, and the gravitational pull toward AdministratorAccess when you're tired at 6 PM.

We've all been there. A developer needs access to S3, so you add s3:* instead of the specific bucket and actions. A Lambda needs to read from DynamoDB, so you grant it full table access instead of just GetItem and Query. Over weeks and months, those shortcuts accumulate into a security posture that looks like Swiss cheese from the inside.

The fix is mechanical but requires discipline. Start with the principle of least privilege, then use policy simulation to validate that your role has exactly what it needs.

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/MyLambdaRole \
  --action-names dynamodb:Query s3:GetObject \
  --resource-arns \
    arn:aws:dynamodb:us-east-1:123456789012:table/Orders \
    arn:aws:s3:::my-bucket/uploads/*

That command tells you whether your role can actually perform the actions you're expecting. Use it before deploying. Better yet, wire it into your CI pipeline so policies are validated automatically.

We recommend building a policy library for common patterns — a read-only S3 bucket role, a Lambda-to-RDS role, an EC2-to-Secrets-Manager role. Version them, document them, and use them as templates. When your junior engineer needs S3 access, they inherit a tested policy instead of inventing one from scratch.

One more thing: rotate your access keys quarterly and delete anything you're not actively using. AWS Access Analyzer will flag over-permissive policies automatically. Use it.

Managed services save teams — Pay AWS to run what isn't your core business

Early in every AWS journey, there's a tempting rabbit hole: running your own Kubernetes cluster, managing your own RDS replica strategy, building a custom caching layer with Redis. It feels like you're being scrappy and cost-conscious. You're usually just accumulating operational debt.

Managed services exist because AWS has already solved the hard problems. RDS handles patching, failover, and backups. ElastiCache handles memory management and eviction policies. ECS Fargate handles container orchestration without making you manage nodes. Lambda handles scaling, concurrency, and the actual server infrastructure. Your job is to use them.

Here's a real scenario: a team built a notification system using an EC2 instance running Node.js with a queue they managed themselves. They wrote alerting. They wrote scaling logic. They wrote remediation playbooks for when the queue fell behind. Eighteen months later, they switched to SQS and Lambda. The Lambda code is simpler, the operational load dropped to nearly zero, and the cost went down. They freed up capacity to build features instead of managing infrastructure.

That trade-off isn't always free. Managed services have limitations — you can't tune every knob, you inherit AWS's scaling characteristics, and sometimes you're locked into their design patterns. But those constraints are features, not bugs. They force you into architectures that actually scale.

The rule of thumb: if it's not your competitive advantage, let AWS run it. Database backups? AWS. TLS certificate renewal? AWS. Message queueing? AWS. Kubernetes cluster lifecycle management? That's the one place we still see value in managing it yourself if you're already running containers at scale. Even then, consider EKS.

Cost is architecture — Every design choice shows up on the bill

AWS pricing is intentionally granular. You pay for compute, storage, data transfer, API calls, and sometimes for the privilege of not paying more. That granularity is a feature. It forces you to think about cost at design time instead of being surprised at the end of the month.

Consider data transfer. Moving data between Availability Zones costs money. Moving data across regions costs more. Moving data out of AWS costs the most. We've seen teams architect around this without even realizing it: they consolidated databases to avoid cross-AZ traffic, they built read replicas in the same region, they cached aggressively to reduce egress. Those decisions happened because the cost structure incentivized them.

Here's a concrete example. A SaaS product generated daily reports, dumped them to S3, and emailed download links to customers. It worked. The problem: every customer clicking the link triggered a DynamoDB query to validate the download token, which then fetched the file from S3. At scale, that's thousands of DynamoDB reads per day, plus S3 request costs, plus data transfer costs if the customer was outside AWS. The bill was high relative to the business value.

The fix required architectural thinking. They generated signed URLs so S3 could validate the download without hitting DynamoDB. They enabled S3 transfer acceleration for international customers. They added CloudFront to cache reports. The same feature, dramatically lower cost. That's cost-aware architecture.

Get Cost Explorer running from day one. Set budget alerts. Tag your resources by team, application, and environment so you can slice the bill however you need to. When something looks expensive, investigate. Sometimes it's a legitimate cost. Sometimes it's a missing index or a caching opportunity.

Automate everything — IaC and CI/CD make AWS predictable

Manual AWS console clicks don't scale. The moment you have two environments, or two regions, or two teams, you'll be copying and pasting configurations, introducing inconsistencies, and eventually breaking something in production because prod looked slightly different from staging.

Infrastructure as Code solves this. CloudFormation, Terraform, CDK — pick one and commit to it. The syntax varies but the principle is identical: your infrastructure lives in version control, changes go through code review, and deployments are repeatable and auditable.

We prefer Terraform for multi-cloud flexibility and CDK for teams comfortable with Python or TypeScript. But we've seen CloudFormation work beautifully in large organizations. The important thing is picking one, learning it well, and using it for everything except throwaway experiments.

Here's what that looks like in practice:

# Developers work on a feature branch
git checkout -b feature/add-analytics-pipeline

# They write Terraform that adds an S3 bucket, Lambda, and Glue job
# terraform/prod.tf gets the new resources

# CI runs validation and a plan
terraform validate
terraform plan -out=tfplan

# Code review happens on the plan output
# Reviewer sees exactly what resources will be created

# Merge, CI applies the plan to prod
# If something breaks, git revert undoes it

The discipline here matters. Treat infrastructure changes like you treat code changes: small, reviewable, testable. A single pull request should do one thing. Deploy frequently instead of waiting for a big bang migration.

Wire it all into CI/CD. GitHub Actions, GitLab CI, CodePipeline — doesn't matter. When code hits main, infrastructure and application updates should flow through your pipeline automatically. When that pipeline is well-designed, deploying to production becomes boring. Boring is good.

Common mistakes

• Assuming managed services are always cheaper. They reduce operational cost, but compute costs the same whether it's EC2 or Fargate. Compare actual bill numbers, not just feel-good narratives.
• Forgetting about cross-region replication until disaster strikes. RTO and RPO aren't theoretical. Calculate them for your use case and architect accordingly before you need it.
• Not monitoring permissions until after an incident. Use AWS Config, Access Analyzer, and CloudTrail to surface permission issues continuously, not in the postmortem.
• Writing one-off scripts instead of Terraform modules. The script works for three weeks, then you need it in another region and it breaks because assumptions were implicit. Spend the extra hour making it a module.
• Treating staging and production as separate snowflakes. If they're different, you're testing the wrong thing. Infrastructure as Code keeps them aligned.

Key takeaways

• IAM is security. Get it right at the start. Use least privilege, validate with policy simulation, and rotate credentials regularly.
• Managed services reduce your operational surface area. If it's not core to your product, let AWS run it. That's not outsourcing, it's focus.
• Cost and architecture are inseparable. Design with pricing in mind. Tag resources, monitor bills, and investigate anomalies.
• Automation separates reliable systems from fragile ones. Infrastructure as Code makes AWS predictable and reproducible. CI/CD makes deployments boring.
• Build in production-like staging. Use the same infrastructure, same deployment process, same monitoring. Find issues there, not in the 3 AM incident.
• The best AWS deployments are invisible. You ship features. You monitor metrics. AWS handles the infrastructure. That's the goal we've been building toward.