Data sources in Terraform for Terraform & IaC developers

Data sources in Terraform for Terraform & IaC developers

Data sources are one of the most underused features in Terraform, yet they solve a critical problem: how do you reference infrastructure that already exists or is managed outside your current state file? Whether you're looking up an AMI ID, finding a VPC created by another team, or querying your AWS account for resource details, data sources let you read existing infrastructure without managing it. Understanding when and how to use them separates teams that fight with hardcoded IDs from those that build flexible, maintainable Terraform configurations.

What data sources do

A data source queries your cloud provider's API and returns information about existing resources. Unlike resources, data sources don't create anything—they only read. During terraform plan and terraform apply, Terraform fetches the current state of the resource you're querying and makes that data available to your configuration.

The fundamental syntax mirrors resource blocks:

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

output "ami_id" {
  value = data.aws_ami.ubuntu.id
}

You reference the returned data the same way you reference computed attributes from resources: data.TYPE.NAME.ATTRIBUTE.

Look up AMIs, VPCs and accounts

The most common use case for data sources is retrieving information about resources that change frequently or have many variants. AMI IDs are a perfect example: they're region-specific, regularly patched, and hardcoding them defeats the purpose of infrastructure-as-code.

Finding the right AMI

Instead of maintaining a spreadsheet of AMI IDs across regions, query AWS for the latest approved image:

data "aws_ami" "app_server" {
  most_recent = true

  filter {
    name   = "name"
    values = ["my-company-app-*"]
  }

  filter {
    name   = "state"
    values = ["available"]
  }

  owners = ["self"]
}

resource "aws_instance" "app" {
  ami           = data.aws_ami.app_server.id
  instance_type = "t3.medium"
}

This works well when your organization maintains a shared AMI pipeline. Every time you run Terraform, it grabs the latest version. You can filter by tags, name patterns, owner, or state to narrow results.

Referencing existing VPCs and subnets

Many teams inherit VPCs created outside Terraform, or they have centralized networking managed by a different team. Rather than importing those resources into your state file, use data sources:

data "aws_vpc" "main" {
  tags = {
    Name = "production"
  }
}

data "aws_subnets" "private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.main.id]
  }

  tags = {
    Tier = "private"
  }
}

resource "aws_security_group" "app" {
  name   = "app-sg"
  vpc_id = data.aws_vpc.main.id
}

This approach keeps your configuration clean: you're not managing the VPC, just consuming it. If the VPC is deleted or modified elsewhere, Terraform will either error or adapt depending on your configuration.

Querying account metadata

Sometimes you need your account ID, current region, or other AWS-wide information:

data "aws_caller_identity" "current" {}

data "aws_region" "current" {}

resource "aws_s3_bucket" "logs" {
  bucket = "app-logs-${data.aws_caller_identity.current.account_id}-${data.aws_region.current.name}"
}

This pattern eliminates region and account assumptions from your code, making it portable across environments.

Reference resources managed elsewhere

In larger organizations, infrastructure spans multiple Terraform projects. Your application Terraform module needs to reference a load balancer provisioned by the platform team, or a database created in a separate stack. Data sources bridge that gap.

Cross-stack references without sharing state

If the other team tags their resources predictably, you can discover them:

data "aws_lb" "platform" {
  tags = {
    ManagedBy = "platform-team"
    Environment = "prod"
  }
}

resource "aws_lb_target_group" "app" {
  load_balancer_arn = data.aws_lb.platform.arn
  name              = "app-targets"
  port              = 8080
  protocol          = "HTTP"
  vpc_id            = data.aws_lb.platform.vpc_id
}

This works because the data source queries AWS's API in real-time. You're not reading from another team's state file—you're asking AWS directly. If they update the load balancer, your next plan reflects that.

Reading outputs from other stacks

For tighter coordination, you can use the terraform_remote_state data source to reference outputs from another state file. We'll cover this in more depth in the "Prefer remote state outputs" section, but the pattern is:

data "terraform_remote_state" "network" {
  backend = "s3"

  config = {
    bucket = "terraform-state"
    key    = "network/prod.tfstate"
    region = "us-east-1"
  }
}

resource "aws_instance" "app" {
  subnet_id = data.terraform_remote_state.network.outputs.private_subnet_id
}

This approach requires the other stack to explicitly output the values you need, but it's explicit and version-controlled—both teams know the contract between stacks.

Avoid data sources that change every plan

Data sources are powerful, but they come with a risk: if you query something that changes constantly, your Terraform plan becomes unstable. You'll see different results on each run, making it hard to reason about what's actually changing.

The unstable data source problem

Consider querying EC2 instances by a tag:

data "aws_instances" "web" {
  filter {
    name   = "tag:Role"
    values = ["web-server"]
  }
}

resource "aws_security_group_rule" "allow_web" {
  for_each                 = toset(data.aws_instances.web.ids)
  type                     = "ingress"
  from_port                = 80
  to_port                  = 80
  protocol                 = "tcp"
  source_security_group_id = each.value
  security_group_id        = aws_security_group.alb.id
}

If someone manually launches or terminates an instance with that tag, your next terraform plan shows different rules. This breaks the principle that infrastructure changes should flow through Terraform. You're now dependent on manual AWS operations.

When to avoid data sources

Use data sources for relatively static infrastructure: VPCs, subnets, security groups managed by other teams, or metadata like AMIs. Avoid them for:

• Instance counts that vary outside Terraform
• Resources created by autoscaling groups
• Infrastructure provisioned by CI/CD pipelines that don't use Terraform
• Any resource whose existence depends on transient operational decisions

When you're tempted to use a data source for something dynamic, ask: does this resource change outside Terraform? If yes, either bring it into your Terraform project or document that dependency clearly in your runbooks.

Prefer remote state outputs for your own stacks

When you control both stacks, remote state outputs are more reliable than querying AWS's API. You're reading from a single source of truth—your state file—rather than the current state of AWS, which may not reflect what Terraform thinks you have.

State outputs vs. data sources

Imagine a networking stack that creates a VPC and outputs its ID:

# networking/main.tf
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name = "prod-vpc"
  }
}

output "vpc_id" {
  value = aws_vpc.main.id
}

Your application stack can consume this in two ways:

# Option 1: Remote state (preferred for your own stacks)
data "terraform_remote_state" "network" {
  backend = "s3"

  config = {
    bucket = "terraform-state"
    key    = "network/prod.tfstate"
    region = "us-east-1"
  }
}

# Option 2: Query AWS directly
data "aws_vpc" "main" {
  tags = {
    Name = "prod-vpc"
  }
}

resource "aws_instance" "app" {
  # Both work, but remote state is more predictable
  subnet_id = data.terraform_remote_state.network.outputs.private_subnet_id
}

Remote state outputs are more reliable because they're in sync with the networking stack's last successful apply. If someone manually deletes the VPC in AWS, the data source query might fail mysteriously, whereas remote state still has the reference—making the error clearer during planning.

Structure for cross-stack communication

Organize your outputs explicitly for consumption by other stacks:

# networking/outputs.tf
output "vpc_id" {
  description = "VPC ID for app stack consumption"
  value       = aws_vpc.main.id
}

output "private_subnet_ids" {
  description = "Private subnet IDs for app deployment"
  value       = aws_subnet.private[*].id
}

output "app_security_group_id" {
  description = "Security group for app servers"
  value       = aws_security_group.app.id
}

Document what you're exporting and why. This makes it clear to other teams what's safe to depend on. If you refactor the networking stack, you know which outputs might break downstream consumers.

For state files stored in different AWS accounts or regions, always use terraform_remote_state over querying AWS directly. It's faster (no API calls) and guaranteed to be consistent with your state.

Common mistakes

• Hardcoding filter values: Avoid queries like "find the VPC tagged with my team's name" if another team might provision it differently. Instead, establish a naming or tagging convention and document it as a prerequisite.
• Using data sources for everything: New users often reach for data sources instead of importing existing resources. If you manage a resource in Terraform, import it; don't query it every run.
• Depending on manual tags: Tags set outside Terraform can disappear or change. Use data sources only for infrastructure managed by stable, documented processes.
• Mixing remote state and data source queries: If you have a choice, pick one pattern per relationship. Mixing makes debugging confusing.
• Forgetting to handle missing data: If a data source query returns no results, Terraform errors. Add validation or default values where appropriate.

Key takeaways

• Data sources read existing infrastructure without managing it—use them to reference resources outside your Terraform project.
• Query AWS directly for AMIs, VPCs, subnets, and other infrastructure managed by other teams or automated processes.
• Avoid data sources for resources that change frequently outside Terraform; they make your plans unstable.
• Prefer terraform_remote_state outputs when both stacks are under your control; it's more reliable than querying AWS's API.
• Tag resources predictably and document your expectations so data source queries remain stable.
• Use data sources to keep your Terraform configurations portable across regions and accounts.