In Terraform, how can I use another deployment's resources?

Question

Terraform v0.12.x

This is a follow up question to my other post How to use Terraform modules for code re-use?.

I have 2 modules that aim to re-use other modules. My dir structure is...

/terraform/
/terraform/blue/main.tf
/terraform/green/main.tf
/terraform/module_snapshot/main.tf
/terraform/module_ebs/main.tf

I wanna re-use module_ebs/main.tf between two deployments, blue/main.tf and green/main.tf. It simply does

resource "aws_ebs_volume" "ebs" {
  availability_zone = "us-east-1a"
  snapshot_id       = "sn-123456abcded"
  size              = 500
  type              = "gp2"
  tags = {
    Name        = "test-ebs"
  }
}

output "ebs_id" {
  value       = aws_ebs_volume.ebs.id
  description = "Volume id of the EBS volume"
}

The idea is green/main.tf creates an EBS volume using module_ebs/main.tf (it has an output called ebs_id).

provider "aws" {
  region = "us-east-1"
}

terraform {
  required_version = ">= 0.12.17, < 0.13"
  backend "s3" {
    bucket = "my-terraform-states"
    key    = "test-modules/terraform.tfstate"
    region = "us-east-1"
  }
}

module "green_ebs" {
  source "../module_ebs"
}
output "green_ebs_id" {
  value = module.green_ebs.ebs_id
}

When I do this, I get the desired EBS volume

$ cd /terraform/green
$ terraform plan -out out.o
$ terraform apply "out.o"
green_ebs_id = "vol-123456abcdef"

Now I want blue/main.tf to take a snapshot of green's EBS volume, so I do

provider "aws" {
  region = "us-east-1"
}

terraform {
  required_version = ">= 0.12.17, < 0.13"
  backend "s3" {
    bucket = "my-terraform-states"
    key    = "test-modules/terraform.tfstate"
    region = "us-east-1"
  }
}

module "green" {
  source "../module"
}
module "snapshot" {
  source "../module_snapshot"
  green_ebs_id = module.green.green_ebs_id
}
output "blue_ebs_id" {
  value = module.blue_ebs.ebs_id
}

However when I run the above script, it also (of course) runs the green/main.tf, which will of course destroy its EBS volume and create another one, which is NOT what I want to do.

$ cd /terraform/blue
$ terraform plan -out out.o
# module.green.aws_ebs_volume.ebs will be destroyed
- resource "aws_ebs_volume" "ebs" {
...
}

How can I use another deployment's resources without destroying and re-creating them?

Answer 1

There are a few different variants of how to achieve this which have some different tradeoffs for considerations like coupling and for implicit relationships vs. explicit interfaces.

One common approach is to establish some conventions by which a downstream configuration can indirectly find the objects created by the upstream configuration, using data sources. In your case, that could involve devising a tagging scheme for your EBS volumes which both configurations agree on, so that the second configuration can find the object created by the first configuration.

In the first configuration:

resource "aws_ebs_volume" "ebs" {
  availability_zone = "us-east-1a"
  snapshot_id       = "sn-123456abcded"
  size              = 500
  type              = "gp2"
  tags = {
    Name = "production-appname"
  }
}

In the second configuration:

data "aws_ebs_volume" "example" {
  filter {
    name   = "tag:Name"
    values = ["production-appname"]
  }
}

The convention in this example is that the "Name" tag will have the value "production-appname". That might not be exactly the right convention for your purposes, but it demonstrates the general idea. The second configuration can then access that id via data.aws_ebs_volume.example.id.

The above approach, as I mentioned in the opening, makes some design tradeoffs:

• The coupling is relatively low because the second module only requires that something previously created an EBS volume with particular tags, and so you could later refactor your system so that EBS volume were created in a different Terraform configuration, or using some different software other than Terraform, without any changes to the downstream.
• However, the connection between these two is implicit because it relies on a shared convention rather than an explicit interface. That might make the overall system architecture harder to understand unless you are careful to document these implicit conventions somewhere that your team knows to look.

---

Another variant of this is to have your upstream configuration explicitly publish the information into a configuration store intended specifically for that purpose. For example, in AWS you might use AWS SSM Parameter Store, which in Terraform is expressed using the aws_ssm_parameter managed resource type and data source:

resource "aws_ssm_parameter" "foo" {
  name  = "appname_ebs_volume_id"
  type  = "String"
  value = aws_ebs_volume.ebs.id
}

data "aws_ssm_parameter" "foo" {
  name = "appname_ebs_volume_id"
}

Again here there is a shared convention between the two configurations, but the convention is to write into a location specifically intended for storing configuration, and so the "rendezvous point" (the SSM Parameter, in this case) is represented clearly in both the upstream and downstream configuration, retaining a similar level of coupling but increasing the explicitness.

---

A final option is to exploit the fact that most "real" Terraform configurations have their state snapshots persisted in a remote network location. The terraform_remote_state data source is a special data source that reads state snapshots from a remote location and extracts the root module outputs stored there, so you can use that data elsewhere. You can therefore make use of the outputs you already declared in your first module to populate resource configurations in your second module, as long as everyone who applies the second module has sufficient access to read the latest state snapshot from the first.

This third option has, I think, the opposite tradeoffs of the first:

• Coupling is high, because the downstream configuration is configured to read this value directly from the state of the other configuration. If you were to refactor later and move the EBS volume to a different Terraform configuration or manage it using other software, you'd need to edit the second configuration to teach it about the new source.
• However, this is perhaps the most explicit option because the EBS volume ID is clearly intentionally exported from the first configuration as a value for use elsewhere, and the second configuration describes exactly which subsystem is responsible for generating that value.

---

None of these options is "right" or "wrong" in all cases, but I personally consider the second option to be a good compromise between the other two, because it moderates both of the competing design considerations. Which one to select will depend on the goals and constraints of the system you're trying to describe, but I think the second one is a good default if there's no clear "winner" in your case.

There's some general guidance on some techniques to reduce coupling and decompose your system in flexible ways in the Terraform documentation guide Module Composition. It's not specifically about splitting infrastructure across multiple separate Terraform configurations, but the techniques described there can help set you up so that you can more easily change your decisions about how to decompose the system later, so that you can defer adding the complexity of multiple separate configurations until you find a real need to do it.

Answer 2

If you are OK with using external "starter" scripts (bash, jq etc), then you can definitely achieve this. After your first run a terraform.tfstate file is created. It contains the description of all the created resources together with their IDs. With your starter script you can iterate through the state file, extract the necessary IDs and import the resources using terraform import {module_2_resource_name} {module_1_resource_id} to your new module. You can also try to reuse the other state file directly with terraform import -state=path. But you should be careful. The definition of resources in module_1 and module_2 should be identical to avoid destruction.