Terraform fires Cycle error when applying

Question

I am trying to build a galera cluster using terraform. To do that I need to render the galera config with the nodes ip, so I use a file template.

When applying, terraform fires an error

Error: Cycle: data.template_file.galera_node_config, hcloud_server.galera_node

It seems there is a circular reference when applying because the servers are not being created before the data template is used.

How may I circumvent this ?

Thanks

galera_node.tf

data "template_file" "galera_node_config" {
  template = file("sys/etc/mysql/mariadb.conf/galera.cnf")

  vars = {
    galera_node0 = hcloud_server.galera_node[0].ipv4_address
    galera_node1 = hcloud_server.galera_node[1].ipv4_address
    galera_node2 = hcloud_server.galera_node[2].ipv4_address
    curnode_ip = hcloud_server.galera_node[count.index].ipv4_address
    curnode = hcloud_server.galera_node[count.index].id
    }
}


resource "hcloud_server" "galera_node" {
  count       = var.galera_nodes
  name        = "galera-${count.index}"
  image       = var.os_type
  server_type = var.server_type
  location    = var.location
  ssh_keys    = [hcloud_ssh_key.default.id]

  labels = {
    type = "cluster"
  }

  user_data = file("galera_cluster.sh")

  provisioner "file" {
    content     = data.template_file.galera_node_config.rendered
    destination = "/tmp/galera_cnf"
    connection {
      type        = "ssh"
      user        = "root"
      host = self.ipv4_address
      private_key = file("~/.ssh/id_rsa")
    }

  }



}

Answer 1

The problem here is that you have multiple nodes that all depend on each other, and so there is no valid order for Terraform to create them: they must all be created before any other one can be created.

To address this will require a different approach. There are a few different options for this, but the one that seems closest to what you were already trying is to use the special resource type null_resource to factor out the provisioning into a separate resource that Terraform can work on only after all of the hcloud_server instances are ready.

Note also that the template_file data source is deprecated in favor of the templatefile function, so this is a good opportunity to simplify the configuration by using the function instead.

Both of those changes together lead to this:

resource "hcloud_server" "galera_node" {
  count       = var.galera_nodes
  name        = "galera-${count.index}"
  image       = var.os_type
  server_type = var.server_type
  location    = var.location
  ssh_keys    = [hcloud_ssh_key.default.id]

  labels = {
    type = "cluster"
  }

  user_data = file("galera_cluster.sh")
}

resource "null_resource" "galera_config" {
  count = length(hcloud_server.galera_node)

  triggers = {
    config_file = templatefile("${path.module}/sys/etc/mysql/mariadb.conf/galera.cnf", {
      all_addresses = hcloud_server.galera_node[*].ipv4_address
      this_address  = hcloud_server.galera_node[count.index].ipv4_address
      this_id       = hcloud_server.galera_node[count.index].id
    })
  }

  provisioner "file" {
    content     = self.triggers.config_file
    destination = "/tmp/galera_cnf"
    connection {
      type        = "ssh"
      user        = "root"
      host        = hcloud_server.galera_node[count.index].ipv4_address
      private_key = file("~/.ssh/id_rsa")
    }
  }
}

The triggers argument above serves to tell Terraform that it must re-run the provisioner each time the configuration file changes in any way, which could for example be because you've added a new node: all of the existing nodes would then be reprovisioned to include that additional node in their configurations.

Provisioners are considered a last resort in the Terraform documentation, but in this particular case the alternatives would likely be considerably more complicated. A typical non-provisioner answer to this would be to use a service discovery system where each node can register itself on startup and then discover the other nodes, for example with HashiCorp Consul's service catalog. But unless you have lots of similar use-cases in your infrastructure which could all share the Consul cluster, having to run another service is likely an unreasonable cost in comparison to just using a provisioner.

Answer 2

You really try to use data.template_file.galera_node_config inside of your resource "hcloud_server" "galera_node" and use hcloud_server.galera_node in your data.template_file.

To avoid this problem:

1. Remove provisioner "file" from your hcloud_server.galera_node
2. Move this provisioner "file" to a new null_resource e.g. like that:

resource "null_resource" template_upload {
  count = var.galera_nodes
  provisioner "file" {
    content     = data.template_file.galera_node_config.rendered
    destination = "/tmp/galera_cnf"
    connection {
      type        = "ssh"
      user        = "root"
      host = hcloud_server.galera_nodes[count.index].ipv4_address
      private_key = file("~/.ssh/id_rsa")
    }
depends_on = [hcloud_server.galera_node]
}