Terraform destroys the instance inside RDS cluster when upgrading

Question

I have created a RDS cluster with 2 instances using terraform. When I am upgrading the RDS from front-end, it modifies the cluster. But when I do the same using terraform, it destroys the instance.

We tried create_before_destroy, and it gives error.

We tried with ignore_changes=engine but that didn't make any changes.

Is there any way to prevent it?

resource "aws_rds_cluster" "rds_mysql" {
  cluster_identifier              = var.cluster_identifier
  engine                          = var.engine
  engine_version                  = var.engine_version
  engine_mode                     = var.engine_mode
  availability_zones              = var.availability_zones
  database_name                   = var.database_name
  port                            = var.db_port
  master_username                 = var.master_username
  master_password                 = var.master_password
  backup_retention_period         = var.backup_retention_period
  preferred_backup_window         = var.engine_mode == "serverless" ? null : var.preferred_backup_window
  db_subnet_group_name            = var.create_db_subnet_group == "true" ? aws_db_subnet_group.rds_subnet_group[0].id : var.db_subnet_group_name
  vpc_security_group_ids          = var.vpc_security_group_ids
  db_cluster_parameter_group_name = var.create_cluster_parameter_group == "true" ? aws_rds_cluster_parameter_group.rds_cluster_parameter_group[0].id : var.cluster_parameter_group
  skip_final_snapshot             = var.skip_final_snapshot
  deletion_protection             = var.deletion_protection
  allow_major_version_upgrade     = var.allow_major_version_upgrade
  lifecycle {
    create_before_destroy = false
    ignore_changes        = [availability_zones]
  }
}

resource "aws_rds_cluster_instance" "cluster_instances" {
  count                      = var.engine_mode == "serverless" ? 0 : var.cluster_instance_count
  identifier                 = "${var.cluster_identifier}-${count.index}"
  cluster_identifier         = aws_rds_cluster.rds_mysql.id
  instance_class             = var.instance_class
  engine                     = var.engine
  engine_version             = aws_rds_cluster.rds_mysql.engine_version
  db_subnet_group_name       = var.create_db_subnet_group == "true" ? aws_db_subnet_group.rds_subnet_group[0].id : var.db_subnet_group_name
  db_parameter_group_name    = var.create_db_parameter_group == "true" ? aws_db_parameter_group.rds_instance_parameter_group[0].id : var.db_parameter_group
  apply_immediately          = var.apply_immediately
  auto_minor_version_upgrade = var.auto_minor_version_upgrade
  lifecycle {
    create_before_destroy = false
    ignore_changes        = [engine_version]
  }
}

Error:

resource \"aws_rds_cluster_instance\" \"cluster_instances\" {\n\n\n\nError: error creating RDS Cluster (aurora-cluster-mysql) Instance: DBInstanceAlreadyExists: DB instance already exists\n\tstatus code: 400, request id: c6a063cc-4ffd-4710-aff2-eb0667b0774f\n\n on 

Plan output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.rds_aurora_create[0].aws_rds_cluster.rds_mysql will be updated in-place
  ~ resource "aws_rds_cluster" "rds_mysql" {
      ~ allow_major_version_upgrade         = false -> true
      ~ engine_version                      = "5.7.mysql_aurora.2.07.1" -> "5.7.mysql_aurora.2.08.1"
        id                                  = "aurora-cluster-mysql"
        tags                                = {}
        # (33 unchanged attributes hidden)
    }

  # module.rds_aurora_create[0].aws_rds_cluster_instance.cluster_instances[0] must be replaced
+/- resource "aws_rds_cluster_instance" "cluster_instances" {
      ~ arn                             = "arn:aws:rds:us-east-1:account:db:aurora-cluster-mysql-0" -> (known after apply)
      ~ availability_zone               = "us-east-1a" -> (known after apply)
      ~ ca_cert_identifier              = "rds-ca-" -> (known after apply)
      ~ dbi_resource_id                 = "db-32432432SDF" -> (known after apply)
      ~ endpoint                        = "aurora-cluster-mysql-0.jkjk.us-east-1.rds.amazonaws.com" -> (known after apply)
      ~ engine_version                  = "5.7.mysql_aurora.2.07.1" -> "5.7.mysql_aurora.2.08.1" # forces replacement
      ~ id                              = "aurora-cluster-mysql-0" -> (known after apply)
      + identifier_prefix               = (known after apply)
      + kms_key_id                      = (known after apply)
      + monitoring_role_arn             = (known after apply)
      ~ performance_insights_enabled    = false -> (known after apply)
      + performance_insights_kms_key_id = (known after apply)
      ~ port                            = 3306 -> (known after apply)
      ~ preferred_backup_window         = "07:00-09:00" -> (known after apply)
      ~ preferred_maintenance_window    = "thu:06:12-thu:06:42" -> (known after apply)
      ~ storage_encrypted               = false -> (known after apply)
      - tags                            = {} -> null
      ~ tags_all                        = {} -> (known after apply)
      ~ writer                          = true -> (known after apply)
        # (12 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Answer 1

I see apply_immediately argument not there in aws_rds_cluster resource , can you add that and try.

Answer 2

Terraform is seeing the engine version change on the instances and is detecting this as an action that forces replacement.

Remove (or ignore changes to) the engine_version input for the aws_rds_cluster_instance resources.

AWS RDS upgrades the engine version for cluster instances itself when you upgrade the engine version of the cluster (this is why you can do an in-place upgrade via the AWS console).

By excluding the engine_version input, Terraform will see no changes made to the aws_rds_cluster_instances and will do nothing.

AWS will handle the engine upgrades for the instances internally.

---

If you decide to ignore changes, use the ignore_changes argument within a lifecycle block:

resource "aws_rds_cluster_instance" "cluster_instance" {
  engine_version     = aws_rds_cluster.main.engine_version
  ...

  lifecycle {
    ignore_changes        = [engine_version]
  }
}