Why is TF build failing with "Error refreshing state: HTTP remote state endpoint requires auth"?

Question

My pipeline builds, plans and applies just fine for my dev branch. When I push to my master branch, I get "Error refreshing state: HTTP remote state endpoint requires auth". See pipeline logs: (and since someone will ask, the token "$onbaord_cloud_account_into_monitoring" has complete read/write access to the scoped project API.)...

Running with gitlab-runner 13.4.1 (REDACTED)
  on runner-gitlab-runner-REDACTED-REDACTED REDACTED
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 ...
Preparing environment
00:03
Waiting for pod gitlab-managed-apps/runner-REDACTED-project-REDACTED-concurrent-REDACTED to be running, status is Pending
Running on runner-REDACTED-project-REDACTED-concurrent-REDACTED via runner-gitlab-runner-REDACTED-REDACTED...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring/.git/
Created fresh repository.
Checking out REDACTED as master...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for REDACTEDREDACTEDREDACTEDREDACTED...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:05
$ git config --global credential.helper cache
$ git fetch
From https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring
 * [new branch]      dev        -> origin/dev
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
$ echo ${TF_ACCOUNT}
REDACTED
$ echo ${TF_ADDRESS}
https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ echo TF_HTTP_ADDRESS=${TF_ADDRESS}
TF_HTTP_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
TF_HTTP_LOCK_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master/lock
$ echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
TF_HTTP_UNLOCK_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master/lock
$ echo TF_ADDRESS=${TF_ADDRESS}
TF_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ export TF_HTTP_ADDRESS=${TF_ADDRESS}
$ export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
$ export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
$ export TF_HTTP_LOCK_METHOD="POST"
$ export TF_HTTP_UNLOCK_METHOD="DELETE"
$ export TF_HTTP_RETRY_WAIT_MIN='5'
$ export TF_HTTP_USERNAME='JOHN.DOE'
$ export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
$ export TF_VAR_ACCOUNT=${TF_ACCOUNT}
$ export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
$ export AWS_DEFAULT_REGION='us-east-1'
$ terraform init
Initializing modules...
Downloading git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/lambda-modules.git?ref=dev for lambda...
- lambda in .terraform/modules/lambda
Downloading git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/iam-modules/lambda-iam.git?ref=dev for lambda_iam...
- lambda_iam in .terraform/modules/lambda_iam
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: HTTP remote state endpoint requires auth
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

A peek at my gitlab-ci.yml:

image:
  name: my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
stages:
  - build
  - plan
  - apply

variables:
  PLAN: plan.tfplan
  JSON_PLAN_FILE: tfplan.json
  WORKSPACE: "dev"
  TF_IN_AUTOMATION: "true"
  ZIP_FILE: "lambda_package.zip"
  STATE_FILE: ${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
  TF_ACCOUNT: ""
  ORG_ACCOUNT: ""
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
  TF_LOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock
  TF_UNLOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock

cache:
  key: "$CI_COMMIT_SHA"
  paths:
    - .terraform

before_script:
  - git config --global credential.helper cache 
  - git fetch 
  - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
  - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
  - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
  - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
  - echo ${TF_ACCOUNT}
  - echo ${TF_ADDRESS}
  - echo TF_HTTP_ADDRESS=${TF_ADDRESS}
  - echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
  - echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
  - echo TF_ADDRESS=${TF_ADDRESS}
  - export TF_HTTP_ADDRESS=${TF_ADDRESS}
  - export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
  - export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
  - export TF_HTTP_LOCK_METHOD="POST"
  - export TF_HTTP_UNLOCK_METHOD="DELETE"
  - export TF_HTTP_RETRY_WAIT_MIN='5'
  - export TF_HTTP_USERNAME='MATTHEW.FETHEROLF'
  - export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
  - export TF_VAR_ACCOUNT=${TF_ACCOUNT}
  - export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
  - export AWS_DEFAULT_REGION='us-east-1'
  - terraform init 

  

# -----BUILD-----
  # 1 build job per lambda function

buildLambda:
  stage: build
  tags:
    - cluster 
  script:
    - echo "Beginning Build"
    - cd lambda_code/
    - echo "Switched into Lambda dir"
    - pip3 install -r requirements.txt --target .
    - echo "Requirements installed"
    - echo $ZIP_FILE
    - zip -r $ZIP_FILE *
    - echo "Zip file packaged up"
  artifacts:
    paths:
      - lambda_code/
  only:
    - dev
    - master
    - merge_requests

# -----PLAN-----

planMerge:
  stage: plan
  tags:
  - cluster 
  script:
    - terraform plan
  dependencies:
    - buildLambda
  only:
    - merge_requests

planCommit:
  stage: plan
  tags:
    - cluster 
  script:
    - terraform plan
  dependencies:
    - buildLambda



# -----APPLY-----
  # dev branch will auto deploy
applyDev:
  stage: apply
  tags:
  - cluster 
  script:
    - echo "Running Terraform Apply"
    - terraform apply -auto-approve
  dependencies:
    - buildLambda
  only:
    - dev 
  #when: manual

    # prod branch will require manual deployment approval
applyProd:
  stage: apply
  tags:
  - cluster 
  script:
    - echo "Running Terraform Apply"
    - terraform apply -auto-approve
  when: manual
  dependencies:
    - buildLambda
  only:
    - master

backend.tf:

terraform {
    backend "http" {
    }
}

main.tf:

locals {
  common_tags = {
    SVC_ACCOUNT_ID = "REDACTED",
    CLOUD_PLATFORM_PROD_ID = "REDACTED"
  }
}

module "lambda" {
    source = "git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/lambda-modules.git?ref=dev" 
    lambda_name = var.name
    lambda_role = "arn:aws:iam::${var.ACCOUNT}:role/${var.lambda_role}"
    lambda_handler = var.handler
    lambda_runtime = var.runtime
    default_lambda_timeout = var.timeout
    env = local.common_tags
    ACCOUNT = var.ACCOUNT
}

module "lambda_iam" {
    source = "git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/iam-modules/lambda-iam.git?ref=dev" 
    lambda_policy = var.lambda_policy
    ACCOUNT = var.ACCOUNT
    lambda_role = var.lambda_role
}

inputs.tf:

variable "handler" {
  type = string
  default = "handler.lambda_handler"
}

variable "runtime" {
  type = string
  default = "python3.8"
}

variable "name" {
  type = string
  default = "onboard-cloud-account-into-monitoring"
}

variable "timeout"{
    type = string
    default = "120"
}

variable "lambda_role" {
  type = string
  default = "cloud-platform-onboard-to-monitoring"
}

variable "ACCOUNT" {
  type = string
}

variable "ORG_ACCOUNT" {
  type = string
}

variable "lambda_policy" {
  default = "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Sid\": \"VisualEditor0\",\"Effect\": \"Allow\",\"Action\": [\"logs:CreateLogStream\",\"logs:CreateLogGroup\"],\"Resource\": \"*\"},{\"Sid\": \"VisualEditor1\",\"Effect\": \"Allow\",\"Action\": \"logs:PutLogEvents\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor2\",\"Effect\": \"Allow\",\"Action\": \"sts:AssumeRole\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor3\",\"Effect\": \"Allow\",\"Action\": \"secretsmanager:GetSecretValue\",\"Resource\": \"*\"}]}"
}

provider.tf:

provider "aws" {
    region          = "us-east-1" 
    assume_role {
        role_arn    ="arn:aws:iam::${var.ACCOUNT}:role/team-platform-onboard-to-monitoring"
    }

    default_tags {
            tags = {
                owner = "[email protected]"
                altowner = "[email protected]"
                blc = "REDACTED"
                costcenter = "REDACTED"
                itemid = "PlatformAutomation"
            }
        }
}

Perhaps I also need to share the terraform modules being invoked?

Answer 1

We found the problem: The pipeline referenced a protected environment variable. The master branch was not a protected branch. The solution was to unprotect the environment variable or protect the branch. Instantly solved the problem.