Using environment properties with files in elastic beanstalk config files

Question

Working with Elastic Beanstalk .config files is kinda... interesting. I'm trying to use environment properties with the files: configuration option in an Elastc Beanstalk .config file. What I'd like to do is something like:

files:
    "/etc/passwd-s3fs" :
        mode: "000640"
        owner: root
        group: root
        content: |
            ${AWS_ACCESS_KEY_ID}:${AWS_SECRET_KEY}

To create an /etc/passwd-s3fs file with content something like:

ABAC73E92DEEWEDS3FG4E:aiDSuhr8eg4fHHGEMes44zdkIJD0wkmd

I.e. use the environment properties defined in the AWS Console (Elastic Beanstalk/Configuration/Software Configuration/Environment Properties) to initialize system configuration files and such.

I've found that it is possible to use environment properties in container-command:s, like so:

container_commands:
    000-create-file:
        command: echo ${AWS_ACCESS_KEY_ID}:${AWS_SECRET_KEY} > /etc/passwd-s3fs

However, doing so will require me to manually set owner, group, file permissions etc. It's also much more of a hassle when dealing with larger configuration files than the Files: configuration option...

Anyone got any tips on this?

Answer 1

I am gravedigging but since I stumbled across this in the course of my travels, there is a "clever" way to do what you describe–at least in 2018, and at least since 2016. You can retrieve an environment variable by key with get-config:

/opt/elasticbeanstalk/bin/get-config environment --key YOUR_ENV_VAR_KEY

And likewise all environment variables with (as JSON or --output YAML)

/opt/elasticbeanstalk/bin/get-config environment

Example usage in a container command:

container_commands:
    00_store_env_var_in_file_and_chmod:
        command: "/opt/elasticbeanstalk/bin/get-config environment --key YOUR_ENV_KEY | install -D /dev/stdin /etc/somefile && chmod 640 /etc/somefile"

Example usage in a file:

files:
    "/opt/elasticbeanstalk/hooks/appdeploy/post/00_do_stuff.sh":
      mode: "000755"
      owner: root
      group: root
      content: |
        #!/bin/bash
        YOUR_ENV_VAR=$(source /opt/elasticbeanstalk/bin/get-config environment --key YOUR_ENV_VAR_KEY)
        echo "Hello $YOUR_ENV_VAR"

I was introduced to get-config by Thomas Reggi in https://serverfault.com/a/771067.

Answer 2

How about something like this. I will use the word "context" for dev vs. qa.

Create one file per context:

dev-envvars

export MYAPP_IP_ADDR=111.222.0.1
export MYAPP_BUCKET=dev

qa-envvars

export MYAPP_IP_ADDR=111.222.1.1
export MYAPP_BUCKET=qa

Upload those files to a private S3 folder, S3://myapp/config.

In IAM, add a policy to the aws-elasticbeanstalk-ec2-role role that allows reading S3://myapp/config.

Add the following file to your .ebextensions directory:

envvars.config

files:
  "/opt/myapp_envvars" :
    mode: "000644"
    owner: root
    group: root
    # change the source when you need a different context
    #source: https://s3-us-west-2.amazonaws.com/myapp/dev-envvars
    source: https://s3-us-west-2.amazonaws.com/myapp/qa-envvars

Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      AWS::CloudFormation::Authentication:
        S3Access:
          type: S3
          roleName: aws-elasticbeanstalk-ec2-role
          buckets: myapp

commands:
  # commands executes after files per 
  # http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
  10-load-env-vars:
    command: . /opt/myapp_envvars

Per the AWS Developer's Guide, commands "run before the application and web server are set up and the application version file is extracted," and before container-commands. I guess the question will be whether that is early enough in the boot process to make the environment variables available when you need them. I actually wound up writing an init.d script to start and stop things in my EC2 instance. I used the technique above to deploy the script.

Credit for the “Resources” section that allows downloading from secured S3 goes to the May 7, 2014 post that Joshua@AWS made to this thread.