Getting custom elastic beanstalk logs into cloudwatch group

Question

Having issues and been unable to create a custom cloudwatch log group from .ebextentions/logs.config

Here are different files I have tried.

1

--- 
files: 
  /opt/elasticbeanstalk/tasks/bundlelogs.d/celery_logs.conf: 
    content: |-
        /var/log/celery_beat.stdout.log
        /var/log/celery_flower.stdout.log
        /var/log/celery_worker.stdout.log
        /var/log/faust_worker.stdout.log
    group: root
    mode: "000755"
    owner: root

2

--- 
files: 
  "/opt/elasticbeanstalk/config/private/logtasks/bundle/applogs.conf" :
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/log/celery_beat.stdout.log
      /var/log/celery_flower.stdout.log
      /var/log/celery_worker.stdout.log
      /var/log/faust_worker.stdout.log

3

packages:
  yum:
    awslogs: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/celery_beat.stdout.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/celery_beat.stdout.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/celery_beat.stdout.log

      [/var/log/celery_flower.stdout.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/celery_flower.stdout.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/celery_flower.stdout.log

      [/var/log/celery_worker.stdout.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/celery_worker.stdout.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/celery_worker.stdout.log

commands:
  "01":
    command: systemctl enable awslogsd.service
  "02":
    command: systemctl restart awslogsd

The logs are properly showing up in the files:

/var/log/celery_beat.stdout.log
/var/log/celery_flower.stdout.log
/var/log/celery_worker.stdout.log
/var/log/faust_worker.stdout.log

But no log group is being created with no logs being transferred to it.

I've tried 15 or more other similar configurations with no luck.

Answer 1

Latest EC 2 does not use awslogs any more. SSH into your instance and check if your instance is using latest cloudwatch agent by typing

sudo amazon-cloudwatch-agent-ctl -a status

if it is running , add following config file to your ebextensions.

container_commands:   01_append_logs_config:
    command: |
      amazon-cloudwatch-agent-ctl -a append-config -m ec2 -c file:/tmp/custom-cloudwatch-config.json -s

files:   "/tmp/custom-cloudwatch-config.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/messages",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/messages"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/cfn-init.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/cfn-init.log"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/cfn-init-cmd.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/cfn-init-cmd.log"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/dnf.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/dnf.log"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/eb-cfn-init.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/eb-cfn-init.log"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/celery_beat.stdout.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/celery_beat.stdout.log"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/celery_worker.stdout.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/celery_worker.stdout.log"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/secure",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/secure"]]}`",
                  "log_stream_name": "{instance_id}"
                },
                {
                  "file_path": "/var/log/cron",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/cron"]]}`",
                  "log_stream_name": "{instance_id}"
                }
              ]
            }
          }
        }
      }

check https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html for appending custom config files that will add custom log files.

config file generation is explained in https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html. You can generate config files that fit your situation by running the wizard and change the content of the file below with the result.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/UseCloudWatchUnifiedAgent.html

lastly make sure your elasticbeanstalk iam profile has the access to write to cloudwatch logs in your iam profile