AWS Elastic Beanstalk default CloudWatch log location for Java applications

Question

I have deployed a JAR file in an AWS Elastic Beanstalk environment, using a load balancer and maximum 2 EC2 instances.

My environment seems to be using the new unified CloudWatch agent, not the legacy CloudWatch Logs Agent

In the environment configuration in the console I turned on "Instance log streaming to CloudWatch Logs", and now in CloudWatch Logs I get the following groups:

• /aws/elasticbeanstalk/myapp-myenv/var/log/eb-engine.log
• /aws/elasticbeanstalk/myapp-myenv/var/log/eb-hooks.log
• /aws/elasticbeanstalk/myapp-myenv/var/log/nginx/access.log
• /aws/elasticbeanstalk/myapp-myenv/var/log/nginx/error.log
• /aws/elasticbeanstalk/myapp-myenv/var/log/web.stdout.log

This is nice. Notably the web server accesses are in access.log, and even the STDOUT console output of my JAR file is in web.stdout.log. So far so good.

Now let's say I want my JAR application to generate a log file and have it picked up automatically in CloudWatch Logs. I've read the Elastic Beanstalk EC2 logging documentation which says that I can bundle special .ebextensions in my application to indicate new logging locations "[i]f your application generates logs in a location that isn't part of the default configuration for your environment's platform".

In the short term I'd rather not create an .ebextensions file. The part about "… a location … part of the default configuration …" intrigues me.

Is there a default location for the Elastic Beanstalk Java platform logs? In other words, is there some default location such as /var/log/app/ that I can simply have my application log to (e.g. via environment variable configuration) and have CloudWatch Logs with the new unified agent automatically pick them up, without needing to create an .ebextensions file? (As a second best solution, is there a way I can configure the location using the AWS console?)

Answer 1

It is very easy to configure this in the new unified CloudWatch agent. The agent picks up any configuration files present in the directory - /etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.d/

You can create a custom config file in that directory,

{
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "path_to_file/app1.log",
                        "log_group_name": "/app/app.log",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    }
}

You can have multiple config files like this. These logs will be available in CloudWatch Logs along with default log groups.

Note - Looking at eb-engine.log, Elastic Beanstalk freshly sets up the unified cloudwatch agent after running commands from .ebextension and prebuild hooks for every deplyment, so any files created in those steps are being deleted. So I suggest you use predeploy hooks to configure this.

A sample hook to do this -

Store the hook file at - .platform/hooks/predeploy/cwa_custom_logs.sh and make sure the file is executable in git.

#!/bin/sh

filepath="/etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.d/custom_logs.json"

# Create the file
cat > $filepath << 'EOL'
{
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "path_to_file/app1.log",
                        "log_group_name": "/app/app.log",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    }
}
EOL

# Change file permissions
chmod 000755 $filepath
chown root:root $filepath

# Restart cloudwatch agent
systemctl restart amazon-cloudwatch-agent.service