Cloudwatch Agent with wildcard file path InvalidParameterException

Question

I'm starting a AWS Cloudwatch Agent on my EC2 and trying to log all files that match "test*" for example: /var/log/test_pyserver & /var/log/test_rserver.

However my current agent.json:

{
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/test*"
                                        }
                                ]
                        }
                }
        }
}

Is throwing this error:

cloudwatchlogs: InvalidParameterException for log group /var/log/test* log stream XXXXXXX, will not retry the request: 1 validation error detected: Value '/var/log/test**' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+

According to the documentation if I don't include the "log_group_name" it will just default to the file name, and the "log_stream_name" should be the "{instance_id}".

Edit: I should point out my goal, would be to have 1 log group with each file as their own log stream. However that doesn't look possible, so then I figured each file going to their own log group was acceptable, but it seems like if I wildcard the file path I have to specify a log group. How are people dynamically watching multiple files?

Answer 1

I think you have two options here:

1. Set the log_group_name in the one collect_list entry you have and have all log files published to the same stream in the same group.
2. List all files separately as individual items on the collect_list, this will give you flexibility to configure log groups and streams.

If the second option is not good for you because you have new log files showing up often from multiple applications, take a look at the option of appending the configuration with application specific entries: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-common-scenarios.html#CloudWatch-Agent-multiple-config-files