refresh EC2 Instance Tags failed: SharedCredsLoad

Question

I have been struggling to get basic metrics from the CloudWatch agent. I've been getting this error and I have no idea what it means nor can I find resources online which talk much about it

refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed

I followed the instructions here and have read through the documentation carefully. Again, the goal is just to read in some basic metrics from my EC2 instance to CloudWatch. Here are the steps I have followed:

• Followed instructions here "To create the IAM role necessary to run the CloudWatch agent on EC2 instances" and then assigned it to my instance.
• wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
  • ami id is ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20190628 (ami-0cfee17793b08a293)
• Install the .deb with command sudo dpkg --install --skip-same-version ./amazon-cloudwatch-agent.deb
  • note --install and --skip-same-version is just -i -E as done in the docs
• generated a config.json with the wizard, located here /opt/aws/amazon-cloudwatch-agent/bin/config.json. I pasted the contents under the error message below.
• modify the /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml file to point to new credentials of cwagent (since not using root user) with the following:

root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/etc# tail -n 4 common-config.toml 
#### BEGIN ANSIBLE MANAGED BLOCK ####
[credentials]
shared_credential_file = "/home/cwagent/.aws/credentials"
#### END ANSIBLE MANAGED BLOCK ####

• fetch config and start agent with sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

---

here's the error I'm seeing in the logs now, and I'm assuming why this is why I can't see any metrics

root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/logs# tail -n 20 amazon-cloudwatch-agent.log 
2019/10/29 22:41:08 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2019/10/29 22:41:08 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json ...
2019/10/29 22:41:08 I! Detected runAsUser: cwagent
2019/10/29 22:41:08 I! Change ownership to cwagent:cwagent
2019/10/29 22:41:08 I! Set HOME: /home/cwagent
2019-10-29T22:41:08Z I! will use file based credentials provider 
2019-10-29T22:41:08Z I! cloudwatch: get unique roll up list []
2019-10-29T22:41:08Z I! Starting AmazonCloudWatchAgent (version 1.230621.0)
2019-10-29T22:41:08Z I! Loaded outputs: cloudwatch
2019-10-29T22:41:08Z I! cloudwatch: publish with ForceFlushInterval: 1m0s, Publish Jitter: 37s
2019-10-29T22:41:08Z I! Loaded inputs: disk mem
2019-10-29T22:41:08Z I! Tags enabled: host=ip-172-31-71-5
2019-10-29T22:41:08Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"ip-172-31-71-5", Flush Interval:1s 
2019-10-29T22:41:08Z I! will use file based credentials provider 
2019-10-29T22:41:08Z E! refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed
2019-10-29T22:42:37Z E! refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed
2019-10-29T22:43:37Z E! refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed
2019-10-29T22:46:37Z E! refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed
2019-10-29T22:49:37Z E! refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed
2019-10-29T22:52:37Z E! refresh EC2 Instance Tags failed: SharedCredsLoad: failed to get profile, metrics will be dropped until it got fixed

and the config.json I used

root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/bin# cat config.json
{
    "agent": {
        "metrics_collection_interval": 10,
        "run_as_user": "cwagent"
    },
    "metrics": {
        "namespace": "TestNamespace",
        "append_dimensions": {
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}"
        },
        "metrics_collected": {
            "disk": {
                "measurement": [
                    "used_percent"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ],
                "metrics_collection_interval": 60
            }
        }
    }
}

---

EDITS

I got it working after I removed the credentials modification

root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/etc# tail -n 4 common-config.toml 
#### BEGIN ANSIBLE MANAGED BLOCK ####
#[credentials]
#shared_credential_file = "/home/cwagent/.aws/credentials"
#### END ANSIBLE MANAGED BLOCK ####

and after I went ahead and copied the config file to the default location it checks (even though the docs say you can pass the file name as I did).

root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/bin# cp config.json /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/bin# cd ../etc/
root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/etc# chown cwagent:cwagent amazon-cloudwatch-agent.json 
root@ip-172-31-71-5:/opt/aws/amazon-cloudwatch-agent/etc# ls -l
total 16
drwxr-xr-x 2 cwagent cwagent 4096 Oct 30 22:05 amazon-cloudwatch-agent.d
-rwxr-xr-x 1 cwagent cwagent  611 Oct 30 22:11 amazon-cloudwatch-agent.json
-rw-rw-r-- 1 cwagent cwagent 1144 Oct 30 22:05 amazon-cloudwatch-agent.toml
-rw-r--r-- 1 cwagent cwagent 1073 Oct 30 22:05 common-config.toml

Answer 1

The error appears to be related to accessing tags that are associated with Amazon EC2 instances.

The installation instructions you linked suggest creating an IAM Role with the CloudWatchAgentServerPolicy policy attached. This policy includes permission to describe tags:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "ec2:DescribeVolumes",
                "ec2:DescribeTags",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams",
                "logs:DescribeLogGroups",
                "logs:CreateLogStream",
                "logs:CreateLogGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
        }
    ]
}

It would appear that the CloudWatch Agent on that server is not receiving such permissions, and is therefore unable to list the tags.

Therefore:

• Confirm that an IAM Role has been created and that it includes the CloudWatchAgentServerPolicy policy
• Confirm that this IAM Role has been assigned to the Amazon EC2 instance that is running the CloudWatch Agent
• If it is still failing, check whether there are any credentials stored locally on the instance that the Agent could be using instead of the IAM Role assigned to the instance