Kubernetes : Cluster-Autoscaler: How to verify autoscaling is working

Question

I am working on our EKS platform, where I have installed Cluster Autoscaler. I can see it running in Kube Dashboard. Yesterday for Load Testing, I triggered 20 replicas of a heavy app we have. The cpu usage per node climbed to 100%, but cluster auto-scaler didn't trigger any additional nodes. I was watching the logs and the logs kept on rotating in main loop, but no changes.

Here are the tags I have added to ASG, worker nodes :

 k8s.io/cluster-autoscaler/enabled : true
  kubernetes.io/cluster/CLUSTER_NAME : owned

I can see the pod running in Dashboard :

./cluster-autoscaler
--v=4
--stderrthreshold=info
--cloud-provider=aws
--skip-nodes-with-local-storage=false
--expander=least-waste
--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/cluster_name

Also, There are no scaling policies added in ASG. Are they required for Cluster Autoscaler? How to verify cluster autoscaler is working properly? What am I missing?

Answer 1

Actually cluster autoscaler checks for any unschedulable pods every 10 seconds and if any pods available in unschedulable state then it will check min and max of autoscaling group. You can check this wonderful FAQ how-does-scale-up-work of autoscaler. If it is not reached max then it will request to aws autoscaling group to add one more.

Now the answer of your question is, you can check or verify autoscaling easily by noticing whether you have any unscheduled pods in your cluster or not. If there is any then autoscaler will try to add one more node which will be reflected in autoscaler log if it is not reached in max limit. For more details you can check this FAQ. You can check also vertical pods scaler to get vertical pods scaling from here