Pod creation in EKS cluster fails with FailedScheduling error

Question

I have created a new EKS cluster with 1 worker node in a public subnet. I am able to query node, connect to the cluster, and run pod creation command, however, when I am trying to create a pod it fails with the below error got by describing the pod. Please guide.

    Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:
      Type     Reason            Age   From               Message
      ----     ------            ----  ----               -------
      Warning  FailedScheduling  81s   default-scheduler  0/1 nodes are available: 1 Too many pods. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Warning  FailedScheduling  16m                 default-scheduler  0/2 nodes are available: 2 Too many pods, 2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 2 node(s) were unschedulable. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  16m                 default-scheduler  0/3 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 2 node(s) were unschedulable, 3 Too many pods. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  14m (x3 over 22m)   default-scheduler  0/2 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 1 node(s) were unschedulable, 2 Too many pods. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  12m                 default-scheduler  0/2 nodes are available: 1 Too many pods, 2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 2 node(s) were unschedulable. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  7m14s               default-scheduler  no nodes available to schedule pods
  Warning  FailedScheduling  105s (x5 over 35m)  default-scheduler  0/1 nodes are available: 1 Too many pods. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

I am able to get status of the node and it looks ready:

kubectl get nodes  
NAME                         STATUS   ROLES    AGE   VERSION
ip-10-0-12-61.ec2.internal   Ready    <none>   15m   v1.24.7-eks-fb459a0

While troubleshooting I tried below options:

1. recreate the complete demo cluster - still the same error
2. try recreating pods with different images - still the same error
3. trying to increase to instance type to t3.micro - still the same error
4. reviewed security groups and other parameters in a cluster - Couldnt come to RCA

Answer 1

it's due to the node's POD limit or IP limit on Nodes.

So if we see official Amazon doc, t3.micro maximum 2 interface you can use and 2 private IP. Roughly you might be getting around 4 IPs to use and 1st IP get used by Node etc, There will be also default system PODs running as Daemon set and so.

Add new instance or upgrade to larger instance who can handle more pods.

Answer 2

The formula for defining the maximum number of Pods per EC2 Node instance is as follows:

N * (M-1) + 2

Where:

N is the number of Elastic Network Interfaces (ENI) of the instance type

M is the number of IP addresses per ENI

So for the instance you used which is t3.micro the number of pods that can be deployed are:

2 * (2-1) + 2 = 4 Pods, the 4 pods capacity is already used by pods in kube-system namespace

Here you can find the calculated max number of pods for each instance type