kubernetes: how to restart pod on memory limit threshold

Question

I have a deployment with memory limits

resources:
  limits:
    memory: 128Mi

But my app starts to fail when it is near to the limit, so, there is any way to restart the pod before it reaches a percentage of the memory limit?

For example if the limit is 128Mi, restart the pod when it reach 85% of it.

Answer 1

I am going to address this question from the Kuberentes side.

As already mentioned by arjain13, the solution you thought of is not the way to go as it is against the idea of Requests and limits:

If you set a memory limit of 4GiB for that Container, the kubelet (and container runtime) enforce the limit. The runtime prevents the container from using more than the configured resource limit. For example: when a process in the container tries to consume more than the allowed amount of memory, the system kernel terminates the process that attempted the allocation, with an out of memory (OOM) error.

You can also find an example of Exceeding a Container's memory limit:

A Container can exceed its memory request if the Node has memory available. But a Container is not allowed to use more than its memory limit. If a Container allocates more memory than its limit, the Container becomes a candidate for termination. If the Container continues to consume memory beyond its limit, the Container is terminated. If a terminated Container can be restarted, the kubelet restarts it, as with any other type of runtime failure.

There are two things I would like to recommend you to try in your current use case:

1. Debug your application in order to eliminate the memory leak which looks like to be the source of this issue.

2. Use a livenessProbe:

Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy.

It can be configured using the fields below:

• initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.

• periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.

• timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1.

• successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness. Minimum value is 1.

• failureThreshold: When a probe fails, Kubernetes will try failureThreshold times before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready. Defaults to 3. Minimum value is 1.

If you set the minimal values for periodSeconds, timeoutSeconds, successThreshold and failureThreshold you can expect more frequent checks and faster restarts.

Below you will find some useful sources and guides:

• Configure Liveness, Readiness and Startup Probes

• Kubernetes best practices: Resource requests and limits

• Kubernetes best practices: Setting up health checks with readiness and liveness probes