helm3 - upgrade does not refresh pod with force

Question

We are deploying Java microservices to AWS 'ECR > EKS' using helm3 and Jenkins CI/CD pipeline. However what we see is, if we re-run Jenkins job to re-install the deployment/pod, then the pod does not re-install if there are no code changes. It still keeps the old running pod as is. Use case considered here is, AWS Secrets Manager configuration for db secret pulled during deployment has changed, so service needs to be redeployed by re-triggering the Jenkins job.

Approach 1 : https://helm.sh/docs/helm/helm_upgrade/

I tried using 'helm upgrade --install --force ....' as suggested in helm3 upgrade documentation but it fails with below error in Jenkins log

"Error: UPGRADE FAILED: failed to replace object: Service "dbservice" is invalid: spec.clusterIP: Invalid value: "": field is immutable"

Approach 2 : using --recreate-pods from earlier helm version

With 'helm upgrade --install --recreate-pods ....', I am getting below warning in Jenkins log

"Flag --recreate-pods has been deprecated, functionality will no longer be updated. Consult the documentation for other methods to recreate pods"

However, the pod gets recreated. But as we know --recreate-pods is not soft-restart. Thus we would have downtime, which breaks the microservice principle.

helm version used

version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}

question

• How to use --force with helm 3 with helm upgrade for above error ?
• How to achieve soft-restart with deprecated --recreate-pods ?

Answer 1

Below is how I configured it - Thanks to @vasili-angapov for redirecting to correct documentation section.

In deployment.yaml, I added annotations and rollme

kind: Deployment
spec:
  template:
    metadata:
      annotations:
        rollme: {{ randAlphaNum 5 | quote }}

As per documentation, each invocation of the template function randAlphaNum will generate a unique random string. Thus random string always changes and causes the deployment to roll.

The other way described in the document is with respect to a changing SHA value for a file.

In the past helm recommended using the --recreate-pods flag as another option. This flag has been marked as deprecated in Helm 3 in favor of the more declarative method above.

Answer 2

This is nicely described in Helm documentation: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments