io.fabric8.kubernetes.client.KubernetesClientException: not ready after 70000 MILLISECONDS

Question

Getting this Error from AKS Jenkins Agent pods. Any idea whats will be reason for this Error? Troubleshooting steps i did.

revert Jenkins to old version => results in same Error upgrade Jenkins to all new Version including plugins in use => Results in same Error. Downgraded Jenkins K8s and K8s API plugins to stable version as per some suggestion in github. => same Error Created Brand new cluster and install Jenkins and Job pod starting giving same Error. => same Error

How to fix this?

18:23:33  [Pipeline] // podTemplate
18:23:33  [Pipeline] End of Pipeline
18:23:33  io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS
18:23:33    at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176)
18:23:33    at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:322)
18:23:33    at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84)
18:23:33    at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:413)
18:23:33    at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:330)
18:23:33    at hudson.Launcher$ProcStarter.start(Launcher.java:507)
18:23:33    at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)
18:23:33    at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
18:23:33    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
18:23:33    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
18:23:33    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
18:23:33    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
18:23:33    at jdk.internal.reflect.GeneratedMethodAccessor6588.invoke(Unknown Source)
18:23:33    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
18:23:33    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
18:23:33    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
18:23:33    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
18:23:33    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
18:23:33    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
18:23:33    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
18:23:33    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
18:23:33    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
18:23:33    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
18:23:33    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
18:23:33    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
18:23:33    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
18:23:33    at WorkflowScript.run(WorkflowScript:114)
18:23:33    at ___cps.transform___(Native Method)
18:23:33    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
18:23:33    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
18:23:33    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
18:23:33    at jdk.internal.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
18:23:33    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
18:23:33    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
18:23:33    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
18:23:33    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
18:23:33    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
18:23:33    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
18:23:33    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
18:23:33    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
18:23:33    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
18:23:33    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
18:23:33    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
18:23:33    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
18:23:33    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)

Answer 1

EDIT: Doesn't seem to make any difference. Still getting 5000 ms timeouts, so not sure this method works (with Environment variables at least). It might work if you are actually able to change timeout, but I haven't got that figured out.

---

Started seeing the same issue after updating Jenkins (and plugins) only - not the K8S cluster.

Except I'm getting 5000 instead of 7000 milliseconds.

io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS

Digging into the stacktrace and source on Github leads back to this default timeout (which haven't changed in 6 years), so somehow you seem to have a non-default.

public static final Long DEFAULT_WEBSOCKET_TIMEOUT = 5 * 1000L;

It seems it can be overriden with environment variable KUBERNETES_WEBSOCKET_TIMEOUT_SYSTEM_PROPERTY on the pod. I just tried to raise mine to 10sec seeing if it makes a difference.

Might be worth a try. If so, it could indicate that the cluster API-server is somehow slower than excepted to respond. I'm not aware of anything that should affect cluster performance around the time of the upgrade in my case, and since the default timeout haven't changed changed for years, it seems odd. Maybe some of the code was refactored somehow no longer ignoring timeouts/retrying etc - Only guessing.

EDIT: I'm on running on bare metal cluster