How can you scale out a Job that is already created?

Question

I have a Kubernetes Job that has, for instance, parallelism set to 4. When this job is created, I might want to scale this out to, say, 8. But it seems like editing the Job and setting parallelism to 8 doesn't actually create more pods in the Job.

Am I missing something? Or is there no way to scale out a Job?

Answer 1

So as per job documentation you can still scale a Job running the following command:

kubectl scale job my-job --replicas=[VALUE]

Simple test shows that this option works right now as expected, but will be really deprecated in a future

kubectl scale job is DEPRECATED and will be removed in a future version.

The ability to use kubectl scale jobs is deprecated. All other scale operations remain in place, but the ability to scale jobs will be removed in a future release.

The reason is: Deprecate kubectl scale job

Use below Job yaml as an example to create job:

apiVersion: batch/v1
kind: Job
metadata:
  name: test-job
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2010)"]
      restartPolicy: Never
  completions: 1000
  parallelism: 5

Now lets test behavior:

kubectl describe jobs.batch test-job 
Parallelism:    5
Completions:    1000
Start Time:     Fri, 17 May 2019 16:58:36 +0200
Pods Statuses:  5 Running / 21 Succeeded / 0 Failed


kubectl get pods | grep test-job | grep Running
test-job-98mlv           1/1     Running     0          13s
test-job-fs2hb           1/1     Running     0          8s
test-job-l8n6v           1/1     Running     0          16s
test-job-lbh46           1/1     Running     0          13s
test-job-m8btl           1/1     Running     0          2s

Changing parallelism with kubectl scale:

kubectl scale jobs.batch test-job --replicas=10

kubectl describe jobs.batch test-job
Parallelism:    10
Completions:    1000
Start Time:     Fri, 17 May 2019 16:58:36 +0200
Pods Statuses:  10 Running / 87 Succeeded / 0 Fail

kubectl get pods | grep test-job | grep Running
test-job-475zf           1/1     Running     0          10s
test-job-5k45h           1/1     Running     0          14s
test-job-8p99v           1/1     Running     0          22s
test-job-jtssp           1/1     Running     0          4s
test-job-ltx8f           1/1     Running     0          12s
test-job-mwnqb           1/1     Running     0          16s
test-job-n7t8b           1/1     Running     0          20s
test-job-p4bfs           1/1     Running     0          18s
test-job-vj8qw           1/1     Running     0          18s
test-job-wtjdl           1/1     Running     0          10s

And the last step that i believe will be the most interesting for you - you can always edit your job using kubectl patch command

kubectl patch job test-job -p '{"spec":{"parallelism":15}}'

kubectl describe jobs.batch test-job
Parallelism:    15
Completions:    1000
Start Time:     Fri, 17 May 2019 16:58:36 +0200
Pods Statuses:  15 Running / 175 Succeeded / 0 Failed

kubectl get pods | grep test-job | grep Running | wc -l
15

Answer 2

kubectl scale doesn't support Job resource anymore. Here is working solution for me:

• kubectl edit job [JOB_NAME]
• set parallelism field to appropriate value for you.

It will create new pods or will terminate existing ones.

Answer 3

There's a scale command:

kubectl scale job my-job --replicas=[VALUE]

From docs:

kubectl scale causes the number of concurrently-running Pods to change. Specifically, it changes the value of parallelism to the [VALUE] you specify.