Set gridSize (number of threads) dynamically in Spring Batch

Question

I've a spring batch inside spring boot project which processes records present in an input file.

The number of records in the input file can vary anywhere between 1 to 1 million.

I want to leverage multi-threading by partitioning the batch, as mentioned here.

But I want the number of threads to be spawned should be decided on the basis of number of records in the input file.

Say, if records are <10,000 then only spawn 10 threads. If they are >10,000 && <50,000 then spawn 20 threads & so on.

But if I'm not wrong, while partitioning the batch you have to provide the gridSize beforehand & implement Partitioner.class on the basis of that.

This is causing me a problem since value of gridSize should be present in PartitionHandler bean, like:

@Bean
  public PartitionHandler masterSlaveHandler() {
    TaskExecutorPartitionHandler handler = new TaskExecutorPartitionHandler();
    handler.setGridSize(****some dynamic value****);
    handler.setTaskExecutor(taskExecutor());
    handler.setStep(slave());
    try {
      handler.afterPropertiesSet();
    } catch (Exception e) {
      e.printStackTrace();
    }
    return handler;
  }

Since I don't know the value beforehand, my @Configuration class will not be build & will throw error.

So how can I set the gridSize dynamically?

Please suggest. Thanks.

Answer 1

You Can use lazy Scopes using @StepScope annotation to set Gride size

Option 1: if you want to set grid size from stepExecutionContext

@Bean
@StepScope
  public PartitionHandler masterSlaveHandler(@Value("#{stepExecutionContext[gridSize]}") int gridSize) {
    TaskExecutorPartitionHandler handler = new TaskExecutorPartitionHandler();
    handler.setGridSize(gridSize);
    handler.setTaskExecutor(taskExecutor());
    handler.setStep(slave());
    try {
      handler.afterPropertiesSet();
    } catch (Exception e) {
      e.printStackTrace();
    }
    return handler;
  }

Option 1: if you want to set grid size from job parameters

@Bean
@StepScope
  public PartitionHandler masterSlaveHandler(@Value("#{jobParameters[gridSize]}") int gridSize) {
    TaskExecutorPartitionHandler handler = new TaskExecutorPartitionHandler();
    handler.setGridSize(gridSize);
    handler.setTaskExecutor(taskExecutor());
    handler.setStep(slave());
    try {
      handler.afterPropertiesSet();
    } catch (Exception e) {
      e.printStackTrace();
    }
    return handler;
  }