Declarative dynamic parallel stages

Question

I figure I’m doing something unorthodox here, but I’d like to stick to declarative for convenience while dynamically generating parallel steps.

I found a way to do something like that, but mixing both paradigms, which doesn’t seem to work well with the BlueOcean UI (multiple stages inside each parallel branch do not show up properly).

The closest I got was with something like this:

def accounts() {
  return ["dynamic", "list"]
}

def parallelJobs() {
  jobs = []

  for (account in accounts()) {
    jobs[] = stage(account) {
      steps {
        echo "Step for $account"
      }
    }
  }

  return jobs
}

# this is inside a shared library, called by my Jenkinsfile, like what is described
# under "Defining Declarative Pipelines in Shared Libraries" in
# https://www.jenkins.io/blog/2017/09/25/declarative-1/
def call() {
  pipeline {
    stages {
      stage('Build all variations') {
        parallel parallelJobs()
      }
    }
  }
}

The problem is Jenkins errors like this:

Expected a block for parallel @ line X, column Y.
           parallel parallelJobs()
           ^

So, I was wondering if there is a way I could transform that list of stages, returned by parallelJobs(), into the block expected by Jenkins...

Answer 1

The answer provided by @ycr is great, but it doesn't take care of some basic housekeeping, mostly in the form of closures. If you run his code "as-is" the line: echo "Step for $account always returns the value Step for list , which is not the desired behavior.

To make sure we get the right output for each stage, we need to take it a step further

def parallelJobs() {
    def jobs = [:] // jobs map is explicitly declared within the function to prevent unintended global scope issues
    for (account in accounts()) {
        def accountID = account // provides correct closure scope, each closure captures the correct value, avoiding unexpected behavior
        jobs[accountID] = {
            stage(accountID) {
                echo "Step for $accountID" // will now return the correct value for each stage
            }
        }
    }
    return jobs
}

Furthermore, in the script call of the pipeline I did something a little redundant and maybe a little nit-picky, but it provides another piece-of-mind bit:

script {
    def jobs = parallelJobs() // parallel block receives jobs properly formatted, ensuring parallel execution of stages.
    parallel jobs
}

Here is the whole shebang...

pipeline {
    agent any

    stages {
        stage('Parallel') {
            steps {
                script {
                    def jobs = parallelJobs() // parallel block receives jobs properly formatted, ensuring parallel execution of stages.
                    parallel jobs
                }
            }
        }
    }
}

def accountIDs() { 
   return ["foo", "bar", "glorp"]
}

def parallelJobs() {
    def jobs = [:] // jobs map is explicitly declared within the function to prevent unintended global scope issues
    for (account in accounts()) {
        def accountID = account // provides correct closure scope, each closure captures the correct value, avoiding unexpected behavior
        jobs[accountID] = {
            stage(accountID) {
                echo "Step for $accountID" // will now return the correct value for each stage
            }
        }
    }
    return jobs
}

Now, when executed you get the parallel runs:

And the values for echo "Step for $accountID" now return the proper values:

Answer 2

Yes, you can. You need to return a map of stages. Following is a working pipeline example.

pipeline {
    agent any

    stages {
        stage('Parallel') {
            steps {
                script {
                    parallel parallelJobs()
                }
            }
        }
    }
}


def accounts() {
  return ["dynamic", "list"]
}

def parallelJobs() {
  jobs = [:]

  for (account in accounts()) {
    jobs[account] = { stage(account) {
       echo "Step for $account"
     }
    }
  }
  return jobs
}