Run multiple Gitlab CI jobs in parallel, but manually

Question

I would like two jobs to run in parallel in Gitlab CI but I want to launch them manually. Basically I have multiple jobs needed to deploy my code to staging (build the server, build the frontend). I want to deploy to staging by clicking ONE button (hence making the jobs manual) but I want both of them to run at the same time in parallel (faster). I don't want both jobs to be manual because it displays two buttons in the Gitlab UI and someone could deploy only one part.

I tried creating a third, empty job, that would be manual, and have the real deploy jobs run automatically when the empty job has completed. However, even an empty job with just an “echo” takes > 30 seconds and I feel this is dumb. Here is my current solution that does not convince me:

start-release:
  stage: build-and-deploy-to-staging
  rules:
    - when: manual
  script:
    - echo "Starting staging deployment"

release-api-staging:
  stage: build-and-deploy-to-staging
  script:
    - "ENV_NAME=staging STEP=release_api release/deploy.sh"
  when: on_success
  needs:
    ["start-release-staging"]

release-frontend-staging:
  stage: build-and-deploy-to-staging
  script:
    - "ENV_NAME=staging STEP=release_frontend release/deploy.sh"
  when: on_success
  needs:
    ["start-release-staging"]

Do you have any idea how I could manage this?

Thanks a lot!

Answer 1

To follow up from the comments, if you were running your own Runners, you can check the pull-policy for the Runner. There are a few values that tell the Runner what to do when trying to find a Docker image for the job.

The first option is always, which means don't look for any local images, and pull from the available repositories every time the Runner processes a job.

The second is if-not-present, which instructs the Runner to use a local image if one is present, otherwise pull from available repositories.

The last is never, which means to never pull from available repositories and only ever look locally. In this case, if an image isn't available locally, any job using that image will fail.

Knowing this, changing the pull policy to if-not-present or even never if you can maintain the available images on the Runner host(s), can greatly speed up your pipelines.

Another factor that can speed up a job that doesn't require any specific software is to use the smallest image possible that still meets any requirements. For your example .yml file, if you were to use an image such as alpine, pulling the image would be much faster as it is considerably smaller than other images (the image's description claims it's <5mb).

As for the pipeline itself, if you define both jobs in the same stage, with the same dependencies/needs, and mark them both as manual, there will be a single button above the Stage label on the Pipeline view to run all manual jobs in that stage, at the same time (as long as there are enough runners).

job1:
  stage: x
  when: manual
  script:
    - ../do_something.sh

job2:
  stage: x
  when: manual
  ...

If all jobs in a stage are manual, you can start them all at once with the 'play' button for the Stage.

Otherwise, you could define it like you have in your question but the first job will need to be in a different stage. Per the docs, the needs keyword will only work if the "needed" jobs are in another stage.

stages:
  - kickoff_build_and_deploy
  - build_and_deploy

kickoff_job:
  when: manual
  image: alpine:latest
  script:
    - echo '' > /dev/null

build_deploy1:
  needs: ['kickoff_job']
  script:
    - ./do_something.sh

build_deploy2:
  needs: ['kickoff_job']
  script:
    - ./do_something.sh

In this example, the build_deploy* jobs will run as soon as kickoff_job finishes and runners are available.