How to define a GitLab CI job to depend on either one or one another previous job?

Question

I want to define a pipeline to compile, deploy to target and test my project.

This should happen in two distinct ways: an incremental (hopefully fast) build at each commit and a full build scheduled at night.

The following .gitlab-ci.yml has all jobs marked "manual" for testing purposes.

stages:
    - build
    - deploy
    - test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-incremental:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    when: manual

build-nightly:
    timeout: 5h
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    tags:
        - yocto
    when: manual

deploy:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    dependencies:
        - build
    when: manual

test:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    dependencies:
        - deploy
    when: manual

This fails with message: deploy job: undefined dependency: build.

How do I explain to GitLab deploy stage needs either build-incremental or build-nightly artifacts?

Later I will have to understand how to trigger build-incremental at commit and build-nightly using a schedule, but that seems to be a different problem.

Answer 1

For your scenario, there are two separate paths through your pipeline depending on the "source": a 'push' or a 'schedule'. You can get the source of the pipeline with the CI_PIPELINE_SOURCE variable. I build these paths separately at first, then combine them:

# First source: push events
stages:
  build
  deploy
  test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-incremental:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: manual
      - when: never

deploy-incremental:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-incremental']
    rules:
      - if $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

test-incremental:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-incremental']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

In this path, if the source is a push, the build step will run upon manual input, otherwise it will never run. Then, the deploy-incremental step will run automatically (without waiting for other jobs or stages) as long as the source is a push, otherwise it will never run. Finally the test-incremental job will run automatically without waiting for other jobs or stages if it's a push like above.

Now we can build the schedule path:

# Scheduled path:

stages:
  build
  deploy
  test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-schedule:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE === 'schedule'
        when: manual
      - when: never

deploy-schedule:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-schedule']
    rules:
      - if $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

test-schedule:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-schedule']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

This works the same way as the push path, but we check to see if the source is schedule.

Now we can combine the two paths:

Combined result:
stages:
  build
  deploy
  test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-incremental:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: manual
      - when: never

build-schedule:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE == 'schedule'
        when: manual
      - when: never

deploy-incremental:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-incremental']
    rules:
      - if $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

deploy-schedule:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-schedule']
    rules:
      - if $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

test-incremental:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-incremental']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

test-schedule:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-schedule']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

A pipeline like this is tedious and takes a bit to build, but works great when you have multiple paths/ways to build the project.