Best way to check app health endpoint after deployment in a YAML deployment job?

Question

Before you shout "gates!" please read.

I have on-prem IIS servers farms, I have created a VM Environment for each (dev, test, prod). In all of my apps, I have a CD YAML like so:

- stage: Deploy_Test
  jobs:
  - deployment: APIDeployTestIISFarm
    displayName: Deploy API to Test IIS
    environment:
      name: Test
      resourceType: VirtualMachine
    strategy:
      runOnce:
        deploy:
          steps:

Because I have 20 different apps all targeting the same environments, I can not use the gates feature. Not ALL apps have the exact same /health endpoint.

I am currently crafting a powershell command to simply call /health and maybe throw on the result if "unhealthy". Although that sounds horribly ugly to me compare to HTTP gate that would simply check 200, and I bet its also fault tolerant/resilient. After all, IIS sites take a second to spin up on first hit after a deployment.

Open to other ideas or tasks that I have not seen yet.

Answer 1

My solution was to create a fault-tolerant PowerShell script to ping the health endpoint. I took this a step further and wrapped it in a template which can be inserted into your main CI/CD template. Doing this lets me develop and test the PS script in isolation with my IDE of choice (VSCode) and also re-use it as a standard task thanks to the template, or use it outside of Azure Pipelines.

Script

Notes on script: The script ultimately decides if the task fails or not via Write-Error combined with the failOnStderr: true for PowerShell@2 task we will see later.

## Base code borrowed from https://karask.com/retry-powershell-invoke-webrequest/
## Slightly modified to remove things like the file logging.

param (
    [string]$URI,
    [string]$Method = 'GET',
    [string]$SuccessTextContent = 'Healthy',
    [string]$Retries = 1,
    [string]$SecondsDelay = 2,
    [string]$TimeoutSec = 120
)

Write-Output "$Method ""$URI"" Retries: $Retries, SecondsDelay $SecondsDelay, TimeoutSec $TimeoutSec";

Function Req {
    Param(
        [Parameter(Mandatory=$True)]
        [hashtable]$Params,
        [int]$Retries = 1,
        [int]$SecondsDelay = 2
    )

    $Params.Add('UserAgent', 'azagent powershell task')

    $method = $Params['Method']
    $url = $Params['Uri']

    $cmd = { Write-Host "$method $url..." -NoNewline; Invoke-WebRequest @Params }

    $retryCount = 0
    $completed = $false
    $response = $null

    while (-not $completed) {
        try {
            $response = Invoke-Command $cmd -ArgumentList $Params
            if ($response.StatusCode -ne 200) {
                throw "Expecting reponse code 200, was: $($response.StatusCode)"
            }
            $completed = $true
        } catch {
            Write-Output "$(Get-Date -Format G): Request to $url failed. $_"
            if ($retrycount -ge $Retries) {
                Write-Error "Request to $url failed the maximum number of $retryCount times."
                throw
            } else {
                Write-Warning "Request to $url failed. Retrying in $SecondsDelay seconds."
                Start-Sleep $SecondsDelay
                $retrycount++
            }
        }
    }

    Write-Host "OK ($($response.StatusCode))"
    return $response
}

$res = Req -Retries $Retries -SecondsDelay $SecondsDelay -Params @{ 'Method'=$Method;'Uri'=$URI;'TimeoutSec'=$TimeoutSec;'UseBasicParsing'=$true }

if($res.Content -ne "$SuccessTextContent")
{
    Write-Error $response.Content
}
else
{
    Write-Host "Helath check validation success."
}

Template

Notes on Template: There is a subtle detail about this task that is easy to miss. - checkout: templates. This will actually check out the repository resource defined in the template in which this template is inserted into. Which seems kind of obvious since the template is inserted and then once its finished its executed as if it were one yaml.

parameters:
  - name: URI
    type: string
  - name: Method
    type: string
    default: 'GET'
  - name: SuccessTextContent
    type: string
    default: 'Healthy'
  - name: Retries
    type: number
    default: 5
  - name: SecondsDelay
    type: number
    default: 2
  - name: TimeoutSec
    type: number
    default: 120

steps:
  - checkout: templates
  - task: PowerShell@2
    displayName: '${{ parameters.Method }} ${{ parameters.URI }}...'    
    inputs:
      failOnStderr: true
      targetType: 'filePath'
      filePath: $(System.DefaultWorkingDirectory)\InvokeRequestWithRetry.ps1
      arguments: > # Use this to avoid newline characters in multi-line string
        -URI "${{ parameters.URI }}"
        -Method "${{ parameters.Method }}"
        -SuccessTextContent "${{ parameters.SuccessTextContent }}"
        -Retries ${{ parameters.Retries }}
        -SecondsDelay ${{ parameters.SecondsDelay }}
        -TimeoutSec ${{ parameters.TimeoutSec }}

Parent CI/CD YAML

Notes on Parent YAML: Finally, we have the usage. I chose to make this its own dedicated deployment job so I can manually click retry if everything fails and not have to re-run the whole deployment. Also, I wanted the PS to run against my VM environments behind my firewall.

resources:
  repositories:
    - repository: templates
      type: git
      name: IAMM/azure-pipelines-templates
      ref: refs/tags/v0.4

## ...
## Removed for brevity
## ...

- deployment: MyAppHealthDevIIS
  dependsOn: MyAppDeployDevIIS
  displayName: 'Hit /health/views endpoint before proceeding'
  environment:
    name: Development
    resourceType: VirtualMachine
  strategy:
    runOnce:
      deploy:
        steps:
        - template: templates/health-check.yaml@templates
          parameters:
            URI: 'https://$(iisHostName)/$(iisTargetApplication)/health/views'
            SecondsDelay: 5