How to run Selenium tests in GitHub Actions

Question

I'm trying to find the equivalent command to run Selenium tests in GitHub actions. In Azure DevOps, I'd use this YAML to run "Visual Studio Test":

- task: VSTest@2
  displayName: 'Run functional smoke tests on website and web service'
  inputs:
    searchFolder: '$(build.artifactstagingdirectory)'
    testAssemblyVer2: |
      **\FeatureFlags.FunctionalTests\FeatureFlags.FunctionalTests.dll
    uiTests: true
    runSettingsFile: '$(build.artifactstagingdirectory)/drop/FunctionalTests/FeatureFlags.FunctionalTests/test.runsettings'
    overrideTestrunParameters: |
     -ServiceUrl "https://$(WebServiceName)-staging.azurewebsites.net/" 
     -WebsiteUrl "https://$(WebsiteName)-staging.azurewebsites.net/"  

In GitHub actions, what is the equivalent task? It doesn't look like the VsTest.Console.exe exists in the GitHub runner/agent, so the answer may involve installing the Visual Studio Test Platform installer - but obviously I'd like to avoid this as this would severely slow down each build.

Answer 1

Thanks to @Eldar who helped point me in the right direction. The basic answer is (running in a Windows runner):

- name: Functional Tests
  run: |
    $vsTestConsoleExe = "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\Common7\\IDE\\Extensions\\TestPlatform\\vstest.console.exe"
    $targetTestDll = "functionaltests\FeatureFlags.FunctionalTests.dll"
    $testRunSettings = "/Settings:`"functionaltests\test.runsettings`" "
    #Note that the `" is an escape character to quote strings, and the `& is needed to start the command
    $command = "`& `"$vsTestConsoleExe`" `"$targetTestDll`" $testRunSettings " 
    Write-Host "$command"
    Invoke-Expression $command
  shell: powershell

Using Invoke-Expression seemed to help with some initial errors I was receiving. The complete yaml, with context, is viewable on GitHub:

Note to find the vstest.console.exe file, I used this yaml to search the GitHub Actions runner. There are folders that are locked, so this throws an error if you search on the root C: folder. With the link provided above, I was able to establish that the root folder for Visual Studio and search for vstest.console.exe there:

- name: search for visual studio test runner
  run: |
    $var1 = Get-Childitem -Path "C:\Program Files (x86)\Microsoft Visual Studio" -Filter "vstest.console.exe" -Recurse | select -ExpandProperty FullName
    Write-Host "VS test runner: $var1"
  shell: powershell

Answer 2

This is now a lot simplier thanks to the Setup VSTeset.console.exe action that is available on the Marketplace.

It still requires running on a Windows agent but the number of YAML lines is significantly reduced.

The following snippet assumes that an artifact named "functional-tests" that contains the compiled assets (DLLs, chromedriver.exe etc) for your Selenium tests has been uploaded then downloaded into "./functional-tests/".

- name: Add VSTest.console.exe to the PATH
  uses: darenm/Setup-VSTest@v1

- name: Run functional tests
  run: vstest.console.exe "functional-tests\<functional-tests-project-dll-name-here>.dll"

My experience has been that if you set environment variables at the job level, these will automatically be picked up by vstest.console.exe. This probably holds true for setting them at the step or workflow level but I have not tested that.