How do you run tests from the command line?

Question

To do this in-editor you open the automation tab, connect to the session and choose which tests to run.

How do you do it from the command line?

(NB. not compiling UnrealEngine/Engine/Build/BatchFiles/* comprehensively covers both building the application and compiling it. Specifically, given that you have code that is 100% happy to compile, how do you kick the test suite off)

--

Here's some more info, from recent testing on 4.10:

Running tests from the editor:

UE4Editor Project.uproject -ExecCmds="Automation RunTests MyTest"

Notice the absence of the -Game flag; this launches the Editor and runs the tests successfully in the editor console.

Running the game engine and using the 'popup log window':

UE4Editor Project.uproject -Game -ExecCmds="Automation RunTests MyTest" -log

This runs the game in 'play' mode, pops up an editor window; however, the logs stop at:

LogAssetRegistry: FAssetRegistry took 0.0004 seconds to start up

...and the game never closes or executes the tests.

Running the game engine and logging to a file:

UE4Editor Project.uproject -Game -ExecCmds="Automation RunTests MyTest" -log=Log.txt

This runs the game in 'play' mode, and then stops and never exists.

It does not appear to run any tests or log to any files.

The folder Saved/Logs does not exist after quitting the running game.

Running in the editor, test types, etc...

see: https://answers.unrealengine.com/questions/358821/hot-reload-does-not-re-compile-automation-tests.html,

Hot reload is not supported for tests; so this isn't an option.

There's also been some suggestion in various places that the test type (eg. ATF_Game, ATF_Editor) has some affect on if runs are or can be run; perhaps this is an issue to, but I've tried all kind of combinations with no success.

--

I've tried all kinds of combinations of things trying to get this working, with no success so it's time for a bounty.

I'll accept an answer which reliably:

• Executes a specific test from the command line
• Logs the output from that test to a file

Answer 1

Right, no one has any idea here or on the issue tracker.

After some serious digging through the UE4 source code, here's the actual deal, which I leave here for the next suffering soul who can't figure this out:

To run tests from the command line, and log the output and exit after the test run use:

UE4Editor.exe path/to/project/TestProject.uproject
              -ExecCmds="Automation RunTests SourceTests"
              -unattended
              -nopause
              -testexit="Automation Test Queue Empty"
              -log=output.txt
              -game

On OSX use UE4Editor.app/Contents/MacOS/UE4Editor.

Notice that the logs will, regardless of what you supply, ultimately be placed in:

WindowsNoEditor/TestProject/Saved/Logs/output.txt

or

~/Library/Logs/TestProject/output.txt 

Notice that for mac this is outside of your project directory, in, for example, /Users/doug/Library/Logs/TestProject. (Who thought that was a good idea?)

(see https://wiki.unrealengine.com/Locating_Project_Logs#Game_Logs)

You can list automation tests using:

-ExecCmds="Automation List"

...and then parse the response to find tests to run; automation commands may be chained, for example:

-ExecCmds="Automation List, Automation RunAll"