xcodebuild ignores testPlan when using test-without-building

Question

Consider this sample project: https://github.com/stremsdoerfer/TestPlan. This is just a Hello World that has two test plans: TestPlanUnit that only runs unit tests and TestPlanUI that only run UI tests.

Running the command below with Xcode 14.3, I'd expect TestPlanUI to be run, but instead only TestPlanUnit is run which is the default.

xcodebuild -scheme TestPlan -destination 'platform=iOS Simulator,id=<sim_id>' -testPlan TestPlanUI test-without-building

It works fine with Xcode 14.2. Using test instead of test-without-building also works fine.

Any thoughts?

Answer 1

Actually I got a working solution that does not require any hacks 🔥

The first invocation of xcodebuild will create a xctestrun file:

xcrun xcodebuild build-for-testing -workspace <project_root>/<project_name>.xcworkspace -scheme <scheme> -destination 'platform=iOS Simulator,id=2253C0D7-C198-44AD-A98B-11301FD88F30'

This file will be located in the DerivedData folder and be named like this:

~/Library/Developer/Xcode/DerivedData/<build_folder>/Build/Products/<scheme>_<test_plan_name>_<device_type><ios-version>-<architecture>.xctestrun

An example would be MyScheme_Screenshots_iphonesimulator16.4-arm64.xctestrun

The second invocation of xcodebuild then uses this file to run the tests. It is not allowed to specify any of [-project, -scheme, -workspace] when using -xctestrun:

xcrun xcodebuild test-without-building -destination 'platform=iOS Simulator,id=2253C0D7-C198-44AD-A98B-11301FD88F30' -xctestrun ~/Library/Developer/Xcode/DerivedData/<build_folder>/Build/Products/<scheme>_<test_plan_name>_<device_type><ios-version>-<architecture>.xctestrun

Hope this helps you folks 👍

PS: Here you can find the Pull Request for my command line tool Snap that integrates this fix. Snap creates screenshots using Xcode TestPlans. Super useful for CI 🔥