Add `dotnet test` integration tests for all Apple platforms by jonathanpeppers · Pull Request #25320 · dotnet/macios

jonathanpeppers · 2026-05-04T21:58:14Z

This PR adds end-to-end dotnet test integration tests for all Apple platforms (iOS, tvOS, Mac Catalyst, macOS) using the MTP (Microsoft Testing Platform) test templates.

Changes

tests/dotnet/UnitTests/DotNetTestTest.cs: New parameterized test that creates a project from dotnet new <template>, replaces the generated tests with a single passing test, and runs dotnet test.
tests/common/DotNet.cs: Added test verb support to DotNet.Execute.

The test reads the MLAUNCH_PATH environment variable to override the mlaunch binary for mobile platforms (iOS/tvOS), enabling testing with the MTP-enabled mlaunch build.

Screenshot

✅ [PR Build #7fb2b4f] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 7fb2b4f27d32fa0d4c43e984839da5007a5c7c72 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-28T19:15:44Z

🔥 [CI Build #7fb2b4f] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

2 tests crashed, 38 tests failed, 154 tests passed.

Failures

❌ dotnettests tests (iOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(iOS,"iostest"): No devices found. Output:
  {
  "Items": {
  "Devices": []
  }
  }

Assert.That(devices, Is.Not.Empty)
Expected: not
...

Html Report (VSDrops) Download

❌ dotnettests tests (MacCatalyst)

🔥 Failed catastrophically on VSTS: test results - dotnettests_maccatalyst (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (tvOS)

1 tests failed, 0 tests passed.

Failed tests

DotNet tests: Failed (Execution failed with exit code 1)
- Xamarin.Tests.DotNetTestTest.DotNetTest(TVOS,"tvostest"): No devices found. Output:
  {
  "Items": {
  "Devices": []
  }
  }

Assert.That(devices, Is.Not.Empty)
Expected: not
...

Html Report (VSDrops) Download

❌ framework tests

🔥 Failed catastrophically on VSTS: test results - framework (no summary found).

Html Report (VSDrops) Download

❌ linker tests (tvOS)

11 tests failed, 0 tests passed.

Failed tests

dont link/tvOS - simulator/Debug: LaunchTimedOut
dont link/tvOS - simulator/Release: LaunchTimedOut
link sdk/tvOS - simulator/Debug: LaunchTimedOut
link sdk/tvOS - simulator/Release: LaunchTimedOut
link all/tvOS - simulator/Debug: LaunchTimedOut
link all/tvOS - simulator/Release: LaunchTimedOut
link all/tvOS - simulator/Debug (don't bundle original resources): LaunchTimedOut
trimmode copy/tvOS - simulator/Debug: LaunchTimedOut
trimmode copy/tvOS - simulator/Release: LaunchTimedOut
trimmode link/tvOS - simulator/Debug: LaunchTimedOut
trimmode link/tvOS - simulator/Release: LaunchTimedOut

Html Report (VSDrops) Download

❌ monotouch tests (tvOS)

24 tests failed, 0 tests passed.

Failed tests

monotouch-test/tvOS - simulator/Debug: LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (CoreCLR): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (CoreCLR, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (CoreCLR, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (CoreCLR, Universal): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (link sdk): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (link all): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (LinkSdk): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, NativeAOT, ARM64): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (managed static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (trimmable static registrar): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (managed static registrar, all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, all optimizations): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (trimmable static registrar, NativeAOT, x64): LaunchTimedOut
monotouch-test/tvOS - simulator/Debug (interpreter): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (interpreter): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (compat inline dlfcn): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (strict inline dlfcn, link sdk): LaunchTimedOut
monotouch-test/tvOS - simulator/Release (NativeAOT, .NET 11 defaults): LaunchTimedOut

Html Report (VSDrops) Download

❌ Tests on macOS Tahoe (26) tests

1 tests failed, 4 tests passed.

Failed tests

monotouch-test: Failed (exit code 2)
- TlsDefaults : ServerName

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. (⚠️ Html Report Publish failed ⚠️) Download
✅ dotnettests (macOS): All 1 tests passed. (⚠️ Html Report Publish failed ⚠️) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. (⚠️ Html Report Publish failed ⚠️) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 9 tests passed. Html Report (VSDrops) Download
✅ linker (iOS): All 11 tests passed. Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 11 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 11 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 24 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 27 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 20 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 7fb2b4f27d32fa0d4c43e984839da5007a5c7c72 [PR build]

The ComputeAvailableDevices MSBuild target returned empty results on CI, likely due to mlaunch not being available yet at that point. Use xcrun simctl list directly which is always available on the CI agents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vs-mobiletools-engineering-service2 · 2026-05-28T19:41:10Z

🔥 [CI Build #b42f06c] Build failed (Build packages) 🔥

Build failed for the job 'Build packages' (with job status 'Failed')

Pipeline on Agent
Hash: b42f06ca91f4e3ff72dcf6fd8c50bf441df63db7 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-28T19:46:09Z

🔥 [CI Build #b42f06c] Build failed (Detect API changes) 🔥

Build failed for the job 'Detect API changes' (with job status 'Failed')

Pipeline on Agent
Hash: b42f06ca91f4e3ff72dcf6fd8c50bf441df63db7 [PR build]

vs-mobiletools-engineering-service2 · 2026-05-28T19:49:09Z

🔥 Unable to find the contents for the comment: D:\a\1\s\change-detection\results\gh-comment.md does not exist :fire

Pipeline on Agent
Hash: b42f06ca91f4e3ff72dcf6fd8c50bf441df63db7 [PR build]

jonathanpeppers changed the base branch from main to net11.0 May 4, 2026 21:58

This comment has been minimized.

Sign in to view

rolfbjarne mentioned this pull request May 11, 2026

[CI Postmortem] Flaky: TestNSUrlSessionEphemeralHandlerCookies on iOS/MacCatalyst #25384

Closed

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch 3 times, most recently from 5c37575 to 4c9418b Compare May 12, 2026 17:35

This comment has been minimized.

Sign in to view

jonathanpeppers force-pushed the jonathanpeppers/add-mlaunch-mtp-tests branch from 992433c to 7fc6857 Compare May 27, 2026 21:29

This comment has been minimized.

Sign in to view

Fix NUnit v4 Assert.That syntax in DotNetTestTest

7fb2b4f

The net11.0 branch upgraded to NUnit v4 which uses Assert.That() instead of Assert.AreEqual(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Conversation

jonathanpeppers commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Screenshot

Related

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented May 28, 2026

✅ [PR Build #7fb2b4f] Build passed (Build macOS tests) ✅

Uh oh!

vs-mobiletools-engineering-service2 commented May 28, 2026

🔥 [CI Build #7fb2b4f] Test results 🔥

Test results

Failures

❌ dotnettests tests (iOS)

Failed tests

❌ dotnettests tests (MacCatalyst)

❌ dotnettests tests (tvOS)

Failed tests

❌ framework tests

❌ linker tests (tvOS)

Failed tests

❌ monotouch tests (tvOS)

Failed tests

❌ Tests on macOS Tahoe (26) tests

Failed tests

Successes

macOS tests

Linux Build Verification

Uh oh!

vs-mobiletools-engineering-service2 commented May 28, 2026

🔥 [CI Build #b42f06c] Build failed (Build packages) 🔥

Uh oh!

vs-mobiletools-engineering-service2 commented May 28, 2026

jonathanpeppers commented May 4, 2026 •

edited

Loading