running-unit-tests

Running MSBuild Unit Tests

This repo uses xUnit v3 with the Microsoft.Testing.Platform (MTP) runner. Test projects are built as OutputType=Exe, so each test assembly is a self-contained host process — not a classic VSTest assembly.

There are two ways to run tests:

Method	When to use
`dotnet test <project>`	Dev loop — run a single test project, optionally filtered
`build.cmd -test` / `build.sh --test`	Final validation — builds everything and runs all test projects via the repo's Arcade harness

Use a fast, scoped dotnet test loop while iterating, and the full build.cmd -test pass before declaring the change done.

Repo-specific knobs to know about

These are configured in Directory.Build.props (repo root), src/Directory.Build.targets, and src/Shared/UnitTests/xunit.runner.json. They apply to both dotnet test and build.cmd -test unless noted otherwise:

Multi-targeting: test projects target net472 and net10.0 on Windows (net10.0 only on Linux/macOS). dotnet test runs the suite once per TFM.
Single-threaded by default: xunit.runner.json sets maxParallelThreads: 1 and parallelizeTestCollections: false. Many tests mutate process-global state (env vars, cwd, SDK resolvers), so this is intentional.
Auto trait filters: platform/TFM-inappropriate tests are filtered out via --filter-not-trait Category=... (e.g., nonwindowstests, failing, nonnetcoreapptests). Don't try to "fix" tests that appear skipped because of these.
Coverage on non-Windows: --coverage --coverage-settings Coverage.config is appended unconditionally to XunitOptions in src/Directory.Build.targets. There is no MSBuild property switch to disable it from dotnet test — to skip coverage, run the test exe directly without --coverage.
Test runner: TestRunnerName=XUnitV3, MTP 1.9.1, xUnit v3 3.2.2 (set in repo-root Directory.Build.props).
DOTNET_HOST_PATH: RunnerUtilities.GetMSBuildEnvironmentVariables (in src/UnitTests.Shared/RunnerUtilities.cs) sets DOTNET_HOST_PATH to the bootstrap dotnet when tests launch MSBuild as a child process, so tasks like RoslynCodeTaskFactory resolve the right host. Don't override DOTNET_HOST_PATH from a test.

Flags — MTP vs VSTest pitfalls

This repo uses MTP with the xUnit v3 MTP runner, not VSTest. Many familiar dotnet test flags silently do nothing. Key differences:

Use --report-trx (not --logger trx), --coverage (not --collect "XPlat Code Coverage"), --filter-method/--filter-class/--filter-trait for xUnit v3 native filtering.
Don't use --filter, --nologo, --blame, --settings *.runsettings, --diag, --collect, or -- RunConfiguration.MaxCpuCount=... — these are VSTest-only and ignored.
Still works: -c, -f, --no-restore, --no-build, -v, -bl:, -p:Property=Value — these are interpreted by dotnet test itself before MTP sees them.

For comprehensive MTP vs VSTest flag reference, see the run-tests skill from the dotnet-test plugin.

Dev loop: fast and scoped

Aim for sub-30s iterations.

Examples below use Windows-style backslashes and PowerShell line continuations. Forward slashes work everywhere with dotnet (src/Tasks.UnitTests/Microsoft.Build.Tasks.UnitTests.csproj); on Linux/macOS use / and shell line continuations (\), and the exe path becomes artifacts/bin/<Proj>/Debug/net10.0/<Proj> (no .exe).

Run via dotnet test with a single TFM and filter (incremental build handles rebuilding automatically):

dotnet test src\Tasks.UnitTests\Microsoft.Build.Tasks.UnitTests.csproj `
  -f net10.0 -- --filter-method "*MyFeature*"

Or run the test exe directly (fastest — no SDK overhead; build first if source changed):

dotnet build src\Tasks.UnitTests\Microsoft.Build.Tasks.UnitTests.csproj -c Debug -f net10.0
artifacts\bin\Microsoft.Build.Tasks.UnitTests\Debug\net10.0\Microsoft.Build.Tasks.UnitTests.exe `
  --filter-method "*MyFeature*" --no-progress

The exe path is TFM-specific: Debug\net10.0\...exe for net10.0, Debug\net472\...exe for net472. Switching TFM without rebuilding silently runs stale binaries.

Speeding up the dev loop further

These trade safety for speed — use during iteration, revert before final validation:

Single TFM: pass -f net10.0. Halves runtime on Windows by skipping net472.
Temporarily relax single-threaded execution: drop a xunit.runner.json next to the test project (or override the existing one) with:
```
{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "longRunningTestSeconds": 60,
  "maxParallelThreads": -1,
  "parallelizeTestCollections": true
}
```
Or pass them as runner args after --: -- xUnit.MaxParallelThreads=-1 xUnit.ParallelizeTestCollections=true. Expect flakes in tests that touch env vars, cwd, the file system, or the global ProjectCollection — don't ship a fix that depends on this being on.
Skip bootstrap packaging for non-bootstrap test projects: dotnet build ... -p:CreateBootstrap=false (useful when the local bootstrap SDK payload is missing).
Narrow with traits: --filter-trait Category=mytraitduringdev if you've tagged a focused subset.
Skip code coverage on Linux/macOS: run the test exe directly without --coverage. This repo does not expose a supported dotnet test property switch to disable the auto-added coverage arguments.

Final validation pass

Before saying "done," run the heavy configuration. Don't skip TFMs and don't keep parallel-overrides.

Restore parallelism settings (revert any local xunit.runner.json change).

Run all TFMs for affected projects:

dotnet test src\Build.UnitTests\Microsoft.Build.Engine.UnitTests.csproj -c Release
dotnet test src\Tasks.UnitTests\Microsoft.Build.Tasks.UnitTests.csproj -c Release
# Add other UnitTests projects whose code paths you touched

For broad changes (engine, framework, shared), run the full repo test suite (~9 minutes — do not cancel):

.\build.cmd -test -c Release       # Windows
./build.sh --test -c Release       # Linux/macOS

Capture a TRX if reporting results to the user or CI:

dotnet test <project> -c Release `
  --report-trx --report-trx-filename validation.trx `
  --results-directory artifacts\TestResults

Reading the summary line

MTP's final summary reports passed, failed, and skipped separately. Always check the skipped count — platform-conditional attributes (WindowsOnlyFact, UnixOnlyFact, etc.) and the auto trait filters cause expected skips, but a sudden jump in skipped count can hide a regression where a test became inapplicable on the current platform without you intending it.

Picking the right project to run

Match the source area you changed to its *.UnitTests project:

Source area	Test project
`src/Build/**` (engine, evaluation, backend)	`src/Build.UnitTests/Microsoft.Build.Engine.UnitTests.csproj`
`src/Framework/**`	`src/Framework.UnitTests/Microsoft.Build.Framework.UnitTests.csproj`
`src/Tasks/**`	`src/Tasks.UnitTests/Microsoft.Build.Tasks.UnitTests.csproj`
`src/Utilities/**`	`src/Utilities.UnitTests/Microsoft.Build.Utilities.UnitTests.csproj`
`src/MSBuild/**` (CLI)	`src/MSBuild.UnitTests/Microsoft.Build.CommandLine.UnitTests.csproj`
`src/Build/BuildCheck/**`	`src/BuildCheck.UnitTests/Microsoft.Build.BuildCheck.UnitTests.csproj`
`src/Shared/**`	Run the consumers above (Build, Tasks, Utilities) — shared code is linked into all of them.

Quick reference

Scenario	Command
Fast scoped dev loop	`dotnet test <proj> -f net10.0 -- --filter-method "X"`
Direct test exe	`artifacts\bin\<Proj>\Debug\net10.0\<Proj>.exe --filter-method "X" --no-progress`
Single test by name	`dotnet test <proj> -- --filter-method "MyTestMethod"`
Final per-project validation	`dotnet test <proj> -c Release` (all TFMs)
Final full validation	`.\build.cmd -test -c Release` / `./build.sh --test -c Release`
TRX report	`--report-trx --report-trx-filename out.trx --results-directory artifacts\TestResults`

About this skill