SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillspipelines-test-results

pipelines-test-results

Testing & quality

View repository

Updated 10-15 21:11

About this skill

Pipeline test results

This Skill is read-only and is centered on thebuild ID , looking at the automated test results published by the test task in the Pipeline (for example, VSTest、PublishTestResults).

Project selection

  • If the user provides a project name (for example, "Contoso project"), use it directly and do not call the project list API.

  • If not provided, ask the user to provide it once.

  • If it is still not provided, list the projects for the user to choose from:

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/_apis/projects?%24top=500&api-version=7.1"

  • Do not continue until the user provides or selects a project.

Build selection

  • If the user provides a build ID (for example, "build 12345"), use it directly.

  • If only a Pipeline name is provided, use GET .../_apis/build/definitions?name={name}&api-version=7.1 to find the definition ID, then get the most recent completed build:

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/build/builds?definitions={definitionId}&statusFilter=completed&%24top=1&queryOrder=finishTimeDescending&api-version=7.1"

    If the user specifies a branch, add branchName=refs/heads/main。

  • If none are provided, ask the user to provide a build ID or Pipeline name once.

Connection and authentication

Always use curl to call the Azure DevOps REST API directly; each command carries its own authentication parameters.

  • Organization: read the environment variable AZURE_DEVOPS_ORG(organization name, for example, contoso). If it is not set, ask the user once; if the user gives a URL, extract the organization name from https://dev.azure.com/{org} or https://{org}.visualstudio.com .
  • Host: in addition to dev.azure.com, the host for the test results summary API is vstmr.dev.azure.com。
  • Token: send the PAT with Basic authentication:-u ":$AZURE_DEVOPS_PAT"; if only AZURE_DEVOPS_EXT_PATis set (the variable used by the Azure CLI azure-devops extension), use that instead. When using a Microsoft Entra ID token instead, replace it with -H "Authorization: Bearer $AZURE_DEVOPS_TOKEN". If neither is set, ask the user to set AZURE_DEVOPS_PAT in their own terminal and retry, and stop. Read tokens only from environment variables: do not ask the user to paste the token into the conversation, and do not print it or write it to a file.
  • PAT permission scopes: Test Management (Read); finding a build by Pipeline name also requires Build (Read).
  • Encoding: URL-encode project and Pipeline names in paths and query strings (write spaces as %20); in the query string, write $ as %24(for example, %24top) to avoid shell expansion.
  • Errors: HTTP 401, 403, 203, or an HTML login page response means the token is invalid, expired, or lacks sufficient permissions; report it and stop. HTTP 404 means the organization, project, build, or test run ID is incorrect (TestObjectNotFoundException means the test run has been deleted). If HTTP 400 is returned with the message Maximum 200 test results can be processed in a single call , lower %24top to 200.
  • Windows PowerShell: call curl.exe(not the curl alias), and write the environment variable as $env:AZURE_DEVOPS_PAT; PowerShell does not support <<'EOF', so first write the JSON body to a temporary file, then send it with --data-binary "@body.json" .

Steps

1. Get the test summary for the entire build

curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://vstmr.dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/testresults/resultsummarybybuild?buildId={buildId}&includeFailureDetails=true&api-version=7.1-preview.1"
  • This resource only has a preview version, and you must include 7.1-preview.1; writing only 7.1 will return HTTP 400.
  • aggregatedResultsAnalysis：
    • totalTests、duration（hh:mm:ss.fffffff）。
    • resultsByOutcome.{Outcome}.count: the count of each outcome (for example Passed、Failed、NotExecuted）。
    • resultsDifference: the increase or decrease compared with the previous build (increaseInTotalTests、increaseInFailures、increaseInPassedTests、increaseInDuration); the comparison target is previousContext.build, and when it is null it means there is no comparable build and the increase or decrease should not be shown.
  • testFailures（includeFailureDetails=true only):newFailures(newly failed this time),existingFailures(already failing before),fixedTests(fixed this time), each with count and testResults[]（testRunId、testResultId); step 3 uses it to mark whether each failure is new or existing.
  • totalTests when it is 0 , explain that this build has not published test results (the pipeline may not have a test job, or the build may have been deleted), and stop.
  • The pass rate is calculated as Passed ÷ totalTests , rounded to one decimal place;NotExecuted and other not-executed counts are listed separately.

2. List the build's test runs

curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/test/runs?buildUri=vstfs:///Build/Build/{buildId}&includeRunDetails=true&%24top=100&api-version=7.1"
  • Each entry has id、name、state（Completed、InProgress、Aborted…）、totalTests、passedTests、unanalyzedTests(failures not yet analyzed),notApplicableTests(not executed, etc.),incompleteTests、webAccessUrl. In addition, includeRunDetails=true also has startedDate、completedDate and pipelineReference.stageReference.stageName、jobReference.jobName, so you can see which stage and job published it.
  • For paging, use %24top and %24skip; when the number of returned entries equals %24top , fetch the next page.
  • state Runs where 255) have usually been deleted by the retention policy and their results can no longer be fetched; mark them and skip them.
  • When there is no run at all, explain that this build has no test runs.

3. Get details of failed tests

For each run where totalTests is greater than passedTests + notApplicableTests , fetch failed results:

curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/test/Runs/{runId}/results?outcomes=Failed,Aborted,Timeout,Error&%24top=1000&api-version=7.1"
  • outcomes separates multiple values with commas. When detailsToInclude is not included, %24top is at most 1000; when detailsToInclude（Iterations、WorkItems、SubResults、Point) is included, it is at most 200. For paging, use %24skip; when the number of returned entries equals %24top , fetch the next page.
  • The default response already includes the required fields, so there is no need to query each entry individually:
    • testCaseTitle、automatedTestName(full name),automatedTestStorage(for example, test component name)
    • outcome、durationInMs(may not exist),computerName
    • errorMessage、stackTrace(may be thousands of characters long; when presenting, take only the first part)
    • failingSince.build.number and failingSince.date: since which build it has been failing
  • Use testFailures.newFailures.testResults[] from step 1 to compare testRun.id with the result id, and mark 🆕 new failure; in existingFailures , mark 🔁 persistent failure.
  • When there are more than 20 failures, first group by the first line of errorMessage (the same error is often the same cause), list the count and the first few test names for each group, and then ask the user to choose the group whose details they want to see.
  • When test attachments or console output are needed, ask the user to view them on the web at webAccessUrl ; to view the build log of the failed step, use pipelines-build-summary。

Examples: "How are the test results for Contoso project build 12345?", "Which tests failed in the latest build of MyApp-CI?", "Did these failures first appear this time?"

Present the results

Display times in the user's local time zone (format YYYY/MM/DD HH:MM); when the time zone cannot be determined, indicate UTC. The build test page link is https://dev.azure.com/{organization}/{project}/_build/results?buildId={buildId}&view=ms.vss-test-web.build-test-results-tab。

  1. Summary: Build number (link), total count, ✅ passed, ❌ failed, ⏭️ not executed, pass rate, total duration; when there is a previous build to compare with, add the change (e.g. "failed +3") and the counts of 🆕 new failures, 🔁 persistent failures, and 🎉 fixed.
  2. Test run: A table listing name, stage/job, status, total count, passed, failed, not executed.
  3. Failed tests: A table listing test name (testCaseTitle, shortened when too long while preserving the class and method name), new/persistent, the build in which the failure started, and the first line of the error message.
  4. Error details: For each (or each group of) failure, present in a code block the first 20 lines of errorMessage and stackTrace .
  5. When there are no failures, explicitly state that all tests passed.