SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillstest-plans-progress

test-plans-progress

Testing & quality

View repository

Updated 10-15 21:11

About this skill

Test Plan Progress

This Skill reads test plans, test suites, and test points (test point, i.e., the execution unit of "test case × configuration") from Azure Test Plans within the scope ofprojectand summarizes the progress of manual tests. It does not execute tests or modify results; for automated test results from a given build, use pipelines-test-results。

Project Selection

  • If the user has provided a project name (for example, "Contoso project"), use it directly without calling the project list API.

  • If not provided, ask the user to provide it once.

  • If still not provided, list the projects for the user to choose from:

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/_apis/projects?%24top=500&api-version=7.1"

  • Do not continue until the user provides or selects a project.

Test Plan Selection

  • If the user has provided test plan IDs or names, report only those plans; when names are given, match them against the list from step 1 using name, and if multiple are found, ask the user to choose one.
  • If not specified, report all in-progress plans listed in step 1; if there are more than 10, list them first and ask the user to select some or confirm all.

Connection and Authentication

Always use curl to call the Azure DevOps REST API directly, with each command carrying its own authentication parameters.

  • Organization: read the environment variable AZURE_DEVOPS_ORG(organization name, for example, contoso). If not set, ask the user once; if the user provides a URL, extract the organization name from https://dev.azure.com/{org} or https://{org}.visualstudio.com .
  • Token: send the PAT using Basic authentication:-u ":$AZURE_DEVOPS_PAT"; if only AZURE_DEVOPS_EXT_PATis set (the variable used by the Azure CLI azure-devops extension), use that instead. When switching to a Microsoft Entra ID token, use -H "Authorization: Bearer $AZURE_DEVOPS_TOKEN"instead. If neither is set, ask the user to set AZURE_DEVOPS_PAT in their own terminal and retry, and stop. Read the token only from environment variables: do not ask the user to paste the token into the conversation, and do not print it or write it to a file.
  • PAT Permission Scopes: Test Management (Read); listing projects additionally requires Project and Team (Read).
  • Encoding: URL-encode project names in paths (write spaces as %20); in query strings, write $ as %24(for example, %24top) to avoid shell expansion.
  • Errors: HTTP 401, 403, 203, or an HTML login page in the response means the token is invalid, expired, or has insufficient permissions; report and stop. HTTP 404 means the organization, project, test plan, or test suite ID is incorrect.
  • Windows PowerShell: call curl.exe(not the curl alias), write the environment variable as $env:AZURE_DEVOPS_PAT; PowerShell does not support <<'EOF'. First write the JSON body to a temporary file, then send it with --data-binary "@body.json" .

Result Classification

The test point's results.outcome is classified by the Test Point - Get Points List of the Outcome enumeration:

Category results.outcome
Passed passed
Failed failed、timeout、aborted、error
Blocked blocked
Not applicable notApplicable
In progress inProgress、paused
Not executed unspecified、none、notExecuted
Other inconclusive、warning、notImpacted and other values
  • For a test point that has never been executed, return outcome as unspecified、results.state as ready, and there is no lastTestRunId(the REST documentation describes unspecified as used to retain the original value on update; when actually read, it means there is no result yet).
  • results.state：ready needs to be executed,completed has passed and does not need to be rerun,inProgress is in progress,notReady needs to be executed but cannot be executed.
  • Execution rate = (total − not executed) ÷ total;Pass rate = passed ÷ total, consistent with the Passed% calculation on the web page Test Plans > Progress Report . Blocked, not applicable, and paused are counted in the execution rate but not in passed or failed, so the pass rate plus the fail rate may be less than the execution rate (see Progress report）。
  • isAutomated is true test points are test cases linked to automated tests; they are still counted, and the number is noted in the results.

Steps

  1. List in-progress test plans：

    curl -sS -u ":$AZURE_DEVOPS_PAT" -D - \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/testplan/plans?filterActivePlans=true&includePlanDetails=true&api-version=7.1"
    • Note id、name、state、iteration、areaPath、startDate、endDate、owner.displayName、rootSuite.id。
    • -D - to look at the response headers: when there is x-ms-continuationtoken , add &continuationToken={token} to get the next page until this header is absent.
    • When there are no plans, clearly state that this project has no in-progress test plans; when the user wants to include non-in-progress plans, remove filterActivePlans=true。

  2. Get the suite hierarchy：

    curl -sS -u ":$AZURE_DEVOPS_PAT" -D - \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/testplan/Plans/{planId}/suites?asTreeView=true&api-version=7.1"
    • The first level of the response is the root suite (with the same name as the plan), and child suites are nested level by level within children[] ; each suite has id、name、suiteType（staticTestSuite、requirementTestSuite、dynamicTestSuite) and hasChildren。
    • Without asTreeView , it returns a flat list, and each suite uses parentSuite.id to point to its parent, which is suitable for directly building a lookup table.
    • When the response header has x-ms-continuationtoken , likewise use continuationToken to get the next page.

  3. Get the test points of the entire plan. Use the root suite ID together with isRecursive=true to retrieve the test points of all child suites at once, without needing to call each suite individually:

    curl -sS -u ":$AZURE_DEVOPS_PAT" -D - \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/testplan/Plans/{planId}/Suites/{rootSuiteId}/TestPoint?includePointDetails=true&isRecursive=true&api-version=7.1"
    • Each page has at most 200 entries; when the response header has x-ms-continuationtoken , add &continuationToken={token} to get the next page until this header is absent.
    • Each test point uses:testSuite.id and testSuite.name(the suite it belongs to),testCaseReference.id and testCaseReference.name、configuration.name、tester.displayName、isActive、isAutomated, as well as results under outcome、state、lastTestRunId、lastRunBuildNumber、lastResultDetails.dateCompleted。
    • lastResultDetails.dateCompleted is 0001-01-01T00:00:00 or there is no lastTestRunId , it means there is no execution record;lastRunBuildNumber is empty or 0 means there is no linked build.
    • When the user only wants a certain suite, replace {rootSuiteId} with that suite ID.

  4. Aggregate: count test points by testSuite.id according to the "result category"; the subtotal of a parent suite includes all descendant suites (summed using the hierarchy from step 2), and the plan total equals the root suite subtotal. A test case has multiple test points under multiple configurations, and the count is based on test points (consistent with the web page).

  5. List test cases that need attention: test points classified as failed or blocked, and additionally those that are in progress but lastResultDetails.dateCompleted Rows older than 7 days are marked as "possibly stuck".

Examples: "What is the test plan progress for the Contoso project?", "How many tests are still not run in Test Plan 101?", "List the failed or blocked test cases in the Sprint 5 test plan"

Present the results

Dates are formatted as YYYY/MM/DD, times are converted to the user's local time zone, and UTC is indicated when it cannot be determined. Present each test plan in order:

  1. Plan Summary: name (link https://dev.azure.com/{organization}/{project}/_testPlans/define?planId={planId}&suiteId={rootSuiteId}), iteration, period (startDate ～ endDate), owner, total test points, execution rate, pass rate; test points linked to automated tests are additionally noted with the count.

  2. Suite Progress Table: list suites indented by hierarchy (prefix child suites with └), and list the root suite as the plan total:

    Suite Total ✅ Passed ❌ Failed 🚫 Blocked ⏳ Not Run Other Execution Rate Pass Rate
    Contoso Release Tests 40 25 3 2 8 2 80% 63%
    └ Login Feature 12 10 1 0 1 0 92% 83%

    The "Other" column totals Not Applicable, In Progress, and other categories; when it has a value, note the composition below the table.

  3. Failed and Blocked Test Cases：

    Test Case Suite Configuration Result Tester Last Run Build
    12345 Login failure displays error message Login Feature Windows 11 ❌ failed Contoso Team 2026/09/01 MyApp_20260901.1

    The test case ID link format is https://dev.azure.com/{organization}/{project}/_workitems/edit/{testCaseId}. If there are no failures or blocked tests, state this explicitly.

  4. Not Run: summarize the number of not run test points by tester (tester.displayName; write "Unassigned" when there is no assignee), making it easy to track who still has tests to run; then list the test points that are "possibly stuck".

Final reminder: the REST API returns real-time status; the web Progress Report is generated by Analytics and has about a 15-minute delay, so the two may show different numbers for a short time.