SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillspipelines-security-audit

pipelines-security-audit

Security & auditing

View repository

Updated 10-15 21:11

About this skill

Pipeline Security Audit

This Skill usesprojectas its scope, and produces a checklist against Microsoft Learn's Azure Pipelines security recommendations. It only calls GET throughout and does not modify any settings; for remediation it always provides web UI steps for the user to handle themselves.

Project selection

  • If the user provides a project name (for example "Contoso project"), use it directly without calling the project list API.

  • If not provided, ask the user to provide it once.

  • If still not provided, list the projects for the user to choose from:

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/_apis/projects?%24top=500&api-version=7.1"

  • Do not continue until the user provides or selects a project.

Connection and authentication

Always use curl to call the Azure DevOps REST API directly; each command carries its own authentication parameters.

  • Organization: read the environment variable AZURE_DEVOPS_ORG(organization name, for example contoso). If it is not set, ask the user once; if the user gives a URL, extract the organization name from https://dev.azure.com/{org} or https://{org}.visualstudio.com .
  • Token: send the PAT using Basic authentication:-u ":$AZURE_DEVOPS_PAT"; if only AZURE_DEVOPS_EXT_PATis set (the variable used by the Azure CLI azure-devops extension), use it instead. When using a Microsoft Entra ID token instead, replace it with -H "Authorization: Bearer $AZURE_DEVOPS_TOKEN". If neither is set, ask the user to set AZURE_DEVOPS_PAT in their own terminal and retry, and stop. Only read the token from environment variables: do not ask the user to paste the token into the conversation, and do not print it or write it to a file.
  • PAT permission scopes: Project and Team (Read), Service Connections (Read), Build (Read), Variable Groups (Read), Agent Pools (Read), Secure Files (Read), Code (Read); listing environments additionally requires Environment (Read & manage) (optional, see step 3). For the scope corresponding to each check, see "Checks". This Skill does not require any write permissions.
  • Encoding: the project name in the path must be URL-encoded (write spaces as %20); in the query string write $ as %24(for example %24top) to avoid expansion by the shell.
  • Errors: HTTP 401, 403, 203, or an HTML login page in the response means the token is invalid, expired, or lacks permissions; report it and stop. HTTP 404 means the organization, project, or resource ID is incorrect. Exception: after the project settings have been retrieved successfully, if the API for a certain check returns 403, mark only that check as ❓ and note the missing permission scope, then continue with the other checks.
  • Windows PowerShell: call curl.exe(not the curl alias), and write the environment variable as $env:AZURE_DEVOPS_PAT; PowerShell does not support <<'EOF', so first write the JSON body to a temporary file, then send it with --data-binary "@body.json" .

Checks

Check API used PAT permission scope Basis (Microsoft Learn)
A. The project's Pipeline settings build/generalsettings Project and Team (Read) Job access tokens、Secure Azure Pipelines、Approach
B. Authentication method, scope, and idleness of service connections serviceendpoint/endpoints、executionhistory Service Connections (Read) Overview、Misc
C. Whether protected resources are open to all pipelines pipelines/pipelinepermissions Build (Read), and also requires the scope of each resource list Resources
D. Approval and checks for production environment resources pipelines/checks/configurations Build (Read) Resources、Overview

Judgment symbols: ✅ meets recommendation, ❌ does not meet, ❓ cannot be determined (field does not exist, insufficient permissions, or requires manual confirmation).

Steps

1. Project's Pipeline settings (A)

curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/build/generalsettings?api-version=7.1"

The recommended value for each field is true; when a field does not exist, mark it ❓, do not treat it as false：

Field UI setting name Basis
enforceJobAuthScope Limit job authorization scope to current project for non-release pipelines access-tokens
enforceJobAuthScopeForReleases Limit job authorization scope to current project for release pipelines access-tokens
enforceReferencedRepoScopedToken Protect access to repositories in YAML pipelines access-tokens
enforceSettableVar Limit variables that can be set at queue time overview
enableShellTasksArgsSanitizing Enable shell tasks arguments parameter validation overview
disableClassicBuildPipelineCreation Disable creation of classic build pipelines approach
disableClassicReleasePipelineCreation Disable creation of classic release pipelines approach
enforceNoAccessToSecretsFromForks Do not let fork builds obtain secrets (see "Don't provide secrets to fork builds" in the overview) overview、misc
  • Organization-level settings have no REST API ({org}/_apis/build/generalsettings returns 404). When enabled at the organization level, it applies to all projects, so items whose project-level value is false are marked ❌, and ask the user to go to Organization settings > Pipelines > Settings（https://dev.azure.com/{org}/_settings/pipelinessettings) to confirm whether it has been enabled at the organization level. Learn recommends limiting the job authorization scope at the organization level.
  • The job authorization scope for public projects is always limited to the project, regardless of the setting value; note this in the results.

2. Service connections (B)

  1. List service connections:

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/serviceendpoint/endpoints?api-version=7.1"

    Note that id、name、type、authorization.scheme、isShared、isReady、isDisabled、serviceEndpointProjectReferences。type may have inconsistent casing (for example, azurerm and AzureRM); comparisons are case-insensitive. The list does not contain any credential contents, and do not attempt to obtain them.

  2. Assess one by one:

    • Authentication method（type is azurerm）：authorization.scheme is WorkloadIdentityFederation mark ✅;ServicePrincipal、PublishProfile and other ways of using secrets are marked ❌;ManagedServiceIdentity does not store secrets, but the permissions of the agent identity need to be confirmed, mark ❓.
    • Scope（azurerm）：authorization.parameters.scope contains /resourcegroups/ mark ✅; only down to the subscription or management group mark ❌ (Learn recommends selecting a specific resource group); if this field is absent (mostly manually created) mark ❓.
    • Readiness：isReady is false mark ❌;isDisabled is true mark as "disabled", skip the remaining checks.
    • Shared across projects：isShared is true or serviceEndpointProjectReferences when there is more than one, note the number of projects sharing it and remind the user to confirm that every project needs it.

  3. Idle check: take the most recent usage record (sorted by time from newest to oldest):

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/serviceendpoint/{endpointId}/executionhistory?top=1&api-version=7.1"

    value[0].data.finishTime(if none, use startTime) mark ❌ if more than 100 days ago;value empty means no usage record can be found, mark ❓ and suggest confirming whether it is still needed.data.planType(for example, Build、Release) and data.definition.name can show the pipeline that last used it. Basis: Learn recommends periodically reviewing Usage history, and explains that Azure Pipelines begins automatically disabling service connections idle for 100 days.

3. List other protected resources

Obtain the lists in order; when a list returns 403, mark that resource type as ❓ (write the missing permission scope in parentheses):

curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/distributedtask/variablegroups?api-version=7.1"
curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/distributedtask/securefiles?api-version=7.1-preview.1"
curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/distributedtask/queues?api-version=7.1"
curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/git/repositories?api-version=7.1"

  • Variable groups (Variable Groups (Read)): note id、name、type（Vsts or AzureKeyVault), and whether there is a variable with variables.*.isSecret as true ; do not output any variable values.

  • Secure Files (Read): The Learn documentation only lists the preview version, here provide 7.1-preview.1; note down id、name。

  • Agent Pools (Read): note down id、name、pool.isHosted（true is Microsoft-hosted).

  • Repositories (Code (Read)): note down id and project.id。

  • Environments (optional): The API for listing environments only accepts the Environment (Read & manage) scope, which has manage permissions; there is no read-only scope available. Call it only when the PAT has this scope; if not, skip the listing and mark ❓, or ask the user to provide the environment ID from the environment URL (/_environments/{id}); steps 4 and 5 only need Build (Read) for environments.

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/distributedtask/environments?%24top=100&api-version=7.1"

4. Open access (C)

For each service connection, variable group, secure file, queue, repository, and environment, query pipeline permissions (only the preview version, must include 7.1-preview.1; including 7.1 returns 400):

curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/pipelines/pipelinepermissions/{resourceType}/{resourceId}?api-version=7.1-preview.1"
Resource resourceType resourceId
Service connection endpoint Service connection id
Variable group variablegroup Variable group id
Secure file securefile Secure file id
Agent queue queue Queue id
Environment environment Environment id
Repository repository {projectId}.{repositoryId}(providing only the repository ID returns 400)
  • allPipelines.authorized is true: the UI's Open access(usable by all pipelines), mark ❌. Learn recommends not enabling it and only explicitly authorizing trusted pipelines. Prioritize variable groups containing secrets, service connections, and production environments.
  • There is no allPipelines field: only the pipelines listed in pipelines[] can use it, mark ✅, and include the number of authorized pipelines.
  • Microsoft-hosted queues (pool.isHosted is true) always return authorized: true, and authorizedOn is the time of the query; this is the default behavior, list it only for reference, do not mark ❌.

5. Approvals and checks for production environment resources (D)

  1. Identify production-like resources: service connections, environments, variable groups, and self-hosted queues whose names contain prod、prd、production、live、正式 ; show the list to the user and ask the user to add or remove items once.

  2. For each resource, query the check settings (only the preview version, must include 7.1-preview.1）；resourceType and resourceId , same as step 4:

    curl -sS -u ":$AZURE_DEVOPS_PAT" \
  "https://dev.azure.com/$AZURE_DEVOPS_ORG/{project}/_apis/pipelines/checks/configurations?resourceType={resourceType}&resourceId={resourceId}&%24expand=settings&api-version=7.1-preview.1"

  3. Determine: skip checks where isDisabled is true .

    • Approvals (type.id is 8c6f20a7-a545-4486-9777-f762fafe0d4d，type.name is Approval), or branch control (type.name is Task Check and settings.definitionRef.name is evaluatebranchProtection，settings.displayName is Branch control) mark ✅; if neither is present, mark ❌.
    • Show other checks as type.name (for example, ExtendsCheck is Required template,ExclusiveLock is Exclusive lock; for the rest Task Check use settings.displayName）。
    • Approvals: show settings.approvers[].displayName the number of approvers and settings.minRequiredApprovers; branch control: show settings.inputs.allowedBranches，settings.inputs.ensureProtectionOfBranch is not true , note "branch protection not required".

Examples: "Audit the pipeline security settings of the Contoso project", "Which service connections are still using service principal secrets?", "Which resources are open to all pipelines?"

Present the results

Times are displayed in the user's local time zone; when the time zone cannot be determined, UTC is indicated; the date format is YYYY/MM/DD. Present in order:

  1. Summary: the project, the ✅/❌/❓ counts for each type of check, and the checks skipped due to insufficient permissions along with the required scopes.

  2. Project settings table: UI setting name, current value, assessment, supporting link.

  3. Service connections table: name, type, authentication method, scope, readiness, last used date, open access, approval or branch control (write "—" for non-production environments).

  4. Other protected resources table: type, name, open access, number of authorized pipelines, approval or branch control, remarks (for example, "contains secrets", "Microsoft-hosted, for reference only").

  5. Remediation steps: grouped by failed items, listing the affected resources and web operations:

    • Project settings:Project settings > Pipelines > Settings open the corresponding setting; it is recommended to open them all at once in Organization settings > Pipelines > Settings .
    • Service connection authentication method:Project settings > Service connections > select the connection, and according to Use an Azure Resource Manager service connection switch to workload identity federation and narrow the scope to a specific resource group.
    • Idle connections: after confirming in Usage history on the same page, disable or remove them (performed by the user).
    • Open access: open the resource's Security page, and in Pipeline permissions turn off Open access, and instead add only trusted pipelines. Locations: service connections in Project settings > Pipelines > Service connections > Connections > More actions > Security; variable groups and secure files in Pipelines > Library > Resources > Security; agent pools in Project settings > Pipelines > Agent pools > Pools > Security; environments in Pipelines > Environments > Environments > More actions > Security; repositories in Project settings > Repositories > Repositories > Security。
    • Approvals and checks: on the same resource page, add More actions > Approvals and checks in Approvals or Branch control(see Define approvals and checks）。

    Finally, note that this report did not modify any settings.