a365-code-validator

A365 Code Validator

Trigger phrases — any of these will activate this skill automatically:

"validate a365 code"

"run a365 code validator"

"check a365 observability code"

"debug a365 activity missing"

"debug MAC activity for this agent"

"validate Agent 365 telemetry"

"check why Agent Activity is empty"

"check A365 exporter flags"

"validate gen_ai agent id"

"check A365 span types"

Overview

This skill performs a report-first validation of an existing agent codebase to answer:

"If this agent runs now, will its spans reach the A365 backend and become eligible for MAC Activity?"

It is designed for incident response. It checks the exact classes of bugs that cause silent or confusing A365 Activity failures:

Exporter configured but not actually enabled.
gen_ai.agent.id set to a Blueprint ID instead of the runtime Agent Identity / Source Agent ID.
Generic HTTP/OpenAI spans emitted without A365 semantic operations.
Delegated (OBO-route) telemetry: the S2S route flag is missing, or an OBO / Agentic User token is fed to the exporter.
Missing token resolver or wrong S2S token shape.
Stale local Blueprint ID, missing Blueprint permissions, or an unregistered agent instance in the target tenant.
Inheritable permission policy configured without grants on the Blueprint service principal.
MAC reporting expectations confused with raw ingest success.

No source code is modified during validation. After the report, the skill asks whether to apply safe fixes, create a fix plan only, or stop.

Output is support-safe (applies to every phase)

This report may be pasted into a customer or partner thread, so it must be safe to share: never print internal cluster/database names, Kusto queries, private endpoint or route templates, correlation IDs, or real tenant/agent IDs. Report file/function names and public, supportable signals only. This is non-negotiable — it holds even when a user asks for "just the raw details."

Phase 0 — Create and Display Task List

Show the user this checklist BEFORE Phase 1. Exactly one task is in progress at a time. Claude Code uses TaskCreate / TaskUpdate; VS Code Copilot Chat / GitHub Copilot CLI should show and update a markdown checklist.

TaskCreate: "Detect stack and A365 artifacts"
TaskCreate: "Run static A365 code validator"
TaskCreate: "Inspect identity binding and semantic spans"
TaskCreate: "Check live blueprint permissions and runtime state"
TaskCreate: "Summarize blockers and next fixes"
TaskCreate: "Offer guided remediation"

Phase 1 — Detect Stack and A365 Artifacts

Mark task in progress: "Detect stack and A365 artifacts"

Read, in parallel when possible:

.a365-workspace-detection.local.json if present
a365.config.json
a365.generated.config.json
.env, .env.example, .env.production, .env.local
appsettings.json, appsettings.Development.json
package.json
requirements.txt, pyproject.toml
*.csproj

Detect:

Signal	Meaning
`microsoft-opentelemetry`	Python A365 distro present
`@microsoft/opentelemetry`	Node.js A365 distro present
`Microsoft.OpenTelemetry`	.NET A365 distro present
`agentBlueprintId` / `agenticAppId`	Distinguish Blueprint ID from runtime Agent Identity
`ENABLE_A365_OBSERVABILITY_EXPORTER` / `EnableAgent365Exporter`	Runtime exporter gate
`authMode`	Selects how the app-only telemetry token is sourced (export is S2S in every mode) and which workload auth applies

Mark task complete.

Phase 2 — Run Static A365 Code Validator

Mark task in progress: "Run static A365 code validator"

Run the validator from the target project root.

When running from the plugin source (Claude Code / marketplace plugin), use:

node ${CLAUDE_PLUGIN_ROOT}/hooks/stop/validate-a365-code-validator.js

If the runtime cannot expand ${CLAUDE_PLUGIN_ROOT}, run with the absolute plugin path:

node /path/to/agent365-skills/plugins/agent365/hooks/stop/validate-a365-code-validator.js

When running from an open-standard .agents/skills/a365-code-validator/ install, use the standalone copy that ships with the skill:

node .agents/skills/a365-code-validator/references/a365-code-validator.js

Parse the JSON output. Treat findings by severity:

Severity	Meaning
`critical`	Known MAC Activity blocker — fix before expecting activity
`high`	Likely export or identity failure
`medium`	Risky or env-dependent — verify before incident closure
`low` / `info`	Useful context

Important: Findings are not stop-hook failures. A validation run is successful when it accurately reports what is wrong.

Mark task complete.

Phase 3 — Inspect Identity Binding and Semantic Spans

Mark task in progress: "Inspect identity binding and semantic spans"

Read the reference checklist:

${CLAUDE_PLUGIN_ROOT}/skills/a365-code-validator/references/validation-checklist.md

Then inspect source for these required MAC Activity inputs:

3.1 Exporter activation

Python must have either explicit code:

use_microsoft_opentelemetry(
    enable_a365=True,
    a365_enable_observability_exporter=True,
    a365_use_s2s_endpoint=True,
)

or a runtime env guarantee:

ENABLE_A365_OBSERVABILITY_EXPORTER=true

Node.js must have:

useMicrosoftOpenTelemetry({
  a365: {
    enabled: true,
    enableObservabilityExporter: true,
    useS2SEndpoint: true,
  },
});

.NET must have:

"EnableAgent365Exporter": true

3.2 Identity binding

Verify the same runtime Agent Identity is used consistently:

token azp/appid/oid == route /agents/{agentId} == span gen_ai.agent.id

The following is wrong for S2S/MAC Activity:

gen_ai.agent.id = <blueprint id>

Blueprint belongs in:

microsoft.a365.agent.blueprint.id

3.3 Semantic span coverage

MAC Activity needs supported A365 operations:

invoke_agent
chat
execute_tool
output_messages

Generic HTTP spans alone are not enough. Flag code that only sets baggage but never creates or auto-generates supported semantic spans.

3.4 Activity/run context coverage

If the developer expects MAC Activity or Defender agent-activity views (not just exporter HTTP 200), inspect whether real runs carry the public Agent 365 activity attributes:

Attribute	What to validate
`gen_ai.agent.name`	Human-readable agent name exists; otherwise UI can show a GUID/blank
`gen_ai.conversation.id`	Present on every span; for non-chat agents use a logical run/job/incident ID
`microsoft.channel.name`	Present on every span; use `msteams` for Teams, or a product/source value such as `icm`, `web`, `scheduler`
`microsoft.session.id`	Prefer a stable run/session ID; optional, but useful for grouping
`user.id`	Human caller OID for `HumanToAgent`; do not confuse it with the agent's own user
`microsoft.agent.user.id`	Agent's own Agent User OID when testing AI teammate / Agent-User / OBO
`gen_ai.input.messages` / `gen_ai.output.messages`	Present on invoke/chat/output spans where applicable
Tool details	`gen_ai.tool.name`, `gen_ai.tool.type`, `gen_ai.tool.call.id`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`
Model details	`gen_ai.request.model`, `gen_ai.provider.name` on `chat` spans

When reviewing a mixed app, separate the two objectives:

S2S/export success proves telemetry plumbing.
AI teammate/HumanToAgent or EventToAgent run context proves Activity/reporting eligibility.

Do not require Teams-specific values for autonomous agents; require equivalent logical run context instead.

Caller identity caveat: for HumanToAgent, user.id is the caller ID used for "who ran this agent" reporting. If it is missing or set to the agent identity / agent user instead of the human caller object ID, export can succeed while caller/user Activity remains blank or incomplete.

3.5 S2S transport mode (every auth mode)

Do not expose or require internal service URLs in the report. Validate only the structural intent. Telemetry export uses the service-to-service (S2S) route with an app-only token for the runtime Agent Identity in every auth mode. authMode only changes how that token is sourced:

Auth mode	Structural expectation
S2S / application	S2S export mode; app-only token from the FMI chain for the configured Agent Identity
OBO / agentic-user (incl. AI Teammate)	S2S export mode; app-only token for the turn's Agent Identity from the hosting connection's blueprint credential. OBO / Agentic User tokens are for workload calls (MCP / Graph) only

Report whether the code selects the S2S transport mode (useS2SEndpoint: true / a365_use_s2s_endpoint=True / o.Agent365.UseS2SEndpoint = true) in every auth mode, without printing backend route templates. For Python, prefer a365_use_s2s_endpoint=True in code rather than relying only on A365_USE_S2S_ENDPOINT=true in the runtime environment.

Flag delegated telemetry as high: the S2S route rejects any token carrying scp, and the legacy delegated route needs admin consent. Signals are:

Node.js AgenticTokenCacheInstance.refreshObservabilityToken(...) (often wrapped in a preloadObservabilityToken helper), or a tokenResolver that reads AgenticTokenCacheInstance.getObservabilityToken(...).
.NET RegisterObservability(..., new AgenticTokenStruct(...), ...), any new AgenticTokenStruct(...) (for example an A365OtelWrapper helper), or an IExporterTokenCache<AgenticTokenStruct> dependency.
Python exchange_token(..., scopes=get_observability_authentication_scope(), ...) feeding a telemetry token cache, cache_agentic_token(...), or an a365_token_resolver backed by AgenticTokenCache / get_cached_agentic_token.
A distro call without the S2S transport flag.
A distro call whose own options pass no token resolver (tokenResolver, a365_token_resolver, or o.Agent365.TokenResolver). The S2S route then has no app-only token to export with. A resolver symbol elsewhere, such as an unused import, does not count.

For Python, also flag (high) an app-only prefetch(self.connection_manager, ...) when nothing assigns self.connection_manager. CloudAdapter does not expose its connection manager, so every prefetch fails.

The fix is the app-only resolver from the instrument-observability references, not an OBS permission grant.

Also check the token shape: the S2S observability token must come from the FMI exchange (principal == runtime Agent Identity, no scp), not a bare client-credentials call — see checklist §5 for the failure signatures.

3.6 Blueprint permission inheritance wording

Do not claim that application app roles can never inherit from blueprints. Current public Entra Agent ID docs describe both inheritableScopes and inheritableRoles. If a repo or tenant shows delegated scope inheritance working but S2S app roles requiring direct assignment, report it as a provisioning/configuration behavior to verify, not as the intended scalable model.

Mark task complete.

Phase 4 — Check Live Blueprint Permissions and Prepare Runtime Verification

Mark task in progress: "Check live blueprint permissions and runtime state"

Live Blueprint permission check (read-only Microsoft Graph)

The static validator can only inspect files. When the project has a non-empty agentBlueprintId and tenantId, complement it with the a365 CLI's read-only Microsoft Graph diagnostics. These commands verify the tenant state without changing permissions:

a365 query-entra blueprint-scopes
a365 query-entra inheritance

blueprint-scopes lists the delegated scopes and application roles actually granted on the Blueprint service principal. This is the authoritative permissions view; do not infer grants from requiredResourceAccess.
inheritance verifies that every configured resource uses kind=allAllowed for scopes and roles and that the Blueprint service principal has something to inherit. It exits non-zero when any resource has Effective inheritance: NONE or BROKEN.

Run these checks by default only when:

a365 query-entra inheritance --help confirms the installed CLI supports the command,
an existing Azure/a365 sign-in is available for the target tenant, and
the check will not trigger a new interactive login.

The validator must not install or update the CLI, start an interactive sign-in, run a365 setup permissions, or issue Graph POST/PATCH/DELETE requests. If a prerequisite is missing, report Blueprint permissions as not checked, explain why, and provide the two commands for the developer to run after a365-setup refreshes the CLI/login.

If the local config is missing, copied from another project, or suspected stale, use the read-only name-based resolver instead:

a365 query-entra blueprint-scopes --agent-name "<agent-name>" --tenant-id "<tenant-id>"
a365 query-entra inheritance --agent-name "<agent-name>" --tenant-id "<tenant-id>"

Compare the live Blueprint ID printed by the command with the local agentBlueprintId, but do not print either real ID in the support-safe report:

Result	Classification
Live and local Blueprint IDs resolve to the same Blueprint	`current`
Local ID is not found, but name-based lookup resolves a different Blueprint	`critical` stale/copied config
Neither lookup resolves a Blueprint	`high` tenant/setup gap
Graph returns 401/403 or the caller lacks the required Entra role	`not checked` authentication/authorization gap, not proof that the Blueprint is broken

Interpret observability authorization for the S2S route, which every auth mode uses:

Tenant state	Classification
Agent instance registered (`agentRegistrationId` present in `a365.generated.config.json`, or registration confirmed by `a365 setup all`)	OK — registered instances are authorized without `Agent365.Observability.OtelWrite` (subject to service policy)
Application role `Agent365.Observability.OtelWrite` granted on the Blueprint (inherited by agent identities)	OK — the S2S route also accepts it
Neither registration nor the application role is evident	`high` — export will likely return 403 `insufficient_scope`. Blueprint agents: `a365 setup all --agent-registration-only`. AI Teammates: complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints
Only the delegated `Agent365.Observability.OtelWrite` scope is granted	Informational — it only matters to legacy delegated-route exporters; flag the code for migration instead (§3.5)

Do not treat a missing OtelWrite grant as a blocker for a registered blueprint agent: newer Agent 365 CLI versions skip Observability API permissions for blueprint agents, and older versions may still grant them harmlessly. Treat a non-zero inheritance result for another required resource as high. Permission names and resource display names are safe to summarize, but redact tenant, Blueprint, application, service-principal, and agent IDs.

For remediation, report the CLI guidance without applying it:

Run a365 setup requirements when the output points to a missing wids claim or CLI consent.
Run a365 setup all --agent-registration-only (idempotent) when a blueprint agent instance is not registered. For AI Teammates, point to the OtelWrite application-role step that a365 setup all --aiteammate prints (the registration-only flag does not apply to them). Run the relevant a365 setup permissions ... flow as a Global Administrator to reconcile other grants and inheritance.
For a stale Blueprint ID, hand off to a365-setup so the developer can explicitly choose the correct reuse/re-run/fresh path. Do not rewrite the ID automatically.

References:

Runtime verification commands

Generate commands appropriate for the project.

Local / App Service env check

For Azure App Service:

az webapp config appsettings list `
  -g <resource-group> `
  -n <app-name> `
  --query "[?name=='ENABLE_A365_OBSERVABILITY_EXPORTER' || name=='EnableAgent365Exporter' || name=='A365_USE_S2S_ENDPOINT']"

For local PowerShell:

$env:ENABLE_A365_OBSERVABILITY_EXPORTER="true"
$env:OTEL_LOG_LEVEL="INFO"
$env:A365_OBSERVABILITY_LOG_LEVEL="debug"

SDK log grep

Select-String .a365-observability.log -Pattern "Agent365Exporter|identity groups|Obtained token|export|No token|No eligible"

Backend verification handoff

If backend verification is required, keep customer-facing validation on public/supportable surfaces:

SDK/exporter logs show export success and no chunk failures,
direct OTel responses have partialSuccess empty/null where applicable,
Microsoft Defender Advanced Hunting CloudAppEvents shows the expected AgentId / TargetAgentId / AlternateId after indexing.

If the user is on an internal service team, they may also use their team's approved dashboard/playbook to confirm request acceptance and downstream delivery, but do not print those internal queries or endpoints in the validator report.

(Report only support-safe signals — see Output is support-safe above.)

If code checks pass but Activity is still empty — verify tenant-side

Most "exporter returns 200/sent but nothing shows in MAC" incidents are not code bugs. When the code checks above pass, have the user confirm, in the target tenant:

Provenance (verified 2026-07-06): the observability resource-SP GUID, the Agent365.Observability.OtelWrite scope, and the ingestion-lag figure below are preview-era Agent 365 facts and will change. Re-verify against current Agent 365 onboarding docs before quoting them to a customer.

Licensing state — confirm the target tenant/user meets the current Agent 365 licensing prerequisites from public onboarding docs.
Tenant enrollment state — confirm Agent 365 is enabled for the target tenant through the supported onboarding or admin process.
Observability resource SP present — az ad sp show --id 9b975845-388f-4429-889e-eab1ef63949c returns a service principal in the tenant. A 404 / AADSTS500011 ("resource principal … not found") means the observability app isn't provisioned there — an onboarding step, not a code fix.
Agent instance registration — a 403 insufficient_scope from the S2S route with a valid app-only token means the instance is not registered (and has no OtelWrite application role). For blueprint agents, registration is the expected fix, not a permission grant: a365 setup all --agent-registration-only. For AI Teammates, complete the OtelWrite application-role step that a365 setup all --aiteammate prints.
Ingestion lag — Defender CloudAppEvents populates before the admin center; give it ~5 min.

Mark task complete: "Check live blueprint permissions and runtime state".

Phase 5 — Summarize Blockers and Next Fixes

Mark task in progress: "Summarize blockers and next fixes"

Return a concise report. Default to the short format below; include the optional details block only when the user asks for deeper debugging or when a blocker needs a code pointer to be actionable.

## A365 Code Validator Result

**Status:** PASS / PASS WITH RISKS / BLOCKED

**Blockers**
1. **<short title>** — <one-sentence impact>.  
   Fix: <one-sentence fix>.
2. **<short title>** — <one-sentence impact>.  
   Fix: <one-sentence fix>.

**High-risk checks**
- <short item> — <why to verify>.

**Identity binding**
- Runtime agent id: <source / missing / from config>
- Blueprint id: <source / missing / from config>
- Verdict: <correct | wrong | unclear>

**Blueprint permissions**
- Live Blueprint: <current | stale/missing | not checked>
- Grants: <verified summary | missing required permission | not checked>
- Effective inheritance: <OK | NONE | BROKEN | not checked>

**Semantic spans**
- Found: <invoke_agent/chat/execute_tool/output_messages or "none">
- Missing: <ops or "none">

**Recommended fix order**
1. <fix root cause first>
2. <then fix config/env>
3. <then verify spans/identity>

**Optional details**
- Files: `<file>:<line or function>`, ...
- Blueprint check: <read-only command result or reason it was skipped>
- Runtime check: <local/App Service env check or SDK log grep>
- Backend verification: use the team's approved telemetry playbook; do not paste internal
  cluster names, private endpoints, or correlation IDs into this report.

Style rules:

Keep the default report to 8 bullets or fewer.
Put details under Optional details, not in the main blocker list.
Use exact file/function names for evidence, but do not paste long code blocks.
Redact per Output is support-safe above (no internal infra, tenant/agent IDs, or correlation IDs).
A skipped live Graph check is a verification gap, not automatically a code blocker.
Do not suggest a365 publish unless the agent is an AI Teammate / M365 package flow; blueprint-based observability does not use a365 publish.

Mark task complete.

Phase 6 — Offer Guided Remediation

Mark task in progress: "Offer guided remediation"

If there are no critical or high findings, say:

No blocking fixes needed. Keep the runtime verification commands handy for live validation.

Then mark the task complete and stop.

If there are critical or high findings, ask the user with AskUserQuestion:

I found blockers. What would you like me to do next?

Options:

Option	Meaning
`apply_safe_fixes`	Apply only low-risk, deterministic code/config-template fixes
`make_fix_plan`	Produce an implementation plan, no edits
`report_only`	Stop after the report

Default: make_fix_plan.

Safe fixes allowed after confirmation

Only apply these automatically when the user chooses apply_safe_fixes:

Blueprint permissions are never a safe code fix. Do not run a365 setup permissions, mutate Graph, grant admin consent, or rewrite a Blueprint ID from this skill. Report the mismatch and hand off to a365-setup or the appropriate administrator.

Python exporter kwarg
- If code calls use_microsoft_opentelemetry(...) or builds kwargs for that call and already sets enable_a365=True, add the explicit exporter flag:
```
a365_enable_observability_exporter=True
```
  or:
```
kwargs["a365_enable_observability_exporter"] = True
```
- Preserve existing code style.
Environment templates
- Add missing, commented or empty template entries only:
```
ENABLE_A365_OBSERVABILITY_EXPORTER=true
```
- Add the S2S transport setting (required in every auth mode) only if the code/reference for that project already names it and the scan reported no delegated-token finding (*-obs-delegated-token), i.e. the exporter already gets an app-only token. Moving an exporter that still sends a delegated token to the S2S route breaks export, because the S2S route rejects scp tokens. In that case, bundle the flag with the resolver swap below, which needs a second confirmation. Do not invent a setting name.
Config field scaffolding for runtime agent identity
- If the project already has a settings/config class, add a clearly named optional field such as agent_identity_app_id / AGENT_IDENTITY_APP_ID.
- Use an empty default or placeholder; never hardcode real IDs.
- Update .env.example if present.

Fixes that require a second explicit question

Ask a focused follow-up before editing:

Which runtime agent identity config name to use
- Suggested default: AGENT_IDENTITY_APP_ID / agent_identity_app_id.
- Ask if the repo already uses a different naming convention.
Token resolver wiring
- If the project already has a token provider with the required exchange flow, ask whether to wire an observability token resolver to it.
- If the exporter is fed a delegated (OBO / Agentic User) token, propose replacing it with the app-only resolver scaffold from the instrument-observability references (hosting-connection resolver for obo / agentic-user, FMI token service for s2s), together with the S2S transport flag. Change both at once, never the flag alone. Do not request an OBS permission grant as the fix.
- If no such provider exists, do not generate a full token service silently. Provide a plan and ask the user to confirm a follow-up implementation.
Semantic spans
- Ask whether to add only top-level invoke_agent / output_messages scopes first, or also wrap tools and LLM calls.
- Prefer incremental fixes: identity/exporter first, semantic span enrichment second.

Fix report

After applying fixes, show:

**Applied fixes**
- ...

**Still needs user input**
- ...

**Validation to rerun**
- `validate a365 code`
- runtime exporter log check

Run the smallest available project tests if the edited repo has an obvious test command and the user did not decline validation. Otherwise, state that no obvious test command was run.

Mark task complete.

About this skill

Overview

Output is support-safe (applies to every phase)

Phase 0 — Create and Display Task List

Phase 1 — Detect Stack and A365 Artifacts

Phase 2 — Run Static A365 Code Validator

Phase 3 — Inspect Identity Binding and Semantic Spans

3.1 Exporter activation

3.2 Identity binding

3.3 Semantic span coverage

3.4 Activity/run context coverage

3.5 S2S transport mode (every auth mode)

3.6 Blueprint permission inheritance wording

Phase 4 — Check Live Blueprint Permissions and Prepare Runtime Verification

Live Blueprint permission check (read-only Microsoft Graph)

Runtime verification commands

Local / App Service env check

SDK log grep

Backend verification handoff

If code checks pass but Activity is still empty — verify tenant-side

Phase 5 — Summarize Blockers and Next Fixes

Phase 6 — Offer Guided Remediation

Safe fixes allowed after confirmation

Fixes that require a second explicit question

Fix report