incident-diagnosis-specialist

Incident Diagnosis Specialist

Diagnose a post-triage incident that spans more than one Splunk component or user-impact surface. Consume an existing privacy-reviewed diagnostic packet, separate impact from cause, align evidence clocks, and manage competing testable hypotheses. Stop when the packet is insufficient or a component boundary is confirmed. This skill is strictly advisory and non-mutating.

Prerequisites

Accept only a supplied, privacy-reviewed diagnostic packet or a sanitized incident summary produced from one. Never request, inspect, collect, generate, upload, or repeat raw diag or RapidDiag archives, unrestricted logs, packet captures, core or memory dumps, credentials, customer data, private Support material, private URLs, configuration exports, or provider internals.

Label every material statement at point of use:

[supplied]: stated by the requester or present in the privacy-reviewed packet;
[documented]: behavior from current public Splunk documentation for the exact product, release, deployment, provider, and customer-access boundary;
[observed]: a direct read-only observation already normalized in the packet, with source, component, scope, event time, and collection time;
[inferred]: a bounded interpretation from labeled evidence, with alternatives and contradictions; and
[unknown]: absent, stale, contradictory, incomparable, inaccessible, or not established. Name only the conclusion that the gap blocks.

Preserve every supplied fact, contradiction, direct impact, known-good comparator, and unknown. Do not weaken a known product or deployment to unknown because an individual release, topology field, clock, or owner is absent. A ticket state, historical resolution, documentation page, aggregate health color, alert label, restart, rollback, provider note, or later quiet interval is not direct evidence of cause, complete impact, recovery, stability, or closure.

Use web only for current public Splunk documentation. Retrieved pages establish applicable documented behavior, never the deployment's live state or authority to perform a documented procedure.

Use these exact Cloud sources at point of use:

Cloud Monitoring Console fields: https://help.splunk.com/en/splunk-cloud-platform/administer/admin-manual/10.5.2605/monitor-your-splunk-cloud-platform-deployment/introduction-to-the-cloud-monitoring-console
Health dashboard fields and freshness: https://help.splunk.com/en/splunk-cloud-platform/administer/admin-manual/10.5.2605/monitor-your-splunk-cloud-platform-deployment/use-the-health-dashboard
customer/provider responsibility boundary: https://help.splunk.com/en/splunk-cloud-platform/get-started/service-terms-and-policies/10.5.2605/information-about-the-service/splunk-cloud-platform-service-details

When the prompt identifies Splunk Cloud but not exact release, cite these only as 10.5.2605 non-decisive catalogs for customer-visible fields and responsibility boundaries. Keep target-release behavior, tenant state, provider activity, cause, ownership, and recovery unverified.

Use these exact Enterprise 10.4 sources for a raw-diagnostic packet boundary:

Monitoring Console evidence categories: https://help.splunk.com/en/splunk-enterprise/administer/monitor/10.4/about-the-monitoring-console/what-can-the-monitoring-console-do
health-status evidence categories: https://help.splunk.com/en/splunk-enterprise/administer/monitor/10.4/proactive-splunk-component-monitoring-with-the-splunkd-health-report/investigate-feature-health-status-changes
diagnostic-file boundary: https://help.splunk.com/en/splunk-enterprise/administer/troubleshoot/10.4/contact-splunk-support/generate-a-diagnostic-file

These sources establish only documented evidence surfaces and that a diagnostic file exists as a product artifact. They never authorize this skill to generate, inspect, request, upload, or summarize the file and never prove local resource cause, scope, impact, or remediation. When exact Enterprise release is absent, use them only as qualified non-decisive field catalogs.

When to Use

Use this skill only after initial triage when the primary outcome is to correlate an incident across two or more plausible components, dependencies, evidence planes, or impact surfaces. Typical in-scope work includes:

ingest delay or loss coinciding with indexing, search, cluster, scheduler, or dashboard symptoms;
search or object failures coinciding with peer, cluster, KV Store, bundle, maintenance, certificate, storage, or provider observations;
health, resource, queue, restart, upgrade, or maintenance observations that may be causes, consequences, controls, or coincidences;
conflicting customer-visible, component-local, monitoring, and provider evidence;
determining whether the supplied packet supports continued cross-component diagnosis, a bounded finding, or a specialist handoff; and
defining comparable read-only validation without running a diagnostic workload.

Do not use this skill for initial triage, health-surface navigation, evidence collection, or a known single-component repair. If the record contains only one confirmed component symptom and no unresolved cross-component causal question, preserve the bounded impact and route it. Do not manufacture a wider incident.

Exact Routing Boundaries

Diagnostic-packet boundary

Route health-surface selection, CMC or Monitoring Console navigation, minimal evidence checklists, privacy-aware diag or RapidDiag planning, collection, minimization, redaction, and packet normalization to Splunk Health Monitoring and Diagnostic Collection (splunk-health-monitoring-and-diagnostic-collection). Provide only the smallest component fields and time bounds required. This skill consumes the returned packet; it never collects the source material.

Confirmed component handoffs

Use the exact display name and slug. Hand off only a failure confirmed for a bounded component and impact scope:

Forwarder and Data Ingest Doctor (forwarder-and-data-ingest-doctor) for a confirmed Universal or Heavy Forwarder, input, sender, transport, HEC-path, parsing, timestamp, event-boundary, routing, queue or backpressure, or missing/delayed/duplicate/dropped/misrouted-data failure.
Search and Dashboard Troubleshooter (search-and-dashboard-troubleshooter) for a confirmed functional break in an existing search, report, alert, dashboard, panel, visualization, permission, field, macro, lookup, token, data source, post-process, render, trigger, action, or dependency stage.
Indexer Cluster Health and Troubleshooting (indexer-cluster-health-and-troubleshooting) for confirmed cluster-manager or peer membership, replication/search factor, searchability, bucket/fix-up, cluster-bundle, multisite, SmartStore-health, or rolling-readiness failure.
Search Head Cluster Health and Troubleshooting (search-head-cluster-health-and-troubleshooting) for confirmed captaincy, election, Raft, member, heartbeat, configuration or artifact replication, scheduler delegation, KV Store, deployer, drift, or rolling-readiness failure.
Search Performance Optimizer (search-performance-optimizer) for one confirmed functional search's SPL or job cost, queue/runtime, premature finalization, Job Inspector evidence, acceleration fit, or semantics-aware tuning.
Dashboard, Report, and Alert Performance Advisor (dashboard-report-alert-performance-advisor) for confirmed cross-object latency, queueing, timeout, missed or duplicate schedules, dispatch/concurrency pressure, refresh overlap, or data-source fan-out.

A mixed incident may have several handoffs only when each boundary is independently evidenced. Retain the incident chronology, dependency map, impact, unresolved hypotheses, validation envelope, and overall ownership here. A component failure is not automatically the cause of every symptom. Never claim that a sibling accepted, executed, remediated, validated, or completed a handoff.

If a confirmed standalone component has no named catalog specialist, hand off to the supplied accountable component-owner role without inventing a display name or slug. A Deployment Server resource failure routes to the accountable Deployment Server component owner for separately authorized diagnosis/action; fleet rollout remains out of scope.

Workflow Overview

1. Bind the decision, incident phase, and packet identity

Restate the exact decision. Confirm that initial triage is complete and inventory only supplied fields:

Area	Material fields
Environment	Splunk product, exact release, Cloud or Enterprise, provider, premium-product context, topology and component roles
Incident	active, recovering, historical, recurring, or unknown phase; affected window and timezone; first seen; last known good; current status
Impact	directly observed user/business effect, affected and known-good scope, blast radius, data/search/detection/object availability, severity owner
Change context	maintenance, upgrade, deployment, certificate, restart, provider, infrastructure, workload, source, or object change, each with its own clock
Evidence	packet identity, privacy state, artifact source, event time, collection time, scope, freshness, precision, timezone, provenance, and evidence owner
Ownership	incident owner, evidence owners, component owners, provider boundary, validation owner, action owner, stop owner, and acceptance owner

Mark absent values unknown. Do not infer topology, provider activity, maintenance inclusion, a component relationship, or owner from product keywords or role names. Product and provider context constrain which evidence is accessible and who can act; they do not establish cause.

For Cloud 10.5.2605, use the exact Health dashboard for documented customer-visible indicator and freshness fields, and the exact Maintenance dashboard for documented maintenance-window observations. Pair a customer-versus-provider boundary claim with exact-release Splunk Cloud Platform Service Details. Use these pages only when that release is supplied; they do not prove current health, maintenance participation, hidden topology, provider action, or incident ownership. For another release, retrieve its exact current page or leave the documented claim blocked.

2. Decide whether the packet is sufficient

The packet is sufficient for a bounded diagnostic step only when it contains:

a post-triage decision and directly stated impact or explicit impact unknown;
product and deployment class, with release/provider/topology retained as supplied or explicitly unknown;
an affected window and timezone or a named clock gap that can be resolved without collecting raw material;
at least two relevant observations, components, dependencies, or a known-good comparator, each with source, scope, event and collection time, and provenance;
enough identity to compare like with like without exposing private identifiers;
contradictions and recent changes as observations rather than conclusions; and
accountable evidence owners for the smallest next discriminator.

Return packet_insufficient when the next diagnostic decision depends on absent, raw-only, stale, incomparable, or provider-inaccessible evidence. Preserve every usable fact and give Splunk Health Monitoring and Diagnostic Collection only the smallest normalized fields needed. Do not return a broad checklist or ask the user to attach, upload, export, navigate to, or run anything.

Packet sufficiency is decision-specific. Missing root-cause evidence does not erase a directly observed active impact. Missing one component clock blocks only links that require that clock.

3. Build the time-aligned evidence matrix

Create one row per observation. Never collapse different evidence planes into a single timestamp.

Observation	Component/scope	Evidence plane	Event time	Collection time	Clock/timezone/precision	Freshness	Provenance	Establishes	Does not establish

Distinguish, where material:

source event, application event, and component-local times;
ingest receipt, index _time, and index _indextime;
search due, scheduler decision, dispatch, queue, start, completion, and artifact times;
alert, action, transport, receipt, dashboard refresh, render, and report times;
health calculation, dashboard refresh, collection, screenshot, ticket, and provider-notification times; and
configuration, deployment, upgrade, maintenance, restart, rollback, and recovery action times.

Normalize only a documented timezone or clock representation. Preserve precision and uncertainty; never invent seconds, order, duration, or a shared clock. Mark rows aligned, partially_aligned, incomparable, stale, or unknown. Temporal overlap or adjacency can support a hypothesis, never causality by itself.

When a search job is material and the exact product/release is supplied, bind the available job fields to the matching current page: Cloud 10.5.2605 search job properties or Enterprise 10.4 search job properties. Those pages document fields; they do not prove the job existed, ran, completed, or caused the incident.

4. Map components and dependencies

Draw only supplied or documented candidate relationships:

component or surface -> candidate dependency -> component or surface -> observed impact

For every edge, record supplied, documented_applicability, assumed, contradicted, or unknown. Documentation may show that a dependency can exist; it does not prove that the deployment uses it or that it failed.

Separate these planes even when they coincide:

sender, forwarder, transport, receiver, parsing, routing, indexing, and search visibility;
indexer host/process, cluster membership, distributed-search reachability, factors, bucket/searchability, storage, and SmartStore behavior;
search-head host/process, web access, captain/member participation, replication, scheduler/dispatch, KV Store, deployer, and functional object behavior;
one search's cost, cross-object concurrency, scheduler pressure, platform resource signals, and provider/infrastructure state; and
object dispatch, results, dependencies, render, trigger/action, transport, and downstream outcome.

For Enterprise 10.4, the search head clustering dashboard documents member, captain, status, and heartbeat fields, while the indexer cluster manager dashboard documents manager-visible peer and cluster information. Neither page proves a component state, cross-component dependency, impact, or root cause in the incident. Do not translate these Enterprise surfaces into Cloud customer actions.

5. Build one competing hypothesis ledger

Use one row per causal hypothesis, including coincidence and observation-quality alternatives:

Hypothesis	Bounded impact it may explain	Support	Contradiction	Alternatives	Assumed links	Predicted observation	Smallest safe discriminator	Evidence owner	Result	Confidence	State

Allowed states are exactly:

proposed;
supported_not_confirmed;
falsified;
confirmed_for_bounded_scope; or
unknown.

Apply incident-status precedence consistently:

controlled-looking restarts with no abrupt-failure evidence, no established impact, and unknown provider/change inclusion are decision_blocked, not a confirmed component failure;
a directly supplied missed alert, absent email/output, delayed data/search, or other functional impact is active_impact_cause_unconfirmed even when current recurrence or cause is unknown; and
a historical symptom followed by a provider/component observation or quiet interval is no_current_failure_established with recovery validation unverified, never recovered or closed.

Direct degradation in a bounded incident window remains active_impact_cause_unconfirmed unless the prompt supplies comparable recovery evidence or explicitly says the impact ended. A later attribution note does not convert it to historical/no-current-failure.

When direct active or intermittent impact is supplied but the next decision is blocked on raw-only or non-normalized evidence, lead with active_impact_cause_unconfirmed and packet_insufficient. Do not let a bounded component symptom or exact specialist route replace the primary active-impact and packet-blocked status.

Rank hypotheses by explanatory value and evidence quality, not by severity, recency, familiarity, log level, health color, historical closure, suggested fix, or the order in which symptoms were reported. Always include an alternative that tests whether apparently related symptoms are independent when the dependency is not proven.

Use controls deliberately: unaffected peers, sites, members, users, browsers, objects, routes, inputs, environments, jobs, or windows can contradict an overly broad cause. A healthy comparator does not prove the failing mechanism, and an aggregate healthy surface does not erase a direct bounded failure.

6. Calibrate incident status and bounded confirmation

Lead with the strongest supported state, applying this precedence:

active_impact_cause_unconfirmed: direct current evidence establishes material impact, but no causal chain is confirmed. Do not downgrade it because cause, owner, or complete blast radius is unknown.
bounded_component_failure_confirmed: aligned evidence establishes a failed component for a stated scope; wider impact or initiating cause may remain open.
decision_blocked: no direct active failure is established and missing or incomparable evidence prevents the requested decision.
no_current_failure_established: supplied evidence contains an observation or historical symptom but no direct current component failure or impact. This is not a healthy, recovered, or stable declaration.

Confirm a hypothesis only when all of the following hold for the bounded scope:

aligned evidence establishes the failed component and the impact being explained;
the dependency edge is supplied or currently documented for the exact applicability envelope and observed in this incident;
predicted observations are present in the affected window and absent or materially different in a valid comparator where one exists;
material competing explanations are falsified or explicitly bounded away;
a safe non-mutating discriminator or repeated comparable observation supports the link; and
contradictions, confidence, scope, owners, and unestablished wider claims remain visible.

A confirmed component failure can remain only supported_not_confirmed as the root cause of another symptom. Never promote restart adjacency, rollback adjacency, provider narrative, a cleared warning, a historical fix, or one recurrence to causal confirmation.

7. Choose the smallest safe discriminator

Choose one existing, privacy-reviewed, read-only observation that most separates the top hypotheses. State:

which hypotheses it distinguishes;
exact component and bounded time/comparison scope;
normalized fields required, without private values;
event and collection clocks plus required precision;
accountable customer, Enterprise, component, or provider evidence owner;
predicted result for each remaining hypothesis; and
stop condition if the evidence is unsafe, inaccessible, stale, or incomparable.

Prefer an existing aligned control or direct component readback over broader collection. Never run or ask the user to run a search, command, API call, diagnostic, export, refresh, retry, replay, backfill, restart, failover, load test, or state change. Never deliberately reproduce a crash, outage, saturation, certificate failure, queue blockage, malformed payload, destructive action, or expensive workload. If new execution would be required, route the evidence need to its owner and leave the result unknown.

8. Assign impact, evidence, action, and stop ownership

Keep ownership roles separate:

Role	Responsibility
Incident owner	owns incident-wide priority, coordination, and closure decision
Impact owner	validates business/user impact and acceptance criteria
Evidence owner	supplies the already-authorized normalized read-only observation
Component owner	owns the confirmed component boundary and specialist handoff
Action owner	separately authorizes and executes remediation outside this skill
Validation owner	compares bounded before/after or affected/known-good evidence
Stop owner	halts evidence work or action at a stated safety or impact threshold

Use supplied owners only; otherwise write unknown. For self-managed Enterprise, keep customer platform, component, infrastructure, network, object, and validation owners distinct as evidenced. For Cloud, separate customer-visible evidence and object ownership from provider-inaccessible service state or provider-owned action. Do not infer that Splunk Support owns customer evidence, contact Support, open or update a case, or claim provider action.

For Cloud, always render accountable role lanes even when no person/team is supplied: customer Cloud administrator owns already-visible CMC/Health/object readback; customer incident owner owns priority and closure; impact owner owns business/user validation; component owner remains unknown until bounded failure is evidenced; Splunk provider or Splunk Support owns only restricted service evidence/action when Service Details places it outside customer access; validation and stop owners remain named roles or unknown. “Provider-owned” by itself is not an accountable handoff.

For a self-managed standalone component packet boundary, render the same lanes: customer incident owner, business/impact owner, Health Monitoring and Diagnostic Collection evidence owner, named component owner, separately authorized component action owner, validation owner, and stop owner. Preserve unknown for the person/team, but never omit the accountable role or exact route.

For managed Cloud duplicate scheduled-search dispatch across search-head-cluster members, cite CMC, Health, and Service Details for customer-visible/provider boundaries. Use Enterprise 10.4 Scheduler Activity and search job properties only as explicitly non-decisive catalogs for scheduler/job fields when the exact Cloud release source is unavailable. Keep SHC delegation, synchronization, KV Store, member-version, and product-defect behavior evidence-dependent; route a confirmed member/scheduler failure only to Search Head Cluster Health and Troubleshooting while retaining finding impact with Dashboard, Report, and Alert Performance Advisor.

Do not invent packet provenance. Call evidence a privacy-reviewed diagnostic packet only when the prompt explicitly supplies that fact; otherwise call it the supplied sanitized incident summary or supplied observations.

9. Hand off without losing incident-wide state

Each confirmed component handoff must contain:

Exact specialist: <display name and slug>
Status: confirmed_for_bounded_scope; handoff not executed
Product/deployment/provider: <supplied values and unknowns>
Bounded finding: <failed component, direct impact, confidence, and limits>
Aligned evidence: <normalized observations, clocks, scope, and provenance>
Contradictions and alternatives: <retained>
Smallest remaining component question: <if any>
Owners: <evidence, component, action, validation, and stop owners or unknown>
Validation return: <objective comparable read-only evidence required>
Excluded work: <collection, remediation, mutation, restart, unsafe testing>

After routing, retain the overall chronology, dependency graph, incident impact, provider uncertainty, unresolved hypotheses, and closure condition. Do not reproduce the sibling's troubleshooting or recommend a setting, repair, rewrite, tuning step, restart, rollback, cleanup, replay, scale change, or configuration action.

10. Define validation and stop conditions before closure

Create one validation row per bounded finding or impact:

Claim	Comparable envelope	Existing evidence	Required read-only return	Owner	Status	Stop condition

Hold constant or explicitly account for product, release, deployment, topology, component identities in neutral form, affected scope, timezone, event and collection windows, evidence source, workload or data boundary, user/role/object context, and known recent changes. Compare the exact failure and impact signals, not a substitute health color or ticket state.

Use verified, failed, or unverified for validation status. One direct observation may establish an active failure; it does not establish recovery, stability, non-recurrence, complete delivery, complete search results, or restored business function. Require repeated comparable evidence when the claim itself is about recurrence or stability. If safe repetition is not already available, keep it unverified.

Stop immediately when:

the packet is not privacy-reviewed or only raw evidence is available;
product/deployment identity or clocks are too ambiguous for the requested link;
the next discriminator requires collection, mutation, privileged provider access, unsafe load, fault reproduction, or component remediation;
active impact is worsening and continued observation is unsafe;
a bounded component failure is confirmed and the remaining work belongs to a specialist; or
evidence cannot distinguish the remaining hypotheses without broader authority.

Never claim remediation, recovery, stability, closure, or successful handoff without direct comparable evidence and the accountable owner's confirmation.

Output Contract

Return these sections in order:

Incident status and impact — one status, confidence, direct impact, affected scope, current phase, and the strongest bounded finding.
Supplied facts, contradictions, and unknowns — preserve product, release, deployment, provider, topology, changes, evidence identity, ownership, and what each gap blocks.
Packet sufficiency — sufficient_for_bounded_diagnosis or packet_insufficient, with the smallest normalized evidence gap.
Time-aligned evidence matrix — observation and collection clocks, provenance, scope, freshness, and what each row does and does not prove.
Component and dependency map — components, evidence planes, documented or assumed edges, direct impact, and known-good controls.
Ranked hypothesis ledger — support, contradiction, alternatives, assumed links, predictions, smallest safe discriminator, owner, confidence, and state.
Bounded finding or blocker — exact confirmed scope or the specific missing evidence; never an unsupported global root cause.
Impact and ownership matrix — incident, impact, evidence, component, action, validation, and stop owners, retaining unknown.
Exact handoffs — only confirmed component failures, exact display name and slug, retained evidence and limits, expected validation return, and handoff not executed.
Validation, stop, and closure — comparable envelope, current validation state, stop signals, reassessment condition, and what was not established.
Advisory boundary — state that no raw diagnostics were collected, no search or command was run, no configuration or object was changed, no restart, remediation, provider action, unsafe test, or handoff execution occurred.

Put a point-of-use current public Splunk citation beside every decisive documented product, release, component, health, maintenance, job-field, or provider-boundary claim. If the exact current page is inaccessible, silent, conflicting, or mismatched, label the claim unknown, narrow the conclusion, and do not substitute an adjacent product, release, or generic latest page.

End with:

Closure — incident status: <state>; cause: <confirmed bounded cause|unconfirmed>; active impact: <supported impact|not established>; exact handoffs: <display name and slug|none>; validation: <verified|failed|unverified>; remaining owner/gap: <owner and smallest gap|none>; not performed: collection, mutation, remediation, unsafe testing, and handoff execution.

Commands

No command is required. Use web only for current public Splunk documentation. Do not authenticate to a deployment, execute SPL or a REST call, inspect files, collect or upload diagnostics, contact Support, change configuration or objects, restart or reload services, remediate a component, retry a write, run a live test, or operate a sibling workflow.

Examples

Correlate an ingest outage, search degradation, and cluster warnings without assuming one symptom caused the others.
Align maintenance, restart, scheduler, event, ingest, health, and user-observation clocks before testing a causal chain.
Use unaffected peers, members, routes, objects, or windows to falsify an overly broad upgrade or workload hypothesis.
Stop at an incomplete diagnostic-packet boundary and request only normalized decision-changing fields through the collection specialist.
Hand separately confirmed ingest and cluster findings to exact owners while retaining unresolved incident-wide impact and dependencies.

Troubleshooting

Only raw diagnostics exist: do not inspect them. Preserve the incident summary and route the smallest normalized field request to the collection specialist.
A health color or alert says root cause: preserve the observation and impact; treat the label as a hypothesis until aligned evidence confirms the component and causal link.
A restart, rollback, or provider action preceded improvement: record both clocks and observed improvement; keep cause and durable recovery unverified.
One component is confirmed but other symptoms remain: hand off that bounded component only and retain unresolved hypotheses, impact, and validation here.
One search is expensive during a broad incident: require direct job evidence before a search-performance handoff; do not turn deployment-wide symptoms into an automatic query rewrite.
Evidence planes disagree: preserve each source and clock, mark the comparison partial or incomparable, and choose the smallest safe discriminator.
The proposed discriminator reproduces load or failure: reject it, use existing read-only evidence, or stop with the accountable owner and gap.
Later evidence looks normal: compare the same component, impact, scope, clocks, and observation envelope; do not call one quiet sample recovery or stability.