mushi-integration

Mushi Integration Smoke Test

Exercises every stage of the Mushi pipeline end-to-end. Run after setup, after a deploy, or any time you need proof that the whole loop works.

Prerequisites

• mushi doctor passes (all green) — run mushi-health first if unsure.
• At least one BYOK key for anthropic and firecrawl is active.
• You have the app URL you want to map stories from.

---

Stage 1 — Bug capture

Send a real test report through the SDK pipeline:

mushi test

Expected: Test report submitted — id: rep_...

Verify in DB (Supabase MCP):

SELECT id, status, severity, category, created_at
FROM reports
ORDER BY created_at DESC LIMIT 1;

Expected: a row with status = classified and a non-null severity within ~30 seconds.
If still pending after 60 s: classify-report edge function failed — check get_logs(service: 'api').

---

Stage 2 — AI triage

Confirm the classifier ran:

mushi reports list --limit 1

Expected output includes severity, category, and blast_radius.

Verify via MCP:

get_report_detail(reportId)

Confirm classification.severity and classification.category are set.

---

Stage 3 — Story mapping

Map user stories from a live URL:

mushi stories map --url https://your-app.com --wait

--wait polls until the crawl finishes (usually 30–90 s). Expected terminal output:

✓  Crawled 12 pages
✓  Claude drafted 8 user stories
✓  Proposal created: prop_...
Open in console → Inventory → Discovery → Past proposals

Verify in DB:

SELECT id, source, status, pages_crawled, created_at
FROM inventory_proposals
ORDER BY created_at DESC LIMIT 1;

Expected: source = 'live_crawl', status = 'pending_review'.

Accept the proposal in the Mushi console (Inventory → Discovery → Past proposals → Accept), or via CLI when the accept command is available.

---

Stage 4 — TDD test generation

Pick a story id from the accepted inventory and generate a Playwright test:

# List available stories from the accepted inventory
mushi tdd pending

# Generate a test (review mode — goes to approval queue)
mushi tdd gen <story-id> --mode review

Expected output:

✓  Test generated: qa_...
✓  Draft PR opened: https://github.com/.../pull/...
   Waiting for approval — run: mushi tdd approve qa_...

Verify in DB:

SELECT id, title, approval_status, source, automation_mode, created_at
FROM qa_stories
ORDER BY created_at DESC LIMIT 1;

Expected: source = 'test_gen_from_story', approval_status = 'pending_review'.

Via MCP:

list_pending_review_stories(projectId)

---

Stage 5 — Approval and execution

Approve the generated test:

mushi tdd approve <qa-story-id>

Trigger a manual run immediately:

mushi tdd run <qa-story-id>

Verify in DB:

SELECT id, status, latency_ms, provider_session_url, created_at
FROM qa_story_runs
WHERE qa_story_id = '<qa-story-id>'
ORDER BY created_at DESC LIMIT 1;

Expected: status = 'completed' (or failed — a failure here is fine; it means the test ran and detected real friction).

Via MCP:

run_qa_story(projectId, qaStoryId)

---

Stage 6 — PDCA improvement cycle

If Stage 5 produced a failure, trigger the PDCA improver:

mushi tdd improve

Expected: Claude analyzes the failure, writes an improved test, and queues it for review.

Verify:

SELECT id, title, source, parent_story_id, approval_status, created_at
FROM qa_stories
WHERE source = 'pdca'
ORDER BY created_at DESC LIMIT 3;

Expected: at least one row with source = 'pdca' and a parent_story_id pointing to the original.

---

Stage 7 — Evolution loop (optional)

Check the full PDCA dashboard to confirm the loop is converging:

Via MCP resource:

project://dashboard

Look for:

• Rising judge_scores over time.
• Falling recurrence_rate (same bugs re-appearing).
• fix_attempts with status = 'completed' outpacing failed.

---

Pass/Fail Summary

Stage	What ran	Status	Notes
1. Bug capture	mushi test → reports row	✅ / ❌
2. AI triage	classify-report → severity/category	✅ / ❌
3. Story mapping	story-mapper → inventory_proposals	✅ / ❌
4. TDD generation	test-gen-from-story → qa_stories	✅ / ❌
5. Approval + run	qa-story-runner → qa_story_runs	✅ / ❌
6. PDCA improve	pdca-runner → qa_stories (source=pdca)	✅ / ❌

All ✅ → Mushi is fully operational end-to-end.
Any ❌ → the relevant edge function failed. Use mushi-debug for targeted diagnosis.

---

Tips

• Fastest smoke test: Stages 1–2 only. Takes ~60 s and confirms bug capture + triage is alive.
• Story map only: Stage 3. Useful after changing the Firecrawl key or updating the story-mapper function.
• TDD-only check: Stages 4–6. Run this after changing test-gen-from-story or pdca-runner.
• Browserbase vs Firecrawl: Stage 5 uses firecrawl_actions by default. To test Browserbase: set provider = 'browserbase' on the QA story in the console first.