bitlysis-testing

Bitlysis — Testing Standards

When to Use

• Writing unit tests for new functions or components
• Adding integration tests for API endpoints
• Creating E2E tests with Playwright
• Reviewing test coverage and identifying gaps
• Setting up test fixtures or factories
• Writing eval tests for LLM features

Instructions

Step 1: Testing Philosophy

• Test behavior, not implementation — test what code does, not how
• Arrange-Act-Assert (AAA) pattern for all tests
• One assertion concept per test — not one assert statement
• Co-locate tests with source files: feature.ts + feature.test.ts
• Test edge cases: empty inputs, null/undefined, max values, error paths

Step 2: Vitest (TypeScript/React)

// components/data-table.test.tsx
import { describe, it, expect, vi, beforeEach } from "vitest";
import { render, screen, fireEvent } from "@testing-library/react";
import { DataTable } from "./data-table";

const mockData = [
  { id: "1", name: "Analysis 1", pValue: 0.03, effectSize: 0.45 },
  { id: "2", name: "Analysis 2", pValue: 0.21, effectSize: 0.12 },
];

describe("DataTable", () => {
  it("renders all rows when data is provided", () => {
    render(<DataTable data={mockData} />);
    expect(screen.getAllByRole("row")).toHaveLength(mockData.length + 1); // +1 header
  });

  it("shows empty state when data is empty", () => {
    render(<DataTable data={[]} />);
    expect(screen.getByText(/no results/i)).toBeInTheDocument();
  });

  it("calls onRowClick with correct data when row is clicked", () => {
    const onRowClick = vi.fn();
    render(<DataTable data={mockData} onRowClick={onRowClick} />);
    fireEvent.click(screen.getByText("Analysis 1"));
    expect(onRowClick).toHaveBeenCalledWith(mockData[0]);
  });
});

Step 3: pytest (Python / FastAPI)

# tests/test_analysis_endpoint.py
import pytest
from httpx import AsyncClient
from app.main import app

@pytest.fixture
def sample_analysis_request():
    return {"file_id": "test-file-uuid", "method": "anova"}

@pytest.mark.asyncio
async def test_run_analysis_success(
    async_client: AsyncClient,
    authenticated_user: dict,
    sample_analysis_request: dict,
):
    """Analysis endpoint returns 202 with result_id for valid request."""
    response = await async_client.post(
        "/api/v1/analyze",
        json=sample_analysis_request,
        headers={"Authorization": f"Bearer {authenticated_user['token']}"},
    )
    assert response.status_code == 202
    data = response.json()
    assert "result_id" in data
    assert 0 <= data["p_value"] <= 1

@pytest.mark.asyncio
async def test_run_analysis_rejects_unauthorized():
    """Analysis endpoint returns 401 without auth token."""
    async with AsyncClient(app=app, base_url="http://test") as client:
        response = await client.post("/api/v1/analyze", json={"file_id": "x"})
    assert response.status_code == 401

@pytest.mark.parametrize("method", ["anova", "ttest", "regression"])
@pytest.mark.asyncio
async def test_all_analysis_methods_accepted(
    async_client: AsyncClient,
    authenticated_user: dict,
    method: str,
):
    response = await async_client.post(
        "/api/v1/analyze",
        json={"file_id": "test-file-uuid", "method": method},
        headers={"Authorization": f"Bearer {authenticated_user['token']}"},
    )
    assert response.status_code == 202

Step 4: Test Factories & Fixtures

# tests/factories.py
import factory
from app.models import User, UploadedFile

class UserFactory(factory.Factory):
    class Meta:
        model = User
    
    id = factory.LazyFunction(lambda: str(uuid4()))
    email = factory.Sequence(lambda n: f"user{n}@test.com")
    name = factory.Faker("name")

class UploadedFileFactory(factory.Factory):
    class Meta:
        model = UploadedFile
    
    id = factory.LazyFunction(lambda: str(uuid4()))
    owner = factory.SubFactory(UserFactory)
    filename = factory.Sequence(lambda n: f"survey_{n}.sav")
    size_bytes = factory.Faker("pyint", min_value=1000, max_value=10_000_000)

Step 5: Playwright E2E (Key User Flows)

// e2e/upload-and-analyze.spec.ts
import { test, expect } from "@playwright/test";

test.describe("Upload and Analysis Flow", () => {
  test.beforeEach(async ({ page }) => {
    await page.goto("/login");
    await page.fill('[data-testid="email"]', "test@bitlysis.com");
    await page.fill('[data-testid="password"]', "test-password");
    await page.click('[data-testid="login-submit"]');
    await expect(page).toHaveURL("/dashboard");
  });

  test("user can upload a CSV and trigger analysis", async ({ page }) => {
    // Upload file
    await page.click('[data-testid="upload-button"]');
    await page.setInputFiles('[data-testid="file-input"]', "fixtures/sample_survey.csv");
    await expect(page.locator('[data-testid="upload-success"]')).toBeVisible();

    // Run analysis
    await page.selectOption('[data-testid="method-select"]', "anova");
    await page.click('[data-testid="run-analysis"]');

    // Wait for result
    await expect(page.locator('[data-testid="analysis-result"]')).toBeVisible({ timeout: 30_000 });
    await expect(page.locator('[data-testid="p-value"]')).toContainText(/0\.\d+/);
  });
});

Step 6: Coverage Requirements

# TypeScript — target > 80% coverage
vitest run --coverage

# Python — target > 80% coverage  
pytest tests -q --cov=app --cov-report=term-missing --cov-fail-under=80

# View HTML report
open coverage/index.html

Step 7: Test Data Management

• Use factories or fixtures — never hardcode user IDs or UUIDs
• Use a separate test database (never test against production data)
• Seed and tear down test data per test, not per suite
• For LLM tests: use deterministic mock responses, not real API calls

Output Format

For each feature tested, provide:

1. Unit tests (pure functions and components)
2. Integration tests (API endpoints)
3. E2E test (if it's a user-facing flow)
4. Coverage report (if requested)

Activate Specialist

@.cursor/.agents/.agents/qa-engineer.md