bitlysis-data-science

Bitlysis — Data Science & Analysis

When to Use

• Writing or reviewing R scripts, R Markdown, or Quarto documents
• Building Python data analysis pipelines
• Working in Jupyter notebooks (*.ipynb)
• Designing data pipeline architecture (raw → processed)
• Statistical modeling, hypothesis testing, or report generation
• Working with SPSS/Stata files (.sav, .dta)

Instructions

Step 1: Ensure Reproducibility

# Always at the top of analysis scripts
set.seed(42)  # Before any stochastic operation
library(renv)  # Confirm renv is active for this project

# Python equivalent
import random, numpy as np
random.seed(42)
np.random.seed(42)

Step 2: Data Pipeline Architecture

Follow strict data lineage — never modify raw files:

data/
├── raw/          ← NEVER modify. Original source files only.
├── interim/      ← Partially processed, transformations in progress
├── processed/    ← Clean, analysis-ready datasets
└── external/     ← Third-party reference data

For SPSS/Stata imports:

library(haven)
raw_data <- read_sav("data/raw/survey.sav")
# Document all missing codes in a companion file: data/raw/survey_codebook.md

Step 3: R Code Standards

# Good: tidyverse pipe-based, explicit, readable
analysis_result <- raw_data |>
  filter(!is.na(response_variable)) |>
  mutate(log_value = log1p(value)) |>
  group_by(segment) |>
  summarise(
    mean = mean(log_value, na.rm = TRUE),
    sd = sd(log_value, na.rm = TRUE),
    n = n(),
    .groups = "drop"
  )

# Format with: styler::style_file("analysis.R")
# Lint with: lintr::lint("analysis.R")

Step 4: Python Data Standards

import pandas as pd
import pandera as pa

# Define schema at pipeline entry points
schema = pa.DataFrameSchema({
    "user_id": pa.Column(int, nullable=False),
    "score": pa.Column(float, pa.Check.between(0, 100)),
    "segment": pa.Column(str, pa.Check.isin(["A", "B", "C"])),
})

def load_and_validate(path: str) -> pd.DataFrame:
    df = pd.read_csv(path)
    return schema.validate(df)

Step 5: Statistical Integrity

• Report effect sizes (Cohen's d, η², r) alongside p-values
• Apply multiple comparison corrections when testing multiple hypotheses:
  • Familywise: Bonferroni
  • FDR: Benjamini-Hochberg
• Distinguish exploratory (EDA) from confirmatory analysis in comments
• For SmartPLS / EViews: document GUI steps with screenshots; never fabricate output

Step 6: Report Generation (Quarto)

# Render headless (CI-compatible)
quarto render reports/analysis.qmd --to html
quarto render reports/analysis.qmd --to pdf

# Check rendering works before committing

Step 7: Quality Gates

# R
cd packages/r-pipeline
Rscript tools/ci_install.R
Rscript -e "testthat::test_dir('tests/testthat')"

# Python
ruff check app tests
mypy app --strict
pytest tests -q

Output Format

Provide reproducible code with:

1. Seed setting at the top
2. Clear data lineage documentation
3. Effect sizes alongside statistical tests
4. Quarto-renderable output where applicable

Activate Specialist

For complex pipeline design: @.cursor/.agents/.agents/data-analyst-r.md