research-design-planner

Research Design Planner

Overview

This skill guides the Research Agent in systematically composing a research methodology design (research design). This design determines the research approach, mapping of constructs/variables, data testing flow, and experimental scenarios (specifically for the fields of computing/AI/ML) to ensure the research proceeds in a structured and reproducible manner (reproducible).

Dependencies

• research-question-builder

Quick Start

Example usage: "Use the research-design-planner skill to design an experimental flow for comparing LLM robustness using the GSM8K benchmark dataset and a GPT-4 baseline."

Workflow

1. Determining the Research Approach

Determine and provide justification for the chosen approach:

• Quantitative: Statistical hypothesis testing, structured surveys, or laboratory experiments.
• Qualitative: In-depth case studies, thematic analysis, semi-structured interviews, or ethnography.
• Mixed-Methods: Sequential or concurrent combination of quantitative and qualitative.
• Computational / AI / ML Experiment: Model development research, performance analysis, robustness testing (robustness), or ablation studies (ablation study).
• Design Science Research: Creation of a new system/software artifact along with its performance evaluation.

2. Operationalization of Variables / Constructs

• For quantitative/social research: Identify Independent Variables (IV), Dependent Variables (DV), Mediator/Moderator Variables, and their measurement metrics.
• For computational/AI research: Determine the baseline model, input data (input data), control variables (control variables), and primary performance metrics (e.g., F1-score, Latency, Accuracy).

3. Data Collection & Sampling Plan

• Design the data sources used (primary vs secondary data).
• Determine the sampling method (probability vs non-probability) or dataset collection details (e.g., scraping, public benchmark datasets, API logs).
• Design the cleaning techniques (preprocessing) and data labeling.

4. Data Analysis & Validation Plan

• Determine the appropriate statistical analysis methods (e.g., ANOVA, SEM, Regression) or model training workflow (model training workflow).
• Design instrument validity/reliability tests or cross-validation plans (cross-validation) for the reliability of results.

5. Special Components of AI / Machine Learning Research (If Relevant)

For AI/ML-based research, the experimental design must describe in detail:

1. Dataset & Data Splitting: Data splitting scenario (train/validation/test split) as well as handling data imbalance (class imbalance).
2. Baseline Model: A standard comparison model that will be tested equally (fair comparison).
3. Robustness & Stress Testing: Testing the model under extreme conditions (data noise, adversarial attacks, or out-of-training distribution/out-of-distribution).
4. Ablation Study: A testing plan to verify the contribution of each new architecture component by gradually deactivating it.

Common Mistakes & Critical Rules

• Design Not Synchronized with RQ: Selecting an analysis method that cannot answer the research question (RQ).
• Biased Experimental Comparison (Unfair Comparison): In ML research, testing a new model in a specially optimized environment while the baseline model is run without fair hyperparameter tuning.
• Absence of a Validation Plan: Not including statistical significance tests or data robustness tests, so results are susceptible to being considered coincidental (cherry-picked).
• Lack of Detail in Reproducibility Procedures: Not detailing the configuration of software, libraries (libraries), hardware, or random seed (random seed) used.