SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsexperiment-design

experiment-design

Design & creation

View repository

Updated 10-16 08:43

About this skill

Experimental Design Skill

Overview

This skill provides a structured experimental design workflow, applicable to academic research in fields such as machine learning, natural language processing, and computer vision. The goal is to help researchers start from vague research ideas and produce a rigorous, reproducible, and persuasive experimental plan.

Core Design Principles

A good experimental design should have the following characteristics:

  • Falsifiability: Experimental results must be able to support or refute the research hypothesis
  • Fairness: All comparison subjects are evaluated under the same conditions
  • Reproducibility: Others can fully reproduce the experiment based on the description
  • Sufficiency: The experiment covers enough aspects to support the paper's conclusions

Experimental Design Pipeline

A complete experimental design follows the following six-step process:

假設 → 變數 → 指標 → Baseline → Ablation → 計算預算

The output of each step is the input of the next step, forming a rigorous chain of derivation.

Step One: Clarifying the Research Hypothesis

Purpose

Transform a vague research motivation into a verifiable, specific hypothesis.

Method

  1. Identify the research question: What question do you want to answer?
  2. Propose the core hypothesis: What is the expected answer to the question?
  3. Clarify the hypothesis: The hypothesis must be measurable and falsifiable
  4. Decompose sub-hypotheses: Break down a complex hypothesis into sub-hypotheses that can be verified one by one

Quality Standards for Hypotheses

Standard Description
Specificity Clearly state the expected direction and magnitude of the effect
Measurability Can be verified using quantitative metrics
Falsifiability There exist experimental results that could refute the hypothesis
Relevance Directly related to the research question

Examples

  • Poor: "Our method is better"
  • Good: "On the SQuAD 2.0 dataset, after adding the cross-attention mechanism, the F1 score improves by at least 2 percentage points compared with the pure self-attention baseline"

See also:Experimental Planning Reference

Step Two: Variable Definition

Independent Variables

Variables that the researcher actively manipulates, that is, "the things that are changed" in the experiment.

  • Variants of model architecture
  • Differences in training strategies
  • Differences in data processing methods

Dependent Variables

Variables used to measure experimental results, that is, "the things that are measured."

  • Model performance metrics (accuracy, F1, BLEU, etc.)
  • Efficiency metrics (inference time, memory usage)
  • Quality metrics (human evaluation scores)

Control Variables

Variables kept constant in the experiment to ensure fairness of comparison.

  • Random seed
  • Training dataset and splitting method
  • Hyperparameters (the parts that are not the research subject)
  • Hardware environment
  • Pretrained model version

Variable control principles

  1. Single-variable principle: Each experiment changes only one independent variable
  2. Complete recording principle: The values of all variables must be recorded
  3. Reasonable range principle: The value range of the independent variable should have a theoretical basis

See:Experiment planning reference

Step 3: Evaluation metric selection

Selection principles

  1. Domain conventions: Prioritize standard metrics recognized in the field
  2. Multi-faceted coverage: Report performance, efficiency, and robustness metrics simultaneously
  3. Statistical significance: Report the mean and standard deviation of multiple experiments
  4. Reasonableness: The metric can truly reflect the aspect concerned by the research hypothesis

Common metric categories

Category Metric examples
Classification tasks Accuracy、Precision、Recall、F1-score、AUC-ROC
Generation tasks BLEU, ROUGE, METEOR, BERTScore, human evaluation
Information extraction MAP、MRR、NDCG、Recall@K
Efficiency metrics FLOPs, number of parameters, inference latency, memory usage
Robustness Cross-dataset performance, adversarial example accuracy

Statistical testing

  • Report the mean and standard deviation of experiments with multiple random seeds
  • Conduct statistical significance tests when necessary (e.g., paired t-test, bootstrap test)
  • Label the statistical significance level (p < 0.05, p < 0.01)

Step 4: Baseline selection and setup

Required baseline types

  1. Classic methods: Important historical methods in the field
  2. Current SOTA: The latest best-performing method
  3. Simple baselines: Simple but reasonable baseline methods (e.g., random, majority class, TF-IDF)

Fair comparison principles

  • Use the same data split
  • Use the same evaluation protocol
  • Use the original authors' code and hyperparameters whenever possible
  • If reimplementation is needed, verify that the reproduced results are consistent with the original paper

Common mistakes

  • Comparing only with weak baselines
  • Not using the latest SOTA as a baseline
  • The baseline's hyperparameters have not been tuned
  • Inconsistent comparison conditions (e.g., different pretrained models)

See:Baseline selection guide

Step 5: Ablation Study design

Ablation Study is a key experiment for verifying the contribution of each component in a method. This skill defines four ablation modes:

Mode 1: Component Ablation

Remove or replace each component in the method one by one and observe the performance changes.

  • Remove only one component at a time
  • Record the performance change after removal
  • Use this to determine the contribution of each component

Mode 2: Hyperparameter Sensitivity

Explore the impact of key hyperparameters on performance.

  • Select the 2-4 most important hyperparameters
  • Vary hyperparameter values within a reasonable range
  • Plot hyperparameter-performance curves

Mode 3: Cross-Dataset Transfer

Verify the method's generalization ability.

  • Test on multiple different datasets
  • Include datasets of different scales and domains
  • Analyze under what conditions the method performs best or worst

Mode 4: Qualitative Analysis

Gain an in-depth understanding of model behavior through visualization and case analysis.

  • Attention weight visualization
  • Success and failure case analysis
  • Feature space visualization (e.g., t-SNE)
  • Error type classification and statistics

See:Ablation Design Guide

Step 6: Computational Resource Estimation

Estimation Item

  1. Single Experiment Cost

    • GPU Hours
    • Memory Requirement
    • Storage Requirement

  2. Total Experiment Calculation

    總 GPU 時數 = 單次時數 × 模型變體數 × 資料集數 × 隨機種子數 × 超參組合數

  3. Safety Factor

    • It is recommended to reserve 1.5-2 times the estimated resources
    • Consider the needs for debugging, preliminary experiments, and additional experiments

Resource Optimization Strategies

  • First conduct preliminary experiments with a small-scale dataset
  • Use early stopping to save training time
  • Make good use of mixed precision training
  • Arrange experiment priorities reasonably

Reproducibility Requirements

The experiment plan must include complete reproducibility information to ensure that others can accurately reproduce the results.

Required Disclosure Items

  1. Hardware Environment

    • GPU model and quantity
    • CPU specifications
    • Memory size

  2. Software Environment

    • Programming language version
    • Deep learning framework version
    • Key package versions

  3. Randomness Control

    • Random seed settings
    • Deterministic algorithm settings
    • List of seeds for multiple experiments

  4. Training Protocol

    • Complete list of hyperparameters
    • Optimizer settings
    • Learning rate schedule
    • Data augmentation strategy
    • Early stopping criteria

  5. Data Processing

    • Dataset version and source
    • Preprocessing steps
    • Data splitting method

  6. Evaluation Protocol

    • Precise definition of evaluation metrics
    • Evaluation frequency
    • Model selection criteria

See:Reproducibility checklist

Output: A structured experiment plan document

The final output of this skill is a structured experiment plan document, containing the following sections:

  1. Research hypotheses and sub-hypotheses
  2. Variable definition table
  3. Evaluation metrics and statistical methods
  4. Baseline list and settings
  5. Ablation Study design matrix
  6. Computational resource estimation and schedule planning
  7. Reproducibility information

Use template:Experiment plan template

Usage workflow

Input

  • Research topic or paper draft
  • Description of the proposed method
  • Available computational resources

Processing

  1. Guide the user to clarify the research hypotheses
  2. Assist in defining independent variables, dependent variables, and control variables
  3. Suggest evaluation metrics based on the task type
  4. Suggest baselines based on the research field
  5. Design an Ablation Study plan
  6. Estimate computational resource requirements

Output

  • Complete experimental plan document (in template format)
  • Experimental priority recommendations
  • Potential risks and mitigation plans

Quality checklist

After completing the experimental plan, please confirm the following items:

  • Each research hypothesis has a corresponding experiment to verify it
  • The value ranges of all independent variables are clearly defined
  • Control variables are fully listed
  • Evaluation metrics cover multiple aspects
  • Baselines include classic methods, SOTA, and simple baselines
  • The Ablation Study covers all proposed components
  • Computational resource estimates are reasonable and include a safety factor
  • Reproducibility information is complete
  • Statistical testing methods have been determined

Reference resources