SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsexperiment-design

experiment-design

Design & creation

View repository

Updated 10-16 01:17

About this skill

Experimental Design Skill

Overview

This skill provides a structured experimental design workflow, applicable to academic research in fields such as machine learning, natural language processing, and computer vision. The goal is to assist researchers in starting from vague research ideas and producing a rigorous, reproducible, and persuasive experimental plan.

Core Design Principles

A good experimental design should possess the following characteristics:

  • Falsifiability: Experimental results must be able to support or refute the research hypothesis
  • Fairness: All comparison targets are evaluated under the same conditions
  • Reproducibility: Others can fully reproduce the experiment based on the description
  • Sufficiency: The experiments cover enough aspects to support the paper's conclusions

Experimental Design Pipeline

A complete experimental design follows the following six-step workflow:

假設 → 變數 → 指標 → Baseline → Ablation → 計算預算

The output of each step is the input of the next step, forming a rigorous chain of derivation.

Step One: Clarifying the Research Hypothesis

Purpose

Transform a vague research motivation into a verifiable, specific hypothesis.

Method

  1. Identify the research question: What question do you want to answer?
  2. Propose the core hypothesis: What is the expected answer to the question?
  3. Clarify the hypothesis: The hypothesis must be measurable and falsifiable
  4. Decompose sub-hypotheses: Break down complex hypotheses into sub-hypotheses that can be verified one by one

Quality Standards for Hypotheses

Standard Description
Specificity Clearly indicate the expected direction and magnitude of the effect
Measurability Can be verified using quantitative metrics
Falsifiability There exist experimental results that could refute the hypothesis
Relevance Directly related to the research question

Example

  • Poor: "Our method is better"
  • Good: "On the SQuAD 2.0 dataset, after adding the cross-attention mechanism, the F1 score improves by at least 2 percentage points compared to the pure self-attention baseline"

See also:Experimental Planning Reference

Step Two: Variable Definition

Independent Variables

Variables that the researcher actively manipulates, i.e., "the things that are changed" in the experiment.

  • Variants of model architecture
  • Differences in training strategies
  • Different data processing methods

Dependent Variables

Variables used to measure experimental results, i.e., "the things that are measured."

  • Model performance metrics (accuracy, F1, BLEU, etc.)
  • Efficiency metrics (inference time, memory usage)
  • Quality metrics (human evaluation scores)

Control Variables

Variables kept constant in the experiment to ensure fairness of comparison.

  • Random seed
  • Training dataset and splitting method
  • Hyperparameters (the parts that are not the research subject)
  • Hardware environment
  • Pretrained model version

Principles of variable control

  1. Single-variable principle: change only one independent variable per experiment
  2. Complete recording principle: the values of all variables must be recorded
  3. Reasonable range principle: the value range of the independent variable should have a theoretical basis

See also:Experimental planning reference

Step 3: Selection of evaluation metrics

Selection principles

  1. Domain conventions: prioritize standard metrics recognized in the field
  2. Multifaceted coverage: report performance, efficiency, and robustness metrics at the same time
  3. Statistical significance: report the mean and standard deviation of multiple experiments
  4. Reasonableness: the metrics can truly reflect the aspects of concern in the research hypothesis

Common metric categories

Category Metric examples
Classification tasks Accuracy、Precision、Recall、F1-score、AUC-ROC
Generation tasks BLEU, ROUGE, METEOR, BERTScore, human evaluation
Information extraction MAP、MRR、NDCG、Recall@K
Efficiency metrics FLOPs, number of parameters, inference latency, memory usage
Robustness Cross-dataset performance, accuracy on adversarial examples

Statistical testing

  • Report the mean and standard deviation of experiments with multiple random seeds
  • Conduct statistical significance tests when necessary (e.g., paired t-test, bootstrap test)
  • Annotate the level of statistical significance (p < 0.05, p < 0.01)

Step 4: Baseline selection and setup

Required baseline types

  1. Classic methods: historically important methods in the field
  2. Current SOTA: the latest best-performing method
  3. Simple baseline: a simple but reasonable baseline method (e.g., random, majority class, TF-IDF)

Fair comparison principles

  • Use the same data split
  • Use the same evaluation protocol
  • Use the original authors' code and hyperparameters whenever possible
  • If reimplementation is required, verify that the reproduced results are consistent with the original paper

Common mistakes

  • Comparing only with weak baselines
  • Not using the latest SOTA as a baseline
  • The baseline's hyperparameters have not been tuned
  • Inconsistent comparison conditions (e.g., different pretrained models)

See also:Baseline selection guide

Step 5: Ablation Study design

Ablation Study is a key experiment for verifying the contribution of each component in a method. This skill defines four Ablation modes:

Mode 1: Component Ablation

Remove or replace each component of the method one by one, and observe the changes in performance.

  • Remove only one component at a time
  • Record the performance change after removal
  • Use this to determine the contribution of each component

Mode 2: Hyperparameter Sensitivity

Investigate the impact of key hyperparameters on performance.

  • Select the 2-4 most important hyperparameters
  • Vary hyperparameter values within a reasonable range
  • Plot hyperparameter-performance curves

Mode 3: Cross-Dataset Transfer

Validate the generalization ability of the method.

  • Test on multiple different datasets
  • Include datasets of different scales and domains
  • Analyze under what conditions the method performs best or worst

Mode 4: Qualitative Analysis

Gain an in-depth understanding of model behavior through visualization and case analysis.

  • Attention weight visualization
  • Analysis of success and failure cases
  • Feature space visualization (e.g., t-SNE)
  • Error type classification and statistics

See also:Ablation Design Guide

Step 6: Computational Resource Estimation

Estimation Item

  1. Cost per experiment

    • GPU hours
    • Memory requirements
    • Storage space requirements

  2. Total experiment volume calculation

    總 GPU 時數 = 單次時數 × 模型變體數 × 資料集數 × 隨機種子數 × 超參組合數

  3. Safety factor

    • It is recommended to reserve 1.5-2 times the estimated resources
    • Consider the needs for debugging, preliminary experiments, and additional experiments

Resource Optimization Strategies

  • Conduct preliminary experiments with small-scale datasets first
  • Use early stopping to save training time
  • Make good use of mixed precision training
  • Arrange experiment priorities reasonably

Reproducibility Requirements

The experiment plan must include complete reproducibility information to ensure that others can accurately reproduce the results.

Required Disclosure Items

  1. Hardware environment

    • GPU model and quantity
    • CPU specifications
    • Memory size

  2. Software environment

    • Programming language version
    • Deep learning framework version
    • Key package versions

  3. Randomness control

    • Random seed settings
    • Deterministic algorithm settings
    • Seed list for multiple experiments

  4. Training protocol

    • Complete hyperparameter list
    • Optimizer settings
    • Learning rate schedule
    • Data augmentation strategy
    • Early stopping criteria

  5. Data processing

    • Dataset version and source
    • Preprocessing steps
    • Data splitting method

  6. Evaluation protocol

    • Precise definition of evaluation metrics
    • Evaluation frequency
    • Model selection criteria

See also:Reproducibility Checklist

Output: Structured Experiment Plan Document

The final output of this skill is a structured experiment plan document, containing the following sections:

  1. Research hypotheses and sub-hypotheses
  2. Variable definition table
  3. Evaluation metrics and statistical methods
  4. Baseline list and settings
  5. Ablation Study design matrix
  6. Computational resource estimation and schedule planning
  7. Reproducibility information

Template used:Experiment Plan Template

Usage Flow

Input

  • Research topic or paper draft
  • Description of the proposed method
  • Available computational resources

Process

  1. Guide the user to clarify the research hypotheses
  2. Help define the independent variables, dependent variables, and control variables
  3. Suggest evaluation metrics based on the task type
  4. Suggest Baselines based on the research field
  5. Design an Ablation Study plan
  6. Estimate computational resource requirements

Output

  • A complete experiment plan document (according to the template format)
  • Suggestions for experiment priorities
  • Potential risks and response plans

Quality Checklist

After completing the experiment plan, please confirm the following items:

  • Each research hypothesis has a corresponding experiment to verify it
  • The value ranges of all independent variables are clearly defined
  • The control variables are fully listed
  • The evaluation metrics cover multiple aspects
  • The Baseline includes classic methods, SOTA, and simple baselines
  • The Ablation Study covers all proposed components
  • The computational resource estimates are reasonable and include a safety factor
  • The reproducibility information is complete
  • The statistical testing methods have been determined

Reference Resources