train-test-split

Source Metadata

Original cangjie-skill frontmatter from the distillation run:

id: f17
title: "训练集/测试集分离 — AI的'科学方法'"
book: "AI for Everyone"
author: "Andrew Ng（吴恩达）"
source: p16
tags: [machine-learning, evaluation, testing, collaboration]
related_skills:
  - slug: ml-workflow, relation: depends-on
  - slug: statistical-acceptance, relation: composes-with

R — Reading (original quote)

"AI teams split data into two main datasets, the first called the training set and the second called the test set" — p16

I — Interpretation (restate in your own words)

Just as a student who only memorizes past exam questions will fail when encountering new ones—if an AI model uses training data to evaluate itself, its performance will look much better than it actually is.

Solution: Split the data into twonon-overlappingsets:

1. Training set: fed to the algorithm to learn, letting the model discover patterns from it.
2. Test set: completely "hidden away," used only for final evaluation, to measure the model's performance onunseen data.

If you don't reserve a test set, you can't know whether the model truly learned the patterns or merely "memorized" the training samples. The test set is AI's "scientific experiment control group"—it tells you whether the model truly has generalization ability.

A1 — Past Application (past application cases)

• Coffee cup defect detection: Collect 10,000 coffee cup images (defective/non-defective), split into a training set (8,000 images) and a test set (2,000 images). Use the training set to teach the model to identify defects, then evaluate with the test set—the model achieves 95% accuracy on the 2,000 unseen images. This 95% is the credible measure of the model's true capability.

A2 — Future Trigger (future trigger conditions)

Trigger scenarios：

• The team reports a very high accuracy, but you want to know whether it is credible
• Starting a new ML project and need to design an evaluation plan
• The business side questions "is this AI actually any good?"

Language signals：

• "What's the AI's accuracy?"
• "How do we know if the AI is good?"
• "How do we evaluate the model?"
• "What's the accuracy?"
• "How are the test results?"
• "How do we do effect evaluation?"

Key distinction: Use this framework when you need to judge an AI model's true capability (rather than surface-level numbers).

E — Execution (execution steps)

1. Ensure the data has been split: Check whether the labeled data has been divided into a training set and a test set, and that the twohave no overlap. The test set must never be "seen" by the model during training.
2. Define acceptance criteria before training: Determine the target before starting training (e.g., "test set accuracy reaches 95%"), to avoid arbitrarily adjusting the standard after training.
3. Use the test set only for final evaluation: Do not use the test set to make any decisions during training (such as tuning parameters or selecting models). The test set can be used only once—for final evaluation.
4. Report test set results: What you report to stakeholders is the performance on the test set, not the performance on the training set.

Completion criteriaThe model meets the predefined acceptance criteria on a test set it has never seen before, and the test set has not been contaminated throughout the entire training process.

B — Boundary (Usage Boundaries)

Not Applicable Scenarios：

• Data science projects (no model to evaluate)
• Situations with extremely little data (cannot set aside a meaningful test set)
• Exploratory analysis stage (not yet at modeling)

Failure Modes：

• Tuning hyperparameters with the test set: repeatedly evaluating and adjusting on the test set is effectively "leaking" test set information
• Overlap between training and test sets: leads to inflated evaluation results
• Test set distribution does not match the real-world scenario: good test set performance but poor performance after deployment

Author's Blind Spot：

• Andrew Ng only discussed the train/test split dichotomy, but real projects typically requirethree sets: training set (training), development set/validation set (tuning and model selection), test set (final evaluation). If a team has only one test set and repeatedly tunes on it, it will lead to a biased estimate of true performance. For serious ML projects, it is recommended to introduce a second "hidden" test set.

Related skills

• ml-workflow — depends-on: Train/test set separation is a key step in the ML workflow; you need to understand the overall workflow first to correctly perform data splitting.
• statistical-acceptance — composes-with: Accuracy results on the test set need to be combined with statistical acceptance criteria to define "what counts as good enough"; only by using both together can a complete evaluation system be established.