statistical-acceptance

Source Metadata

Original cangjie-skill frontmatter from the distillation run:

id: f18
title: "统计化验收标准 — 不追求100%准确"
book: "AI for Everyone"
author: "Andrew Ng（吴恩达）"
source: p16, p24
tags: [project-management, evaluation, acceptance-criteria, communication]
related_skills:
  - slug: train-test-split, relation: depends-on
  - slug: ml-workflow, relation: depends-on
  - slug: iterate-not-perfect, relation: composes-with

R — Reading (original quote)

"One innovation of AI systems is that performance is usually defined statistically. Rather than requiring AI systems to perform perfectly, we more often require AI systems to achieve a specific accuracy rate" — p16

I — Interpretation (restate in your own words)

Requiring AI to achieve 100% accuracy is one of the most common misconceptions in the business world. Pursuing perfection is usually impossible, for three reasons:

1. ML has fundamental limitations: Any statistical learning method has an irreducible error floor.
2. The data itself is noisy: Labeling errors, ambiguous samples, edge cases—the data itself is imperfect.
3. Human experts also disagree: For ambiguous cases, different experts may give different answers, and there is no "standard answer."

The right approach: Define acceptance criteria as statistical targets (such as "achieve 95% accuracy on the test set"), and recognize that a system with 95% accuracy can already create enormous value. The key is toagree in advanceon acceptable standards, rather than demanding perfection after the fact.

A1 — Past Application (past application cases)

• Coffee cup defect detection: 95% accuracy means 5% of defective products will be missed. But this is still very valuable—it can be paired with manual sampling inspection as a fallback. If you pursue 100%, the project may never launch, while a 95% system can already intercept the vast majority of defects and greatly reduce quality inspection costs.

A2 — Future Trigger (future trigger conditions)

Trigger scenarios：

• The business side insists that "AI cannot make mistakes"
• The team rejects the entire project because AI made a few mistakes
• Disputes arise during acceptance over "what counts as qualified"

Language signals：

• "Must be 100% accurate"
• "Zero errors"
• "AI made a mistake"
• "Perfect"
• "100%"
• "Zero defects"

Key distinction: When someone (including yourself) imposes a "zero errors" requirement on an AI system, use this framework to reset reasonable expectations.

E — Execution (execution steps)

1. Discuss achievable accuracy with the AI team: Understand what a reasonable accuracy target is under the current data and technical conditions.
2. Define statistical acceptance criteria: Quantify the target into specific numbers (such as "test set accuracy ≥95%") and write them into the project document.
3. Accept that errors will inevitably occur: Clearly inform stakeholders—the system will make mistakes, and this is normal.
4. Design error handling mechanisms: Plan responses for that 5% of errors (manual review, fallback process, user feedback channel).

Completion criteria: The team and business side reach consensus on the "acceptable accuracy rate" and establish a response plan for when errors occur.

B — Boundary (usage boundaries)

Not applicable scenarios：

• Safety-critical systems (medical diagnosis, autonomous driving, aviation control—in these fields errors can cause personal injury and require standards far higher than 95%)
• Scenarios where 95% is genuinely insufficient (such as financial anti-fraud, where a 1% missed detection rate can cause enormous losses)
• Fields where legal and regulatory requirements demand zero tolerance

Failure Modes：

• Blindly accepting 95% without evaluating the cost of errors: if the cost of each error is extremely high, 95% may be far from enough
• Failing to design fallback plans for errors: telling the business side "there will be 5% errors" without any countermeasures
• Using statistical standards to mask technical shortcomings: packaging "cannot do it" as "do not need to do it"

Author's Blind Spots：

• Andrew Ng did not discuss in depthhow to combine AI with human review to handle that 5% of errors. In real projects, the hybrid model of "AI initial screening + human review" is often the best practice, but the book says little about this. In addition, different types of errors (false positives vs false negatives) have different costs, and a simple accuracy number may mask important differences.

Related skills

• train-test-split — depends-on: Statistical acceptance criteria need to be defined based on accuracy on an independent test set; without correct data splitting, credible acceptance criteria cannot be established.
• ml-workflow — depends-on: Acceptance criteria are the core output of the evaluation stage in the ML workflow and need to be understood as embedded in the overall workflow.
• iterate-not-perfect — composes-with: Accepting statistical acceptance criteria and the philosophy of "iterate rather than perfect" complement each other—first reach an acceptable accuracy and launch, then continuously iterate and improve.