SuperAPI AI Directory

pol-probe

General & other

View repository

Updated 10-16 09:59

About this skill

Validation Experiment Design (PoL Probe)

Reference source:deanpeters/Product-Manager-Skills 's pol-probe skill

PoL = Proof of Learning. The goal is not to prove that a solution "can be done," but to prove that the solution "should be done."

Applicable scenarios

  • You have an idea but are unsure whether users really need it
  • Multiple candidate solutions, unsure which to choose
  • High investment but high cost of failure
  • Proposals that need data support
  • Small-scale testing before launching a new feature

Inapplicable scenarios

  • Requirements have already been validated and are entering the development phase
  • Urgent fixes or compliance requirements (must be done)
  • Internal tool improvements (no external validation needed)

Core principles

1. 验证假设,不是验证方案
   错:测试"语义搜索是否能做出来"
   对:测试"用户是否真的因为找不到商品而流失"

2. 最小成本,最大信息量
   能 30 分钟解决的不要花 3 小时

3. 验证失败 ≠ 失败
   失败的实验也是有价值的发现

4. 提前定义成功标准
   不能事后凑数据

5 PoL Probe types

1. Feasibility Probe

Question: Is it technically possible to do?

Applicable:

  • Uncertain solutions involving AI/algorithms/performance
  • Requires third-party APIs or data sources
  • Cross-system integration

Approach:

  • Write a minimal prototype/spike
  • Do not pursue experience; only validate "whether it can run through"
  • Time budget: 30-90 minutes

Example:

假设:能用现有数据训练出准确率 > 80% 的推荐模型
实验:用历史数据跑一次,看准确率
成功标准:准确率 ≥ 75%
失败后:放弃方案 / 换数据源 / 换算法

2. Task Probe

Question: Can users complete the operation?

Applicable:

  • Features with complex flows
  • New interaction patterns
  • Information architecture redesign

Approach:

  • Paper prototype / Figma static prototype
  • Wizard of Oz (manually simulate system responses)
  • Have 3-5 real users attempt to complete the task

Example:

假设:用户能在 3 分钟内完成新版下单流程
实验:5 个用户测试,记录耗时和卡点
成功标准:4/5 用户在 3 分钟内完成
失败后:简化流程 / 重新设计

3. Narrative Probe

Question: Do users understand the value proposition?

Applicable:

  • New product positioning
  • New feature promotion
  • Marketing copy validation

Approach:

  • Landing Page
  • Fake Door: place a button but after clicking, tell them "coming soon"
  • A/B test different value propositions

Example:

假设:用户对"AI 智能推荐"的接受度高于"个性化推荐"
实验:两个落地页 A/B 测试,对比转化率
成功标准:A 版转化率 > B 版 20%
失败后:调整价值主张

4. Data Probe

Question: Does the data support the hypothesis?

Applicable:

  • The proposal needs data support
  • User behavior analysis
  • Market capacity assessment

Approach:

  • SQL query of historical data
  • Log analysis
  • Questionnaire survey
  • Third-party data sources

Example:

假设:每月有 2000+ 用户因为找不到商品而流失
实验:分析最近 30 天的搜索失败 + 退出率数据
成功标准:数据支持月均 1500+
失败后:重新评估机会优先级

5. Vibe-Coded Probe

Question: Can it run through end to end? Does it feel right to users?

Applicable:

  • Products with an entirely new concept
  • Fast MVP with AI-assisted development
  • Internal Demo or roadshow

Approach:

  • AI-generated code + existing component library for rapid building
  • Do not pursue production quality; pursue "can run and can demo"
  • Time budget: 1–3 hours

Example:

假设:用 AI 推荐 + 现有商品库,能产生让用户感兴趣的商品流
实验:用 AI 生成一个原型 demo,给 10 个用户试用
成功标准:6/10 用户表示"会再来用"
失败后:调整方向或放弃

Decision tree selection

问题类型?
  ├─ 技术能不能做 → Feasibility
  ├─ 用户能不能用 → Task
  ├─ 用户买不买账 → Narrative
  ├─ 数据支不支持 → Data
  └─ 整体感觉对不对 → Vibe-Coded

成本预算?
  ├─ 30 分钟内 → Data / Narrative
  ├─ 1 小时左右 → Feasibility / Task
  └─ 半天 → Vibe-Coded

风险等级?
  ├─ 低(可逆) → Vibe-Coded(快速做)
  └─ 高(不可逆) → Data + Task 双重验证

Experiment template

## PoL Probe: [实验名称]

**类型**:Feasibility / Task / Narrative / Data / Vibe-Coded

**待验证假设**:
如果 [条件],那么 [结果]

**验证方式**:
[具体怎么测,包括工具/数据/对象]

**成功标准**:
[必须是可观察、可量化的指标]

**失败后行动**:
- 如果失败:[放弃 / 调整 / 重新设计]
- 如果部分成功:[继续验证 / 缩小范围]

**时间预算**:
[最多花多少时间]

**所需资源**:
[需要哪些数据/工具/人员]

**实验记录**:
- 开始时间:
- 结束时间:
- 实际耗时:
- 结果数据:
- 结论:
- 后续行动:

Workflow

1. 识别需要验证的假设(来自 OST 或 PRD)
   ↓
2. 选择 PoL Probe 类型(用决策树)
   ↓
3. 设计实验(用模板)
   ↓
4. 设定成功标准(可量化、可观察)
   ↓
5. 设定时间预算(不超过预算就停)
   ↓
6. 执行实验
   ↓
7. 记录结果(成功/失败/部分成功)
   ↓
8. 决策:继续/调整/放弃
   ↓
9. 沉淀到 field-journal

Quality self-check

□ 实验类型是否选对了?(不要用 Feasibility 验证用户接受度)
□ 假设是否具体可测试?(不是"用户喜欢"而是"用户点击率 > X%")
□ 成功标准是否提前定义?(不能事后凑数据)
□ 失败后行动是否提前规划?(避免"再做一个实验"的循环)
□ 时间预算是否合理?(PoL 不应该比正式开发还久)
□ 是否最小成本?(能 Data 就不 Vibe-Coded)

Common pitfalls

  1. Confusing Probe types— using Feasibility ("can it be built") instead of Task ("can users use it")
  2. No success criteria— after finishing, you don't know whether it succeeded or failed
  3. Time budget overrun— a PoL shouldn't take longer than formal development
  4. Results biased toward what you want— selectively looking at data
  5. Failed results not captured— failed experiments are also valuable experience
  6. Over-validation— not every idea needs a PoL; do the obvious things directly
  7. No decision after validation— after finishing the experiment, you just leave it there and don't move to the next step

Supporting templates

  • templates/pol-probe-template.md — Experiment design template
  • templates/probe-result-template.md — Experiment result recording template

Collaboration with other skills

上游:
  opportunity-tree → 提供需要验证的假设

平行:
  user-story → 验证某个用户故事的核心假设
  prioritization → 验证高分但低置信度的方案

下游:
  实验通过 → mvp-scoping 进入 MVP 范围
  实验失败 → 调整 OST 或放弃
  实验沉淀 → field-journal