SuperAPI AI Directory

weighted-scoring

General & other

View repository

Updated 11-06 16:22

About this skill

Decision Matrix — Weighted Scoring Decision Framework

A systematic multi-criteria decision methodology. Suitable for scenarios that require comparing multiple options across multiple dimensions, such as technology selection, vendor selection, Build vs Buy, tool comparison, and hiring evaluation.

Quick Start

Guide the user through 5 steps:

  1. List candidate options(2-7)
  2. Define evaluation dimensions(4-8)
  3. Assign weights(total = 100%)
  4. Score each item(1-5 scale)
  5. Calculate weighted total score + sensitivity analysis

The user only needs to say:

"Help me do a technology selection comparing PostgreSQL, MySQL, and MongoDB for our e-commerce order system"

The Agent will guide the user step by step through the entire decision process.


I. Detailed Explanation of the Complete Process

Step 1: Clarify the decision context

Before starting scoring, first clarify the following information (guide the user by asking questions):

Question Purpose
What problem is this decision meant to solve? Focus on real needs and avoid comparing for the sake of comparison
What are the candidate options? Determine the objects of comparison (2-7 recommended)
Who are the key stakeholders? Different roles focus on different dimensions
What is the decision timeline? Affects the depth of information collection
Are there any veto criteria? Identify hard constraints and first filter out infeasible options

Hard constraint filtering: If an option does not satisfy any veto criterion (such as budget cap, compliance requirements, or technical compatibility), exclude it directly and do not enter the scoring stage.

Step 2: Define evaluation dimensions

Principles for selecting dimensions

  • MECE principle: Dimensions do not overlap with one another and together provide complete coverage
  • Evaluability: Each dimension must be able to produce a score with discriminative power
  • Quantity control: 4-8 dimensions is optimal. Fewer than 4 makes the decision too coarse, while more than 8 dilutes the signal with noise

Common dimension templates

Technology selection:

Dimension Description Evaluation method
Feature fit The degree to which core requirements are met List the requirements and compare item by item
Performance Latency, throughput, concurrency capability Benchmark data or official documentation
Scalability Horizontal/vertical scaling capability Architecture design analysis
Community and ecosystem Documentation, community activity, third-party integrations GitHub stars/issues, Stack Overflow activity
Learning cost Difficulty for the team to get started Match with the team's existing skills
Operational complexity Difficulty of deployment, monitoring, and troubleshooting Assessment based on operational experience or documentation
Cost License fees, infrastructure, and labor costs TCO (Total Cost of Ownership) estimate
Long-term risk Vendor lock-in and technology obsolescence risk Open source/closed source, market trends

Vendor selection:

Dimension Description
Product capabilities Feature coverage, ease of use, customizability
Price and commercial terms Total cost, payment methods, discount conditions
Technical support SLA, response time, support channels
Security and compliance Data security, certifications, compliance capabilities
Integration capabilities API, SDK, compatibility with existing systems
Company strength Financial stability, market share, customer cases

Build vs Buy:

Dimension Description
Requirement fit Degree to which customization requirements are met
Time to launch Time from decision to availability
Total cost (3 years) Development/purchase + maintenance + opportunity cost
Team capability fit Whether building in-house is within the team's capabilities
Strategic value Whether it builds core competitiveness
Maintenance burden Long-term maintenance labor and complexity

Step 3: Assign weights

Weights reflect the relative importance of each dimension. The total must be 100%.

Method A: Direct allocation method (fast)

Suitable for scenarios with few dimensions (≤5) or where the decision-maker has a clear judgment of priorities.

Operation: Have the user directly assign a percentage to each dimension, with the total = 100%.

Suggested prompt to guide the user:

"Please distribute 100 points across these dimensions. Give more points to the most important dimensions. It doesn't need to be precise; just go with your intuition first, and we can adjust later."

Method B: Rank order allocation method (recommended)

Suitable for scenarios where the user does not have a clear numerical sense of the weights.

Steps:

  1. First have the user rank the dimensions by importancerank(most important first)
  2. Use the ROC (Rank Order Centroid) formula to automatically calculate weights:
第 k 个维度的权重 = (1/k + 1/(k+1) + ... + 1/n) / n

其中 n = 维度总数,k = 该维度的排名(1 = 最重要)
Rank 4-dimension weights 6-dimension weights 8-dimension weights
1 52.1% 40.8% 34.0%
2 27.1% 24.2% 21.5%
3 14.6% 15.8% 15.2%
4 6.3% 10.3% 11.1%
5 — 6.1% 7.9%
6 — 2.8% 5.4%
7 — — 3.3%
8 — — 1.6%

No manual calculation is needed; you can use scripts/decision_matrix.py to generate it automatically.

Method C: Pairwise comparison method (precise)

Suitable for team decision-making, many dimensions (>5), and scenarios where subjective bias needs to be reduced.

Steps:

  1. Pair all dimensions with each other
  2. For each pair, determine which is more important (win = 1, lose = 0, tie = 0.5)
  3. Each dimension's score = number of wins / total number of comparisons

Example (4 dimensions: A, B, C, D):

Comparison Result
A vs B A wins
A vs C C wins
A vs D A wins
B vs C C wins
B vs D Tie
C vs D C wins

Result: A=2/6=33%, B=0.5/6=8%, C=3/6=50%, D=0.5/6=8%. Normalized to 100%.

Weight Quality Check

After assigning weights, perform the following checks:

  • Extreme check: Is the highest-weighted dimension really that much more important than the lowest?
  • Discrimination check: Is any dimension's weight < 5%? If so, consider merging or removing it
  • Coverage check: Is the sum of the weights of the top two dimensions > 70%? If so, are the other dimensions really unimportant
  • Stakeholder check: If a different role assigned the weights, would they differ greatly?

Step 4: Score item by item

Scoring scale (1-5 point scale)

The meaning of each score must be defined before scoring, otherwise the standards among scorers will be inconsistent:

Score General meaning Example (using the "performance" dimension as an example)
5 Excellent, significantly exceeds requirements Latency <10ms, easily supports 10x expected traffic
4 Good, meets requirements with margin Latency <50ms, can support 3x expected traffic
3 Meets standard, just meets current requirements Latency <100ms, meets current traffic
2 Insufficient, requires extra work to meet requirements Latency 100-500ms, needs optimization
1 Severely insufficient, basically unusable Latency >500ms, cannot meet basic requirements

Scoring Guidelines

  • Set anchors first: For each dimension, first quickly determine which option is best (5 points) and which is worst (1-2 points), then score the middle options
  • Dimension priority: Score column by column by dimension (rather than row by row by option), so that cross-option comparisons are fairer
  • Evidence-based: Attach a one-sentence reason to each score to avoid forgetting the basis for the score later
  • Differentiation: If all options receive the same score on a dimension, that dimension has no differentiating value; consider removing it

Step 5: Calculation and Analysis

Weighted total score calculation

方案 X 的总分 = Σ(维度 i 的权重 × 方案 X 在维度 i 的评分)

Output format

Present the results in a standard table:

| 维度 | 权重 | 方案A | 方案B | 方案C |
|------|------|------|------|------|
| 功能匹配度 | 30% | 4 (1.20) | 5 (1.50) | 3 (0.90) |
| 性能 | 25% | 5 (1.25) | 3 (0.75) | 4 (1.00) |
| 成本 | 25% | 3 (0.75) | 4 (1.00) | 5 (1.25) |
| 学习成本 | 20% | 4 (0.80) | 2 (0.40) | 3 (0.60) |
| **加权总分** | **100%** | **4.00** | **3.65** | **3.75** |
| **排名** | | **1** | **3** | **2** |

The parenthetical value is the weighted score for that dimension (weight × score).


II. Sensitivity analysis (key step, cannot be skipped)

The weighted scoring result depends on the assumptions of weights and scores.If the total score gap between 1st place and 2nd place is < 0.3 points (on a 5-point scale), a sensitivity analysis must be performed。

Analysis methods

1. Weight fluctuation test

For each key dimension, change its weight by ±10% (adjusting the other dimensions proportionally), and observe whether the ranking changes:

原始权重:功能 30%, 性能 25%, 成本 25%, 学习 20%
测试 1:功能 40%, 性能 21.4%, 成本 21.4%, 学习 17.1%  → 排名变了吗?
测试 2:功能 20%, 性能 28.6%, 成本 28.6%, 学习 22.9%  → 排名变了吗?

2. Score fluctuation test

Change disputed score cells by ±1 point and observe whether the ranking changes.

3. Reversal point analysis

Calculate "how much the weight/score of a certain dimension needs to change for 2nd place to overtake 1st place." If only a tiny change is needed to reverse the result, the two options are essentially hard to distinguish.

Analysis conclusion template

Conclusion: Option A leads Option C by 4.00 points to 3.75 points (a gap of 0.25 points). Sensitivity analysis shows that when the weight of the "cost" dimension is increased to 35% (+10%), Option C overtakes Option A. It is recommended to examine whether the weight of the cost dimension is reasonable, or to further collect cost data to improve scoring accuracy.


III. Cognitive Bias Checklist

After completing the scoring, check the following common biases one by one:

Bias Manifestation Countermeasure
Anchoring effect The first option evaluated receives a higher score After scoring all options, go back and re-examine the first one
Halo effect A good overall impression of an option leads to high scores in all dimensions Score column by column by dimension, not row by row by option
Status quo bias Tendency to choose the option currently in use Assume choosing from scratch, eliminating interference from migration costs
Confirmation bias Reaching a conclusion first, then finding evidence to support it Have people with different positions score separately and compare the differences
Sunk cost Unwilling to switch options because of what has already been invested Clearly distinguish between "past investment" and "future value"
Availability bias Recently hearing that an option is good/bad affects judgment Require each score to be accompanied by objective evidence

IV. Script Tools

Provided scripts/decision_matrix.py for automated calculation:

Features:

  • Input dimensions, weights, and scores, and output a weighted scoring table (Markdown format)
  • Automatically perform sensitivity analysis (weight ±10% fluctuation test)
  • Automatically calculate weights using the ROC ranking method
  • Detect scoring consistency issues (such as zero discrimination in a dimension)

Usage:

# 完整评分计算 + 敏感性分析
python3 scripts/decision_matrix.py --json '{
  "dimensions": ["功能", "性能", "成本", "学习成本"],
  "weights": [30, 25, 25, 20],
  "options": ["PostgreSQL", "MySQL", "MongoDB"],
  "scores": {
    "PostgreSQL": [4, 5, 3, 4],
    "MySQL":      [5, 3, 4, 2],
    "MongoDB":    [3, 4, 5, 3]
  }
}'

# ROC 权重计算:只需提供维度排名(从最重要到最不重要)
python3 scripts/decision_matrix.py --roc '["功能", "性能", "成本", "学习成本"]'

V. Complete Case: E-commerce System Database Selection

Background

  • E-commerce order system, daily order volume of 100,000
  • Team of 5, with main experience in relational databases
  • Limited budget, prioritize open-source options

Step 1: Candidate Options

PostgreSQL, MySQL, MongoDB (hard constraint filtering: Oracle excluded due to license fees exceeding budget)

Step 2: Dimension Definition

Feature fit, performance, scalability, community ecosystem, learning cost, operational complexity, cost

Step 3: Weights (rank allocation method)

User ranking: features > performance > cost > learning cost > community > operations > scalability

ROC weights: features 37.0%, performance 22.8%, cost 15.6%, learning 10.9%, community 7.3%, operations 4.4%, scalability 2.0%

Step 4: Scoring

Dimension Weight PostgreSQL MySQL MongoDB
Feature fit 37.0% 5 4 3
Performance 22.8% 4 4 5
Cost 15.6% 5 5 4
Learning cost 10.9% 3 4 2
Community ecosystem 7.3% 5 5 4
Operational complexity 4.4% 3 4 2
Scalability 2.0% 4 3 5

Step 5: Results

  • PostgreSQL: 4.45 points → Ranked 1st
  • MySQL: 4.21 points → Ranked 2nd
  • MongoDB: 3.57 points → Ranked 3rd

Sensitivity Analysis: The gap between PostgreSQL and MySQL is 0.24 points. The sensitivity analysis shows that the ranking remains unchanged after all dimension weights fluctuate by ±10%. Although the score difference is small, the conclusion is insensitive to weight changes, so its credibility is relatively high.


VI. Applicable Scenarios and Limitations

Applicable Scenarios

  • 2-7 candidate options, 4-8 evaluation dimensions
  • The decision requires participation by multiple people or needs to be kept on record
  • Need to explain to stakeholders "why this one was chosen"

Inapplicable Scenarios

  • Only 2 options with obvious differences → a direct Pros/Cons list is more efficient
  • The decision depends highly on a single factor → a multi-dimensional framework is not needed
  • Information is extremely incomplete → scoring is meaningless; first collect information
  • Personal decisions involving strong emotional factors → a rational framework may obscure true preferences

Common Misconceptions

  1. Too many dimensions: More than 8 dimensions causes the weight of each dimension to be very low, reducing discrimination
  2. False precision: 3.72 vs 3.68 does not mean A is truly better than B—pay attention to sensitivity analysis
  3. Ignoring qualitative judgment: The matrix is an auxiliary tool, not an algorithm that replaces intuition
  4. Once decided, final forever: When decision conditions change, it should be re-evaluated

Reference Documents

  • Keeney, R. & Raiffa, H. — Decisions with Multiple Objectives(classic textbook on multi-objective decision making)
  • Belton, V. & Stewart, T. — Multiple Criteria Decision Analysis(practical guide to MCDA)
  • ROC weighting method — Barron & Barrett (1996), "Decision quality using ranked attribute weights"