A systematic multi-criteria decision methodology. Suitable for scenarios that require comparing multiple options across multiple dimensions, such as technology selection, vendor selection, Build vs Buy, tool comparison, and hiring evaluation.
Quick Start
Guide the user through 5 steps:
- List candidate options(2-7)
- Define evaluation dimensions(4-8)
- Assign weights(total = 100%)
- Score each item(1-5 scale)
- Calculate weighted total score + sensitivity analysis
The user only needs to say:
"Help me do a technology selection comparing PostgreSQL, MySQL, and MongoDB for our e-commerce order system"
The Agent will guide the user step by step through the entire decision process.
I. Detailed Explanation of the Complete Process
Step 1: Clarify the decision context
Before starting scoring, first clarify the following information (guide the user by asking questions):
| Question | Purpose |
|---|---|
| What problem is this decision meant to solve? | Focus on real needs and avoid comparing for the sake of comparison |
| What are the candidate options? | Determine the objects of comparison (2-7 recommended) |
| Who are the key stakeholders? | Different roles focus on different dimensions |
| What is the decision timeline? | Affects the depth of information collection |
| Are there any veto criteria? | Identify hard constraints and first filter out infeasible options |
Hard constraint filtering: If an option does not satisfy any veto criterion (such as budget cap, compliance requirements, or technical compatibility), exclude it directly and do not enter the scoring stage.
Step 2: Define evaluation dimensions
Principles for selecting dimensions
- MECE principle: Dimensions do not overlap with one another and together provide complete coverage
- Evaluability: Each dimension must be able to produce a score with discriminative power
- Quantity control: 4-8 dimensions is optimal. Fewer than 4 makes the decision too coarse, while more than 8 dilutes the signal with noise
Common dimension templates
Technology selection:
| Dimension | Description | Evaluation method |
|---|---|---|
| Feature fit | The degree to which core requirements are met | List the requirements and compare item by item |
| Performance | Latency, throughput, concurrency capability | Benchmark data or official documentation |
| Scalability | Horizontal/vertical scaling capability | Architecture design analysis |
| Community and ecosystem | Documentation, community activity, third-party integrations | GitHub stars/issues, Stack Overflow activity |
| Learning cost | Difficulty for the team to get started | Match with the team's existing skills |
| Operational complexity | Difficulty of deployment, monitoring, and troubleshooting | Assessment based on operational experience or documentation |
| Cost | License fees, infrastructure, and labor costs | TCO (Total Cost of Ownership) estimate |
| Long-term risk | Vendor lock-in and technology obsolescence risk | Open source/closed source, market trends |
Vendor selection:
| Dimension | Description |
|---|---|
| Product capabilities | Feature coverage, ease of use, customizability |
| Price and commercial terms | Total cost, payment methods, discount conditions |
| Technical support | SLA, response time, support channels |
| Security and compliance | Data security, certifications, compliance capabilities |
| Integration capabilities | API, SDK, compatibility with existing systems |
| Company strength | Financial stability, market share, customer cases |
Build vs Buy:
| Dimension | Description |
|---|---|
| Requirement fit | Degree to which customization requirements are met |
| Time to launch | Time from decision to availability |
| Total cost (3 years) | Development/purchase + maintenance + opportunity cost |
| Team capability fit | Whether building in-house is within the team's capabilities |
| Strategic value | Whether it builds core competitiveness |
| Maintenance burden | Long-term maintenance labor and complexity |
Step 3: Assign weights
Weights reflect the relative importance of each dimension. The total must be 100%.
Method A: Direct allocation method (fast)
Suitable for scenarios with few dimensions (≤5) or where the decision-maker has a clear judgment of priorities.
Operation: Have the user directly assign a percentage to each dimension, with the total = 100%.
Suggested prompt to guide the user:
"Please distribute 100 points across these dimensions. Give more points to the most important dimensions. It doesn't need to be precise; just go with your intuition first, and we can adjust later."
Method B: Rank order allocation method (recommended)
Suitable for scenarios where the user does not have a clear numerical sense of the weights.
Steps:
- First have the user rank the dimensions by importancerank(most important first)
- Use the ROC (Rank Order Centroid) formula to automatically calculate weights:
第 k 个维度的权重 = (1/k + 1/(k+1) + ... + 1/n) / n
其中 n = 维度总数,k = 该维度的排名(1 = 最重要)
| Rank | 4-dimension weights | 6-dimension weights | 8-dimension weights |
|---|---|---|---|
| 1 | 52.1% | 40.8% | 34.0% |
| 2 | 27.1% | 24.2% | 21.5% |
| 3 | 14.6% | 15.8% | 15.2% |
| 4 | 6.3% | 10.3% | 11.1% |
| 5 | — | 6.1% | 7.9% |
| 6 | — | 2.8% | 5.4% |
| 7 | — | — | 3.3% |
| 8 | — | — | 1.6% |
No manual calculation is needed; you can use
scripts/decision_matrix.pyto generate it automatically.
Method C: Pairwise comparison method (precise)
Suitable for team decision-making, many dimensions (>5), and scenarios where subjective bias needs to be reduced.
Steps:
- Pair all dimensions with each other
- For each pair, determine which is more important (win = 1, lose = 0, tie = 0.5)
- Each dimension's score = number of wins / total number of comparisons
Example (4 dimensions: A, B, C, D):
| Comparison | Result |
|---|---|
| A vs B | A wins |
| A vs C | C wins |
| A vs D | A wins |
| B vs C | C wins |
| B vs D | Tie |
| C vs D | C wins |
Result: A=2/6=33%, B=0.5/6=8%, C=3/6=50%, D=0.5/6=8%. Normalized to 100%.
Weight Quality Check
After assigning weights, perform the following checks:
- Extreme check: Is the highest-weighted dimension really that much more important than the lowest?
- Discrimination check: Is any dimension's weight < 5%? If so, consider merging or removing it
- Coverage check: Is the sum of the weights of the top two dimensions > 70%? If so, are the other dimensions really unimportant
- Stakeholder check: If a different role assigned the weights, would they differ greatly?
Step 4: Score item by item
Scoring scale (1-5 point scale)
The meaning of each score must be defined before scoring, otherwise the standards among scorers will be inconsistent:
| Score | General meaning | Example (using the "performance" dimension as an example) |
|---|---|---|
| 5 | Excellent, significantly exceeds requirements | Latency <10ms, easily supports 10x expected traffic |
| 4 | Good, meets requirements with margin | Latency <50ms, can support 3x expected traffic |
| 3 | Meets standard, just meets current requirements | Latency <100ms, meets current traffic |
| 2 | Insufficient, requires extra work to meet requirements | Latency 100-500ms, needs optimization |
| 1 | Severely insufficient, basically unusable | Latency >500ms, cannot meet basic requirements |
Scoring Guidelines
- Set anchors first: For each dimension, first quickly determine which option is best (5 points) and which is worst (1-2 points), then score the middle options
- Dimension priority: Score column by column by dimension (rather than row by row by option), so that cross-option comparisons are fairer
- Evidence-based: Attach a one-sentence reason to each score to avoid forgetting the basis for the score later
- Differentiation: If all options receive the same score on a dimension, that dimension has no differentiating value; consider removing it
Step 5: Calculation and Analysis
Weighted total score calculation
方案 X 的总分 = Σ(维度 i 的权重 × 方案 X 在维度 i 的评分)
Output format
Present the results in a standard table:
| 维度 | 权重 | 方案A | 方案B | 方案C |
|------|------|------|------|------|
| 功能匹配度 | 30% | 4 (1.20) | 5 (1.50) | 3 (0.90) |
| 性能 | 25% | 5 (1.25) | 3 (0.75) | 4 (1.00) |
| 成本 | 25% | 3 (0.75) | 4 (1.00) | 5 (1.25) |
| 学习成本 | 20% | 4 (0.80) | 2 (0.40) | 3 (0.60) |
| **加权总分** | **100%** | **4.00** | **3.65** | **3.75** |
| **排名** | | **1** | **3** | **2** |
The parenthetical value is the weighted score for that dimension (weight × score).
II. Sensitivity analysis (key step, cannot be skipped)
The weighted scoring result depends on the assumptions of weights and scores.If the total score gap between 1st place and 2nd place is < 0.3 points (on a 5-point scale), a sensitivity analysis must be performed。
Analysis methods
1. Weight fluctuation test
For each key dimension, change its weight by ±10% (adjusting the other dimensions proportionally), and observe whether the ranking changes:
原始权重:功能 30%, 性能 25%, 成本 25%, 学习 20%
测试 1:功能 40%, 性能 21.4%, 成本 21.4%, 学习 17.1% → 排名变了吗?
测试 2:功能 20%, 性能 28.6%, 成本 28.6%, 学习 22.9% → 排名变了吗?
2. Score fluctuation test
Change disputed score cells by ±1 point and observe whether the ranking changes.
3. Reversal point analysis
Calculate "how much the weight/score of a certain dimension needs to change for 2nd place to overtake 1st place." If only a tiny change is needed to reverse the result, the two options are essentially hard to distinguish.
Analysis conclusion template
Conclusion: Option A leads Option C by 4.00 points to 3.75 points (a gap of 0.25 points). Sensitivity analysis shows that when the weight of the "cost" dimension is increased to 35% (+10%), Option C overtakes Option A. It is recommended to examine whether the weight of the cost dimension is reasonable, or to further collect cost data to improve scoring accuracy.
III. Cognitive Bias Checklist
After completing the scoring, check the following common biases one by one:
| Bias | Manifestation | Countermeasure |
|---|---|---|
| Anchoring effect | The first option evaluated receives a higher score | After scoring all options, go back and re-examine the first one |
| Halo effect | A good overall impression of an option leads to high scores in all dimensions | Score column by column by dimension, not row by row by option |
| Status quo bias | Tendency to choose the option currently in use | Assume choosing from scratch, eliminating interference from migration costs |
| Confirmation bias | Reaching a conclusion first, then finding evidence to support it | Have people with different positions score separately and compare the differences |
| Sunk cost | Unwilling to switch options because of what has already been invested | Clearly distinguish between "past investment" and "future value" |
| Availability bias | Recently hearing that an option is good/bad affects judgment | Require each score to be accompanied by objective evidence |
IV. Script Tools
Provided scripts/decision_matrix.py for automated calculation:
Features:
- Input dimensions, weights, and scores, and output a weighted scoring table (Markdown format)
- Automatically perform sensitivity analysis (weight ±10% fluctuation test)
- Automatically calculate weights using the ROC ranking method
- Detect scoring consistency issues (such as zero discrimination in a dimension)
Usage:
# 完整评分计算 + 敏感性分析
python3 scripts/decision_matrix.py --json '{
"dimensions": ["功能", "性能", "成本", "学习成本"],
"weights": [30, 25, 25, 20],
"options": ["PostgreSQL", "MySQL", "MongoDB"],
"scores": {
"PostgreSQL": [4, 5, 3, 4],
"MySQL": [5, 3, 4, 2],
"MongoDB": [3, 4, 5, 3]
}
}'
# ROC 权重计算:只需提供维度排名(从最重要到最不重要)
python3 scripts/decision_matrix.py --roc '["功能", "性能", "成本", "学习成本"]'
V. Complete Case: E-commerce System Database Selection
Background
- E-commerce order system, daily order volume of 100,000
- Team of 5, with main experience in relational databases
- Limited budget, prioritize open-source options
Step 1: Candidate Options
PostgreSQL, MySQL, MongoDB (hard constraint filtering: Oracle excluded due to license fees exceeding budget)
Step 2: Dimension Definition
Feature fit, performance, scalability, community ecosystem, learning cost, operational complexity, cost
Step 3: Weights (rank allocation method)
User ranking: features > performance > cost > learning cost > community > operations > scalability
ROC weights: features 37.0%, performance 22.8%, cost 15.6%, learning 10.9%, community 7.3%, operations 4.4%, scalability 2.0%
Step 4: Scoring
| Dimension | Weight | PostgreSQL | MySQL | MongoDB |
|---|---|---|---|---|
| Feature fit | 37.0% | 5 | 4 | 3 |
| Performance | 22.8% | 4 | 4 | 5 |
| Cost | 15.6% | 5 | 5 | 4 |
| Learning cost | 10.9% | 3 | 4 | 2 |
| Community ecosystem | 7.3% | 5 | 5 | 4 |
| Operational complexity | 4.4% | 3 | 4 | 2 |
| Scalability | 2.0% | 4 | 3 | 5 |
Step 5: Results
- PostgreSQL: 4.45 points → Ranked 1st
- MySQL: 4.21 points → Ranked 2nd
- MongoDB: 3.57 points → Ranked 3rd
Sensitivity Analysis: The gap between PostgreSQL and MySQL is 0.24 points. The sensitivity analysis shows that the ranking remains unchanged after all dimension weights fluctuate by ±10%. Although the score difference is small, the conclusion is insensitive to weight changes, so its credibility is relatively high.
VI. Applicable Scenarios and Limitations
Applicable Scenarios
- 2-7 candidate options, 4-8 evaluation dimensions
- The decision requires participation by multiple people or needs to be kept on record
- Need to explain to stakeholders "why this one was chosen"
Inapplicable Scenarios
- Only 2 options with obvious differences → a direct Pros/Cons list is more efficient
- The decision depends highly on a single factor → a multi-dimensional framework is not needed
- Information is extremely incomplete → scoring is meaningless; first collect information
- Personal decisions involving strong emotional factors → a rational framework may obscure true preferences
Common Misconceptions
- Too many dimensions: More than 8 dimensions causes the weight of each dimension to be very low, reducing discrimination
- False precision: 3.72 vs 3.68 does not mean A is truly better than B—pay attention to sensitivity analysis
- Ignoring qualitative judgment: The matrix is an auxiliary tool, not an algorithm that replaces intuition
- Once decided, final forever: When decision conditions change, it should be re-evaluated
Reference Documents
- Keeney, R. & Raiffa, H. — Decisions with Multiple Objectives(classic textbook on multi-objective decision making)
- Belton, V. & Stewart, T. — Multiple Criteria Decision Analysis(practical guide to MCDA)
- ROC weighting method — Barron & Barrett (1996), "Decision quality using ranked attribute weights"