Assessment Engineering — Assessment Engineering Methodology
Expert knowledge used by the quiz-maker agent when designing formative and summative assessments.
Why Assessment Engineering
Good assessment accurately measures learner understanding while simultaneously reinforcing learning. Poor assessment rewards memorization; good assessment rewards understanding.
Item Type Design Guide
1. Multiple Choice
| Element |
Rule |
| Stem |
Must be a clear question when read independently |
| Key (Correct Answer) |
Must be unambiguously the best answer |
| Distractors |
Plausible but clearly incorrect (see below) |
| Number of Options |
4 (5 adds meaningless wrong answers; 3 gives a 33% guess rate) |
| Option Length |
Keep similar length (longest option being correct is a clue leak) |
Distractor Design Psychology
| Distractor Type |
Purpose |
Example (Q: "Which HTTP method deletes a resource in REST?") |
| Common Misconception |
Diagnose incorrect knowledge |
"POST" (confusing create and delete) |
| Partial Answer |
Diagnose superficial understanding |
"PUT" (confusing update and delete) |
| Related Concept |
Diagnose concept discrimination |
"PATCH" (partial update, similar concept) |
| Correct: |
|
"DELETE" |
2. Code Writing Problems
[Scenario Setup]
Write a function that meets the following requirements.
[Input/Output Specification]
- Input: [Type and range]
- Output: [Type and conditions]
- Constraints: [Time/space complexity, restrictions]
[Example]
Input: [Example input]
Output: [Example output]
[Grading Criteria]
- [ ] Correctness: Passes all test cases
- [ ] Code Quality: Readability, naming
- [ ] Efficiency: Time/space complexity
3. Scenario-Based Problems
[Scenario]
You are in the following situation. [Concrete situation description]
[Question]
What is the most appropriate [action/design/solution] in this situation?
[Evaluation Perspective]
- Problem identification ability
- Solution strategy appropriateness
- Trade-off awareness
Formative vs. Summative Assessment
| Attribute |
Formative |
Summative |
| Timing |
During learning (every lesson) |
After learning (end of module) |
| Purpose |
Check understanding, immediate correction |
Measure achievement, assign grades |
| Atmosphere |
Low-stakes, practice |
Formal, final |
| Item Count |
3-5 |
10-20 |
| Feedback |
Immediate, detailed |
Post-completion, aggregate |
| Retakes |
Unlimited |
1-2 |
| Bloom's Level |
1-3 (Remember, Understand, Apply) |
3-6 (Apply through Create) |
Formative Assessment Design Principles
- 3-5 items at the end of each lesson: Confirm key concept understanding
- Immediate feedback: Correct/incorrect + "Why this is the answer" explanation
- Learning-directed wrong answers: "Review this section: [link]"
- Difficulty ordering: Easy to hard (builds confidence)
Summative Assessment Design Principles
- 100% learning objective coverage: Every objective mapped to at least one item
- Bloom's distribution: Remember 20% + Understand 20% + Apply 30% + Analyze/Evaluate/Create 30%
- Time allocation: 2-3 minutes per item, 30-60 minutes total
- Practical weighting: Labs/projects should account for 40-60% of the total score
Rubric Construction
Analytic Rubric Template
| Criterion |
Excellent (4) |
Good (3) |
Satisfactory (2) |
Needs Improvement (1) |
| Functionality |
All requirements met |
90% met |
70% met |
Below 50% |
| Code Quality |
Clean, consistent style |
Mostly clean |
Some inconsistency |
Low readability |
| Error Handling |
All edge cases |
Major cases |
Basic cases only |
No error handling |
| Documentation |
Detailed comments + README |
Key function comments |
Minimal comments |
No comments |
Rubric Writing Rules
- Describe observable behaviors: Instead of "understood well," use "can explain..."
- Clear distinctions between levels: Each level must be unambiguously different
- 3-5 criteria: Too many makes assessment impractical
- Share with learners in advance: Showing the rubric before assessment sets learning direction
Feedback Design
Effective Feedback Formula: EEC
E (Evidence): "Your code lacks error handling"
E (Effect): "In this case, the program will crash if the user enters an empty value"
C (Change): "Wrap it in a try-except block and show the user a guidance message"
Feedback Rules
- Specific: Instead of "Good job," say "Your variable names clearly describe their purpose"
- Immediate: As soon as possible after submission (automated tests preferred)
- Action-oriented: Tell the learner what to do next
- Positive first: What was done well -> Areas for improvement -> Next steps
