CDLA · Module 3 — Outcomes and authentic assessment
Scenario item construction, the evidence-based item-writing rules, three-option items, and setting a pass mark with a modified Angoff panel instead of picking 80% by habit.
About 24 min
Practice in the lesson player
Sorting, matching, quizzes, and the module gate stay behind a free account so scored answers are not public. The teaching notes below are the same cited curriculum.
01
A bank's annual conduct exam has a 98% pass rate. Complaints about mis-selling are unchanged. Reviewing the items, you find the longest option is correct in 70% of them, several distractors are obviously silly, and two questions can be answered from the wording of the stem. The test does not measure judgment. It measures test-taking.
02
Stem: A short, realistic situation with only the details needed to decide. Ends in a clear question: "What do you do first?"
Key: The response an expert panel agrees is best in this context, not merely acceptable.
Distractors: Real mistakes people make, found in your CTA, QA data or SME interviews. Each wrong for a different reason.
Rationale: One sentence for each option saying why it is right or wrong. Reviewers check this, not just the key.
03
Rule | Why
Three options are usually enough | Rodriguez (2005) found three-option items perform about as well as four or five, and a third bad distractor adds nothing
Make options similar in length and grammar | Test-wise learners pick the longest or most qualified option
Avoid "all of the above" and "none of the above" | They reward partial knowledge and guessing
Put the question in the stem | Learners should be able to answer without seeing options
No negatives unless the job uses them | "Which is NOT." measures reading care, not judgment
Vary the position of the key | Patterns are learned quickly
04
Situational judgment tests
05
Most pass marks are habits. A defensible pass mark comes from a standard-setting method. In a modified Angoff study, a panel of five to eight people who know the job imagine a minimally competent performer: someone you would just trust to do the job unsupervised. For each item, each panellist estimates the probability that this person would answer correctly. Average across panellists and sum across items: that is the cut score. Discuss the items with the widest disagreement and repeat once.
06
Item | Panellist A | Panellist B | Panellist C | Mean
1 | 0.9 | 0.8 | 0.9 | 0.87
2 | 0.6 | 0.7 | 0.5 | 0.60
3 | 0.8 | 0.8 | 0.7 | 0.77
4 | 0.7 | 0.9 | 0.8 | 0.80
5 | 0.5 | 0.6 | 0.6 | 0.57
Cut score | | | Sum | 3.61 of 5 (72%)
07
Write five scenario items for the highest-risk action in your assessment map, three options each, with rationales. Ask three people who know the job to rate each for a minimally competent performer. Calculate the cut score. Add the items and the worksheet to your module artifact. A note for this academy: our gates use 80% as a starting rule; we are running panels to replace it module by module.