Evaluation Rubrics

Evaluation Rubrics

Table of Contents

• Workflow
• Common Patterns
• Guardrails
• Quick Reference

Example

Scenario: Evaluating technical blog posts (1-5 scale)

Scoring: Post A [4, 5, 3, 2] = 3.5 avg. Post B [5, 4, 5, 4] = 4.5 avg. Feedback for Post A: "Strong clarity (5) and good accuracy (4), but needs more practical examples (3) and offers less original insight (2)."

Workflow

Copy this checklist and track your progress:

Rubric Development Progress:
- [ ] Step 1: Define purpose and scope
- [ ] Step 2: Identify evaluation criteria
- [ ] Step 3: Design the scale
- [ ] Step 4: Write performance descriptors
- [ ] Step 5: Test and calibrate
- [ ] Step 6: Use and iterate

Step 1: Define purpose and scope

Clarify what you're evaluating, who evaluates, who uses results, what decisions depend on scores. See resources/template.md for scoping questions.

Step 2: Identify evaluation criteria

Brainstorm quality dimensions, prioritize most important/observable, balance coverage vs. simplicity (4-8 criteria typical). See resources/template.md for brainstorming framework.

Step 3: Design the scale

Choose number of levels (1-5, 1-4, 1-10), scale type (numeric, qualitative), anchors (what does each level mean?). See resources/methodology.md for scale selection guidance.

Step 4: Write performance descriptors

For each criterion × level, write observable description of what that performance looks like. See resources/template.md for writing guidelines.

Step 5: Test and calibrate

Have multiple reviewers score sample work, compare scores, discuss discrepancies, refine rubric. See resources/methodology.md for inter-rater reliability testing.

Step 6: Use and iterate

Apply rubric, collect feedback from evaluators and evaluatees, revise criteria/descriptors as needed. Validate using resources/evaluators/rubric_evaluation_rubrics.json. Minimum standard: Average score ≥ 3.5.

Common Patterns

Pattern 1: Analytic Rubric (Most Common)

• Structure: Multiple criteria (rows), multiple levels (columns), descriptor for each cell
• Use case: Detailed feedback needed, want to see performance across dimensions, diagnostic assessment
• Pros: Specific feedback, identifies strengths/weaknesses by criterion, high reliability
• Cons: Time-consuming to create and use, can feel reductive
• Example: Code review rubric (Correctness, Efficiency, Readability, Maintainability × 1-5 scale)

Pattern 2: Holistic Rubric

• Structure: Single overall score, descriptors integrate multiple criteria
• Use case: Quick overall judgment, summative assessment, criteria hard to separate
• Pros: Fast, intuitive, captures gestalt quality
• Cons: Less actionable feedback, lower reliability, can't diagnose specific weaknesses
• Example: Essay holistic scoring (1=poor essay, 3=adequate essay, 5=excellent essay with detailed descriptors)

Pattern 3: Single-Point Rubric

• Structure: Criteria listed with only "meets standard" descriptor, space to note above/below
• Use case: Growth mindset feedback, encourage self-assessment, less punitive feel
• Pros: Emphasizes improvement not deficit, simpler to create, encourages dialogue
• Cons: Less precision, requires written feedback to supplement
• Example: Design critique (list criteria like "Visual hierarchy", "Accessibility", note "+Clear focal point, -Poor contrast")

Pattern 4: Checklist (Binary)

• Structure: List of yes/no items, must-haves for acceptance
• Use case: Compliance checks, minimum quality gates, pass/fail decisions
• Pros: Very clear, objective, easy to use
• Cons: No gradations, misses quality beyond basics, can feel rigid
• Example: Pull request checklist (Tests pass? Code linted? Documentation updated? Security review?)

Pattern 5: Standards-Based Rubric

• Structure: Criteria tied to learning objectives/competencies, levels = degree of mastery
• Use case: Educational assessment, skill certification, training evaluation, criterion-referenced
• Pros: Aligned to standards, shows progress toward mastery, diagnostic
• Cons: Requires clear standards, can be complex to design
• Example: Data science skills (Proficiency in: Data cleaning, Modeling, Visualization, Communication × Novice/Competent/Expert)

Guardrails

1. Criteria should be observable and measurable: Not "good attitude" (subjective), but "arrives on time, volunteers for tasks, helps teammates" (observable). Test: Can two independent reviewers score this criterion consistently?

2. Descriptors should distinguish levels clearly: Each level needs concrete differences from adjacent levels. Avoid "5=very good, 4=good, 3=okay". Better: "5=zero bugs, meets all requirements, 4=1-2 minor bugs, meets 90% requirements."

3. Use appropriate scale granularity: 1-3 is too coarse, 1-10 is too fine. Sweet spot: 1-4 (forced choice, no middle) or 1-5 (allows neutral middle). Match granularity to actual observable differences.

4. Balance comprehensiveness with simplicity: Aim for 4-8 criteria covering essential quality dimensions. If >10 criteria, consider grouping or prioritizing.

5. Calibrate for inter-rater reliability: Have multiple reviewers score same work, measure agreement (Kappa, ICC). If <70% agreement, refine descriptors.

6. Provide examples at each level: Include concrete examples of work at each level (anchor papers, reference designs, code samples) to calibrate reviewers.

7. Share rubric before evaluation: If evaluatees see the rubric only after being scored, it is grading not guidance. Share upfront so people know expectations and can self-assess.

8. Weight criteria appropriately: If "Security" matters more than "Code style", weight it (Security x3, Style x1). Or use thresholds (score >=4 on Security to pass, regardless of other scores).

Common pitfalls:

• ❌ Subjective language: "Shows effort", "creative", "professional" - not observable without concrete descriptors
• ❌ Overlapping criteria: "Clarity" and "Organization" often conflated - define boundaries clearly
• ❌ Hidden expectations: Rubric doesn't mention X, but evaluators penalize for missing X - document all criteria
• ❌ Central tendency bias: Reviewers avoid extremes (always score 3/5) - use even-number scales (1-4) to force choice
• ❌ Halo effect: High score on one criterion biases other scores up - score each criterion independently before looking at others
• ❌ Rubric drift: Descriptors erode over time, reviewers interpret differently - periodic re-calibration required

Quick Reference

Key resources:

• [resources/template.md](resources/template.md): Purpose definition, criteria brainstorming, scale selection, descriptor templates, rubric formats
• [resources/methodology.md](resources/methodology.md): Scale design principles, descriptor writing techniques, inter-rater reliability testing, bias mitigation
• [resources/evaluators/rubric_evaluation_rubrics.json](resources/evaluators/rubric_evaluation_rubrics.json): Quality criteria for rubric design (criteria clarity, scale appropriateness, descriptor specificity)

Scale Selection Guide:

Criteria Types:

• Product criteria: Evaluate the artifact itself (correctness, clarity, completeness, aesthetics, performance)
• Process criteria: How work was done (methodology followed, collaboration, iteration, time management)
• Impact criteria: Outcomes/effects (user satisfaction, business value, learning achieved)
• Meta criteria: Quality of quality (documentation, testability, maintainability, scalability)

Inter-Rater Reliability Benchmarks:

• <50% agreement: Rubric unreliable, needs major revision
• 50-70% agreement: Marginal, refine descriptors and calibrate reviewers
• 70-85% agreement: Good, acceptable for most uses
• >85% agreement: Excellent, highly reliable scoring

Typical Rubric Development Time:

• Simple rubric (3-5 criteria, 1-4 scale, known domain): 2-4 hours
• Standard rubric (5-7 criteria, 1-5 scale, some complexity): 6-10 hours + calibration session
• Complex rubric (8+ criteria, multiple scales, novel domain): 15-25 hours + multiple calibration rounds

When to escalate beyond rubrics:

• High-stakes decisions (hiring, admissions, awards) → Add structured interviews, portfolios, multi-method assessment
• Subjective/creative work (art, poetry, design) → Supplement rubric with critique, discourse, expert judgment
• Complex holistic judgment (leadership, cultural fit) → Rubrics help but don't capture everything, use thoughtfully

→ Rubrics are tools not replacements for human judgment. Use to structure thinking, not mechanize decisions.

Inputs required:

• Artifact type (what are we evaluating? essays, code, designs, proposals?)
• Criteria (quality dimensions to assess, 4-8 most common)
• Scale (1-5 default, or specify 1-4, 1-10, qualitative labels)

Outputs produced:

• evaluation-rubrics.md: Purpose, criteria definitions, scale with descriptors, usage instructions, weighting/thresholds, calibration notes

Evaluation Rubrics Methodology

Comprehensive guidance on scale design, descriptor writing, calibration, bias mitigation, and advanced rubric design techniques.

Workflow

Rubric Development Progress:
- [ ] Step 1: Define purpose and scope
- [ ] Step 2: Identify evaluation criteria
- [ ] Step 3: Design the scale
- [ ] Step 4: Write performance descriptors
- [ ] Step 5: Test and calibrate
- [ ] Step 6: Use and iterate

Step 1: Define purpose and scope → See resources/template.md

Step 2: Identify evaluation criteria → See resources/template.md

Step 3: Design the scale → See 1. Scale Design Principles

Step 4: Write performance descriptors → See 2. Descriptor Writing Techniques

Step 5: Test and calibrate → See 3. Calibration Techniques

Step 6: Use and iterate → See 4. Bias Mitigation and 6. Common Pitfalls

---

1. Scale Design Principles

Choosing Appropriate Granularity

The granularity dilemma: Too few levels (1-3) miss meaningful distinctions; too many levels (1-10) create false precision and inconsistency.

Scale characteristics (See SKILL.md Quick Reference for detailed comparison):

• 1-3: Fast, coarse, high reliability. Use for quick screening.
• 1-4: Forces choice (no middle), avoids central tendency. Use when bias observed.
• 1-5: Most common, allows neutral, good balance. General purpose.
• 1-10: Fine gradations, statistical analysis. Use for large samples (100+), expert reviewers.
• Qualitative (Novice/Proficient/Expert): Intuitive for skills, growth-oriented. Educational contexts.

Central Tendency and Response Biases

Central tendency bias: Reviewers avoid extremes, cluster around middle (most get 3/5).

Causes: Uncertainty, social pressure, lack of calibration.

Mitigations: 1. Even-number scales (1-4, 1-6) force choice above/below standard 2. Anchor examples at each level (what does 1 vs 5 look like?) 3. Calibration sessions where reviewers score same work, discuss discrepancies 4. Forced distributions (controversial): Require X% in each category. Use sparingly.

Other response biases:

• Halo effect: Overall impression biases individual criteria scores.
• Mitigation: Vertical scoring (all work on Criterion 1, then Criterion 2), blind scoring.

• Leniency/severity bias: Reviewer consistently scores higher/lower than others.
• Mitigation: Calibration sessions, normalization across reviewers.

• Range restriction: Reviewer uses only part of scale (always 3-4, never 1-2 or 5).
• Mitigation: Anchor examples at extremes, forced distribution (cautiously).

Numeric vs. Qualitative Scales

Numeric (1-5, 1-10): Easy to aggregate, quantitative comparison, ranking. Numbers feel precise but may be arbitrary.

Qualitative (Novice/Proficient/Expert, Below/Meets/Exceeds): Intuitive labels, less false precision. Harder to aggregate, ordinal only.

Hybrid approach (best of both): Numeric with labels (1=Poor, 2=Fair, 3=Adequate, 4=Good, 5=Excellent). Labels anchor meaning, numbers enable analysis.

Unipolar vs. Bipolar:

• Unipolar: 1 (None) → 5 (Maximum). Measures amount or quality. Use for rubrics.
• Bipolar: 1 (Strongly Disagree) → 5 (Strongly Agree), 3=Neutral. Measures agreement.

---

2. Descriptor Writing Techniques

Observable, Measurable Language

Core principle: Two independent reviewers should score the same work consistently based on descriptors alone.

Test for observability: Could two reviewers count/measure this? (Yes → observable). Does this require mind-reading? (Yes → subjective).

Techniques: 1. Quantification: "All 5 requirements met" vs. "Most requirements met" 2. Explicit features: "Includes abstract, intro, methods, results, discussion" vs. "Complete structure" 3. Behavioral indicators: "Asks clarifying questions, proposes alternatives" vs. "Critical thinking" 4. Comparison to standards: "WCAG AA compliant" vs. "Accessible"

Parallel Structure Across Levels

Parallel structure: Each level addresses the same aspects, making differences clear.

Example: Code Review, "Readability" criterion

Benefits: Easy comparison (what changes 3→5?), diagnostic (pinpoint weakness), fair (same dimensions).

Examples and Anchors at Each Level

Anchor: Concrete example of work at a specific level, calibrates reviewers.

Types: 1. Exemplar work samples: Actual submissions scored at each level (authentic, requires permission) 2. Synthetic examples: Crafted to demonstrate each level (controlled, no permission needed) 3. Annotated excerpts: Sections highlighting what merits that score (focused, may miss holistic quality)

Best practices:

• Anchor at extremes and middle (minimum: 1, 3, 5)
• Diversity of anchors (different ways to achieve a level)
• Update anchors as rubric evolves
• Make accessible to evaluators and evaluatees

Avoiding Hidden Expectations

Hidden expectation: Quality dimension reviewers penalize but isn't in rubric.

Example: Rubric has "Technical Accuracy", "Clarity", "Practical Value". Reviewer scores down for "poor visual design" (not a criterion). Problem: Evaluatee had no way to know design mattered.

Mitigation: 1. Comprehensive criteria: If it matters, include it. If not in rubric, don't penalize. 2. Criterion definitions: Explicitly state what is/isn't included. 3. Feedback constraints: Suggestions outside rubric don't affect score. 4. Rubric review: Ask evaluatees what's missing, update accordingly.

---

3. Calibration Techniques

Inter-Rater Reliability Measurement

Inter-rater reliability (IRR): Degree to which independent reviewers give consistent scores.

Target IRR thresholds:

• <50%: Unreliable, major revision needed
• 50-70%: Marginal, refine descriptors, more calibration
• 70-85%: Good, acceptable for most uses
• >85%: Excellent, highly reliable

Measurement methods:

1. Percent Agreement

• Calculation: (# items where reviewers agree exactly) / (total items)
• Pros: Simple, intuitive. Cons: Inflated by chance agreement.
• Variant: Within-1 agreement: Scores within 1 point count as agree. Target: ≥80%.

2. Cohen's Kappa (κ)

• Calculation: (Observed agreement - Expected by chance) / (1 - Expected by chance)
• Range: -1 to 1 (0=chance, 1=perfect agreement)
• Interpretation: <0.20 Poor, 0.21-0.40 Fair, 0.41-0.60 Moderate, 0.61-0.80 Substantial, 0.81-1.00 Almost perfect
• Pros: Corrects for chance. Cons: Only 2 raters, affected by prevalence.

3. Intraclass Correlation Coefficient (ICC)

• Use when: More than 2 raters, continuous scores
• Range: 0 to 1. Interpretation: <0.50 Poor, 0.50-0.75 Moderate, 0.75-0.90 Good, >0.90 Excellent
• Pros: Handles multiple raters, gold standard. Cons: Requires statistical software.

4. Krippendorff's Alpha

• Use when: Multiple raters, missing data, various data types
• Range: 0 to 1. Interpretation: α≥0.80 acceptable, ≥0.67 tentatively acceptable
• Pros: Most flexible, robust to missing data. Cons: Less familiar.

Calibration Session Design

Pre-calibration: 1. Select 3-5 samples spanning quality range (low, medium, high, edge cases) 2. Independent scoring: Each reviewer scores all samples alone, no discussion 3. Calculate IRR: Baseline reliability (percent agreement, Kappa)

During calibration: 4. Discuss discrepancies (focus on differences >1 point): "I scored Sample 1 as 4 because... What led you to 3?" 5. Identify ambiguities: Descriptor unclear? Criterion boundaries fuzzy? Missing cases? 6. Refine rubric: Clarify descriptors (add specificity, numbers, examples), add anchors, revise criteria 7. Re-score: Independently re-score same samples using refined rubric

Post-calibration: 8. Calculate final IRR: If ≥70%, proceed. If <70%, iterate (more refinement + re-calibration). 9. Document: Date, participants, samples, IRR metrics (before/after), rubric changes, scoring decisions 10. Schedule ongoing calibration: Monthly or quarterly check-ins (prevents rubric drift)

Resolving Discrepancies

When reviewers disagree:

• Option 1: Discussion to consensus: Reviewers discuss, agree on final score. Ensures consistency but time-consuming.
• Option 2: Averaged scores: Mean of reviewers' scores. Fast but can mask disagreement (4+2=3).
• Option 3: Third reviewer: If A and B differ by >1, C scores as tie-breaker. Resolves impasse but requires extra reviewer.
• Option 4: Escalation: Discrepancies >1 escalated to lead reviewer or committee. Quality control but bottleneck.

Recommended: Average for small discrepancies (1 point), discussion for large (2+ points), escalate if unresolved.

---

4. Bias Mitigation

Halo Effect

Halo effect: Overall impression biases individual criterion scores. "Excellent work" → all criteria high, or "poor work" → all low.

Example: Code has excellent documentation (5/5) but poor performance (should be 2/5). Halo: Reviewer scores performance 4/5 due to overall positive impression.

Mitigation: 1. Vertical scoring: Score all submissions on Criterion 1, then all on Criterion 2 (focus on one criterion at a time) 2. Blind scoring: Reviewers don't see previous scores when scoring new criterion 3. Separate passes: First pass for overall sense (don't score), second pass to score each criterion 4. Criterion definitions: Clear, narrow definitions reduce bleed-over

Anchoring and Order Effects

Anchoring: First information biases subsequent judgments. First essay scored 5/5 → second (objectively 4/5) feels worse → scored 3/5.

Mitigation: 1. Randomize order: Review in random order, not alphabetical or submission time 2. Calibration anchors: Review rubric and anchors before scoring (resets mental baseline) 3. Batch scoring: Score all on one criterion at once (easier to compare)

Order effects: Position in sequence affects score (first/last reviewed scored differently).

Mitigation: Multiple reviewers score in different random orders (order effect averages out).

Leniency and Severity Bias

Leniency: Reviewer consistently scores higher than others (generous). Severity: Consistently scores lower (harsh).

Detection: Calculate mean score per reviewer. If Reviewer A averages 4.2 and Reviewer B averages 2.8 on same work → bias present.

Mitigation: 1. Calibration sessions: Show reviewers their bias, discuss differences 2. Normalization (controversial): Convert to z-scores (adjust for reviewer's mean). Changes scores, may feel unfair. 3. Multiple reviewers: Average scores (bias cancels out) 4. Threshold-based: Focus on "meets standard" (yes/no) vs numeric score

---

5. Advanced Rubric Design

Weighted Criteria

Weighting approaches:

1. Multiplicative weights:

• Score × weight, sum weighted scores, divide by sum of weights
• Example: Security (4×3=12), Performance (3×2=6), Style (5×1=5). Total: 23/6 = 3.83

2. Percentage weights:

• Assign % to each criterion (sum to 100%)
• Example: Security 4×50%=2.0, Performance 3×30%=0.9, Style 5×20%=1.0. Total: 3.9/5.0

When to weight: Criteria have different importance, regulatory/compliance criteria, developmental priorities.

Cautions: Adds complexity, can obscure deficiencies (low critical score hidden in average). Alternative: Threshold scoring.

Threshold Scoring

Threshold: Minimum score required on specific criteria regardless of overall average.

Example:

• Overall average ≥3.0 to pass
• AND Security ≥4.0 (critical threshold)
• AND No criterion <2.0 (floor threshold)

Benefits: Ensures critical criteria meet standard, prevents "compensation" (high Style masking low Security), clear requirements.

Use cases: Safety-critical systems, compliance requirements, competency gatekeeping.

Combination Rubrics

Hybrid approaches:

• Analytic + Holistic: Analytic for diagnostic detail, holistic for overall judgment. Use when want both.
• Checklist + Rubric: Checklist for must-haves (gatekeeping), rubric for quality gradations (among passing). Use for gatekeeping then ranking.
• Self-Assessment + Peer + Instructor: Same rubric used by student, peers, instructor. Compare scores, discuss. Use for metacognitive learning.

---

6. Common Pitfalls

Overlapping Criteria

Problem: Criteria not distinct, same dimension scored multiple times.

Example: "Organization" (structure, flow, coherence) + "Clarity" (easy to understand, well-structured, logical). Overlap: "well-structured" in both.

Detection: High correlation between criteria scores. Difficulty explaining difference.

Fix: Define boundaries explicitly ("Organization = structure. Clarity = language."), combine overlapping criteria, or split into finer-grained distinct criteria.

Rubric Drift

Problem: Over time, reviewers interpret descriptors differently, rubric meaning changes.

Causes: No ongoing calibration, staff turnover, system changes.

Detection: IRR declines (was 80%, now 60%), scores inflate/deflate (average was 3.5, now 4.2 with no quality change), inconsistency complaints.

Prevention: 1. Periodic calibration: Quarterly sessions even with experienced reviewers 2. Anchor examples: Maintain library, use same anchors over time 3. Documentation: Record scoring decisions, accessible to new reviewers 4. Version control: Date rubric versions, note changes, communicate updates

False Precision

Problem: Numeric scores imply precision that doesn't exist. 10-point scale but difference between 7 vs 8 arbitrary.

Fix:

• Reduce granularity (10→5 or 3 categories)
• Add descriptors for each level
• Report confidence intervals (Score = 3.5 ± 0.5)
• Be transparent: "Scores are informed judgments, not objective measurements"

No Consequences for Ignoring Rubric

Problem: Rubric exists but reviewers don't use it or override scores based on gut feeling. Rubric becomes meaningless.

Fix: 1. Require justification: Reviewers must cite rubric descriptors when scoring 2. Audit scores: Spot-check scores against rubric, challenge unjustified deviations 3. Training: Emphasize rubric as contract (if wrong, change rubric, don't ignore) 4. Accountability: Reviewers who consistently deviate lose review privileges

---

Summary

Scale design: Choose granularity matching observable differences. Mitigate central tendency with even-number scales or anchors.

Descriptor writing: Use observable language, parallel structure, examples at each level. Test: Can two reviewers score consistently?

Calibration: Measure IRR (Kappa, ICC), conduct calibration sessions, refine rubric, prevent drift with ongoing calibration.

Bias mitigation: Vertical scoring for halo effect, randomize order for anchoring, normalize or average for leniency/severity.

Advanced design: Weight critical criteria, use thresholds to prevent compensation, combine rubric types.

Pitfalls: Define distinct criteria, prevent drift with documentation and re-calibration, avoid false precision, ensure rubric has teeth.

Final principle: Rubrics structure judgment, not replace it. Use to increase consistency and transparency, not mechanize evaluation.

Evaluation Rubrics Templates

Quick-start templates for purpose definition, criteria selection, scale design, descriptor writing, and rubric formats.

Workflow

Rubric Development Progress:
- [ ] Step 1: Define purpose and scope
- [ ] Step 2: Identify evaluation criteria
- [ ] Step 3: Design the scale
- [ ] Step 4: Write performance descriptors
- [ ] Step 5: Test and calibrate
- [ ] Step 6: Use and iterate

Step 1: Define purpose and scope

Use Purpose Definition Template to clarify evaluation context and constraints.

Step 2: Identify evaluation criteria

Brainstorm and prioritize quality dimensions using Criteria Identification Template.

Step 3: Design the scale

Select scale type and levels using Scale Selection Template.

Step 4: Write performance descriptors

Write clear, observable descriptors using Descriptor Writing Template.

Step 5: Test and calibrate

Conduct inter-rater reliability testing and refine rubric.

Step 6: Use and iterate

Apply rubric, collect feedback, revise as needed.

---

Purpose Definition Template

What are we evaluating?

• Artifact type: [e.g., code pull requests, research proposals, design mockups, student essays]
• Specific context: [e.g., internal code review, grant competition, course assignment]

Who will evaluate?

• Number of evaluators: [Single reviewer or multiple?]
• Evaluator expertise: [Subject matter experts, peers, instructors, automated systems]
• Evaluator availability: [Time per evaluation? Total volume?]

Who are the evaluatees?

• Audience: [Students, employees, vendors, applicants]
• Skill level: [Novice, intermediate, expert]
• Will they see rubric before evaluation? [Yes/No - if yes, rubric serves as guide]

What decisions depend on scores?

• High stakes: [Pass/fail, hiring, funding, promotion, grades]
• Medium stakes: [Feedback for improvement, prioritization, awards]
• Low stakes: [Self-assessment, informal feedback]

Success criteria for rubric:

• [ ] Enables consistent scoring across evaluators (inter-rater reliability >70%)
• [ ] Provides actionable feedback for improvement
• [ ] Takes reasonable time to use (target: X minutes per evaluation)
• [ ] Acceptable to evaluators (not overly complex or rigid)
• [ ] Acceptable to evaluatees (perceived as fair and transparent)

---

Criteria Identification Template

Brainstorming Quality Dimensions

Product criteria (artifact itself):

• Correctness/Accuracy: [Is it right? Factually accurate? Meets requirements?]
• Completeness: [Covers all necessary elements? No major gaps?]
• Clarity: [Easy to understand? Well-organized? Clear communication?]
• Quality/Craftsmanship: [Attention to detail? Polished? Professional?]
• Originality/Creativity: [Novel approach? Innovative? Goes beyond expected?]
• Performance: [Fast? Efficient? Scalable? Meets technical specs?]

Process criteria (how it was made):

• Methodology: [Followed appropriate process? Research methods sound?]
• Collaboration: [Teamwork? Communication? Used feedback?]
• Iteration: [Multiple drafts? Refinement? Responsiveness to critique?]
• Time management: [Completed on time? Paced work appropriately?]

Impact criteria (effects/outcomes):

• Usability: [User-friendly? Accessible? Intuitive?]
• Value: [Solves problem? Addresses need? Business impact?]
• Learning demonstrated: [Shows understanding? Growth from previous work?]

Meta criteria (quality of quality):

• Maintainability: [Can others work with this? Documented? Modular?]
• Testability: [Can be verified? Validated? Measured?]
• Extensibility: [Can be built upon? Flexible? Adaptable?]

Prioritization

Rate each candidate criterion:

Selection rules:

• Must be High or Medium importance
• Must be Observable (can two reviewers score consistently?)
• Must be Distinct (not overlapping with other criteria)
• Aim for 4-8 criteria (balance coverage vs. simplicity)

Final criteria (4-8 selected): 1. [Criterion]: [Brief definition] 2. [Criterion]: [Brief definition] 3. [Criterion]: [Brief definition] 4. [Criterion]: [Brief definition]

---

Scale Selection Template

Scale type options:

Numeric Scales

1-3 scale (Low/Medium/High)

• Use when: Quick categorization, clear tiers sufficient
• Levels: 1=Below standard, 2=Meets standard, 3=Exceeds standard

1-4 scale (Forced choice, no middle)

• Use when: Want to avoid central tendency, need clear differentiation
• Levels: 1=Poor, 2=Fair, 3=Good, 4=Excellent

1-5 scale (Most common, allows neutral)

• Use when: General purpose, familiar to evaluators
• Levels: 1=Poor, 2=Fair, 3=Adequate, 4=Good, 5=Excellent

1-10 scale (Fine gradations)

• Use when: Large sample, need statistical analysis, can distinguish subtle differences
• Levels: 1-2=Poor, 3-4=Fair, 5-6=Adequate, 7-8=Good, 9-10=Excellent

Qualitative Scales

Developmental: Novice → Developing → Proficient → Expert Standards-based: Below Standard → Approaching → Meets → Exceeds Competency: Not Yet Competent → Partially Competent → Competent → Highly Competent

Binary

Pass/Fail, Yes/No, Present/Absent

• Use when: Compliance checks, minimum thresholds, clear criteria

Selected scale for this rubric: [Choose one]

• Type: [Numeric 1-5, Qualitative, etc.]
• Levels: [List with labels]
• Rationale: [Why this scale fits purpose]

---

Descriptor Writing Template

For each criterion, write descriptors at each scale level.

Criterion: [Name]

Definition: [What does this criterion assess? 1-2 sentences]

Why it matters: [Importance to overall quality]

Scale descriptors:

Level 5 (or highest): [Label]

Observable characteristics:

• [Concrete, observable feature 1]
• [Concrete, observable feature 2]
• [Concrete, observable feature 3]

Example: [Specific instance of work at this level]

Level 4: [Label]

Observable characteristics:

• [How this differs from Level 5 - what's missing or less strong]
• [Concrete observable feature]

Example: [Specific instance]

Level 3: [Label] (Baseline/Adequate)

Observable characteristics:

• [Minimum acceptable performance]
• [Observable feature]

Example: [Specific instance]

Level 2: [Label]

Observable characteristics:

• [What's lacking compared to Level 3]
• [Observable deficiency]

Example: [Specific instance]

Level 1 (or lowest): [Label]

Observable characteristics:

• [Significant deficiencies]
• [Observable problems]

Example: [Specific instance]

---

Descriptor Writing Guidelines

DO:

• Use observable, measurable language ("Contains 3+ bugs" not "poor quality")
• Provide concrete examples or anchors for each level
• Focus on what IS present at each level, not just "less than" higher level
• Use parallel structure across levels (same aspects addressed at each level)
• Specify quantities when possible ("All 5 requirements met" vs "Most requirements met")

DON'T:

• Use subjective terms without definition ("creative", "professional", "excellent effort")
• Rely on comparative language only ("better than", "more sophisticated")
• Make assumptions about process ("spent time", "worked hard" - unless observable)
• Penalize for things not mentioned in descriptor (hidden expectations)

---

Analytic Rubric Template

Most common format: Multiple criteria (rows) × Multiple levels (columns)

Rubric for: [Artifact Type]

Purpose: [Brief description]

Scale: [1-5, 1-4, etc. with labels]

Scoring:

• Calculate: (Score1 × Weight1) + (Score2 × Weight2) + ... / Total Weights
• Threshold: [e.g., Must average ≥3.0 to pass, ≥4 on critical criteria]

Usage notes:

• Score each criterion independently before looking at others (avoid halo effect)
• Provide brief justification for each score
• Flag areas for improvement in feedback

---

Holistic Rubric Template

Single overall score integrating multiple criteria.

Rubric for: [Artifact Type]

Purpose: [Brief description]

Level 5: Excellent

Overall quality: [Integrated description touching all important aspects]

• Criterion A: [How it manifests at this level]
• Criterion B: [How it manifests at this level]
• Criterion C: [How it manifests at this level]

Example: [Work that exemplifies this level]

Level 4: Good

Overall quality: [Integrated description]

• Differences from Level 5: [What's less strong]
• Key characteristics: [Observable features]

Example: [Work that exemplifies this level]

Level 3: Adequate

Overall quality: [Integrated description of baseline acceptable]

• Meets minimum standards: [What's required]
• May have: [Acceptable weaknesses]

Example: [Work that exemplifies this level]

Level 2: Weak

Overall quality: [Integrated description of below standard]

• Falls short because: [Key deficiencies]
• Problems include: [Observable issues]

Example: [Work that exemplifies this level]

Level 1: Poor

Overall quality: [Integrated description of unacceptable]

• Major problems: [Significant deficiencies across multiple aspects]

Example: [Work that exemplifies this level]

---

Single-Point Rubric Template

Lists criteria with "meets standard" description only, space to note exceeds/below.

Rubric for: [Artifact Type]

Usage:

• Check if work meets standard for each criterion
• Note specific strengths in "Advanced" column (e.g., "+Exceptionally clear examples")
• Note specific areas for improvement in "Concerns" column (e.g., "-Missing citations for 3 claims")

---

Checklist Template

Binary yes/no for must-have requirements.

Checklist for: [Artifact Type]

Category 1: [e.g., Completeness]

• [ ] [Specific requirement 1]
• [ ] [Specific requirement 2]
• [ ] [Specific requirement 3]

Category 2: [e.g., Quality]

• [ ] [Specific requirement 4]
• [ ] [Specific requirement 5]

Category 3: [e.g., Compliance]

• [ ] [Specific requirement 6]
• [ ] [Specific requirement 7]

Pass/Fail Criteria:

• Pass: All items checked OR All items in critical categories + X% of others
• Fail: Any critical item unchecked OR <Y% total items checked

---

Weighted Scoring Template

When criteria have different importance.

Weighted Rubric for: [Artifact Type]

Weight categories:

• ×3 = Critical (must be strong, threshold: ≥4 required)
• ×2 = Important (significant impact on overall quality)
• ×1 = Desirable (nice to have, less critical)

---

Calibration Session Template

Pre-calibration: 1. Select 3-5 sample works spanning quality range (low, medium, high) 2. Have each reviewer independently score samples using rubric 3. Record scores without discussion

During calibration: 4. Compare scores across reviewers for each sample 5. For discrepancies (>1 point difference):

• Discuss what each reviewer saw
• Identify ambiguous descriptors
• Clarify criterion boundaries
• Refine rubric language

6. Re-score samples using refined rubric

Post-calibration: 7. Calculate inter-rater reliability (% agreement, Kappa) 8. Target: ≥70% agreement (within 1 point) or Kappa ≥0.6 9. If below target: Iterate with more refinement + calibration 10. Document calibration decisions and rubric changes

---

Feedback Template

For: [Evaluatee Name]

Overall Score: [X.X / 5.0 or Level]

Criterion-by-Criterion Scores:

Summary:

• Greatest strengths: [2-3 specific strengths]
• Priority improvements: [2-3 most important areas to address]
• Next steps: [Actionable recommendations]

Overall assessment: [Pass/Fail or qualitative judgment]