Usability Tester

Usability Tester

Overview

Validates that users can successfully complete core tasks through systematic observation and expert evaluation. Covers moderated and unmoderated testing, heuristic evaluation, accessibility checks, and issue severity scoring. Not a substitute for analytics or A/B testing -- those measure what happens, usability testing reveals why.

When to use: Testing user flows, validating designs, identifying friction points, running heuristic evaluations, ensuring users can complete core tasks, planning and executing usability test sessions.

When NOT to use: Analytics or A/B test setup, visual design critique without task-based evaluation, automated UI testing (use a testing framework), performance benchmarking.

Quick Reference

Core Metrics

Issue Severity

Severity equals Impact (1-3) multiplied by Frequency (1-3). Critical (8-9): fix before release. High (6-7): fix before release. Medium (4-5): next release. Low (1-3): backlog.

Common Mistakes

Delegation

• Test plan and scenario creation: Use Task agent to draft objectives, research questions, and realistic task scenarios for each persona
• Heuristic evaluation: Use Task agent to walk through interfaces against Nielsen's 10 heuristics and document violations with severity ratings
• Metrics analysis and severity scoring: Use Task agent to calculate SUS scores, task success rates, and severity ratings from raw test data
• Test report generation: Use Task agent to compile findings into structured reports with prioritized recommendations
• Test infrastructure planning: Use Plan agent to select testing methods, define recruitment criteria, and schedule test waves

If the quality-auditor skill is available, delegate full codebase quality audits and scoring to it.

Otherwise, recommend: pnpm dlx skills add oakoss/agent-skills -s quality-auditor -a claude-code -y

References

• Test Planning -- objectives, research questions, recruitment, task scenario templates, screening criteria
• Conducting Tests -- think-aloud protocol, facilitation rules, post-task questions, session structure
• Testing Methods -- unmoderated, guerrilla, first-click, cognitive walkthrough, method selection by lifecycle stage
• Heuristic Evaluation -- Nielsen's 10 heuristics, evaluation process, severity rating, combining with user testing
• Metrics and Severity -- success rate, time on task, errors, satisfaction, SUS scoring, severity formula
• Reporting -- test report template, key insights format, stakeholder presentation, recommended actions
• Accessibility Testing -- inclusive recruitment, assistive technology testing, WCAG alignment, accessibility heuristics
• Remote and Tools -- remote vs in-person comparison, testing tools, test frequency, checklists

Accessibility Testing

Accessibility testing validates that people with disabilities can use the product effectively. It combines automated checks, expert review against WCAG guidelines, and usability testing with participants who use assistive technologies.

When to Test for Accessibility

Pre-Launch (required):
  - Run automated accessibility scanner (catches ~30% of issues)
  - Conduct expert WCAG audit against Level AA criteria
  - Test with screen reader on key user flows
  - Test keyboard-only navigation on all interactive elements

Post-Launch (recommended):
  - Usability test with 3-5 participants with disabilities
  - Test new features with assistive technologies before release
  - Quarterly accessibility audit of high-traffic pages

Compliance Checkpoints:
  - Before any public-facing release
  - When adding new interactive components
  - When redesigning navigation or forms
  - When changing color schemes or typography

Inclusive Recruitment

Recruit participants with disabilities who represent your actual user base.

Participant Mix (minimum 3-5 participants):
  - At least 1 screen reader user (JAWS, NVDA, or VoiceOver)
  - At least 1 keyboard-only user (motor impairment)
  - At least 1 user with low vision (magnification, high contrast)
  - Consider: cognitive disabilities, hearing impairments (for video/audio content)

Recruitment Sources:
  - Disability advocacy organizations
  - Specialized recruitment panels (Fable, AbilityNet)
  - University disability services
  - Existing user base (add accessibility questions to screener)

Screening Considerations:
  - Ask about assistive technology used daily (not just occasionally)
  - Ask about experience level with the assistive technology
  - Confirm the participant's setup matches the test environment
  - Schedule extra time (sessions may run 25-50% longer)

Incentives:
  - Pay at least the same rate as other participants (never less)
  - Offer flexible scheduling (transportation may take longer)
  - For remote tests: verify the participant's setup supports screen sharing

Test Adaptations

Usability tests with participants with disabilities require protocol modifications.

Session Adjustments:
  - Allow 75-90 minutes instead of 60 (tasks take longer with AT)
  - Reduce to 3-4 tasks instead of 5
  - Provide task instructions in multiple formats (verbal, written, large print)
  - Be prepared for the session to run at a different pace
  - Have backup plans if the participant's AT has compatibility issues

Think-Aloud Modifications:
  - Screen reader users may find concurrent think-aloud harder (AT already produces audio)
  - Consider retrospective think-aloud for screen reader users
  - Ask screen reader users to describe what they hear, not just what they do
  - For keyboard users: ask them to describe their navigation strategy

Facilitation Notes:
  - Do not assume what the participant can or cannot do
  - Let the participant use their own device and AT setup when possible
  - Ask how they normally approach similar tasks before starting
  - If using remote testing, test the screen sharing setup in advance
  - Describe any visual elements if the participant cannot see them

Assistive Technology Testing

Even without users with disabilities, the team should test with assistive technologies directly.

Screen Reader Testing

Test With:
  - VoiceOver (macOS/iOS, built-in)
  - NVDA (Windows, free)
  - JAWS (Windows, enterprise standard)

Key Checks:
  - All interactive elements are reachable and labeled
  - Images have meaningful alt text (or empty alt for decorative)
  - Form fields have associated labels
  - Error messages are announced when they appear
  - Page structure uses proper heading hierarchy (h1 through h6)
  - Dynamic content updates are announced (ARIA live regions)
  - Modal dialogs trap focus correctly
  - Tables have proper headers for data relationships

Common Issues Found:
  - Unlabeled buttons (icon-only with no accessible name)
  - Custom components not announcing their role or state
  - Focus moves to unexpected locations after interactions
  - Dynamic content changes silently (no announcement)

Keyboard Navigation Testing

Test All Pages For:
  - Every interactive element reachable via Tab key
  - Logical tab order (left-to-right, top-to-bottom)
  - Visible focus indicator on every focusable element
  - Escape key closes modals and dropdowns
  - Enter or Space activates buttons and links
  - Arrow keys navigate within components (menus, tabs, radio groups)
  - No keyboard traps (focus gets stuck in a component)
  - Skip navigation link available to bypass repeated content

Common Issues Found:
  - Custom dropdowns not keyboard accessible
  - Focus indicator hidden by CSS (outline: none without replacement)
  - Tab order jumps around the page illogically
  - Modal does not return focus to trigger element on close

Visual Testing

Color Contrast:
  - Normal text: 4.5:1 contrast ratio minimum (WCAG AA)
  - Large text (18px+ or 14px+ bold): 3:1 contrast ratio minimum
  - UI components and graphical objects: 3:1 contrast ratio minimum
  - Check with browser DevTools or contrast checker tools

Zoom and Magnification:
  - Content remains usable at 200% browser zoom
  - No horizontal scrolling at 200% zoom on standard viewports
  - Text resizes properly without breaking layout
  - Touch targets remain accessible when zoomed

Color Independence:
  - Information is not conveyed by color alone
  - Error states use icons or text in addition to red color
  - Charts and graphs have patterns or labels, not just colors
  - Links are distinguishable from surrounding text without color

Accessibility Heuristics

Five principles for evaluating accessibility, complementing Nielsen's general usability heuristics.

1. Perceivable:
  - All content available in text or text alternatives
  - Captions for video, transcripts for audio
  - Content adaptable to different presentations (reflow, zoom)
  - Sufficient contrast for all text and UI components

2. Operable:
  - All functionality available via keyboard
  - Users have enough time to read and interact
  - No content that flashes more than 3 times per second
  - Multiple ways to navigate (search, sitemap, breadcrumbs)

3. Understandable:
  - Text is readable and comprehensible
  - Pages behave predictably (no unexpected changes)
  - Users can avoid and correct mistakes
  - Instructions do not rely solely on sensory characteristics

4. Robust:
  - Content works with current and future assistive technologies
  - Valid HTML with proper ARIA attributes
  - Custom components follow WAI-ARIA authoring practices
  - Compatible across browsers and devices

5. Inclusive Interaction:
  - Multiple input methods supported (mouse, keyboard, touch, voice)
  - Error recovery is accessible to all users
  - Help and documentation accessible with assistive technology
  - Timeout warnings give users enough time to respond

WCAG Quick Reference

Focus on the most commonly violated criteria.

Most Frequently Failed WCAG AA Criteria:
  - 1.1.1 Non-text Content (missing alt text)
  - 1.3.1 Info and Relationships (missing form labels, heading structure)
  - 1.4.3 Contrast Minimum (text below 4.5:1 ratio)
  - 2.1.1 Keyboard (interactive elements not keyboard accessible)
  - 2.4.4 Link Purpose (generic "click here" or "read more" links)
  - 3.3.2 Labels or Instructions (form fields without labels)
  - 4.1.2 Name, Role, Value (custom components without ARIA)

Automated Tools (catch ~30% of issues):
  - axe DevTools (browser extension, free)
  - WAVE (browser extension, free)
  - Lighthouse (built into Chrome DevTools)
  - pa11y (command line, CI integration)

Manual Checks Required For:
  - Logical reading order
  - Meaningful alt text quality
  - Keyboard focus management
  - ARIA implementation correctness
  - Cognitive load and readability

Reporting Accessibility Findings

Issue Format:
  Title: [Short description]
  WCAG Criterion: [e.g., 1.4.3 Contrast Minimum]
  Level: [A | AA | AAA]
  Severity: [Critical | High | Medium | Low]
  Affected Users: [Which disability groups]
  Location: [Page and component]
  Current Behavior: [What happens now]
  Expected Behavior: [What should happen]
  Remediation: [Specific fix with code example if relevant]

Example:
  Title: Search button has no accessible name
  WCAG Criterion: 4.1.2 Name, Role, Value
  Level: A
  Severity: Critical
  Affected Users: Screen reader users
  Location: Global header, search component
  Current Behavior: Screen reader announces "button" with no label
  Expected Behavior: Screen reader announces "Search button"
  Remediation: Add aria-label="Search" to the button element

Conducting Tests

Think-Aloud Protocol

The think-aloud protocol is the primary method for understanding user mental models during testing. There are two variants.

Concurrent Think-Aloud

The participant verbalizes thoughts while performing tasks. Use for most moderated tests.

Key instruction to participant:

"Please think aloud as you work. Tell me what you're looking for,
what you're thinking, what you're trying to do. There are no
wrong answers -- we're testing the product, not you."

If the participant goes silent for more than 10-15 seconds:

"What are you thinking right now?"
"What are you looking for?"
"Tell me what's going through your mind."

Retrospective Think-Aloud

The participant completes the task silently, then reviews a recording and explains their thought process. Use when concurrent verbalization would interfere with the task (timing-sensitive interactions, complex calculations).

"Now let's watch the recording together. Please tell me what you
were thinking at each point. I'll pause if you want to explain
something in more detail."

What to Listen For

Facilitation Rules

Do

• Observe silently while the participant works
• Take timestamped notes on behavior, not just outcomes
• Let them struggle -- this reveals real discoverability issues
• Ask follow-up questions AFTER each task, not during
• Stay physically neutral (no nodding, no frowning)
• Use the "echo" technique: repeat their words back as a question
• Redirect if they ask you a question: "What would you do if I weren't here?"

Do Not

• Help or explain the interface
• Lead them ("maybe try clicking the menu...")
• Defend design choices ("that's because we designed it to...")
• Interrupt during a task
• Show frustration, surprise, or disappointment
• Answer questions about the interface during tasks
• Rush participants who are taking longer than expected

The Echo Technique

When participants ask questions or make statements, reflect them back instead of answering.

Participant: "Should I click this button?"
Moderator: "What do you think that button does?"

Participant: "Is this the right page?"
Moderator: "What were you expecting to see?"

Participant: "I don't know what to do next."
Moderator: "What would you try if I weren't here?"

Participant: "This doesn't make sense."
Moderator: "Tell me more about what doesn't make sense."

Note-Taking Framework

Use a structured format to capture observations consistently across sessions.

Note Template:
  Timestamp: [MM:SS from session start]
  Task: [Which task number]
  Observation Type: [action | verbalization | error | recovery | abandonment]
  Description: [What happened]
  Participant Quote: [Verbatim if notable]
  Severity Estimate: [low | medium | high | critical]

Example Notes:
  - "02:15 | Task 2 | action | Clicked 'Settings' looking for invite feature"
  - "02:30 | Task 2 | verbalization | 'I expected invite to be on the project page'"
  - '03:00 | Task 2 | error | Opened wrong menu, had to backtrack'
  - '03:45 | Task 2 | recovery | Found invite in team settings after browsing'

Post-Task Questions

Ask these immediately after each task while the experience is fresh.

Completion Questions:
  - 'On a scale of 1-5, how easy was that task?'
  - 'What were you expecting to see?'
  - 'What was confusing about that process?'
  - 'If you could change one thing about that experience, what would it be?'

Discovery Questions:
  - 'Where did you expect to find that feature?'
  - 'What do you think this [element] does?'
  - 'Why did you click there first?'
  - 'Was there a point where you felt lost?'

Comparison Questions (for redesigns):
  - 'How does this compare to how you usually do this task?'
  - 'Was anything easier or harder than you expected?'

Post-Session Questions

Ask at the end of all tasks for overall impressions.

Overall Experience:
  - 'What was the most frustrating part of your experience?'
  - 'What was the easiest part?'
  - 'If you were describing this product to a friend, what would you say?'
  - 'Would you use this product? Why or why not?'
  - 'Is there anything else you want to share about your experience?'

Feature Prioritization:
  - 'If you could add one feature, what would it be?'
  - "Was there anything you expected to find but didn't?"

Session Management

When to Intervene

Intervention should be rare but sometimes necessary.

Intervene When:
  - Participant is visibly distressed (not just frustrated)
  - Participant has been stuck for more than the time limit
  - Technical failure prevents task completion (bug, crash)
  - Participant explicitly asks to stop

Do Not Intervene When:
  - Participant is frustrated but still trying
  - Participant is taking a wrong path (this is valuable data)
  - Participant pauses to think
  - Participant makes an error they can recover from

Handling Common Situations

Participant asks for help:
  Response: "What would you do if I weren't here?"
  If stuck beyond time limit: "Let's move on to the next task."

Participant gets emotional:
  Response: "Remember, we're testing the product, not you. It's completely
  fine to find things difficult -- that's exactly why we're testing."

Participant wants to quit a task:
  Response: "That's fine, let's move on. Can you tell me what you found
  difficult about that?"

Technical failure:
  Response: Note the failure, skip the task, move on. Do not debug
  during the session.

Between-Session Checklist

After Each Session:
  - Save and label the recording
  - Review notes while memory is fresh
  - Flag top 3 issues observed
  - Note any changes needed to the test script
  - Reset the test environment to starting state
  - Take a 10-minute break before the next session

Heuristic Evaluation

Heuristic evaluation is an expert-based inspection method where evaluators examine an interface against established usability principles. It finds approximately 50% of usability issues that user testing would uncover, making it a cost-effective complement to (not replacement for) user testing.

Nielsen's 10 Usability Heuristics

These 10 principles, developed by Jakob Nielsen, serve as the standard checklist for interface evaluation.

1. Visibility of System Status

The system should keep users informed about what is going on through appropriate feedback within a reasonable amount of time.

Check For:
  - Loading indicators for operations longer than 1 second
  - Progress bars for multi-step processes
  - Confirmation messages after user actions
  - Clear indication of current location in navigation
  - Real-time validation on form inputs

Common Violations:
  - No feedback after clicking a submit button
  - No indication of upload progress
  - User cannot tell which step they are on in a wizard
  - No visual change when an item is added to a cart

2. Match Between System and Real World

The system should speak the user's language with familiar words, phrases, and concepts rather than system-oriented terms.

Check For:
  - Labels use language the target audience understands
  - Icons match real-world conventions
  - Information appears in a natural and logical order
  - Date formats match locale expectations
  - Error messages use plain language, not error codes

Common Violations:
  - Technical jargon in user-facing labels ("null reference", "404")
  - Unfamiliar abbreviations without explanation
  - Non-standard icons for common actions
  - System-generated IDs shown to users

3. User Control and Freedom

Users often perform actions by mistake and need a clearly marked exit to leave the unwanted state without going through an extended process.

Check For:
  - Undo and redo support for destructive actions
  - Clear cancel buttons on forms and dialogs
  - Back navigation that preserves user input
  - Ability to dismiss modals and overlays easily
  - Confirmation prompts before irreversible actions

Common Violations:
  - No undo after deleting content
  - Modal dialogs with no close button
  - Multi-step form that loses data on back navigation
  - No way to cancel a long-running operation

4. Consistency and Standards

Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

Check For:
  - Same action uses the same label everywhere
  - Visual design is consistent across pages
  - Interaction patterns match platform conventions
  - Terminology is consistent throughout the product
  - Similar elements behave the same way

Common Violations:
  - "Save" on one page, "Submit" on another for the same action
  - Different button styles for the same action type
  - Inconsistent navigation placement across pages
  - Mixed terminology ("account" vs "profile" for the same thing)

5. Error Prevention

A design that prevents problems from occurring is better than good error messages.

Check For:
  - Constraints that prevent invalid input (date pickers, dropdowns)
  - Confirmation dialogs before destructive actions
  - Inline validation before form submission
  - Sensible defaults that reduce user decisions
  - Disabled states for unavailable actions with explanation

Common Violations:
  - Free text input where a dropdown would prevent errors
  - No confirmation before permanent deletion
  - Validation only after form submission
  - Allowing users to submit incomplete forms

6. Recognition Rather Than Recall

Minimize the user's memory load by making elements, actions, and options visible. Users should not have to remember information from one part of the interface to another.

Check For:
  - Recently used items are easily accessible
  - Labels and instructions are visible, not hidden in tooltips
  - Search suggestions and autocomplete
  - Context preserved when navigating between pages
  - Related information displayed together

Common Violations:
  - User must remember a code from a previous page
  - No recent items or search history
  - Hidden labels that only appear on hover
  - Multi-step process without summary of previous selections

7. Flexibility and Efficiency of Use

The interface should cater to both novice and expert users. Experienced users should be able to use shortcuts to speed up frequent actions.

Check For:
  - Keyboard shortcuts for frequent actions
  - Customizable interface elements
  - Shortcuts and advanced features that do not confuse beginners
  - Bulk actions for repetitive tasks
  - Default values that match common use cases

Common Violations:
  - No keyboard shortcuts for power users
  - No bulk selection or batch operations
  - Forced multi-step wizards with no quick path for experts
  - Settings that require multiple clicks to reach frequently

8. Aesthetic and Minimalist Design

Every extra unit of information competes with relevant information and diminishes its relative visibility.

Check For:
  - Content hierarchy prioritizes important information
  - Minimal use of decorative elements that do not aid comprehension
  - White space used effectively to reduce visual noise
  - Only essential information shown by default, details on demand
  - Visual design supports rather than competes with content

Common Violations:
  - Cluttered dashboards showing everything at once
  - Decorative animations that slow down task completion
  - Dense text without visual hierarchy
  - Too many competing calls to action on one page

9. Help Users Recognize, Diagnose, and Recover from Errors

Error messages should be expressed in plain language, precisely indicate the problem, and constructively suggest a solution.

Check For:
  - Error messages explain what went wrong in plain language
  - Error messages suggest how to fix the problem
  - Errors are displayed near the relevant input field
  - Error state is visually distinct (color, icon, position)
  - Users can fix errors without re-entering all data

Common Violations:
  - Generic "Something went wrong" with no details
  - Error codes instead of human-readable messages
  - Error message at top of page, far from the problematic field
  - Form clears all fields after a validation error

10. Help and Documentation

The interface should provide help and documentation that is easy to search, focused on the user's task, and concise.

Check For:
  - Contextual help available where users need it
  - Searchable help documentation
  - Onboarding or guided tours for new users
  - Tooltips for complex or unfamiliar features
  - FAQ or knowledge base for common questions

Common Violations:
  - No in-app help or documentation
  - Help content that does not match the current product version
  - Documentation that requires leaving the application
  - No onboarding for new users

Running a Heuristic Evaluation

Process: 1. Select 3-5 evaluators (mix of UX expertise and domain knowledge)
  2. Brief evaluators on the product, target users, and key tasks
  3. Each evaluator independently inspects the interface
  4. Each evaluator documents issues with heuristic violated and severity
  5. Compile all findings, removing duplicates
  6. Prioritize by severity and frequency across evaluators
  7. Present findings to the team with recommendations

Time Estimate:
  - Individual evaluation: 1-2 hours per evaluator
  - Compilation and debrief: 2-3 hours
  - Total: 1-2 days for a complete evaluation

Tips:
  - Evaluators should work independently to avoid groupthink
  - Walk through key user tasks, not just browse the interface
  - Document the specific location and context of each issue
  - Take screenshots to illustrate issues
  - Rate severity independently before comparing with other evaluators

Severity Rating for Heuristic Issues

0 - Not a usability problem: Cosmetic only, no impact on task completion

1 - Cosmetic problem: Fix if time permits, does not affect usability

2 - Minor usability problem: Low priority, causes slight delay or confusion

3 - Major usability problem: High priority, causes significant difficulty for users

4 - Usability catastrophe: Must fix before release, prevents task completion

Combining with User Testing

Heuristic evaluation and user testing are complementary methods that find different types of issues.

Recommended Order: 1. Run heuristic evaluation first (cheaper, faster)
  2. Fix critical and major issues found
  3. Then run user testing to find issues experts missed
  4. Focus user testing time on areas heuristic evaluation flagged as uncertain

What Heuristic Evaluation Finds Better:
  - Consistency violations across the interface
  - Missing standard features (undo, error messages)
  - Accessibility violations
  - Platform convention deviations

What User Testing Finds Better:
  - Mental model mismatches
  - Terminology confusion
  - Task flow problems
  - Unexpected user strategies
  - Emotional responses to design decisions

Metrics and Severity

Task Success Rate

The most fundamental usability metric. Measures whether users can complete the task at all.

Measurement:
  - Completed: User achieved goal without any help
  - Partial: User achieved goal with hints or after errors
  - Failed: User could not complete task or gave up

Calculation: Task Success Rate = (Completed Tasks / Total Attempts) x 100

Targets:
  - Core tasks (sign up, checkout): 90% or higher
  - Secondary tasks (settings, export): 80% or higher
  - Advanced tasks (integrations, admin): 70% or higher

Reporting:
  - Report both unassisted and assisted success rates separately
  - Track partial completions to identify where users need help
  - Compare across participant segments (new vs returning users)

Time on Task

Measures how long users take to complete a task. Longer times indicate confusion or inefficiency.

Measurement:
  - Start timer when task instructions are given
  - Stop when user completes the task or gives up
  - Record both completed and abandoned times separately

Analysis:
  - Calculate median time (more robust than mean for small samples)
  - Compare to baseline from previous tests or competitor benchmarks
  - Identify outliers: very fast may indicate skipping, very slow indicates struggle
  - Plot distribution to find bimodal patterns (easy for some, hard for others)

Targets (guidelines, adjust per product):
  - Simple task (log in, toggle setting): under 30 seconds
  - Medium task (create item, fill form): 1-2 minutes
  - Complex task (configure integration, set up workflow): 3-5 minutes

Red Flags:
  - Time increases across test rounds (regression)
  - Wide variance between participants (inconsistent experience)
  - Completed tasks taking 3x or more the expected time

Error Rate

Counts mistakes, wrong paths, and backtracking during task completion.

What Counts as an Error:
  - Clicked wrong element
  - Navigated to wrong page
  - Had to backtrack or undo
  - Entered invalid data
  - Gave up and tried a different approach
  - Misinterpreted a label or instruction

Calculation: Errors per Task = Total Errors / Number of Participants

Targets:
  - Simple tasks: 0-1 errors per user
  - Medium tasks: fewer than 2 errors per user
  - Complex tasks: fewer than 3 errors per user

Analysis:
  - Categorize errors by type (navigation, input, comprehension)
  - Map errors to specific interface elements
  - Identify the most common first error (often reveals the root cause)
  - Track error recovery rate (how many users recover without help)

Post-Task Satisfaction

Collected immediately after each task using a standardized scale.

Single Ease Question (SEQ)

The simplest post-task metric. One question, one number.

Question: 'Overall, how easy or difficult was this task?'

Scale: 1 = Very Difficult
  2 = Difficult
  3 = Somewhat Difficult
  4 = Neither Easy nor Difficult
  5 = Somewhat Easy
  6 = Easy
  7 = Very Easy

Interpretation:
  - Average SEQ score across studies: 5.5
  - Below 5.5: task is harder than average, investigate
  - Above 5.5: task is easier than average
  - Below 4.0: significant usability problem

5-Point Satisfaction Scale

Alternative to SEQ, commonly used for simpler reporting.

Question: 'How satisfied are you with completing this task?'

Scale: 1 = Very Dissatisfied
  2 = Dissatisfied
  3 = Neutral
  4 = Satisfied
  5 = Very Satisfied

Target: 4.0 or higher average

System Usability Scale (SUS)

Standardized 10-question post-test survey. Administered after all tasks are complete, not after individual tasks. Produces a score from 0 to 100.

Questions (alternate positive and negative):
  1. I think I would like to use this product frequently
  2. I found the product unnecessarily complex
  3. I thought the product was easy to use
  4. I think I would need support to use this product
  5. I found the various functions well integrated
  6. I thought there was too much inconsistency
  7. I imagine most people would learn this quickly
  8. I found the product cumbersome to use
  9. I felt very confident using the product
  10. I needed to learn a lot before getting going

Scoring Steps:
  1. For odd-numbered questions (positive): subtract 1 from the score
  2. For even-numbered questions (negative): subtract the score from 5
  3. Sum all 10 adjusted scores
  4. Multiply by 2.5
  5. Result is a score from 0 to 100

Score Interpretation:
  - 80.3 or above: A grade (top 10% of products)
  - 68-80: B-C grade (average to above average)
  - 51-67: D grade (below average, needs improvement)
  - Below 51: F grade (significant usability problems)
  - Industry average: 68

Percentile Benchmarks:
  - 90th percentile: 80.3
  - 75th percentile: 75.0
  - 50th percentile: 68.0
  - 25th percentile: 51.0

Minimum Sample: 8-12 participants for reliable SUS scores

Issue Severity Rating

Severity Formula

Severity = Impact x Frequency

Impact Scale (1-3)

1 - Low Impact:
  - Minor inconvenience or cosmetic issue
  - User can easily recover or work around it
  - Does not affect task completion
  - Example: slightly confusing label that users figure out quickly

2 - Medium Impact:
  - Causes noticeable delay or confusion
  - User eventually completes the task but with frustration
  - Moderate impact on efficiency
  - Example: important feature buried in unexpected location

3 - High Impact:
  - Blocks task completion for some or all users
  - User cannot proceed without external help
  - Critical to core product functionality
  - Example: submit button does not work, no error feedback on failed action

Frequency Scale (1-3)

1 - Rare:
  - Only 1-2 participants encountered the issue
  - Edge case or specific conditions required
  - Affects a small subset of users

2 - Occasional:
  - 3-5 participants encountered the issue
  - Somewhat common, specific user types or paths
  - Affects a moderate portion of users

3 - Frequent:
  - Most or all participants encountered the issue
  - Consistent across user types and approaches
  - Affects nearly all users

Combined Severity Matrix

Critical (score 9):
  - Impact 3, Frequency 3
  - Blocks most users from completing a core task
  - Action: Fix immediately, do not release without fix

High (score 6-8):
  - Impact 3 Frequency 2, or Impact 2 Frequency 3, or Impact 3 Frequency 1 (if core task)
  - Significant delay for many users or complete blocker for some
  - Action: Fix before release

Medium (score 4-5):
  - Impact 2 Frequency 2, or Impact 3 Frequency 1
  - Moderate frustration or rare blocker
  - Action: Fix in next release cycle

Low (score 1-3):
  - Impact 1 Frequency 1-3, or Impact 2 Frequency 1
  - Minor inconvenience, cosmetic issue
  - Action: Add to backlog, fix when convenient

Benchmarking and Comparison

Tracking Over Time:
  - Test the same tasks with each product release
  - Compare metrics to your own baseline, not just industry averages
  - Track trends: improving, stable, or regressing
  - Flag any metric that drops more than 10% between rounds

Competitor Benchmarking:
  - Run the same tasks on competitor products
  - Compare task success rate, time on task, and SUS scores
  - Identify areas where competitors outperform your product
  - Use findings to prioritize improvements

Statistical Considerations:
  - 5 participants: qualitative patterns, not statistically significant
  - 8-12 participants: SUS scores become reliable
  - 20+ participants: task success rates become statistically meaningful
  - Use confidence intervals when reporting to stakeholders

Remote Testing and Tools

Remote vs In-Person Testing

Remote Moderated Testing

Setup Checklist

Before the Session:
  - Test screen sharing on the video platform (Zoom, Google Meet, Teams)
  - Verify recording permissions and storage
  - Prepare backup communication method (phone number, alternative platform)
  - Send participant clear instructions with video link, tech requirements
  - Do a dry run with a colleague to test the full flow
  - Prepare the test environment (stable build, test accounts, sample data)

Platform Selection:
  - Zoom: widely available, good recording, breakout rooms for observers
  - Google Meet: no install required, good for quick sessions
  - Microsoft Teams: enterprise environments, built-in recording
  - Lookback: purpose-built for research, timestamped notes, highlight reels
  - dscout: mobile-first, diary studies, longitudinal research

Technical Requirements for Participants:
  - Stable internet connection
  - Screen sharing capability
  - Camera enabled (recommended, not required)
  - Microphone and speakers or headphones
  - Browser or device matching the test environment

Best Practices

Session Management:
  - Start 5 minutes early to resolve technical issues
  - Have the participant share their screen before starting tasks
  - Ask them to close other tabs and notifications
  - Record both the participant's screen and the video call
  - Have a note-taker on the call (camera and mic off)

Rapport Building (harder remotely):
  - Spend 2-3 minutes on casual conversation before tasks
  - Use the participant's name throughout the session
  - Acknowledge technical difficulties without frustration
  - Keep your camera on so the participant can see you are engaged

Handling Technical Issues:
  - If screen sharing fails: switch to the participant describing what they see
  - If audio cuts out: use chat as a backup channel
  - If the connection drops: have a phone number to call back
  - If the prototype crashes: note the failure, skip to next task

In-Person Testing

Lab Setup

Room Requirements:
  - Quiet room with minimal distractions
  - Desk with computer or device under test
  - Chair positioned so moderator sits beside or behind the participant
  - Screen recording software running (OBS, Morae, Silverback)
  - External camera for facial expressions (optional)
  - One-way mirror or video feed for observers (optional)

Supplies:
  - Printed consent forms
  - Printed task cards (one task per card)
  - Note-taking sheets for moderator
  - Water and snacks for participants
  - Gift cards or cash for incentives
  - Backup device in case of hardware failure

Observer Guidelines:
  - Observers watch via video feed or one-way mirror
  - No entering the testing room during sessions
  - Take notes independently, compare after all sessions
  - Save questions for the debrief, not during the session

Testing Tools by Category

Usability Testing Platforms

Moderated Testing:
  - Lookback: live moderated sessions with timestamped notes and highlights
  - UserTesting.com: recruit and test, video recordings, moderated and unmoderated
  - dscout: mobile-first research, diary studies, video responses

Unmoderated Testing:
  - Maze: task-based testing with Figma integration, click heatmaps
  - Lyssna (formerly UsabilityHub): first-click, preference, five-second tests
  - UserTesting.com: unmoderated option with think-aloud recording
  - Loop11: task-based testing with analytics

Prototype Testing:
  - Figma prototypes: interactive prototypes that work directly in Maze and Lyssna
  - InVision: clickable prototypes with hotspot analytics
  - Axure: high-fidelity prototypes with conditional logic

Information Architecture Tools

Card Sorting:
  - Optimal Workshop (OptimalSort): open, closed, and hybrid card sorts
  - Maze: integrated card sorting within broader test plans
  - UserZoom: enterprise card sorting with analytics

Tree Testing:
  - Optimal Workshop (Treejack): text-based navigation testing
  - UserZoom: tree testing with task-based analysis

First-Click Testing:
  - Lyssna: click heatmaps on static designs
  - Maze: first-click analysis within task flows
  - Optimal Workshop (Chalkmark): standalone first-click testing

Analysis and Reporting

Qualitative Analysis:
  - Dovetail: tag and organize qualitative data, generate themes
  - EnjoyHQ: research repository with tagging and search
  - Condens: collaborative analysis with video highlights

Session Recording:
  - Zoom: screen and audio recording, cloud storage
  - Loom: quick screen recordings for sharing highlights
  - OBS Studio: advanced recording with multiple sources (free)

Collaboration:
  - Miro: affinity mapping, journey mapping, collaborative analysis
  - FigJam: collaborative whiteboarding integrated with Figma
  - Notion: research templates, shared databases, team wikis

Metrics Tracking:
  - Spreadsheets (Google Sheets, Excel): flexible, custom formulas
  - Airtable: structured data with views and formulas
  - Dovetail: quantitative analysis alongside qualitative data

Accessibility Testing Tools

Automated Scanners:
  - axe DevTools: browser extension, WCAG checks (free)
  - WAVE: visual accessibility report in browser (free)
  - Lighthouse: built into Chrome DevTools (free)
  - pa11y: command-line tool for CI integration (free)

Screen Readers:
  - VoiceOver: built into macOS and iOS (free)
  - NVDA: Windows screen reader (free)
  - JAWS: enterprise Windows screen reader (paid)

Contrast Checkers:
  - WebAIM Contrast Checker: web-based ratio calculator
  - Stark: Figma plugin for contrast and color blindness simulation
  - Color Contrast Analyzer: desktop application (free)

Test Frequency

Product Lifecycle Testing Schedule:
  Pre-Launch:
    - Heuristic evaluation of wireframes
    - Prototype testing with 5 users
    - Accessibility audit before development
    - Final usability test on staging environment

  Launch:
    - Monitor support tickets and analytics for UX issues
    - Quick guerrilla test on top 3 user flows

  Post-Launch (ongoing):
    - Test every major feature release with 5 users
    - Quarterly full usability audit (5-8 users, all core tasks)
    - Annual accessibility audit with participants with disabilities

Continuous Testing Cadence:
  Week 1: Test with 5 users on identified problem areas
  Week 2: Fix top issues found
  Week 3: Test with 5 new users to verify fixes
  Week 4: Report results and plan next test cycle
  Repeat until task success rate meets targets (80%+ for core tasks)

Quick Start Checklists

Planning Phase

• [ ] Define 3-5 measurable test objectives
• [ ] Write research questions
• [ ] Write task scenarios with realistic context
• [ ] Create screening questionnaire
• [ ] Recruit 5-8 participants per persona
• [ ] Prepare consent form
• [ ] Write test script with timing
• [ ] Set up recording tools
• [ ] Run 1-2 pilot sessions

Testing Phase

• [ ] Welcome participant and collect consent
• [ ] Ask background questions
• [ ] Explain think-aloud protocol with practice task
• [ ] Conduct tasks (do not help)
• [ ] Ask post-task questions after each task
• [ ] Administer SUS survey after all tasks
• [ ] Debrief with open questions
• [ ] Thank participant and provide incentive

Analysis Phase

• [ ] Review all session recordings and notes
• [ ] Calculate task success rates per task
• [ ] Calculate time on task (median)
• [ ] Calculate error rates per task
• [ ] Calculate SUS score
• [ ] List all issues with severity ratings
• [ ] Identify patterns and key insights
• [ ] Create prioritized recommendations
• [ ] Write report with executive summary
• [ ] Share findings with stakeholders
• [ ] Create fix tickets for critical and high issues
• [ ] Schedule follow-up test to verify fixes

Reporting

Report Structure

A usability test report serves two audiences: the immediate team that needs to fix issues, and stakeholders who need to understand the business impact.

Report Sections: 1. Executive Summary (1 page)
  2. Test Overview (objectives, methodology, participants)
  3. Task Results (per-task metrics and findings)
  4. Issue List (all issues with severity ratings)
  5. Key Insights (patterns across tasks)
  6. Recommendations (prioritized action items)
  7. Appendix (raw data, session recordings, SUS scores)

Executive Summary

Write this last. It should stand alone for stakeholders who read nothing else.

Template:
  Purpose: [Why the test was conducted]
  Method: [Number of participants, testing method, dates]
  Top Finding: [Single most important discovery]
  Overall Score: [SUS score with interpretation]
  Critical Issues: [Count of critical and high severity issues]
  Recommendation: [Single most important next step]

Example:
  Purpose: Evaluate the onboarding flow for new users before launch.
  Method: 8 participants, moderated remote testing, January 15-17.
  Top Finding: 6 of 8 users could not find the team invite feature.
  Overall Score: SUS 62 (below average, needs improvement).
  Critical Issues: 2 critical, 3 high severity.
  Recommendation: Redesign the invite flow before launch.

Test Overview Section

Test Details:
  Objectives:
    - 'Can new users complete onboarding in under 5 minutes?'
    - 'Can users find and use the team invite feature?'
    - 'Do users understand the difference between workspaces and projects?'

  Methodology: Moderated remote usability testing with think-aloud protocol

  Participants:
    Total: 8
    Profile: New users, age 25-45, tech-savvy professionals
    Recruitment: UserTesting.com panel, screened for no prior product use
    Incentive: $75 gift card per participant

  Tasks Tested: 1. Create a new project
    2. Invite a team member
    3. Assign a task
    4. Export project data
    5. Change account settings

  Test Environment: Production application, Chrome browser, participant's own device

Task Results Template

Report each task with quantitative metrics and qualitative observations.

Task: "Create a new project"
  Success Rate: 87.5% (7/8 completed without help)
  Median Time: 1m 24s
  Errors per User: 1.2
  Post-Task Satisfaction: 4.3/5
  SEQ Score: 5.8/7

  Key Observations:
    - 5 users found the "New Project" button immediately
    - 2 users looked in the sidebar first before finding the button
    - 1 user confused "project" with "workspace" and created the wrong item

  Issues Found:
    - "New Project" button not visible without scrolling on small screens (medium)
    - Project vs workspace distinction unclear (high)

Task: "Invite a team member"
  Success Rate: 62.5% (5/8 completed without help)
  Median Time: 2m 45s
  Errors per User: 2.8
  Post-Task Satisfaction: 3.1/5
  SEQ Score: 3.9/7

  Key Observations:
    - 6 users expected the invite feature on the project page
    - 3 users looked in account settings before finding team settings
    - 2 users gave up and could not find the feature at all

  Issues Found:
    - Invite button hidden in team settings, not on project page (critical)
    - "Team" and "Members" labels used inconsistently (medium)

Issue List Format

Every issue gets a structured entry for tracking and prioritization.

Issue Entry Template:
  ID: [UST-001]
  Title: [Short description]
  Severity: [Critical | High | Medium | Low]
  Impact: [1-3]
  Frequency: [1-3]
  Affected Users: [X/Y participants]
  Task: [Which task(s) affected]
  Description: [What happened and why it is a problem]
  Evidence: [Quote, screenshot, or timestamp reference]
  Recommendation: [Specific fix suggestion]
  Owner: [Team or person responsible]

Example:
  ID: UST-001
  Title: Users cannot find the invite feature
  Severity: Critical
  Impact: 3
  Frequency: 3
  Affected Users: 6/8
  Task: Task 2 (Invite a team member)
  Description: The invite button is located in Team Settings, but users
    expect it on the project page or in the top navigation. Most users
    never found team settings without help.
  Evidence: "I keep looking for an invite button on the project page but
    I can't find one anywhere." (P3, 02:30)
  Recommendation: Add an invite button to the project page header and
    include invite in the top navigation dropdown.
  Owner: Product team

Key Insights

Insights are patterns that emerge across multiple tasks and participants. They go beyond individual issues to reveal systemic problems or opportunities.

Insight Format:
  Finding: [What the pattern is]
  Evidence: [Which tasks and how many users]
  Impact: [Why it matters for the business]
  Recommendation: [What to do about it]

Example Insights:
  - Finding: Users expect collaboration features to be accessible from the project page.
    Evidence: 6/8 users looked for invite on the project page (Task 2).
      4/8 users expected task assignment on the project page (Task 3).
    Impact:
      Core collaboration workflows are harder to discover than they should be,
      likely reducing team adoption.
    Recommendation: Surface all collaboration actions on the project page.

  - Finding: Users do not understand the workspace vs project hierarchy.
    Evidence:
      3/8 users created a workspace when asked to create a project (Task 1).
      2/8 users could not explain the difference in the debrief.
    Impact: Confused mental model leads to errors in all project-related tasks.
    Recommendation: Add contextual explanation during onboarding and consider
      whether both concepts are necessary.

  - Finding: Onboarding is effective for basic tasks but fails for team features.
    Evidence: Task 1 success rate 87.5%, but Task 2 only 62.5%.
    Impact: Users can get started but cannot bring their team, reducing product stickiness.
    Recommendation: Add team setup as part of the onboarding flow.

Presenting to Stakeholders

Presentation Tips:
  - Lead with the SUS score and overall success rate (executives want numbers)
  - Show 2-3 video clips of users struggling (emotional impact drives action)
  - Focus on business impact: "6/8 users cannot invite teammates" not "invite button is in wrong place"
  - Present recommendations as a prioritized list with effort estimates
  - End with a clear next step and timeline

Slide Structure (15 minutes):
  1. Test overview and goals (1 slide)
  2. Overall metrics: SUS score, average success rate (1 slide)
  3. Top 3 findings with video clips (3 slides)
  4. Issue severity breakdown chart (1 slide)
  5. Prioritized recommendations (1 slide)
  6. Next steps and timeline (1 slide)

Common Objections:
  - "Only 8 users, that's not statistically significant"
    Response: 5 users find 85% of usability issues (Nielsen). We found consistent
    patterns across participants.
  - "Users will figure it out eventually"
    Response: Show time-on-task data and abandonment rates. Every extra second
    of confusion costs conversion.
  - "We already know about that issue"
    Response: Now you have evidence with severity scores to prioritize the fix.

Action Item Prioritization

Priority Matrix:
  Fix Now (before release):
    - Critical severity issues (score 9)
    - High severity issues affecting core tasks (score 6-8)

  Fix Next Sprint:
    - High severity issues affecting secondary tasks
    - Medium severity issues on core tasks (score 4-5)

  Backlog:
    - Low severity issues (score 1-3)
    - Medium severity issues on non-critical tasks

  Track and Monitor:
    - Issues that appeared in only 1-2 sessions
    - Issues that may resolve with a broader redesign already planned

Action Item Format:
  - Issue ID and title
  - Severity score
  - Specific fix recommendation
  - Assigned team or owner
  - Target sprint or release
  - Verification plan (how to confirm the fix works)

Test Planning

Define Test Objectives

Objectives must be behavioral and measurable, not subjective.

Good Objectives:
  - "Can users complete onboarding in under 5 minutes?"
  - "Can users find and use the export feature without hints?"
  - "Do users understand what each pricing tier includes?"
  - "Can users recover from a payment error without contacting support?"

Bad Objectives:
  - "Test the UI" (too vague, no success criteria)
  - "See if users like it" (subjective, not behavioral)
  - "Validate the new design" (confirmation bias framing)
  - "Check if it works" (not specific enough to measure)

Research Questions

Research questions identify specific unknowns the test should resolve.

Navigation:
  - Can users find the settings page from the dashboard?
  - Where do users expect account management features to live?

Task Completion:
  - Where do users get stuck during sign-up?
  - What errors do users encounter when creating a project?
  - Can users complete checkout without abandoning the cart?

Comprehension:
  - Do users understand the difference between workspaces and projects?
  - Do users understand what each plan tier includes?

Recovery:
  - Can users recover from a failed payment?
  - Do users know how to undo a destructive action?

Identify Core Tasks

Choose 3-5 tasks that represent key user journeys. Order them from simple to complex to build participant confidence.

Example Tasks (Project Management Tool): 1. Sign up and create your account
  2. Create your first project
  3. Invite a team member to collaborate
  4. Assign a task to someone on your team
  5. Export project data as a CSV file

Task Selection Criteria:
  - Covers the critical path (what most users must do)
  - Includes at least one discoverability task (find a hidden feature)
  - Includes at least one error recovery task
  - Avoids tasks that require prior domain knowledge
  - Each task is independent (failure on one does not block the next)

Task Scenario Template

Scenarios provide realistic context without revealing the solution path.

Template:
  Context: [Why the user needs to do this]
  Goal: [What they need to accomplish]
  Success Criteria: [How to know they completed it]
  Time Limit: [Maximum before marking abandoned]

Example:
  Context: You are preparing for a client meeting tomorrow and need to review past conversations.
  Goal: Find all conversations with "Acme Corp" from the last 30 days.
  Success Criteria: User successfully uses search or filter to find the conversations.
  Time Limit: 3 minutes

Good vs Bad Scenarios

Good task scenario:

"Your team is launching a new project next week. Create a project
called 'Q2 Launch' and invite john@example.com to collaborate."

Why it works: realistic context, clear goal, natural language, tests discoverability without giving step-by-step instructions.

Bad task scenario:

"Click the 'New Project' button, then enter 'Q2 Launch', then
click Settings, then click Invite, then enter the email."

Why it fails: step-by-step instructions remove discoverability testing, no context for user motivation, user just follows orders.

Recruit Participants

Sample Size:
  - 5-8 users per persona for moderated testing
  - 20-50 users for unmoderated testing
  - After 5 users, diminishing returns on new findings (Nielsen)
  - Test in waves: 5 users, fix top issues, test 5 more

Recruitment Criteria:
  - Match target persona demographics and technical proficiency
  - Have not used the product before (for onboarding tests)
  - Active users with 1-3 months experience (for feature tests)
  - Mix of technical skill levels within the persona
  - Exclude UX professionals (they behave differently)

Incentives:
  - B2C: $50-100 per hour
  - B2B professionals: $100-200 per hour
  - Gift cards work well for remote sessions
  - Pay even if participant cannot complete the test

Screening Questionnaire

Screen participants before scheduling to ensure they match the target persona.

Screening Questions:
  - "How often do you use [product category] tools?" (daily/weekly/monthly/never)
  - "What is your role?" (match target persona)
  - "Have you used [product name] before?" (exclude for onboarding tests)
  - "What device do you primarily use for work?" (match test environment)
  - "Are you comfortable sharing your screen during a video call?" (remote tests)

Disqualification Criteria:
  - Works in UX, design, or usability (unless testing for expert users)
  - Has participated in a test for this product in the last 6 months
  - Does not match the target persona demographics
  - Cannot commit to the full session duration

Pilot Testing

Run 1-2 pilot sessions before the real test to catch problems with the test itself.

Pilot Checklist:
  - Task scenarios are clear and unambiguous
  - Time limits are realistic (not too short or too long)
  - Recording setup works (screen, audio, camera)
  - Think-aloud instructions are understood
  - Post-task questions flow naturally
  - Session fits within the scheduled time slot
  - Prototype or product is stable enough to test

Common Pilot Findings:
  - Tasks are too easy or too hard
  - Scenario wording reveals the answer
  - Time limits are unrealistic
  - Technical setup has friction (screen sharing, recording)
  - Session runs over the scheduled time

Test Script Structure

Session Structure (60 minutes typical):
  - Welcome and consent (5 minutes)
  - Background questions (5 minutes)
  - Think-aloud instructions (3 minutes)
  - Practice task (2 minutes)
  - Test tasks with post-task questions (35 minutes)
  - SUS questionnaire (5 minutes)
  - Debrief and open questions (5 minutes)

Consent Items:
  - Permission to record screen and audio
  - Permission to share anonymized clips with team
  - Explanation that the product is being tested, not the participant
  - Right to stop at any time without penalty
  - Confirmation that incentive is paid regardless of completion

Testing Methods

Method Comparison

Unmoderated Testing

Participants complete tasks on their own schedule without a moderator present. Software records their interactions, clicks, and optionally their screen and voice.

When to Use

• Need quantitative data from a large sample
• Testing straightforward tasks that do not require probing
• Validating findings from moderated testing at scale
• Budget or timeline does not allow moderated sessions

When NOT to Use

• Testing complex flows that need follow-up questions
• Exploring new product concepts where context matters
• Tasks require domain knowledge that needs explanation
• You need to understand the "why" behind user behavior

Setup

Platform Options:
  - Maze (task-based, integrates with Figma)
  - UserTesting.com (recruit and test, video recording)
  - Lyssna (formerly UsabilityHub, multiple test types)
  - Lookback (asynchronous recording)

Test Configuration:
  - Write clear, unambiguous task instructions
  - Set realistic time limits per task
  - Include a practice task so participants understand the format
  - Add screening questions to filter participants
  - Enable screen recording for qualitative review

Sample Size:
  - Minimum 20 participants for statistical patterns
  - 50+ for quantitative confidence
  - Up to 150 for A/B comparison between designs

Analyzing Unmoderated Results

Quantitative Metrics:
  - Task completion rate per task
  - Average time on task
  - Misclick rate (clicks on wrong elements)
  - Drop-off points (where users abandon tasks)

Qualitative Review:
  - Watch recordings of failed tasks to understand why
  - Look for common wrong paths
  - Note where users hesitate (long pauses before clicking)
  - Compare fast completers vs slow completers for pattern differences

Guerrilla Testing

Quick, informal testing with random participants in public places. Sessions last 5-15 minutes and cover 2-3 tasks maximum.

When to Use

• Very early concept validation
• Testing a single interaction or screen
• Need feedback today, not next week
• Budget is minimal

When NOT to Use

• Product requires specific domain expertise
• Tasks take more than 5 minutes
• Need consistent, comparable results across sessions
• Testing with a specific persona matters

How to Run

Setup:
  - Prepare a laptop or tablet with the prototype
  - Write 2-3 simple tasks on index cards
  - Choose a high-traffic location (coffee shop, coworking space, campus)
  - Offer a small incentive (coffee, gift card)

Session Flow:
  - Approach: "We're testing a new product. Can I get 5 minutes of your time?"
  - Brief intro: "I'll show you something and ask you to try a couple tasks."
  - Give task card, observe, take notes
  - Quick debrief: 'What was that like? Anything confusing?'
  - Thank and provide incentive

Tips:
  - Keep sessions under 10 minutes
  - Do not explain the product before the task
  - Test the same tasks with each participant for comparison
  - Bring a partner to take notes while you moderate
  - 5-10 participants is enough for guerrilla testing

First-Click Testing

Measures where users click first when given a task. If the first click is correct, users complete the task successfully 87% of the time (based on research by Bob Bailey).

When to Use

• Evaluating navigation and menu structure
• Testing button placement and labeling
• Comparing two layout options
• Validating information architecture changes

How to Run

Setup:
  - Create a screenshot or static mockup of the page
  - Write task-based questions: 'Where would you click to invite a team member?'
  - Use a tool that records click coordinates (Lyssna, Maze, Optimal Workshop)

Analysis:
  - Generate a click heatmap showing where users clicked
  - Calculate first-click success rate per task
  - Identify competing click targets (multiple areas getting clicks)
  - Compare results across design variations

Success Criteria:
  - 80%+ of users click the correct area on first try: navigation is clear
  - 60-79%: navigation works but could be improved
  - Below 60%: significant navigation or labeling problem

Cognitive Walkthrough

Expert evaluators step through a task flow, asking four questions at each step to identify where users might struggle.

The Four Questions

At each step in the task flow, the evaluator asks:

1. "Will the user try to achieve the right effect?"
Does the user understand what action is needed at this step?

2. "Will the user notice the correct action is available?"
Is the right button, link, or control visible and discoverable?

3. "Will the user associate the correct action with the desired effect?"
Does the label or appearance of the control match what the user expects?

4. "If the correct action is performed, will the user see progress?"
Does the system provide feedback that the action was successful?

How to Run

Preparation:
  - Define the target user persona (experience level, goals)
  - List the tasks to evaluate
  - Document the "happy path" steps for each task
  - Gather 2-3 evaluators (can include designers and developers)

Process:
  - Walk through each step of each task
  - At each step, answer all four questions
  - Document any "no" answers as potential usability issues
  - Rate severity of each issue found
  - Compile findings into a prioritized list

Output:
  - List of steps where users are likely to fail or hesitate
  - Specific reasons for each predicted failure
  - Severity ratings for prioritization
  - Recommended fixes for each issue

Card Sorting

Participants organize content items into groups that make sense to them. Reveals how users expect information to be structured.

Open Card Sort:
  - Users create their own categories and label them
  - Use for: discovering user mental models, building new IA

Closed Card Sort:
  - Users sort cards into predefined categories
  - Use for: validating an existing or proposed IA

Hybrid Card Sort:
  - Predefined categories plus option to create new ones
  - Use for: refining an IA with known structure

Tools: Optimal Workshop, Maze, UserZoom
Sample Size: 15-30 participants per sort

Tree Testing

Participants navigate a text-based tree structure to find where content lives. Tests the information architecture without visual design influence.

Setup:
  - Create a text hierarchy of your site structure
  - Write task-based questions: 'Where would you find the refund policy?'
  - Participants click through the tree to find the answer

Metrics:
  - Task success rate: did they find the right location?
  - Directness: did they go straight there or backtrack?
  - Time to complete: how long did navigation take?

Tools: Optimal Workshop, Treejack, UserZoom
Sample Size: 30-50 participants

Method Selection by Lifecycle Stage

Early Concept Stage:
  - Guerrilla testing (quick validation of core concept)
  - Card sorting (information architecture discovery)
  - Cognitive walkthrough (expert review of proposed flows)

Design and Prototyping Stage:
  - Moderated testing (deep insights on task flows)
  - First-click testing (navigation validation)
  - Tree testing (IA validation)
  - Heuristic evaluation (expert review against principles)

Pre-Launch Stage:
  - Unmoderated testing (scale validation, quantitative data)
  - Accessibility testing (inclusive design verification)
  - First-click testing (final navigation check)

Post-Launch Stage:
  - Unmoderated testing (ongoing monitoring)
  - Moderated testing (investigating specific problem areas)
  - Analytics review (complement qualitative findings with data)