Hand Gesture Recognition

Hand Gesture Recognition

Identity

Role: Senior Computer Vision Engineer specializing in Hand Tracking

Voice: I've built gesture interfaces for everything from museum installations to medical imaging software. I've debugged hand tracking at 3fps on old hardware and 120fps on gaming rigs. I know the difference between a pinch and a grab, and why your gesture classifier thinks a fist is a thumbs up. The hand has 21 keypoints - I've memorized all of them.

Personality:

• Detail-oriented about hand anatomy (it matters for accuracy)
• Patient with calibration issues (everyone's hands are different)
• Excited about touchless futures (but realistic about current limits)
• Always thinking about edge cases (literally - hands at frame edges)

Expertise

• Core Areas:
• MediaPipe Hands integration
• Custom gesture classification
• Real-time hand landmark processing
• Gesture-to-action mapping
• Multi-hand tracking
• Sign language recognition basics
• Touchless interface design

• Battle Scars:
• Spent weeks on a demo that broke when someone wore rings
• Learned hand detection drops when fingers overlap the hard way
• Built beautiful gestures nobody could reliably perform
• Discovered webcam quality matters more than algorithm quality
• Had users try gestures for 5 minutes before I realized lighting was wrong
• Optimized from 200ms latency to 16ms - makes all the difference

• Contrarian Opinions:
• Simple gestures beat complex ones - swipe > complex finger spelling
• False positives are worse than false negatives for UX
• 2D landmark positions are often enough - don't overcomplicate with 3D
• Train on diverse hands or your app is racist/ageist/ableist
• Gesture interfaces should have keyboard fallbacks - always

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Hand Gesture Recognition

Patterns

---

Name

MediaPipe Hands Setup

Context

Getting hand tracking running in browser or Python

Approach

Use MediaPipe's pre-trained model for 21 landmark detection. Handle loading, processing, and rendering efficiently.

Example

// Browser - MediaPipe Hands with Webcam import { Hands, HAND_CONNECTIONS } from '@mediapipe/hands'; import { drawConnectors, drawLandmarks } from '@mediapipe/drawing_utils'; import { Camera } from '@mediapipe/camera_utils';

class HandTracker { constructor(videoElement, canvasElement) { this.video = videoElement; this.canvas = canvasElement; this.ctx = canvasElement.getContext('2d'); this.landmarks = []; this.onGesture = null;

this.initMediaPipe(); }

initMediaPipe() { this.hands = new Hands({ locateFile: (file) => { return https://cdn.jsdelivr.net/npm/@mediapipe/hands/${file}; } });

this.hands.setOptions({ maxNumHands: 2, modelComplexity: 1, // 0=lite, 1=full minDetectionConfidence: 0.7, minTrackingConfidence: 0.5 });

this.hands.onResults(this.onResults.bind(this)); }

start() { this.camera = new Camera(this.video, { onFrame: async () => { await this.hands.send({ image: this.video }); }, width: 1280, height: 720 }); this.camera.start(); }

onResults(results) { // Clear and draw video frame this.ctx.save(); this.ctx.clearRect(0, 0, this.canvas.width, this.canvas.height); this.ctx.drawImage(results.image, 0, 0);

// Store landmarks this.landmarks = results.multiHandLandmarks || [];

// Draw hand landmarks for (const landmarks of this.landmarks) { drawConnectors(this.ctx, landmarks, HAND_CONNECTIONS, { color: '#00FF00', lineWidth: 2 }); drawLandmarks(this.ctx, landmarks, { color: '#FF0000', lineWidth: 1, radius: 3 }); }

this.ctx.restore();

// Process gestures if (this.onGesture && this.landmarks.length > 0) { const gesture = this.detectGesture(this.landmarks[0]); if (gesture) { this.onGesture(gesture); } } }

detectGesture(landmarks) { // Basic gesture detection - override for custom const fingers = this.getFingerStates(landmarks);

if (fingers.every(f => f === 'extended')) { return { name: 'open_hand', confidence: 0.9 }; } if (fingers.every(f => f === 'folded')) { return { name: 'fist', confidence: 0.9 }; } if (fingers[0] === 'extended' && fingers.slice(1).every(f => f === 'folded')) { return { name: 'thumbs_up', confidence: 0.85 }; } if (fingers[1] === 'extended' && fingers.filter(f => f === 'folded').length === 4) { return { name: 'pointing', confidence: 0.85 }; }

return null; }

getFingerStates(landmarks) { // Finger tip indices: thumb=4, index=8, middle=12, ring=16, pinky=20 // Finger MCP indices: thumb=2, index=5, middle=9, ring=13, pinky=17 const tips = [4, 8, 12, 16, 20]; const mcps = [2, 5, 9, 13, 17];

return tips.map((tip, i) => { const tipY = landmarks[tip].y; const mcpY = landmarks[mcps[i]].y;

// For thumb, check x instead (it moves sideways) if (i === 0) { const tipX = landmarks[tip].x; const mcpX = landmarks[mcps[i]].x; // Assume right hand - flip logic for left return tipX < mcpX ? 'extended' : 'folded'; }

// For other fingers, tip above MCP = extended return tipY < mcpY ? 'extended' : 'folded'; }); }

stop() { this.camera?.stop(); } }

// Usage const tracker = new HandTracker( document.getElementById('video'), document.getElementById('canvas') ); tracker.onGesture = (gesture) => { console.log('Detected:', gesture.name); }; tracker.start();

---

Name

Custom Gesture Classifier

Context

Training custom gestures beyond basic detection

Approach

Collect landmark data, normalize it, and train a classifier. Use distance/angle features for robust recognition.

Example

// Custom gesture classifier with landmark features class GestureClassifier { constructor() { this.gestures = new Map(); this.samples = []; }

// Extract features from landmarks extractFeatures(landmarks) { const features = [];

// Normalize to wrist position const wrist = landmarks[0]; const normalizedLandmarks = landmarks.map(lm => ({ x: lm.x - wrist.x, y: lm.y - wrist.y, z: lm.z - wrist.z }));

// Palm size for scale normalization const palmSize = this.distance( normalizedLandmarks[0], normalizedLandmarks[9] );

// Finger tip distances from wrist const tipIndices = [4, 8, 12, 16, 20]; for (const tip of tipIndices) { features.push( this.distance(normalizedLandmarks[0], normalizedLandmarks[tip]) / palmSize ); }

// Finger curls (tip to MCP distance) const mcpIndices = [2, 5, 9, 13, 17]; for (let i = 0; i < 5; i++) { features.push( this.distance( normalizedLandmarks[tipIndices[i]], normalizedLandmarks[mcpIndices[i]] ) / palmSize ); }

// Finger spreads (angles between fingers) for (let i = 0; i < 4; i++) { const angle = this.angleBetweenFingers( normalizedLandmarks, tipIndices[i], tipIndices[i + 1] ); features.push(angle / Math.PI); }

// Thumb-index pinch distance features.push( this.distance(normalizedLandmarks[4], normalizedLandmarks[8]) / palmSize );

return features; }

distance(p1, p2) { return Math.sqrt( (p1.x - p2.x) 2 + (p1.y - p2.y) 2 + (p1.z - p2.z) ** 2 ); }

angleBetweenFingers(landmarks, tip1, tip2) { const wrist = landmarks[0]; const v1 = { x: landmarks[tip1].x - wrist.x, y: landmarks[tip1].y - wrist.y }; const v2 = { x: landmarks[tip2].x - wrist.x, y: landmarks[tip2].y - wrist.y };

const dot = v1.x v2.x + v1.y v2.y; const mag1 = Math.sqrt(v1.x 2 + v1.y 2); const mag2 = Math.sqrt(v2.x 2 + v2.y 2);

return Math.acos(dot / (mag1 * mag2)); }

// Record a training sample addSample(gestureName, landmarks) { const features = this.extractFeatures(landmarks); this.samples.push({ name: gestureName, features });

// Update gesture prototype (mean of all samples) if (!this.gestures.has(gestureName)) { this.gestures.set(gestureName, { samples: [], prototype: null }); }

const gesture = this.gestures.get(gestureName); gesture.samples.push(features); gesture.prototype = this.computePrototype(gesture.samples); }

computePrototype(samples) { if (samples.length === 0) return null;

const numFeatures = samples[0].length; const prototype = new Array(numFeatures).fill(0);

for (const sample of samples) { for (let i = 0; i < numFeatures; i++) { prototype[i] += sample[i]; } }

return prototype.map(v => v / samples.length); }

// Classify a gesture classify(landmarks, threshold = 0.3) { const features = this.extractFeatures(landmarks); let bestMatch = null; let bestDistance = Infinity;

for (const [name, gesture] of this.gestures) { if (!gesture.prototype) continue;

const distance = this.euclideanDistance(features, gesture.prototype); if (distance < bestDistance) { bestDistance = distance; bestMatch = name; } }

if (bestDistance > threshold) { return { name: 'unknown', confidence: 0, distance: bestDistance }; }

const confidence = 1 - (bestDistance / threshold); return { name: bestMatch, confidence, distance: bestDistance }; }

euclideanDistance(a, b) { let sum = 0; for (let i = 0; i < a.length; i++) { sum += (a[i] - b[i]) ** 2; } return Math.sqrt(sum); }

// Export/import for persistence export() { const data = {}; for (const [name, gesture] of this.gestures) { data[name] = gesture.samples; } return JSON.stringify(data); }

import(json) { const data = JSON.parse(json); for (const [name, samples] of Object.entries(data)) { this.gestures.set(name, { samples, prototype: this.computePrototype(samples) }); } } }

---

Name

Gesture Smoothing and Debouncing

Context

Preventing jittery gesture recognition

Approach

Use temporal smoothing, confidence thresholds, and state machines to provide stable gesture detection.

Example

// Smoothed gesture detector with debouncing class SmoothGestureDetector { constructor(classifier) { this.classifier = classifier; this.history = []; this.historySize = 5; this.currentGesture = null; this.gestureStartTime = 0; this.minHoldTime = 200; // ms to confirm gesture this.onGestureStart = null; this.onGestureEnd = null; this.onGestureHold = null; }

update(landmarks, timestamp) { // Classify current frame const result = this.classifier.classify(landmarks);

// Add to history this.history.push({ gesture: result.name, confidence: result.confidence, timestamp });

// Keep history limited while (this.history.length > this.historySize) { this.history.shift(); }

// Get majority vote from history const stableGesture = this.getMajorityGesture();

// State machine if (stableGesture !== this.currentGesture) { // Gesture changed if (this.currentGesture) { this.onGestureEnd?.(this.currentGesture, timestamp); }

if (stableGesture && stableGesture !== 'unknown') { this.currentGesture = stableGesture; this.gestureStartTime = timestamp; this.onGestureStart?.(stableGesture, timestamp); } else { this.currentGesture = null; } } else if (this.currentGesture) { // Gesture held const holdTime = timestamp - this.gestureStartTime; if (holdTime >= this.minHoldTime) { this.onGestureHold?.(this.currentGesture, holdTime, timestamp); } }

return { gesture: stableGesture, confidence: this.getAverageConfidence(), isStable: this.history.length >= this.historySize }; }

getMajorityGesture() { if (this.history.length < 3) return null;

const counts = {}; for (const entry of this.history) { if (entry.confidence > 0.5) { counts[entry.gesture] = (counts[entry.gesture] || 0) + 1; } }

let maxCount = 0; let majority = null;

for (const [gesture, count] of Object.entries(counts)) { if (count > maxCount) { maxCount = count; majority = gesture; } }

// Require majority (more than half) if (maxCount > this.history.length / 2) { return majority; }

return null; }

getAverageConfidence() { if (this.history.length === 0) return 0;

const sum = this.history.reduce((acc, h) => acc + h.confidence, 0); return sum / this.history.length; }

reset() { this.history = []; this.currentGesture = null; } }

// Usage const detector = new SmoothGestureDetector(classifier);

detector.onGestureStart = (gesture) => { console.log('Gesture started:', gesture); triggerFeedback('start'); };

detector.onGestureHold = (gesture, holdTime) => { if (holdTime > 1000) { console.log('Long press detected:', gesture); executeAction(gesture); } };

detector.onGestureEnd = (gesture) => { console.log('Gesture ended:', gesture); };

---

Name

Pinch and Grab Detection

Context

Detecting pinch gestures for manipulation

Approach

Track thumb-finger distances and velocities for precise pinch/grab detection with hysteresis.

Example

// Pinch detector with hysteresis class PinchDetector { constructor() { this.isPinching = false; this.pinchStartPos = null; this.pinchThreshold = 0.05; // Distance to start pinch this.releaseThreshold = 0.08; // Distance to release (hysteresis) this.smoothing = 0.3; this.smoothedDistance = 0;

this.onPinchStart = null; this.onPinchMove = null; this.onPinchEnd = null; }

update(landmarks) { // Thumb tip (4) and index tip (8) const thumb = landmarks[4]; const index = landmarks[8];

// Calculate pinch distance const rawDistance = Math.sqrt( (thumb.x - index.x) 2 + (thumb.y - index.y) 2 + (thumb.z - index.z) ** 2 );

// Smooth the distance this.smoothedDistance = this.smoothedDistance (1 - this.smoothing) + rawDistance this.smoothing;

// Pinch midpoint const midpoint = { x: (thumb.x + index.x) / 2, y: (thumb.y + index.y) / 2, z: (thumb.z + index.z) / 2 };

// State machine with hysteresis if (!this.isPinching && this.smoothedDistance < this.pinchThreshold) { // Start pinch this.isPinching = true; this.pinchStartPos = { ...midpoint }; this.onPinchStart?.({ position: midpoint, distance: this.smoothedDistance }); } else if (this.isPinching && this.smoothedDistance > this.releaseThreshold) { // End pinch this.isPinching = false; this.onPinchEnd?.({ position: midpoint, startPosition: this.pinchStartPos, delta: { x: midpoint.x - this.pinchStartPos.x, y: midpoint.y - this.pinchStartPos.y, z: midpoint.z - this.pinchStartPos.z } }); this.pinchStartPos = null; } else if (this.isPinching) { // Continue pinch this.onPinchMove?.({ position: midpoint, startPosition: this.pinchStartPos, delta: { x: midpoint.x - this.pinchStartPos.x, y: midpoint.y - this.pinchStartPos.y, z: midpoint.z - this.pinchStartPos.z }, distance: this.smoothedDistance }); }

return { isPinching: this.isPinching, distance: this.smoothedDistance, position: midpoint }; }

// Grab detection (all fingers closing) detectGrab(landmarks) { const tips = [4, 8, 12, 16, 20]; // Finger tips const palm = landmarks[0]; // Wrist as palm reference

let totalDistance = 0; for (const tip of tips) { totalDistance += Math.sqrt( (landmarks[tip].x - palm.x) 2 + (landmarks[tip].y - palm.y) 2 ); }

const avgDistance = totalDistance / tips.length; const isGrabbing = avgDistance < 0.15; // Threshold

return { isGrabbing, openness: avgDistance }; } }

Anti-Patterns

---

Name

Ignoring Hand Laterality

Description

Not accounting for left vs right hand differences

Wrong

// Assumes right hand only function isThumbsUp(landmarks) { return landmarks[4].x < landmarks[2].x; // Wrong for left hand! }

Right

// Account for handedness function isThumbsUp(landmarks, handedness) { const isRightHand = handedness === 'Right';

if (isRightHand) { return landmarks[4].x < landmarks[2].x; } else { return landmarks[4].x > landmarks[2].x; } }

---

Name

No Confidence Threshold

Description

Acting on every detection regardless of confidence

Wrong

hands.onResults((results) => { if (results.multiHandLandmarks.length > 0) { executeGesture(detectGesture(results.multiHandLandmarks[0])); } });

Right

hands.onResults((results) => { if (results.multiHandLandmarks.length > 0) { const gesture = detectGesture(results.multiHandLandmarks[0]);

// Only act on high-confidence detections if (gesture.confidence > 0.8) { executeGesture(gesture); } } });

---

Name

Creating Complex Gestures

Description

Designing gestures that are hard to perform reliably

Wrong

// Gesture: pinky and thumb extended, others folded, rotated 45 degrees // Users will fail 80% of the time

Right

// Gesture: open hand vs closed fist // Users can do this reliably every time

// Keep gestures: // - Distinct (not easily confused) // - Natural (comfortable to hold) // - Visible (camera can see them)

---

Name

Not Handling Frame Edge Cases

Description

Failing when hands are partially visible

Wrong

function processHand(landmarks) { const gesture = classify(landmarks); executeAction(gesture); // Crashes when landmarks are incomplete at frame edges }

Right

function processHand(landmarks) { // Check if all required landmarks are visible const requiredLandmarks = [0, 4, 8, 12, 16, 20]; const allVisible = requiredLandmarks.every(i => landmarks[i] && landmarks[i].visibility > 0.5 );

if (!allVisible) { return { gesture: 'partial', confidence: 0 }; }

return classify(landmarks); }

Hand Gesture Recognition - Sharp Edges

Hand Detection Is Extremely Sensitive to Lighting

Id

lighting-sensitivity

Severity

CRITICAL

Description

Poor lighting will make even the best model fail

Symptoms

• Detection works in lab, fails in real world
• Flickering detections
• Complete loss of tracking in shadows
• Overexposed hands cause false negatives

Detection Pattern

mediapipe|handpose|hand.*detect

Solution

Lighting Is Everything in Hand Tracking:

Common failures:

• Backlit hands (window behind user)
• Strong shadows (overhead lighting)
• Very dark skin in low light (model bias)
• Overexposed hands (direct sunlight)

Solutions:

  // 1. Check for adequate lighting
  function checkLighting(imageData) {
    let totalBrightness = 0;
    const data = imageData.data;
  
    for (let i = 0; i < data.length; i += 4) {
      const brightness = (data[i] + data[i+1] + data[i+2]) / 3;
      totalBrightness += brightness;
    }
  
    const avgBrightness = totalBrightness / (data.length / 4);
  
    if (avgBrightness < 50) {
      return { ok: false, message: 'Too dark - need more light' };
    }
    if (avgBrightness > 200) {
      return { ok: false, message: 'Too bright - reduce exposure' };
    }
    return { ok: true };
  }
  
  // 2. Guide users to good lighting
  function showLightingGuide(lighting) {
    if (!lighting.ok) {
      showOverlay(`⚠️ ${lighting.message}`);
      return false;
    }
    return true;
  }
  
  // 3. Adjust camera exposure if possible
  async function optimizeCamera(videoTrack) {
    const capabilities = videoTrack.getCapabilities();
  
    if (capabilities.exposureMode) {
      await videoTrack.applyConstraints({
        advanced: [{ exposureMode: 'continuous' }]
      });
    }
  
    if (capabilities.brightness) {
      // Boost brightness slightly
      await videoTrack.applyConstraints({
        advanced: [{
          brightness: capabilities.brightness.max * 0.6
        }]
      });
    }
  }

Best practices:

• Test with diverse skin tones
• Test in various lighting conditions
• Provide real-time lighting feedback
• Consider IR-based solutions for consistent lighting

References

• Computer vision lighting requirements

Webcam Quality Varies Wildly

Id

webcam-quality-variance

Severity

HIGH

Description

Your MacBook Pro camera is not what users have

Symptoms

• Works on dev machine, fails on user devices
• Massive latency on some cameras
• Low resolution breaks detection
• Auto-focus hunting causes detection drops

Detection Pattern

getUserMedia|webcam|video.*stream

Solution

Webcam Reality Check:

Your camera: 1080p, good low-light, fast autofocus Average user: 720p laptop cam from 2018, noisy, slow

Handle the variance:

  // 1. Adaptive resolution
  async function getOptimalStream() {
    // Try high res first, fallback gracefully
    const configs = [
      { width: 1280, height: 720 },
      { width: 960, height: 540 },
      { width: 640, height: 480 }
    ];
  
    for (const config of configs) {
      try {
        const stream = await navigator.mediaDevices.getUserMedia({
          video: {
            width: { ideal: config.width },
            height: { ideal: config.height },
            facingMode: 'user'
          }
        });
  
        console.log(`Got ${config.width}x${config.height} stream`);
        return stream;
      } catch (e) {
        console.log(`${config.width}x${config.height} failed, trying lower`);
      }
    }
  
    // Last resort - any video
    return navigator.mediaDevices.getUserMedia({ video: true });
  }
  
  // 2. Measure actual camera latency
  async function measureCameraLatency(video) {
    const frames = [];
    const startTime = performance.now();
  
    return new Promise((resolve) => {
      let frameCount = 0;
      const checkFrame = () => {
        if (video.readyState >= 2) {
          frames.push(performance.now());
          frameCount++;
  
          if (frameCount >= 30) {
            const avgInterval = (frames[29] - frames[0]) / 29;
            const fps = 1000 / avgInterval;
            resolve({ fps, latency: avgInterval });
            return;
          }
        }
        requestAnimationFrame(checkFrame);
      };
      checkFrame();
    });
  }
  
  // 3. Adjust processing based on capabilities
  function adjustProcessingForDevice(capabilities) {
    if (capabilities.fps < 20) {
      // Slow camera - reduce model complexity
      hands.setOptions({ modelComplexity: 0 });
    }
  
    if (capabilities.fps < 15) {
      // Very slow - skip frames
      return { skipFrames: 2 };
    }
  
    return { skipFrames: 0 };
  }

References

• WebRTC camera handling

Hand Detection Models Have Skin Tone Bias

Id

skin-tone-bias

Severity

HIGH

Description

Models trained on light skin fail on darker skin tones

Symptoms

• Lower detection rates for dark-skinned users
• Higher false negatives for certain ethnicities
• Works for developers but not diverse users
• Customer complaints about "broken" feature

Detection Pattern

hand.*detect|mediapipe|handpose

Solution

Bias Is Real - Test for It:

Most public hand datasets are:

• 70%+ light-skinned hands
• Mostly young adult hands
• Few elderly hands
• Few hands with conditions (arthritis, etc.)

Mitigation strategies:

  // 1. Test with diverse samples
  const testImages = [
    'hands_light_skin.jpg',
    'hands_medium_skin.jpg',
    'hands_dark_skin.jpg',
    'hands_elderly.jpg',
    'hands_with_rings.jpg',
    'hands_with_tattoos.jpg'
  ];
  
  async function runBiasTest(detector) {
    const results = [];
  
    for (const image of testImages) {
      const detections = await detector.detect(image);
      results.push({
        image,
        detected: detections.length > 0,
        confidence: detections[0]?.confidence || 0
      });
    }
  
    // Flag if any category has significantly lower detection
    const avgConfidence = results.reduce((a, r) => a + r.confidence, 0) / results.length;
    const biasedImages = results.filter(r =>
      r.confidence < avgConfidence * 0.7
    );
  
    return { results, biasedImages };
  }
  
  // 2. Adjust detection thresholds
  function adaptiveThreshold(skinTone) {
    // Lower confidence threshold for harder cases
    // This is a band-aid, not a solution
    const baseThreshold = 0.7;
  
    // Could use histogram analysis to detect challenging conditions
    return baseThreshold * 0.9;
  }
  
  // 3. Augment training data if training custom model
  // - Various lighting conditions
  // - Diverse skin tones
  // - Age ranges
  // - Accessories (rings, watches, gloves)

Best practices:

• Test with 10+ diverse hand samples
• Monitor detection rates by user demographics
• Provide fallback input methods
• Be transparent about limitations

References

• Algorithmic fairness in computer vision

Overlapping Fingers Break Detection

Id

occlusion-handling

Severity

HIGH

Description

When fingers cross or overlap, landmarks become unreliable

Symptoms

• Jittery landmarks when making fist
• Wrong finger identified
• Landmark jumping between fingers
• Gestures with crossed fingers fail

Detection Pattern

landmarks|finger.*tip|gesture

Solution

Occlusion Is the Hard Problem:

Hand tracking models see 2D - they guess 3D. When fingers overlap, the guess is often wrong.

Strategies:

  // 1. Detect occlusion and reduce confidence
  function detectOcclusion(landmarks) {
    const fingerTips = [4, 8, 12, 16, 20];
    const minDistance = 0.03; // Normalized units
  
    let occlusionScore = 0;
  
    for (let i = 0; i < fingerTips.length; i++) {
      for (let j = i + 1; j < fingerTips.length; j++) {
        const dist = distance2D(
          landmarks[fingerTips[i]],
          landmarks[fingerTips[j]]
        );
  
        if (dist < minDistance) {
          occlusionScore++;
        }
      }
    }
  
    return {
      isOccluded: occlusionScore > 0,
      occlusionLevel: occlusionScore / 10 // Normalize
    };
  }
  
  // 2. Use temporal consistency
  class OcclusionSmoother {
    constructor() {
      this.history = [];
      this.maxHistory = 5;
    }
  
    smooth(landmarks, occlusion) {
      if (occlusion.isOccluded && this.history.length > 0) {
        // Use historical data when occluded
        const lastGood = this.history[this.history.length - 1];
        return this.interpolate(landmarks, lastGood, occlusion.occlusionLevel);
      }
  
      // Store good frames
      if (!occlusion.isOccluded) {
        this.history.push([...landmarks]);
        if (this.history.length > this.maxHistory) {
          this.history.shift();
        }
      }
  
      return landmarks;
    }
  
    interpolate(current, previous, factor) {
      return current.map((lm, i) => ({
        x: lm.x * (1 - factor) + previous[i].x * factor,
        y: lm.y * (1 - factor) + previous[i].y * factor,
        z: lm.z * (1 - factor) + previous[i].z * factor
      }));
    }
  }
  
  // 3. Avoid gestures requiring finger overlap
  const RELIABLE_GESTURES = [
    'open_hand',      // All fingers extended
    'fist',           // All fingers closed
    'thumbs_up',      // Thumb up, others closed
    'pointing',       // Index extended
    'peace',          // Index + middle extended
    'pinch'           // Thumb + index together
  ];
  
  // Avoid these (high occlusion probability):
  const UNRELIABLE_GESTURES = [
    'crossed_fingers',
    'finger_gun',     // Overlapping middle/ring
    'complex_signs'   // Multiple finger crossings
  ];

References

• Hand occlusion research

Even 100ms Latency Feels Broken

Id

latency-perception

Severity

MEDIUM

Description

Gesture interfaces need <50ms response to feel natural

Symptoms

• Users feel disconnected from interface
• Gestures feel "laggy" or "floaty"
• Users overshoot targets
• Frustration with precise movements

Detection Pattern

gesture|hand.*track|interactive

Solution

Latency Budget for Natural Feel:

Camera capture: ~33ms (30fps) Model inference: ~20-50ms Gesture processing: ~5ms Rendering: ~16ms Total: 74-104ms

That's already too slow!

Optimization strategies:

  // 1. Measure your actual latency
  class LatencyTracker {
    constructor() {
      this.samples = [];
    }
  
    startFrame() {
      this.frameStart = performance.now();
    }
  
    endProcessing() {
      const latency = performance.now() - this.frameStart;
      this.samples.push(latency);
  
      if (this.samples.length > 100) {
        this.samples.shift();
      }
    }
  
    getStats() {
      const sorted = [...this.samples].sort((a, b) => a - b);
      return {
        avg: this.samples.reduce((a, b) => a + b, 0) / this.samples.length,
        p50: sorted[Math.floor(sorted.length * 0.5)],
        p95: sorted[Math.floor(sorted.length * 0.95)],
        p99: sorted[Math.floor(sorted.length * 0.99)]
      };
    }
  }
  
  // 2. Use prediction to hide latency
  class MotionPredictor {
    constructor() {
      this.velocities = [];
    }
  
    predict(currentPos, deltaTime) {
      if (this.velocities.length < 2) {
        return currentPos;
      }
  
      // Average recent velocities
      const avgVelocity = {
        x: this.velocities.reduce((a, v) => a + v.x, 0) / this.velocities.length,
        y: this.velocities.reduce((a, v) => a + v.y, 0) / this.velocities.length
      };
  
      // Predict ahead by ~50ms
      const predictionTime = 0.05;
      return {
        x: currentPos.x + avgVelocity.x * predictionTime,
        y: currentPos.y + avgVelocity.y * predictionTime
      };
    }
  
    update(currentPos, lastPos, deltaTime) {
      if (lastPos) {
        const velocity = {
          x: (currentPos.x - lastPos.x) / deltaTime,
          y: (currentPos.y - lastPos.y) / deltaTime
        };
  
        this.velocities.push(velocity);
        if (this.velocities.length > 5) {
          this.velocities.shift();
        }
      }
    }
  }
  
  // 3. Use lite model for lower latency
  hands.setOptions({
    modelComplexity: 0  // 0 = lite, faster but less accurate
  });
  
  // 4. Skip processing on some frames
  let frameCount = 0;
  function processFrame() {
    frameCount++;
    if (frameCount % 2 === 0) {
      return lastResult; // Reuse previous result
    }
    return detectHands();
  }

References

• Real-time interaction latency

ML Models Take Forever to Load

Id

model-loading-time

Severity

MEDIUM

Description

Users stare at blank screen while model downloads/initializes

Symptoms

• 5+ seconds before tracking starts
• Users think it's broken and leave
• Mobile users on slow connections fail
• No feedback during loading

Detection Pattern

mediapipe|tensorflow|load.*model

Solution

Model Loading UX:

MediaPipe Hands model: ~5-10MB First load: Can be 5-30 seconds on slow connections

Better loading experience:

  // 1. Show meaningful progress
  class ModelLoader {
    constructor() {
      this.onProgress = null;
      this.onReady = null;
    }
  
    async load() {
      // Show we're starting
      this.onProgress?.({ stage: 'downloading', percent: 0 });
  
      const hands = new Hands({
        locateFile: (file) => {
          // Could track download progress here
          return `https://cdn.jsdelivr.net/npm/@mediapipe/hands/${file}`;
        }
      });
  
      this.onProgress?.({ stage: 'initializing', percent: 50 });
  
      await hands.initialize();
  
      this.onProgress?.({ stage: 'warming_up', percent: 80 });
  
      // Warm up with dummy inference
      await this.warmUp(hands);
  
      this.onProgress?.({ stage: 'ready', percent: 100 });
      this.onReady?.();
  
      return hands;
    }
  
    async warmUp(hands) {
      // Run a few inferences to warm up GPU
      const dummyCanvas = document.createElement('canvas');
      dummyCanvas.width = 640;
      dummyCanvas.height = 480;
  
      for (let i = 0; i < 3; i++) {
        await hands.send({ image: dummyCanvas });
      }
    }
  }
  
  // 2. Progressive UI
  function renderLoadingUI(progress) {
    const messages = {
      downloading: 'Downloading hand tracking model...',
      initializing: 'Preparing hand tracking...',
      warming_up: 'Almost ready...',
      ready: 'Hand tracking ready!'
    };
  
    showMessage(messages[progress.stage]);
    showProgressBar(progress.percent);
  }
  
  // 3. Preload during idle time
  // If you know user will need hand tracking,
  // start loading it before they click the button
  const preloadPromise = import('@mediapipe/hands')
    .then(({ Hands }) => new Hands(config));

References

• ML model loading optimization

WebGL/WASM Requirements Aren't Universal

Id

browser-compatibility

Severity

MEDIUM

Description

Hand tracking needs WebGL2 and WASM - not everyone has them

Symptoms

• Black screen on older browsers
• "WebGL not supported" errors
• Crashes on iOS Safari versions
• Works on Chrome, breaks on Firefox

Detection Pattern

webgl|wasm|mediapipe

Solution

Check Before You Wreck:

Requirements:

• WebGL2 (or WebGL with extensions)
• WebAssembly (WASM)
• SharedArrayBuffer (for some models)
• Sufficient GPU memory

  // Feature detection
  function checkHandTrackingSupport() {
    const issues = [];
  
    // WebGL check
    const canvas = document.createElement('canvas');
    const gl = canvas.getContext('webgl2') ||
               canvas.getContext('webgl');
  
    if (!gl) {
      issues.push('WebGL not supported');
    } else if (!canvas.getContext('webgl2')) {
      issues.push('WebGL2 not available - may have reduced performance');
    }
  
    // WASM check
    if (typeof WebAssembly === 'undefined') {
      issues.push('WebAssembly not supported');
    }
  
    // Camera check
    if (!navigator.mediaDevices?.getUserMedia) {
      issues.push('Camera access not available');
    }
  
    // SharedArrayBuffer (needed for SIMD)
    if (typeof SharedArrayBuffer === 'undefined') {
      // Might work, but slower
      console.warn('SharedArrayBuffer not available - reduced performance');
    }
  
    return {
      supported: issues.length === 0,
      issues,
      canFallback: issues.every(i => !i.includes('not supported'))
    };
  }
  
  // Graceful degradation
  async function initHandTracking() {
    const support = checkHandTrackingSupport();
  
    if (!support.supported) {
      if (support.canFallback) {
        showMessage('Limited hand tracking available');
        return initLiteMode();
      } else {
        showMessage('Hand tracking not supported on this device');
        showAlternativeInput();
        return null;
      }
    }
  
    return initFullMode();
  }

Browser-specific gotchas:

• Safari: Needs HTTPS for camera
• Firefox: Some WASM features slower
• Mobile Chrome: Check GPU memory
• iOS Safari: Version-dependent WASM support

References

• WebGL browser compatibility

Hand Gesture Recognition - Validations

Gesture Confidence Threshold

Id

check-confidence-threshold

Description

Gestures should have confidence thresholds to prevent false positives

Pattern

detectGesture|classify.*gesture

File Glob

*/.{js,ts,jsx,tsx,py}

Match

present

Context Pattern

confidence|threshold|score

Message

Add confidence threshold for gesture detection to prevent false positives

Severity

warning

Autofix

Camera Permission Handling

Id

check-camera-permissions

Description

getUserMedia should have proper error handling

Pattern

getUserMedia|navigator.mediaDevices

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

catch|NotAllowedError|NotFoundError

Message

Handle camera permission errors gracefully

Severity

error

Autofix

Model Loading State

Id

check-loading-state

Description

Show loading state while ML model initializes

Pattern

new Hands|handpose|hand.*model

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

loading|progress|initialize

Message

Show loading state during model initialization

Severity

warning

Autofix

Fallback Input Method

Id

check-fallback-input

Description

Provide alternative input when gestures aren't available

Pattern

gesture.control|hand.interface

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

keyboard|mouse|fallback|alternative

Message

Provide keyboard/mouse fallback for accessibility

Severity

warning

Autofix

WebGL Feature Detection

Id

check-feature-detection

Description

Check for WebGL/WASM support before using hand tracking

Pattern

mediapipe|handpose|tensorflow

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

webgl|WebAssembly|getContext

Message

Check WebGL/WASM support before initializing hand tracking

Severity

warning

Autofix

Resource Cleanup

Id

check-cleanup

Description

Clean up camera and model resources on unmount

Pattern

getUserMedia|new Hands

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

stop|dispose|cleanup|close

Message

Clean up camera stream and model when component unmounts

Severity

warning

Autofix

Gesture Smoothing

Id

check-gesture-smoothing

Description

Raw gesture detection should be smoothed for stability

Pattern

detectGesture|onResults

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

smooth|history|buffer|debounce

Message

Apply smoothing to prevent jittery gesture detection

Severity

info

Autofix

Handedness Handling

Id

check-handedness

Description

Account for left vs right hand differences

Pattern

thumb|finger.*extended

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

handedness|left|right|hand

Message

Account for left vs right hand when detecting gestures

Severity

info

Autofix

Lighting Quality Feedback

Id

check-lighting-feedback

Description

Provide feedback about lighting conditions

Pattern

hand.track|gesture.detect

File Glob

*/.{js,ts,jsx,tsx}

Match

present

Context Pattern

lighting|brightness|exposure

Message

Consider providing lighting quality feedback to users

Severity

info

Autofix

HTTPS for Camera Access

Id

check-https-camera

Description

Camera requires HTTPS in production

Pattern

getUserMedia

File Glob

*/.{js,ts,jsx,tsx,html}

Match

present

Context Pattern

https|localhost|secure

Message

Camera access requires HTTPS in production

Severity

info

Autofix