Orcahand

OrcaHand

Full-stack skill for the ORCA Hand — 17-DOF tendon-driven dexterous robotic hand. Compounds on agentic-control-kernel (plant/shield/trace) and bstack (governance).

Plant Interface

Dual-backend: physical (orca_core) and simulated (orca_sim) share identical typed schemas.

observe() → OrcaHandState (schemas/orcahand-state.schema.json)
  measured:
    joint_positions: {thumb_mcp, thumb_abd, thumb_pip, thumb_dip,
                      index_abd, index_mcp, index_pip,
                      middle_abd, middle_mcp, middle_pip,
                      ring_abd, ring_mcp, ring_pip,
                      pinky_abd, pinky_mcp, pinky_pip, wrist}  # degrees
    motor_currents:      17 motors, mA
    motor_temperatures:  17 motors, celsius
    tactile_readings:    per-sensor [fx, fy, fz] in N (touch model only)
  estimated:
    grasp_state:  "open" | "contact" | "secured" | "slipping"
  context:
    backend: "physical" | "simulated"
    control_mode: "position" | "current" | "current_based_position"
    torque_enabled: bool

apply(action) → ActuationResult (schemas/orcahand-action.schema.json)
  directive_type:  setpoint_update | experiment_request | mode_switch
  target_controller: "orca_core" | "orca_sim"
  payload:
    joint_targets: {joint_name: degrees}   # partial dict OK
    num_steps: 25, step_size: 0.001s
    grasp_type: "power" | "precision" | "pinch"

reset(seed?) → neutral position (physical) or env.reset (simulated)

constraints():
  joint_roms: per-joint [min_deg, max_deg] from config.yaml
  max_current: 200mA, max_temperature: 70°C

Safety Shields

Implements kernel SafetyShield contract: filter(), feasible(), fallback().

Emergency fallback: hand.disable_torque() — hand goes limp. Safe due to popping joints. Cascade: filter() -> feasible() -> if infeasible -> emergency fallback + alert outer loop.

Multi-Rate Loop Mapping

SERVO (ms)     Dynamixel PID firmware. Agent never touches this.
     |
MID (10-100ms) Retargeter @ 30Hz / RL policy @ 60Hz / Replay @ 60Hz
               Safety shields run HERE: ROM clamp + current check per frame
     |
OUTER (sec)    LLM supervisory: grasp strategy, mode switch, task goals
               Outputs ControlDirective -> mid-loop controller
     |
META (min-day) EGRI: problem-spec -> train in sim -> evaluate -> promote to physical
               Runs on remote GPU, validated on local macOS

Quick Starts

• Build a hand: Read references/hardware-build.md — BOM, 3D printing, Dynamixel sourcing, assembly, wiring
• Simulate: Read references/simulation-setup.md — pip install orca_sim, MuJoCo on macOS, environment catalog
• Teleoperate: Read references/teleoperation.md — AVP / Rokoko / MediaPipe -> retargeter -> hand
• Train RL policies: Read references/rl-training.md — local CPU/MPS or remote GPU, reward design
• Improve controllers: Read references/egri-controller-loop.md — EGRI problem-spec, evaluator, promotion

Scope Router

Load the relevant reference based on user intent. Max 3 references at once.

Scripts

• scripts/orcahand_init.py — Bootstrap workspace: clone repos, install deps, detect serial, generate .control/plant.yaml
• scripts/orcahand_check.py — Health check for bstack integration (JSON output, exit 0/1)

Schemas

• schemas/orcahand-state.schema.json — extends kernel state.schema.json
• schemas/orcahand-action.schema.json — extends kernel action.schema.json
• schemas/orcahand-trace.schema.json — extends kernel trace.schema.json

Templates

• assets/templates/problem-spec.orcahand.yaml — EGRI template for grasp optimization
• assets/templates/config.orcahand.yaml — starter config for new hand builds

API Reference

Table of Contents

• OrcaHand Class
• MockOrcaHand
• FastAPI REST Endpoints
• Joint Naming Convention
• Config File Schemas

OrcaHand Class

Main controller class in orca_core/core.py:

from orca_core import OrcaHand

hand = OrcaHand("orca_core/models/orcahand_v1_right")

Connection

hand.connect()                  # Open serial connection
hand.disconnect()               # Close serial connection

Torque Control

hand.enable_torque()            # Enable all motors
hand.disable_torque()           # Disable all motors (hand goes limp)

Joint Control

# Move specific joints (partial dict OK)
hand.set_joint_pos(
    {"index_mcp": 90, "middle_pip": 30},
    num_steps=25,       # interpolation steps (smoother = more steps)
    step_size=0.001     # seconds between steps
)

# Read current joint positions
positions = hand.get_joint_pos()  # Returns dict: {joint_name: degrees}

Motor Telemetry

motor_pos = hand.get_motor_pos()        # Raw Dynamixel positions
motor_cur = hand.get_motor_current()    # Current draw in mA
motor_tmp = hand.get_motor_temp()       # Temperature in celsius

Calibration

hand.calibrate()                # Run auto-calibration sequence
hand.set_neutral_position()     # Move to neutral (open hand)
hand.set_zero_position()        # Move to zero position

Control Mode

hand.set_control_mode("position")              # Pure position control
hand.set_control_mode("current_based_position") # Recommended: position + current limit
hand.set_max_current(200)                       # Set current limit in mA

MockOrcaHand

For testing without hardware:

from orca_core import MockOrcaHand

hand = MockOrcaHand("orca_core/models/orcahand_v1_right")
hand.connect()  # No serial required
hand.set_joint_pos({"thumb_mcp": 45})  # Simulated, no hardware

Same API as OrcaHand. Useful for unit tests and CI.

FastAPI REST Endpoints

Start server: python -m orca_core.api.api (port 8000)

Joint Naming Convention

Format: {finger}_{joint_type}

Joint types:

• mcp: Metacarpophalangeal (knuckle)
• pip: Proximal interphalangeal (first bend)
• dip: Distal interphalangeal (fingertip bend, thumb only)
• abd: Abduction (side-to-side spread)

Config File Schemas

config.yaml (per model)

Primary configuration. Contains: baudrate, port, max_current, control_mode, motor_ids (17), joint_ids (17), joint_to_motor_map (with sign for tendon inversion), joint_roms, neutral_position, calibration_sequence.

calibration.yaml (per model)

Written by calibrate.py. Contains: calibrated (bool), timestamp, motor_limits, joint_to_motor_ratios.

hand_scheme.yaml (retargeter)

Kinematic structure. Contains: gc_tendons, finger_to_tip, finger_to_base, gc_limits_lower, gc_limits_upper, wrist_name.

retargeter.yaml (retargeter)

Optimization tuning. Contains: lr, use_scalar_distance_palm, loss_coeffs, mano_adjustments, joint_regularizers.

Calibration Pipeline

Table of Contents

• Serial Port Detection
• 3-Step Pipeline
• config.yaml Schema
• calibration.yaml Output
• When to Re-Calibrate
• Troubleshooting

Serial Port Detection

# macOS
ls /dev/tty.usbserial-*

# Linux
ls /dev/ttyUSB*

The U2D2 adapter uses FTDI VID 0x0403. If multiple serial devices, check with:

# macOS
system_profiler SPUSBDataType | grep -A5 "U2D2"

Update config.yaml with the detected port path.

3-Step Pipeline

Run from the orca_core/ directory. Each script takes the model path as argument.

Step 1: Tension Tendons

python scripts/tension.py orca_core/models/orcahand_v1_right

Applies light current to each motor to take up tendon slack. Hold the hand steady during this step.

Step 2: Auto-Calibrate

python scripts/calibrate.py orca_core/models/orcahand_v1_right

Runs a 28-step flex/extend sequence across all joints. Records motor positions at extreme positions to compute joint_to_motor_ratios. Writes results to calibration.yaml.

Important: The hand will move through its full range of motion. Ensure nothing is blocking the fingers.

Step 3: Move to Neutral

python scripts/neutral.py orca_core/models/orcahand_v1_right

Moves all joints to the neutral (open hand) position defined in config.yaml. Verifies calibration is correct visually.

config.yaml Schema

baudrate: 3000000
port: /dev/tty.usbserial-XXXXX
max_current: 200              # mA
control_mode: 5               # current_based_position (recommended)
motor_ids: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
joint_ids: [thumb_mcp, thumb_abd, thumb_pip, thumb_dip,
            index_abd, index_mcp, index_pip,
            middle_abd, middle_mcp, middle_pip,
            ring_abd, ring_mcp, ring_pip,
            pinky_abd, pinky_mcp, pinky_pip, wrist]
joint_to_motor_map:
  thumb_mcp: {motor: 0, sign: 1}
  thumb_abd: {motor: 1, sign: -1}   # negative sign = inverted tendon
  # ... (17 mappings total)
joint_roms:
  thumb_mcp: [0, 90]
  thumb_abd: [-30, 30]
  # ... (17 joint ROMs)
neutral_position:
  thumb_mcp: 0
  thumb_abd: 0
  # ... (17 neutral positions)

calibration.yaml Output

Written by calibrate.py:

calibrated: true
timestamp: 2026-03-25T10:00:00
motor_limits:
  0: {min: 1024, max: 3072}    # raw Dynamixel positions
  # ... (17 motors)
joint_to_motor_ratios:
  thumb_mcp: 22.75             # degrees per Dynamixel unit
  # ... (17 ratios)

When to Re-Calibrate

• After replacing a tendon
• After significant use (>1000 grasp cycles)
• If grasps feel imprecise or joints drift
• After any mechanical repair
• Weekly if running experiments (calibration drift is real)

Troubleshooting

Motor not found: Check U2D2 connection, verify baudrate matches config, try scripts/check_motor.py.

Calibration drift: Tendons stretch over time. Re-run tension.py + calibrate.py. If persistent, replace the tendon.

Motor overheating: Reduce max_current in config.yaml. Check for tendon friction at routing points.

Dependency Graph

Table of Contents

• Repo Relationships
• Version Matrix
• Install Order
• Python Requirements
• Known Conflicts

Repo Relationships

                    orcahand_description
                    (URDF/MJCF models)
                   /         |          \
                  /          |           \
            orca_sim    orca_retargeter   rwr_system
         (Gymnasium     (Teleoperation    (ROS2 pipeline)
          + MuJoCo)      retargeting)
              |              |                |
              v              v                v
          [RL training]  [Human input]    [Deployment]
                          (AVP/Rokoko)
                              |
                              v
                         orca_core
                     (Hardware control)
                              |
                              v
                    [Physical ORCA Hand]

Version Matrix

Install Order

Must be installed in dependency order:

1. orcahand_description — no deps, just git clone 2. orca_sim — depends on orcahand_description for MJCF models 3. orca_core — independent, but install after sim for testing 4. orca_retargeter — depends on orcahand_description for URDF 5. rwr_system — depends on ROS2, optional

Use scripts/orcahand_init.py to automate this.

Python Requirements

• Python 3.10+ (all repos)
• MuJoCo: pip install mujoco (3.1+)
• PyTorch: 2.2.0 specifically for orca_retargeter
• ROS2: Required only for rwr_system (Humble or newer)

Known Conflicts

numpy version conflict

• orca_core requires numpy >=2.2.6
• orca_retargeter requires numpy <2
• Resolution: Use separate virtual environments, or install in a single env with numpy==1.26.4 (works for both with a warning from orca_core)

torch version pinning

• orca_retargeter pins torch ==2.2.0
• If you need a newer torch for training, use a separate venv for training vs retargeting

Recommended setup

# Option A: Single venv (some version warnings)
python -m venv .venv && source .venv/bin/activate
pip install numpy==1.26.4  # compromise version
pip install -e orca_core/ -e orca_sim/

# Option B: Separate venvs (cleaner, recommended)
python -m venv .venv-sim && .venv-sim/bin/pip install -e orca_sim/
python -m venv .venv-core && .venv-core/bin/pip install -e orca_core/
python -m venv .venv-retarget && .venv-retarget/bin/pip install -e orca_retargeter/

EGRI Controller Improvement Loop

Table of Contents

• EGRI Overview for OrcaHand
• Problem-Spec Template
• Evaluator Design
• Promotion Policy
• Integration with autoany
• Running an EGRI Cycle

EGRI Overview for OrcaHand

EGRI (Evaluator-Governed Recursive Improvement) applied to the OrcaHand:

Problem-spec → Train in sim (remote GPU) → Evaluate (sim metrics)
                     ↑                            ↓
              Update artifact              Gate: pass/fail?
                     ↑                            ↓
              Rollback if failed     Deploy to physical (human approval)
                                            ↓
                                     Evaluate (real metrics)
                                            ↓
                                     Update sim-to-real gap

Mutable artifacts (what EGRI improves):

• RL policy weights (.pt files)
• Retargeter configuration (retargeter.yaml loss_coeffs, lr, mano_adjustments)
• Grasp primitive parameters (pre-defined joint angle sequences)

Problem-Spec Template

Use assets/templates/problem-spec.orcahand.yaml as your starting point. Key sections to customize:

• objective.metric: What you're optimizing (grasp_success_rate, reorientation_accuracy, grasp_stability_duration)
• objective.target: Threshold for success (0.85 = 85%)
• artifacts: Which files the EGRI loop can modify
• evaluator.outputs: Metrics to compute and their thresholds
• execution.budget: Time and compute limits

Evaluator Design

Sim evaluator (runs on every iteration):

def evaluate_sim(policy, env, n_episodes=100):
    successes = 0
    for _ in range(n_episodes):
        obs, info = env.reset()
        done = False
        while not done:
            action, _ = policy.predict(obs, deterministic=True)
            obs, reward, terminated, truncated, info = env.step(action)
            done = terminated or truncated
        if info.get("success", False):
            successes += 1
    return {"sim_success_rate": successes / n_episodes}

Real evaluator (runs only after sim gate passes):

def evaluate_real(policy, hand, n_grasps=20):
    # Requires human supervision
    successes = 0
    for i in range(n_grasps):
        print(f"Trial {i+1}/{n_grasps} — place object and press Enter")
        input()
        # Execute policy...
        result = input("Success? (y/n): ")
        if result.lower() == "y":
            successes += 1
    return {"real_success_rate": successes / n_grasps}

Composite evaluator:

def evaluate(sim_results, real_results):
    gap = abs(sim_results["sim_success_rate"] - real_results["real_success_rate"])
    return {
        "sim_success_rate": sim_results["sim_success_rate"],
        "real_success_rate": real_results["real_success_rate"],
        "sim_to_real_gap": gap,
        "promoted": sim_results["sim_success_rate"] > 0.85 and gap < 0.15,
    }

Promotion Policy

IF sim_success_rate > 0.85:
    → Request human approval for physical trial
    IF human_approved:
        → Run real evaluator (20 grasps, supervised)
        IF sim_to_real_gap < 0.15:
            → PROMOTE: save as production policy
        ELSE:
            → ROLLBACK: revert to previous checkpoint
            → Increase domain randomization, re-train
    ELSE:
        → HOLD: keep training in sim
ELSE:
    → CONTINUE: more training iterations

Human-in-the-loop: Physical deployment always requires human approval. This is enforced by the approval: manual field in the problem-spec.

Integration with autoany

Wire into the existing autoany EGRI harness:

# From the autoany workspace
autoany egri run --problem-spec path/to/problem-spec.orcahand.yaml

The autoany harness handles:

• Iteration management (train → evaluate → decide)
• Artifact versioning (checkpoint management)
• Ledger persistence (via Lago)
• Budget enforcement (max_trials, max_hours)

Running an EGRI Cycle

1. Setup: Copy problem-spec template, customize objective and budget 2. Train: python train.py --config problem-spec.orcahand.yaml 3. Evaluate (sim): Automatic after training completes 4. Gate check: Script reports pass/fail 5. Deploy (if passed): Human approves, run real-world trials 6. Record: All results logged to Lago trace entries 7. Iterate: Adjust hyperparameters, domain randomization, re-run

Hardware Build Guide

Table of Contents

• Bill of Materials
• 3D Printing
• Sourcing Servos
• Assembly
• Wiring
• First Power-On

Bill of Materials

Budget alternative: Feetech SCS0009 servos (~$8 each vs $70 for Dynamixel). Lower quality, less precise, but functional. Use FeetechClient in orca_core.

Pre-assembled: $3,500 from orcahand.com or ROBOTIS (orcahand standard).

3D Printing

STL files: orcahand_description/ repo, under v1/meshes/ or v2/meshes/.

Recommended settings:

• Material: PLA or PETG (PETG preferred for durability)
• Layer height: 0.2mm
• Infill: 30-40% (structural parts), 20% (cosmetic)
• Supports: required for finger segments and palm
• Estimated print time: ~24-30 hours total

Key parts: main tower (~15K faces), base (~2K), finger segments, skin covers.

Popping joints: Print these at 100% infill — they must survive repeated dislocation/relocation cycles without deforming.

Sourcing Servos

• ROBOTIS direct: Best price, reliable shipping. Order XC330-T288-T (TTL protocol).
• Amazon/ROBOTIS resellers: Faster shipping, slight markup.
• Feetech alternative: SCS0009 — budget option, requires FeetechClient driver in orca_core.

All 17 servos must be the same model. Do not mix Dynamixel and Feetech.

Assembly

Official video guides available at orcahand.com.

Key steps: 1. Assemble finger segments — thread tendons through channels 2. Mount servos into the palm/tower structure 3. Route tendons through low-friction channels at joint rotation centers 4. Attach finger assemblies to palm 5. Wire servo daisy-chain (TTL bus) 6. Connect U2D2 adapter

Tendon routing: Route tendons through joint rotation centers to minimize friction. This is the most critical assembly step — poor routing causes tendon wear and calibration drift.

Assembly time: ~6-8 hours for first build, ~3-4 hours with experience.

Wiring

12V PSU → U2D2 Power Hub → Servo daisy-chain (TTL bus)
                            Motor 0 → Motor 1 → ... → Motor 16
                                                         |
U2D2 USB adapter ← ─────────────── TTL data bus ─────────┘
     |
  USB to Mac/PC

• Protocol: Dynamixel Protocol 2.0 (TTL)
• Baudrate: 3,000,000 (set via scripts/configure_motor_chain.py)
• Motor IDs: 0-16, pre-configured or set with configure_motor_chain.py

First Power-On

1. Connect U2D2 USB to your computer 2. Detect serial port: ls /dev/tty.usbserial-* (macOS) or ls /dev/ttyUSB* (Linux) 3. Update config.yaml with your serial port 4. Test motor detection: python scripts/check_motor.py orca_core/models/orcahand_v1_right 5. Proceed to calibration: see calibration-pipeline.md

RL Training

Table of Contents

• Local Training (macOS)
• Remote GPU Setup
• Framework Choice
• Reward Design
• Curriculum Training
• Checkpointing

Local Training (macOS)

Good for prototyping small policies. Apple Silicon MPS backend available for PyTorch.

pip install stable-baselines3 torch

from stable_baselines3 import PPO
from orca_sim import OrcaHandRightCubeOrientation

env = OrcaHandRightCubeOrientation(version="v2", render_mode="rgb_array")
model = PPO("MlpPolicy", env, verbose=1, device="mps")  # or "cpu"
model.learn(total_timesteps=100_000)
model.save("checkpoints/grasp_policy_v1")

Limitations: CPU/MPS is 5-20x slower than GPU for training. Fine for <500K timesteps, use remote GPU for serious training.

Remote GPU Setup

1. Provision a GPU box (Lambda, Vast.ai, RunPod, etc.) with CUDA + PyTorch 2. Mirror the environment:

   ssh gpu-box "pip install orca_sim stable-baselines3 torch"

3. Start training:

   ssh gpu-box "python train.py --timesteps 10_000_000 --device cuda"

4. Sync checkpoints back:

   rsync -av gpu-box:~/checkpoints/ ./checkpoints/

Alternative: Use a Jupyter notebook on Colab/Lambda for interactive development.

Framework Choice

Recommended: Start with SB3 PPO, move to CleanRL or custom if you need specific modifications.

Reward Design

CubeOrientation task (built-in):

# orca_sim/task_envs.py reward logic:
# reward = alignment_to_target + lift_bonus - drop_penalty
# terminated on: success (15° tolerance) or drop

Custom grasp rewards — design principles:

• Alignment: Reward reducing angle between current and target object pose
• Stability: Reward maintaining grasp over multiple timesteps
• Efficiency: Penalize energy (motor current) usage
• Safety: Large penalty for exceeding joint ROM or temperature limits

def compute_reward(self):
    alignment = self._compute_alignment()    # 0-1
    stability = self._grasp_duration / 100   # 0-1
    energy = -0.01 * self._total_current     # penalty
    return alignment + 0.5 * stability + energy

Curriculum Training

Progressive difficulty for complex manipulation tasks:

1. Stage 1: Large, easy-to-grasp objects (cube, sphere), wide success tolerance 2. Stage 2: Smaller objects, tighter tolerance, varied starting positions 3. Stage 3: In-hand manipulation (rotation, reorientation) 4. Stage 4: Domain randomization (friction, mass, motor delay)

Advance to next stage when success rate > 80% for 1000 episodes.

Checkpointing

# Save during training
model.save(f"checkpoints/grasp_policy_step_{step}")

# Load for evaluation
model = PPO.load("checkpoints/grasp_policy_v1")
obs, info = env.reset()
action, _ = model.predict(obs, deterministic=True)

Best practice: Save checkpoints every 100K steps. Keep the last 5 + the best-performing one.

Evaluation:

from stable_baselines3.common.evaluation import evaluate_policy
mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=100)
print(f"Success rate proxy: {mean_reward:.2f} +/- {std_reward:.2f}")

Sim-to-Real Transfer

Table of Contents

• Joint Reordering Map
• Domain Randomization
• Action Scaling
• Validation Protocol
• Common Failure Modes
• Zero-Shot vs Fine-Tuned

Joint Reordering Map

Simulation joint ordering differs from physical motor ordering. Apply this map when deploying sim-trained policies to real hardware:

# Sim index → Real index (from rwr_system)
SIM_TO_REAL = [0, 13, 14, 15, 16, 10, 11, 12, 4, 5, 6, 1, 2, 3, 7, 8, 9]

def reorder_sim_to_real(sim_actions):
    """Reorder action vector from sim ordering to real motor ordering."""
    return [sim_actions[i] for i in SIM_TO_REAL]

Why they differ: Simulation models joints by kinematic chain (thumb → index → middle → ring → pinky), while the physical motor IDs follow the wiring daisy-chain topology.

Domain Randomization

Randomize these parameters during training to improve transfer robustness:

Apply in your training script:

# Randomize at episode reset
env.model.geom_friction[:] *= np.random.uniform(0.5, 1.5)
env.model.dof_damping[:] *= np.random.uniform(0.5, 2.0)

Action Scaling

Simulation actions are in actuator control range. Convert to degrees for OrcaHand.set_joint_pos():

def sim_action_to_degrees(action, joint_roms):
    """Scale normalized [-1, 1] action to joint ROM in degrees."""
    degrees = {}
    for i, (joint, (lo, hi)) in enumerate(joint_roms.items()):
        degrees[joint] = lo + (action[i] + 1) / 2 * (hi - lo)
    return degrees

Validation Protocol

Before deploying a sim-trained policy to physical hardware:

1. Sim evaluation (automated):

• Run 100 episodes in sim with deterministic policy
• Record success rate, average reward, episode length
• Threshold: success rate > 85%

2. Supervised real evaluation (human present):

• Run 20 grasps on physical hand
• Human monitors for anomalies (servo overheating, tendon issues)
• Record success rate, compare to sim

3. Gap metric:

   sim_to_real_gap = abs(sim_success_rate - real_success_rate)

• Gap < 15%: deploy
• Gap 15-30%: more domain randomization needed
• Gap > 30%: fundamental sim-real mismatch, investigate

Common Failure Modes

Zero-Shot vs Fine-Tuned

Zero-shot: Train only in sim with heavy domain randomization. Directly deploy to real. Simpler pipeline, works for robust policies.

Fine-tuned: Train in sim, then collect real-world episodes and fine-tune. Better performance, requires data collection infrastructure (rwr_system).

Recommendation: Start zero-shot. Only fine-tune if the sim-to-real gap exceeds 15% after thorough domain randomization.

Simulation Setup

Table of Contents

• Installation
• MuJoCo on macOS (Apple Silicon)
• Environment Catalog
• Custom Environments
• Version Pinning
• Headless Rendering

Installation

# From PyPI (stable)
pip install orca_sim

# From source (dev)
git clone https://github.com/orcahand/orca_sim.git
cd orca_sim && pip install -e .

Dependencies: gymnasium >=0.29, mujoco >=3.1, numpy >=1.26.

MuJoCo on macOS (Apple Silicon)

MuJoCo runs natively on Apple Silicon. For interactive rendering (render_mode="human"), use mjpython:

pip install mujoco
# Interactive viewer requires mjpython on macOS:
mjpython your_script.py

For headless rendering (render_mode="rgb_array"), regular python works fine.

Visualize hand model directly:

python -m mujoco.viewer --mjcf=$(pwd)/orcahand_description/v2/scene_combined.xml

Environment Catalog

All environments registered as Gymnasium envs. Default version: v2.

Basic usage:

from orca_sim import OrcaHandRight

env = OrcaHandRight(version="v2", render_mode="human")
obs, info = env.reset(seed=42)

for _ in range(1000):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()

env.close()

Observation space: [qpos, qvel] — joint positions and velocities. Action space: Actuator control range (continuous). Frame skip: 5 physics steps per env step.

Custom Environments

Subclass BaseOrcaHandEnv to create task-specific environments:

from orca_sim.envs import BaseOrcaHandEnv

class MyGraspEnv(BaseOrcaHandEnv):
    def __init__(self, **kwargs):
        super().__init__(hand="right", version="v2", **kwargs)

    def _get_reward(self):
        # Define your reward function
        return 0.0

    def _get_terminated(self):
        # Define termination conditions
        return False

Add objects by modifying the MJCF XML or using MuJoCo's runtime API.

Version Pinning

# Use v1 models (legacy)
env = OrcaHandRight(version="v1")

# Use v2 models (default, recommended)
env = OrcaHandRight(version="v2")

# Check available versions
from orca_sim.versions import list_versions, latest_version
print(list_versions())   # ["v1", "v2"]
print(latest_version())  # "v2"

Headless Rendering

For recording videos or remote servers:

env = OrcaHandRight(render_mode="rgb_array")
obs, info = env.reset()
frame = env.render()  # returns numpy array (H, W, 3)

Combine with imageio or cv2 for video recording.

Teleoperation

Table of Contents

• Architecture Overview
• Apple Vision Pro Setup
• Rokoko Glove Setup
• MediaPipe Webcam Fallback
• Retargeter Configuration
• End-to-End Demo

Architecture Overview

Human hand input → Retargeter → OrcaHand
                   (gradient-based optimization)
                   MANO keypoints → joint angles

Three input sources, one retargeter, one output: 1. Apple Vision Pro via avp-stream — best quality, wireless 2. Rokoko glove via ROS2 ingress — professional, wired/wireless 3. MediaPipe webcam — no hardware required, lower accuracy

Apple Vision Pro Setup

pip install avp-stream

from avp_stream import VisionProStreamer
from orca_retargeter import Retargeter
from orca_core import OrcaHand

# Initialize
streamer = VisionProStreamer(ip="<AVP_IP>")
retargeter = Retargeter("orca_retargeter/models/orcahand_v1_right")
hand = OrcaHand("orca_core/models/orcahand_v1_right")
hand.connect()
hand.enable_torque()

# Stream loop
while True:
    avp_data = streamer.get_latest()
    if avp_data is not None:
        joint_angles = retargeter.retarget(avp_data)
        hand.set_joint_pos(joint_angles, num_steps=5, step_size=0.001)

Latency: ~30-50ms end-to-end (AVP tracking → WiFi → retarget → serial → servo).

Rokoko Glove Setup

Requires ROS2 and rwr_system:

# In ROS2 workspace
cd ~/ros2_ws/src
git clone https://github.com/orcahand/rwr_system.git
pip install -e .[all]
colcon build --symlink-install
source install/setup.bash

# Run teleoperation
ros2 launch experiments record_demonstration.launch.py
ros2 run experiments run_teleop_rokoko.py

Calibration (3 steps in the Rokoko teleop script): 1. Robot init — move to neutral 2. Robot calibrate — map joint limits 3. Glove calibrate — map human hand range

ROS2 topics:

• /ingress/mano (Float32MultiArray, 21x3) — MANO keypoints
• /ingress/wrist (PoseStamped) — wrist pose
• /hand/policy_output (Float32MultiArray) — joint angles in radians

MediaPipe Webcam Fallback

No special hardware — just a webcam. Lower accuracy, higher latency.

Available via rwr_system/ingress/webcam or standalone with MediaPipe:

import mediapipe as mp
import cv2

mp_hands = mp.solutions.hands.Hands(max_num_hands=1)
cap = cv2.VideoCapture(0)

while cap.isOpened():
    ret, frame = cap.read()
    results = mp_hands.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    if results.multi_hand_landmarks:
        landmarks = results.multi_hand_landmarks[0]
        # Convert to MANO format and feed to retargeter

Limitations: 2D tracking (depth estimated), sensitive to lighting, ~100ms latency.

Retargeter Configuration

Three config files in orca_retargeter/models/<model_name>/:

hand_scheme.yaml

Defines tendon-joint coupling and kinematic structure:

gc_tendons:        # tendon → joint coupling matrix
finger_to_tip:     # finger name → tip link name
finger_to_base:    # finger name → base link name
gc_limits_lower:   # generalized coordinate lower bounds
gc_limits_upper:   # generalized coordinate upper bounds
wrist_name: wrist  # wrist joint name

retargeter.yaml

Tuning parameters for the optimization:

lr: 2.5                      # RMSprop learning rate
use_scalar_distance_palm: true
loss_coeffs:                  # per-finger loss weights
  thumb: 1.0
  index: 1.0
  middle: 0.8
  ring: 0.6
  pinky: 0.4
mano_adjustments:             # per-finger corrections
  thumb:
    translation: [0, 0, 0]
    rotation: [0, 0, 0]
    scale: 1.0
joint_regularizers:           # prevent extreme angles
  thumb_mcp: 0.01

Tuning tips

• Increase lr for faster tracking (may overshoot)
• Adjust loss_coeffs to prioritize certain fingers (thumb > pinky for most tasks)
• Modify mano_adjustments if mapping feels offset for your hand size

End-to-End Demo

The ret_demo.py script in orca_retargeter/ connects everything:

cd orca_retargeter
python ret_demo.py --model models/orcahand_v1_right --input avp

Options: --input avp (Apple Vision Pro), --input replay (recorded data).

Troubleshooting

Table of Contents

• Hardware Issues
• Simulation Issues
• Calibration Issues
• Teleoperation Issues
• Common Error Messages

Hardware Issues

Motor not found

DynamixelError: Motor ID X not found on port /dev/tty.usbserial-XXXXX

• Check U2D2 USB connection
• Verify serial port: ls /dev/tty.usbserial-* (macOS)
• Verify baudrate matches config.yaml (must be 3000000)
• Run python scripts/check_motor.py to scan all IDs
• Check power supply is connected and on (motors need 12V)

Servo overheating

• Reduce max_current in config.yaml (try 150mA)
• Check for tendon friction at routing points — re-route if needed
• Allow cooldown period (temperature shield triggers at 70°C)
• If persistent: check for mechanical binding in the joint

Tendon snapped

• Replace with Dyneema fishing line (same gauge)
• Re-route through low-friction channels
• Run full calibration after replacement: tension → calibrate → neutral

Popping joint stuck

• Gently push joint back into socket (designed to pop back)
• If repeatedly stuck: check 3D print quality, reprint at 100% infill

Simulation Issues

MuJoCo segfault on macOS

Segmentation fault: 11

• Use mjpython instead of python for interactive rendering
• For headless: use render_mode="rgb_array" (no display needed)
• Update MuJoCo: pip install --upgrade mujoco

Environment won't load

FileNotFoundError: MJCF file not found

• Ensure orcahand_description is cloned in the expected location
• Check orca_sim version matches description version (v1 vs v2)
• Reinstall: pip install -e orca_sim/

Rendering blank/black

• macOS: Must use mjpython for render_mode="human"
• Remote server: Use render_mode="rgb_array" (no display)
• Check: python -c "import mujoco; print(mujoco.glfw)" — should not error

Calibration Issues

Calibration drift

• Tendons stretch over time — re-tension periodically
• Run tension.py → calibrate.py weekly during active use
• If drift is severe: replace tendon

Motor position jumps during calibration

• Motor may be in wrong control mode — ensure control_mode: 5 in config.yaml
• Check motor daisy-chain wiring for loose connections
• Verify motor IDs match config.yaml ordering

calibration.yaml not updating

• Check file permissions on the model directory
• Verify calibrate.py completed without errors
• Look for calibrated: false — indicates calibration failed

Teleoperation Issues

Apple Vision Pro not connecting

• Verify AVP and Mac are on the same WiFi network
• Check IP address: streamer = VisionProStreamer(ip="<correct_IP>")
• Restart AVP streaming app

Retargeter output looks wrong

• Check hand_scheme.yaml — joint coupling may be misconfigured
• Verify URDF path in retargeter matches your hand version (v1/v2)
• Adjust mano_adjustments in retargeter.yaml for hand size differences
• Increase optimization steps for better convergence (reduce lr)

High latency during teleoperation

• Reduce num_steps in set_joint_pos (try 5 instead of 25)
• Check WiFi latency (AVP): should be <10ms
• Serial latency: U2D2 at 3M baud is ~0.5ms per command

Common Error Messages