Pilotty

Terminal Automation with pilotty

CRITICAL: Argument Positioning

All flags (`--name`, `-s`, `--format`, etc.) MUST come BEFORE positional arguments:

# CORRECT - flags before command/arguments
pilotty spawn --name myapp vim file.txt
pilotty key -s myapp Enter
pilotty snapshot -s myapp --format text

# WRONG - flags after command (they get passed to the app, not pilotty!)
pilotty spawn vim file.txt --name myapp   # FAILS: --name goes to vim
pilotty key Enter -s myapp                # FAILS: -s goes nowhere useful

This is the #1 cause of agent failures. When in doubt: flags first, then command/args.

---

Quick start

pilotty spawn vim file.txt        # Start TUI app in managed session
pilotty wait-for "file.txt"       # Wait for app to be ready
pilotty snapshot                  # Get screen state with UI elements
pilotty key i                     # Enter insert mode
pilotty type "Hello, World!"      # Type text
pilotty key Escape                # Exit insert mode
pilotty kill                      # End session

Core workflow

1. Spawn: pilotty spawn <command> starts the app in a background PTY 2. Wait: pilotty wait-for <text> ensures the app is ready 3. Snapshot: pilotty snapshot returns screen state with detected UI elements 4. Understand: Parse elements[] to identify buttons, inputs, toggles 5. Interact: Use keyboard commands (key, type) to navigate and interact 6. Re-snapshot: Check content_hash to detect screen changes

Commands

Session management

pilotty spawn <command>           # Start TUI app (e.g., pilotty spawn htop)
pilotty spawn --name myapp <cmd>  # Start with custom session name (--name before command)
pilotty kill                      # Kill default session
pilotty kill -s myapp             # Kill specific session
pilotty list-sessions             # List all active sessions
pilotty daemon                    # Manually start daemon (usually auto-starts)
pilotty shutdown                  # Stop daemon and all sessions
pilotty examples                  # Show end-to-end workflow example

Screen capture

pilotty snapshot                  # Full JSON with text content and elements
pilotty snapshot --format compact # JSON without text field
pilotty snapshot --format text    # Plain text with cursor indicator
pilotty snapshot -s myapp         # Snapshot specific session

# Wait for screen to change (eliminates need for sleep!)
HASH=$(pilotty snapshot | jq '.content_hash')
pilotty key Enter
pilotty snapshot --await-change $HASH           # Block until screen changes
pilotty snapshot --await-change $HASH --settle 50  # Wait for 50ms stability

Input

pilotty type "hello"              # Type text at cursor
pilotty type -s myapp "text"      # Type in specific session

pilotty key Enter                 # Press Enter
pilotty key Ctrl+C                # Send interrupt
pilotty key Escape                # Send Escape
pilotty key Tab                   # Send Tab
pilotty key F1                    # Function key
pilotty key Alt+F                 # Alt combination
pilotty key Up                    # Arrow key
pilotty key -s myapp Ctrl+S       # Key in specific session

# Key sequences (space-separated, sent in order)
pilotty key "Ctrl+X m"            # Emacs chord: Ctrl+X then m
pilotty key "Escape : w q Enter"  # vim :wq sequence
pilotty key "a b c" --delay 50    # Send a, b, c with 50ms delay
pilotty key -s myapp "Tab Tab Enter"  # Sequence in specific session

Interaction

pilotty click 5 10                # Click at row 5, col 10
pilotty click -s myapp 10 20      # Click in specific session
pilotty scroll up                 # Scroll up 1 line
pilotty scroll down 5             # Scroll down 5 lines
pilotty scroll up 10 -s myapp     # Scroll in specific session

Terminal control

pilotty resize 120 40             # Resize terminal to 120 cols x 40 rows
pilotty resize 80 24 -s myapp     # Resize specific session

pilotty wait-for "Ready"          # Wait for text to appear (30s default)
pilotty wait-for "Error" -r       # Wait for regex pattern
pilotty wait-for "Done" -t 5000   # Wait with 5s timeout
pilotty wait-for "~" -s editor    # Wait in specific session

Global options

Environment variables

PILOTTY_SESSION="mysession"       # Default session name
PILOTTY_SOCKET_DIR="/tmp/pilotty" # Override socket directory
RUST_LOG="debug"                  # Enable debug logging

Snapshot Output

The snapshot command returns structured JSON with detected UI elements:

{
  "snapshot_id": 42,
  "size": { "cols": 80, "rows": 24 },
  "cursor": { "row": 5, "col": 10, "visible": true },
  "text": "Settings:\n  [x] Notifications  [ ] Dark mode\n  [Save]  [Cancel]",
  "elements": [
    { "kind": "toggle", "row": 1, "col": 2, "width": 3, "text": "[x]", "confidence": 1.0, "checked": true },
    { "kind": "toggle", "row": 1, "col": 20, "width": 3, "text": "[ ]", "confidence": 1.0, "checked": false },
    { "kind": "button", "row": 2, "col": 2, "width": 6, "text": "[Save]", "confidence": 0.8 },
    { "kind": "button", "row": 2, "col": 10, "width": 8, "text": "[Cancel]", "confidence": 0.8 }
  ],
  "content_hash": 12345678901234567890
}

Use --format text for a plain text view with cursor indicator:

--- Terminal 80x24 | Cursor: (5, 10) ---
bash-3.2$ [_]

The [_] shows cursor position. Use the text content to understand screen state and navigate with keyboard commands.

---

Element Detection

pilotty automatically detects interactive UI elements in terminal applications. Elements provide read-only context to help understand UI structure.

Element Kinds

Element Fields

Confidence Levels

Wait for Screen Changes (Recommended)

Stop guessing sleep durations! Use --await-change to wait for the screen to actually update:

# Capture baseline hash
HASH=$(pilotty snapshot | jq '.content_hash')

# Perform action
pilotty key Enter

# Wait for screen to change (blocks until hash differs)
pilotty snapshot --await-change $HASH

# Or wait for screen to stabilize (for apps that render progressively)
pilotty snapshot --await-change $HASH --settle 100

Flags:

Why this is better than sleep:

• sleep 1 is a guess - too short causes race conditions, too long slows automation
• --await-change waits exactly as long as needed - no more, no less
• --settle handles apps that render progressively (show partial, then complete)

Waiting for Streaming AI Responses

When interacting with AI-powered TUIs (like opencode, etc.) that stream responses, you need a longer --settle time since the screen keeps updating as tokens arrive:

# 1. Capture hash before sending prompt
HASH=$(pilotty snapshot -s myapp | jq -r '.content_hash')

# 2. Type prompt and submit
pilotty type -s myapp "write me a poem about ai agents"
pilotty key -s myapp Enter

# 3. Wait for streaming response to complete
#    - Use longer settle (2-3s) since AI apps pause between chunks
#    - Extend timeout for long responses (60s+)
pilotty snapshot -s myapp --await-change "$HASH" --settle 3000 -t 60000

# 4. Response may be scrolled - scroll up if needed to see full output
pilotty scroll -s myapp up 10
pilotty snapshot -s myapp --format text

Key parameters for streaming:

• --settle 2000-3000: AI responses have pauses between chunks; 2-3 seconds ensures streaming is truly done
• -t 60000: Extend timeout beyond the 30s default for longer generations
• The settle timer resets on each screen change, so it naturally waits until streaming stops

Manual Change Detection

For manual polling (not recommended), use content_hash directly:

# Get initial state
SNAP1=$(pilotty snapshot)
HASH1=$(echo "$SNAP1" | jq -r '.content_hash')

# Perform action
pilotty key Tab

# Check if screen changed
SNAP2=$(pilotty snapshot)
HASH2=$(echo "$SNAP2" | jq -r '.content_hash')

if [ "$HASH1" != "$HASH2" ]; then
  echo "Screen changed - re-analyze elements"
fi

Using Elements Effectively

Elements are read-only context for understanding the UI. Use keyboard navigation for reliable interaction:

# 1. Get snapshot to understand UI structure
pilotty snapshot | jq '.elements'
# Output shows toggles (checked/unchecked) and buttons with positions

# 2. Navigate and interact with keyboard (reliable approach)
pilotty key Tab          # Move to next element
pilotty key Space        # Toggle checkbox
pilotty key Enter        # Activate button

# 3. Verify state changed
pilotty snapshot | jq '.elements[] | select(.kind == "toggle")'

Key insight: Use elements to understand WHAT is on screen, use keyboard to interact with it.

---

Navigation Approach

pilotty uses keyboard-first navigation, just like a human would:

# 1. Take snapshot to see the screen
pilotty snapshot --format text

# 2. Navigate using keyboard
pilotty key Tab           # Move to next element
pilotty key Enter         # Activate/select
pilotty key Escape        # Cancel/back
pilotty key Up            # Move up in list/menu
pilotty key Space         # Toggle checkbox

# 3. Type text when needed
pilotty type "search term"
pilotty key Enter

# 4. Click at coordinates for mouse-enabled TUIs
pilotty click 5 10        # Click at row 5, col 10

Key insight: Parse the snapshot text and elements to understand what's on screen, then use keyboard commands to navigate. This works reliably across all TUI applications.

---

Example: Edit file with vim

# 1. Spawn vim
pilotty spawn --name editor vim /tmp/hello.txt

# 2. Wait for vim to load and capture baseline hash
pilotty wait-for -s editor "hello.txt"
HASH=$(pilotty snapshot -s editor | jq '.content_hash')

# 3. Enter insert mode
pilotty key -s editor i

# 4. Type content
pilotty type -s editor "Hello from pilotty!"

# 5. Wait for screen to update, then exit (no sleep needed!)
pilotty snapshot -s editor --await-change $HASH --settle 50
pilotty key -s editor "Escape : w q Enter"

# 6. Verify session ended
pilotty list-sessions

Alternative using individual keys:

pilotty key -s editor Escape
pilotty type -s editor ":wq"
pilotty key -s editor Enter

Example: Dialog checklist interaction

# 1. Spawn dialog checklist (--name before command)
pilotty spawn --name opts dialog --checklist "Select features:" 12 50 4 \
    "notifications" "Push notifications" on \
    "darkmode" "Dark mode theme" off \
    "autosave" "Auto-save documents" on \
    "telemetry" "Usage analytics" off

# 2. Wait for dialog to render (use await-change, not sleep!)
pilotty snapshot -s opts --settle 200  # Wait for initial render to stabilize

# 3. Get snapshot and examine elements, capture hash
SNAP=$(pilotty snapshot -s opts)
echo "$SNAP" | jq '.elements[] | select(.kind == "toggle")'
HASH=$(echo "$SNAP" | jq '.content_hash')

# 4. Navigate to "darkmode" and toggle it
pilotty key -s opts Down      # Move to second option
pilotty key -s opts Space     # Toggle it on

# 5. Wait for change and verify
pilotty snapshot -s opts --await-change $HASH | jq '.elements[] | select(.kind == "toggle") | {text, checked}'

# 6. Confirm selection
pilotty key -s opts Enter

# 7. Clean up
pilotty kill -s opts

Example: Form filling with elements

# 1. Spawn a form application
pilotty spawn --name form my-form-app

# 2. Get snapshot to understand form structure
pilotty snapshot -s form | jq '.elements'
# Shows inputs, toggles, and buttons with positions for click command

# 3. Tab to first input (likely already focused)
pilotty type -s form "myusername"

# 4. Tab to password field
pilotty key -s form Tab
pilotty type -s form "mypassword"

# 5. Tab to remember me and toggle
pilotty key -s form Tab
pilotty key -s form Space

# 6. Tab to Login and activate
pilotty key -s form Tab
pilotty key -s form Enter

# 7. Check result
pilotty snapshot -s form --format text

Example: Monitor with htop

# 1. Spawn htop
pilotty spawn --name monitor htop

# 2. Wait for display
pilotty wait-for -s monitor "CPU"

# 3. Take snapshot to see current state
pilotty snapshot -s monitor --format text

# 4. Send commands
pilotty key -s monitor F9    # Kill menu
pilotty key -s monitor q     # Quit

# 5. Kill session
pilotty kill -s monitor

Example: Interact with AI TUI (opencode, etc.)

AI-powered TUIs stream responses, requiring special handling:

# 1. Spawn the AI app
pilotty spawn --name ai opencode

# 2. Wait for the prompt to be ready
pilotty wait-for -s ai "Ask anything" -t 15000

# 3. Capture baseline hash
HASH=$(pilotty snapshot -s ai | jq -r '.content_hash')

# 4. Type prompt and submit
pilotty type -s ai "explain the architecture of this codebase"
pilotty key -s ai Enter

# 5. Wait for streaming response to complete
#    - settle=3000: Wait 3s of no changes to ensure streaming is done
#    - timeout=60000: Allow up to 60s for long responses
pilotty snapshot -s ai --await-change "$HASH" --settle 3000 -t 60000 --format text

# 6. If response is long and scrolled, scroll up to see full output
pilotty scroll -s ai up 20
pilotty snapshot -s ai --format text

# 7. Clean up
pilotty kill -s ai

Gotchas with AI apps:

• Use --settle 2000-3000 because AI responses pause between chunks
• Extend timeout with -t 60000 for complex prompts
• Long responses may scroll the terminal; use scroll up to see the beginning
• The settle timer resets on each screen update, so it waits for true completion

---

Sessions

Each session is isolated with its own:

• PTY (pseudo-terminal)
• Screen buffer
• Child process

# Run multiple apps (--name must come before the command)
pilotty spawn --name monitoring htop
pilotty spawn --name editor vim file.txt

# Target specific session
pilotty snapshot -s monitoring
pilotty key -s editor Ctrl+S

# List all
pilotty list-sessions

# Kill specific
pilotty kill -s editor

The first session spawned without --name is automatically named default.

Important: The --name flag must come before the command. Everything after the command is passed as arguments to that command.

Daemon Architecture

pilotty uses a background daemon for session management:

• Auto-start: Daemon starts on first command
• Auto-stop: Shuts down after 5 minutes with no sessions
• Session cleanup: Sessions removed when process exits (within 500ms)
• Shared state: Multiple CLI calls share sessions

You rarely need to manage the daemon manually.

Error Handling

Errors include actionable suggestions:

{
  "code": "SESSION_NOT_FOUND",
  "message": "Session 'abc123' not found",
  "suggestion": "Run 'pilotty list-sessions' to see available sessions"
}

{
  "code": "SPAWN_FAILED",
  "message": "Failed to spawn process: command not found",
  "suggestion": "Check that the command exists and is in PATH"
}

---

Common Patterns

Reliable action + wait (recommended)

# The pattern: capture hash, act, await change
HASH=$(pilotty snapshot | jq '.content_hash')
pilotty key Enter
pilotty snapshot --await-change $HASH --settle 50

# This replaces fragile patterns like:
# pilotty key Enter && sleep 1 && pilotty snapshot  # BAD: guessing

Wait then act

pilotty spawn my-app
pilotty wait-for "Ready"    # Ensure app is ready
pilotty snapshot            # Then snapshot

Check state before action

pilotty snapshot --format text | grep "Error"  # Check for errors
pilotty key Enter                               # Then proceed

Check for specific element

# Check if the first toggle is checked
pilotty snapshot | jq '.elements[] | select(.kind == "toggle") | {text, checked}' | head -1

# Find element at specific position
pilotty snapshot | jq '.elements[] | select(.row == 5 and .col == 10)'

Retry on timeout

pilotty wait-for "Ready" -t 5000 || {
  pilotty snapshot --format text   # Check what's on screen
  # Adjust approach based on actual state
}

---

Deep-dive Documentation

For detailed patterns and edge cases, see:

Ready-to-use Templates

Executable workflow scripts:

Usage:

./templates/vim-workflow.sh /tmp/myfile.txt "File content here"
./templates/dialog-interaction.sh
./templates/multi-session.sh
./templates/element-detection.sh

Element Detection

pilotty automatically detects interactive UI elements in terminal applications. Elements provide read-only context to help agents understand UI structure.

Overview

pilotty analyzes terminal screen content and detects:

• Toggles: Checkboxes like [x], [ ], [*], ☑, ☐
• Buttons: Action elements like [OK], <Cancel>, (Submit)
• Inputs: Text fields marked by underscores ____ or cursor position

Each detected element includes:

• Kind, position (row, col), width, text content
• Confidence score (0.0-1.0)
• State information (checked for toggles, focused for inputs/buttons)

Detection Rules

Priority Order (Highest to Lowest)

1. Cursor Position - Input (confidence: 1.0, focused: true) 2. Checkbox Patterns - Toggle (confidence: 1.0) 3. Inverse Video - Button (confidence: 1.0, focused: true) 4. Bracket Patterns - Button (confidence: 0.8) 5. Underscore Fields - Input (confidence: 0.6)

Toggle Detection

Toggles are detected from checkbox patterns:

Example detection:

{
  "kind": "toggle",
  "row": 5,
  "col": 2,
  "width": 3,
  "text": "[x]",
  "confidence": 1.0,
  "checked": true
}

Button Detection

Buttons are detected from:

1. Inverse video (highest confidence)

• Text with reversed foreground/background colors
• Common in dialog, whiptail, and ncurses apps
• Confidence: 1.0, focused: true

2. Bracket patterns (medium confidence)

• Square brackets: [OK], [Cancel], [Save]
• Angle brackets: <Yes>, <No>
• Parentheses: (Submit), (Reset)
• Confidence: 0.8

Example detection:

{
  "kind": "button",
  "row": 10,
  "col": 5,
  "width": 6,
  "text": "[Save]",
  "confidence": 0.8
}

Input Detection

Inputs are detected from:

1. Cursor position (highest confidence)

• The cell where the cursor is located
• Confidence: 1.0, focused: true

2. Underscore runs (lower confidence)

• 3+ consecutive underscores: ___, __________
• Common in form-style TUIs
• Confidence: 0.6

Example detection:

{
  "kind": "input",
  "row": 8,
  "col": 12,
  "width": 10,
  "text": "__________",
  "confidence": 0.6
}

Non-Interactive Patterns (Filtered)

The following patterns are recognized but NOT returned as interactive elements:

Element Fields Reference

Width Calculation

Element width uses Unicode display width:

• ASCII characters: width 1
• CJK characters (Chinese, Japanese, Korean): width 2
• Emoji: width 2
• Zero-width characters: width 0

This matches terminal column alignment.

Content Hash

Each snapshot includes a content_hash field for change detection:

{
  "content_hash": 12345678901234567890,
  ...
}

The hash is computed from the visible screen text content. Use it to:

• Detect if the screen changed between snapshots
• Avoid re-processing unchanged screens

HASH1=$(pilotty snapshot | jq -r '.content_hash')
pilotty key Tab
HASH2=$(pilotty snapshot | jq -r '.content_hash')
[ "$HASH1" != "$HASH2" ] && echo "Screen changed"

Best Practices

1. Elements for Understanding, Keyboard for Interaction

Elements tell you WHAT is on screen. Use keyboard to interact:

# See what's on screen
pilotty snapshot | jq '.elements[] | {kind, text, row, col, checked}'

# Navigate with keyboard
pilotty key Tab    # Move between elements
pilotty key Space  # Toggle checkboxes
pilotty key Enter  # Activate buttons

2. Check Confidence Levels

Higher confidence means more reliable detection:

# Filter to high-confidence elements only
pilotty snapshot | jq '.elements[] | select(.confidence >= 0.8)'

3. Find Elements by Content or Position

# Find element by text content
pilotty snapshot | jq '.elements[] | select(.text | contains("Save"))'

# Find element at specific position
pilotty snapshot | jq '.elements[] | select(.row == 5 and .col == 10)'

# Get first toggle
pilotty snapshot | jq '[.elements[] | select(.kind == "toggle")][0]'

Limitations

What Detection Does NOT Find

1. Menu items without markers - Plain text menus need keyboard navigation 2. Custom widgets - Non-standard UI patterns may not be recognized 3. Color-only highlighting - Elements must have text patterns or inverse video 4. Disabled elements - No distinction between enabled/disabled

What Detection Cannot Do

1. Click elements directly by name - Use row/col with click command 2. Track elements across screens - Elements may move; use text content to re-find

Troubleshooting

No Elements Detected

1. Check if the app uses standard patterns:

   pilotty snapshot --format text  # View raw screen

2. Look for inverse video (may show elements on button/input):

   pilotty snapshot | jq '.elements[] | select(.confidence == 1.0)'

Wrong Element Kind

The classifier uses heuristics. If [x] is detected as a button instead of toggle: 1. Check for surrounding context 2. Use text field to identify element purpose

Elements Missing After Action

Element positions may change between snapshots. Track elements by:

• Text content (most reliable)
• Element kind
• Approximate row/column position

Example: Complete Workflow

#!/bin/bash
SESSION="form"

# 1. Spawn application
pilotty spawn --name $SESSION dialog --checklist "Options:" 15 50 4 \
    "opt1" "Feature A" on \
    "opt2" "Feature B" off \
    "opt3" "Feature C" on \
    "opt4" "Feature D" off

sleep 0.5

# 2. Analyze initial state
echo "Initial state:"
pilotty snapshot -s $SESSION | jq '.elements[] | select(.kind == "toggle") | {text, checked}'

# 3. Find unchecked toggles
UNCHECKED=$(pilotty snapshot -s $SESSION | jq '[.elements[] | select(.kind == "toggle" and .checked == false)] | length')
echo "Unchecked toggles: $UNCHECKED"

# 4. Navigate and toggle opt2
pilotty key -s $SESSION Down   # Move to opt2
pilotty key -s $SESSION Space  # Toggle it

# 5. Verify change via content_hash
HASH1=$(pilotty snapshot -s $SESSION | jq -r '.content_hash')
echo "Hash after toggle: $HASH1"

# 6. Confirm and check final state
pilotty key -s $SESSION Enter
sleep 0.3

echo "Final state:"
pilotty snapshot -s $SESSION | jq '.elements[] | select(.kind == "toggle") | {text, checked}'

# 7. Cleanup
pilotty kill -s $SESSION

Key Input Reference

Complete reference for key combinations supported by pilotty key.

Basic Usage

pilotty key <key>                 # Send single key to default session
pilotty key -s myapp <key>        # Send to specific session
pilotty key "key1 key2 key3"      # Send key sequence (space-separated)
pilotty key "key1 key2" --delay 50  # Sequence with 50ms delay between keys

Named Keys

Arrow Keys

Navigation Keys

Function Keys

Modifier Combinations

Ctrl Combinations

Alt Combinations

Shift Combinations

Combined Modifiers

Special Characters

Key Sequences

Send multiple keys in order with a single command. Keys are space-separated:

# Emacs-style chords
pilotty key "Ctrl+X Ctrl+S"       # Save file
pilotty key "Ctrl+X Ctrl+C"       # Exit Emacs
pilotty key "Ctrl+X m"            # Compose mail

# vim command sequences
pilotty key "Escape : w q Enter"  # Save and quit
pilotty key "Escape : q ! Enter"  # Quit without saving
pilotty key "g g d G"             # Delete entire file

# Navigation sequences
pilotty key "Tab Tab Enter"       # Tab twice then Enter
pilotty key "Down Down Space"     # Move down twice and select

Inter-key Delay

Use --delay for TUIs that need time between keys:

pilotty key "Tab Tab Enter" --delay 100   # 100ms between each key
pilotty key "F9 Down Enter" --delay 50    # htop kill menu navigation

When to Use Sequences vs Individual Keys

Use sequences for:

• Emacs/vim chords that must be sent together
• Predictable navigation patterns
• Reducing command overhead

Use individual keys when:

• You need to check screen state between keys
• Timing is unpredictable
• Different paths based on UI state

Common TUI Patterns

Dialog/Whiptail

pilotty key Tab       # Move between buttons
pilotty key Enter     # Activate button
pilotty key Space     # Toggle checkbox
pilotty key Escape    # Cancel dialog

Vim

pilotty key i         # Insert mode (use pilotty type for text)
pilotty key Escape    # Normal mode
pilotty key Ctrl+C    # Also exits insert mode
pilotty type ":wq"    # Command (then Enter)
pilotty key Enter

# Using sequences for common operations
pilotty key "Escape : w q Enter"     # Save and quit
pilotty key "Escape : q ! Enter"     # Force quit
pilotty key "Escape d d"             # Delete line
pilotty key "Escape g g"             # Go to top

Htop

pilotty key F1        # Help
pilotty key F2        # Setup
pilotty key F5        # Tree view
pilotty key F9        # Kill process
pilotty key F10       # Quit
pilotty key q         # Also quit

Less/More

pilotty key Space     # Page down
pilotty key b         # Page up
pilotty key q         # Quit
pilotty key /         # Search (then type pattern)
pilotty key n         # Next match
pilotty key N         # Previous match

Nano

pilotty key Ctrl+O    # Save
pilotty key Ctrl+X    # Exit
pilotty key Ctrl+K    # Cut line
pilotty key Ctrl+U    # Paste
pilotty key Ctrl+W    # Search

# Using sequences
pilotty key "Ctrl+O Enter"    # Save with default filename
pilotty key "Ctrl+X n"        # Exit without saving (answer 'n' to save prompt)

Tmux (default prefix)

pilotty key Ctrl+B    # Prefix key
# Then send the command key:
pilotty key c         # New window
pilotty key n         # Next window
pilotty key p         # Previous window
pilotty key d         # Detach

# Using sequences for tmux commands
pilotty key "Ctrl+B c"    # Prefix + new window
pilotty key "Ctrl+B n"    # Prefix + next window
pilotty key "Ctrl+B d"    # Prefix + detach

Readline/Bash

pilotty key Ctrl+A    # Beginning of line
pilotty key Ctrl+E    # End of line
pilotty key Ctrl+U    # Clear line
pilotty key Ctrl+R    # Reverse search
pilotty key Ctrl+L    # Clear screen
pilotty key Up        # Previous history
pilotty key Down      # Next history

Case Sensitivity

• Named keys are case-insensitive: Enter, ENTER, enter all work
• Letter keys with Ctrl/Alt are case-insensitive: Ctrl+c = Ctrl+C
• Plain letters: Use pilotty type for text, not pilotty key

Escaping

The + character is the modifier separator. To type a literal +:

pilotty key Plus      # Sends the + character
# Or use type for text:
pilotty type "2+2"    # Types "2+2"

Troubleshooting

Key Not Recognized

# Check if it's a named key or text
pilotty key Enter     # Named key
pilotty type "hello"  # Text input

Modifier Not Working

Some apps intercept modifiers before the terminal sees them. Try:

# Check raw terminal behavior
pilotty spawn cat
pilotty key Ctrl+C    # Should show ^C or exit

Timing Issues

Some TUIs need time to process input:

pilotty key F9        # Opens menu
pilotty wait-for "SIGTERM"  # Wait for menu
pilotty key Enter     # Then select

Session Management

pilotty manages multiple isolated terminal sessions, each running its own TUI application with independent state.

CRITICAL: Flag Positioning

All flags MUST come BEFORE positional arguments. This applies to --name, -s/--session, and all other options:

# CORRECT
pilotty spawn --name myapp vim file.txt
pilotty key -s myapp Enter
pilotty snapshot -s myapp --format text

# WRONG - flags after positional args get passed to the command, not pilotty
pilotty spawn vim file.txt --name myapp   # --name goes to vim, session uses "default"
pilotty key Enter -s myapp                # -s is ignored, targets wrong session

---

Session Basics

Each session has:

• PTY: Pseudo-terminal for the application
• Screen buffer: Terminal emulator state
• Child process: The running application

Creating Sessions

Default Session

The first spawn without --name creates the default session:

pilotty spawn htop
# Creates session named "default"

pilotty snapshot
# Snapshots the "default" session

Named Sessions

Use --name for multiple concurrent sessions. Note: --name must come before the command:

pilotty spawn --name monitoring htop
pilotty spawn --name editor vim file.txt
pilotty spawn --name git lazygit

Session Naming Rules

• Names are sanitized (alphanumeric, hyphens, underscores)
• Path traversal attempts (../) are rejected
• Names must be unique per daemon instance

Targeting Sessions

Use -s or --session to target a specific session:

# Snapshot specific session
pilotty snapshot -s monitoring

# Send key to specific session
pilotty key -s editor Ctrl+S

# Send key in specific session
pilotty key -s git Enter

# Kill specific session
pilotty kill -s monitoring

Without -s, commands target the most recently used session (or default).

Listing Sessions

pilotty list-sessions

Output:

{
  "sessions": [
    { "id": "abc123", "name": "monitoring", "command": "htop" },
    { "id": "def456", "name": "editor", "command": "vim file.txt" },
    { "id": "ghi789", "name": "git", "command": "lazygit" }
  ]
}

Session Lifecycle

Spawn

pilotty spawn --name myapp my-command arg1 arg2

1. Daemon creates PTY 2. Forks child process with command 3. Initializes terminal emulator (default: 80x24) 4. Returns session ID

Active Use

While a session is active:

• Screen buffer updates on process output
• Cursor position is tracked
• Terminal size can be changed with resize

Process Exit

When the child process exits:

• Session is marked for cleanup
• Cleanup happens within 500ms
• Session is removed from list

Manual Kill

pilotty kill -s myapp

Sends SIGTERM to the child process, then cleans up.

Multi-Session Patterns

Parallel Monitoring

Run multiple apps and switch between them:

# Start apps (--name before command)
pilotty spawn --name cpu htop
pilotty spawn --name io iotop
pilotty spawn --name net nethogs

# Check each
pilotty snapshot -s cpu --format text
pilotty snapshot -s io --format text
pilotty snapshot -s net --format text

# Clean up
pilotty kill -s cpu
pilotty kill -s io
pilotty kill -s net

Editor + Preview

Edit a file while watching output:

# Start editor (--name before command)
pilotty spawn --name editor vim main.py

# Start file watcher
pilotty spawn --name preview watch -n1 python main.py

# Edit
pilotty key -s editor i
pilotty type -s editor "print('hello')"
pilotty key -s editor Escape
pilotty type -s editor ":w"
pilotty key -s editor Enter

# Check preview
pilotty snapshot -s preview --format text

Pipeline Workflow

Sequential operations across sessions:

# Setup (--name before command)
pilotty spawn --name worker bash

# Run commands
pilotty type -s worker "curl -s https://api.example.com > data.json"
pilotty key -s worker Enter
pilotty wait-for -s worker "$"  # Wait for prompt

pilotty type -s worker "jq '.items[]' data.json"
pilotty key -s worker Enter
pilotty wait-for -s worker "$"

# Get output
pilotty snapshot -s worker --format text

Session Isolation

Sessions are fully isolated:

• Separate PTY file descriptors
• Independent screen buffers
• Independent cursor positions
• No shared state between sessions

This means:

• Killing session A doesn't affect session B
• Each session can have different terminal sizes
• Snapshots from one session don't affect others

Daemon Lifecycle

The daemon manages all sessions:

Auto-Start

The daemon starts automatically on the first command:

pilotty spawn vim  # Starts daemon if not running

Auto-Stop

After 5 minutes with no active sessions, the daemon shuts down automatically.

Manual Control

pilotty daemon     # Manually start daemon
pilotty stop       # Stop daemon and all sessions

Socket Location

The daemon creates a Unix socket at (in priority order):

1. $PILOTTY_SOCKET_DIR/pilotty.sock 2. $XDG_RUNTIME_DIR/pilotty/pilotty.sock 3. ~/.pilotty/pilotty.sock 4. /tmp/pilotty/pilotty.sock

Environment Variables

Example:

export PILOTTY_SESSION=editor
pilotty snapshot  # Targets "editor" session without -s flag

Error Handling

Session Not Found

{
  "code": "SESSION_NOT_FOUND",
  "message": "Session 'myapp' not found",
  "suggestion": "Run 'pilotty list-sessions' to see available sessions"
}

Session Already Exists

Attempting to spawn with a name that's already in use:

{
  "code": "SESSION_EXISTS",
  "message": "Session 'myapp' already exists",
  "suggestion": "Use a different name or kill the existing session first"
}

Best Practices

1. Put --name before command: pilotty spawn --name myapp cmd (not after) 2. Use meaningful names: --name editor is better than --name s1 3. Clean up when done: Kill sessions you're finished with 4. Don't rely on default: For multi-session work, always name your sessions 5. Check session exists: Use list-sessions before targeting 6. Handle process exit: Sessions auto-cleanup, but check if your command is still running