Heterogeneous Agent

Heterogeneous Agent Development

Use this skill when the bug or feature lives in the external CLI agent pipeline, not the normal server-side agent runtime.

Use This Skill For

• Adding or changing a driver under apps/desktop/src/main/modules/heterogeneousAgent/drivers/
• Editing an adapter under packages/heterogeneous-agents/src/adapters/
• Debugging heteroAgentRawLine transport, window.__HETERO_AGENT_TRACE, or executeHeterogeneousAgent
• Fixing Claude Code stream-json bugs such as duplicate partial/full chunks, broken message.id boundaries, missing tool_result, TodoWrite state drift, or subagent thread routing
• Fixing Codex JSONL bugs such as mixed multi-tool messages, broken turn boundaries, or missing tool-result mapping
• Fixing step-boundary, tool persistence, subagent thread, or resume bugs in Claude Code / Codex flows
• Reproducing multi-tool mixing, orphan tool messages, or stuck tool-result loading

Pipeline Map

1. CLI raw stdout / JSONL 2. Electron main spawns the CLI and broadcasts heteroAgentRawLine 3. Adapter maps raw provider events into HeterogeneousAgentEvent 4. executeHeterogeneousAgent persists assistant/tool messages and forwards stream events 5. createGatewayEventHandler hydrates the UI 6. Only after this path looks correct should you move on to agent-tracing or context-engine debugging

Read These Files First

• apps/desktop/src/main/controllers/HeterogeneousAgentCtr.ts
• apps/desktop/src/main/modules/heterogeneousAgent/drivers/claudeCode.ts
• apps/desktop/src/main/modules/heterogeneousAgent/drivers/codex.ts
• packages/heterogeneous-agents/src/adapters/claudeCode.ts
• packages/heterogeneous-agents/src/adapters/codex.ts
• src/store/chat/slices/aiChat/actions/heterogeneousAgentExecutor.ts
• src/store/chat/slices/aiChat/actions/__tests__/heterogeneousAgentExecutor.test.ts

Default Debug Order

1. Prove whether the raw CLI output is correct before touching UI code. 2. If raw output is correct, compare it with adapter output. In dev, executeHeterogeneousAgent exposes window.__HETERO_AGENT_TRACE. 3. If adapted events look correct, inspect persistToolBatch, persistToolResult, step transitions, and subagent routing. 4. Turn the repro into a focused test before fixing. 5. Only after the transport/adapter/executor path looks sound should you debug later-stage message processing.

Critical Invariants

• One raw tool item must map to one stable ToolCallPayload.id.
• A new main-agent step must emit a boundary signal before events are forwarded to the new assistant.
• In Claude Code, multiple assistant events with the same message.id are one turn, not multiple turns.
• In Claude Code, tool_result lives in type: 'user' events, not assistant events.
• In Claude Code partial mode, message_delta.usage is authoritative; do not trust echoed usage on every assistant block.
• persistToolBatch must pre-register assistant tools[] before creating tool messages.
• Every tool message must keep parentId equal to the owning assistant and tool_call_id equal to the tool id.
• tool_result must resolve an existing toolMsgIdByCallId.
• Subagent chunks must stay in thread scope and must not be forwarded into the main assistant stream.
• Never clear the global toolMsgIdByCallId map at main step boundaries.

Common Bug Patterns

• Claude Code duplicates text or thinking:

check whether partial deltas and the later full assistant block are both being emitted.

• Claude Code opens too many assistant messages:

check whether the adapter is cutting steps on every assistant event instead of only on message.id changes.

• Claude Code tool results never land:

check whether type: 'user' tool_result blocks are being ignored because the code only inspects assistant events.

• Claude Code TodoWrite cards look stale:

check whether synthesized pluginState.todos is being attached at tool-result time.

• Claude Code subagent transcript leaks into the main bubble:

check parent_tool_use_id handling and whether subagent chunks are being forwarded to the main gateway handler.

• Multiple Codex tools collapse into one assistant message:

first check whether the adapter emits a usable step boundary such as newStep or an equivalent turn-change signal.

• Orphan tool messages:

first check step-transition ordering and whether persistToolBatch Phase 1 ran before tool message creation.

• Tool bubble stays loading:

look for tool_result for unknown toolCallId and missing result_msg_id backfill.

• Subagent tools show up in the main bubble:

check for subagent chunks reaching the main gateway handler.

References

• For commands, trace capture, invariants, and focused test commands, read references/debug-workflow.md.

Heterogeneous Agent Debug Workflow

Contents

1. Pipeline map 2. Capture raw CLI traces first 3. Compare raw and adapted events 4. Check step boundaries before persistence 5. Check tool persistence invariants 6. Focused tests 7. Repro-to-fix workflow

1. Pipeline Map

CLI raw stdout
  -> HeterogeneousAgentCtr (Electron main)
  -> heteroAgentRawLine broadcast
  -> createAdapter(...)
  -> executeHeterogeneousAgent(...)
  -> persistToolBatch / persistToolResult
  -> createGatewayEventHandler(...)
  -> UI hydration

Start at the leftmost broken layer. Do not jump straight to UI rendering unless raw and adapted events already look correct.

2. Capture Raw CLI Traces First

Codex raw JSONL

Use a read-only prompt and save traces under the repo-local scratch directory .heerogeneous-tracing/.

ts=$(date +%Y%m%d-%H%M%S)
out=".heerogeneous-tracing/codex-${ts}.jsonl"
last=".heerogeneous-tracing/codex-${ts}.last.txt"

cat << 'EOF' | codex exec --json --skip-git-repo-check --sandbox read-only -C "$PWD" -o "$last" - > "$out"
You are being run only to collect a raw Codex JSON event trace.
Do not modify any files.
Use at least 4 separate shell tool invocations, one invocation per command.
Run a short sequence of read-only repo checks and then reply with a one-sentence summary.
EOF

What to look for in the JSONL:

• thread.started
• turn.started
• item.started / item.completed
• item.type === 'command_execution'
• item.type === 'agent_message'
• turn.completed

If raw Codex already merges tools into one item, the adapter is innocent. If raw Codex emits independent items but UI collapses them, the bug is downstream.

If the repo already contains useful traces under .heerogeneous-tracing/, inspect them before reproducing.

Claude Code raw NDJSON

Mirror the arguments from apps/desktop/src/main/modules/heterogeneousAgent/drivers/claudeCode.ts.

• -p
• --input-format stream-json
• --output-format stream-json
• --verbose
• --include-partial-messages
• --permission-mode bypassPermissions

You can capture a local raw trace like this:

ts=$(date +%Y%m%d-%H%M%S)
out=".heerogeneous-tracing/claude-${ts}.ndjson"

cat << 'EOF' | claude -p \
  --input-format stream-json \
  --output-format stream-json \
  --verbose \
  --include-partial-messages \
  --permission-mode bypassPermissions \
  > "$out"
{"type":"user","message":{"role":"user","content":[{"type":"text","text":"Do a few read-only repo checks, use several tool calls, and then summarize briefly."}]}}
EOF

What to look for in Claude Code raw traces:

• type: 'system', subtype: 'init'
• type: 'assistant' blocks for thinking, tool_use, and text
• type: 'user' blocks containing tool_result
• type: 'stream_event' with message_start, content_block_delta, and message_delta
• type: 'result'
• type: 'rate_limit_event'

Important Claude Code semantics:

• Each content block often arrives as its own assistant event.
• Multiple assistant events can share the same message.id; that is still one turn.
• message.id change is the main-step boundary.
• Partial deltas arrive before the later full assistant block.
• message_delta.usage is the authoritative per-turn usage.
• Subagent events are tagged with parent_tool_use_id.

If the repo already contains useful references, inspect these first:

• .heerogeneous-tracing/cc-monitor-real-trace.jsonl
• .heerogeneous-tracing/cc-stream-chain-reference.md

If you only need boundary semantics or tool persistence behavior, prefer existing adapter tests under:

• packages/heterogeneous-agents/src/adapters/claudeCode.test.ts
• packages/heterogeneous-agents/src/adapters/claudeCode.e2e.test.ts

3. Compare Raw And Adapted Events

In dev builds, executeHeterogeneousAgent stores raw lines plus adapted events on:

• window.__HETERO_AGENT_TRACE

Use that trace to compare:

• raw item.started / item.completed
• adapted stream_chunk { chunkType: 'tools_calling' }
• adapted tool_result
• adapted tool_end

For Codex, the usual mapping is:

• raw item.started(command_execution) -> tools_calling + tool_start
• raw item.completed(command_execution) -> tool_result + tool_end
• raw item.completed(agent_message) -> stream_chunk(text)

If the raw trace is right but adapted events are wrong, fix the adapter before touching persistence.

4. Check Step Boundaries Before Persistence

This is the first thing to verify for "mixed tools in one assistant" bugs.

Claude Code

Claude Code step boundaries are keyed off assistant message.id changes. The adapter should emit:

• stream_end
• stream_start { newStep: true }

Also verify these Claude-specific invariants:

• the first assistant after init does not open a new step
• repeated assistant events with the same message.id do not open a new step
• partial content_block_delta text/thinking does not get duplicated by the later full assistant event
• tool_result from type: 'user' updates the matching tool row
• parent_tool_use_id creates thread-scoped subagent chunks instead of main-stream chunks
• TodoWrite tool_use.input is converted into synthesized pluginState.todos on tool_result

Good references:

• packages/heterogeneous-agents/src/adapters/claudeCode.ts
• packages/heterogeneous-agents/src/adapters/claudeCode.test.ts

Codex

Codex raw traces usually provide turn-level boundaries through:

• turn.started
• turn.completed

The executor only cuts a new assistant message when it receives a step-boundary signal it understands. If the adapter emits stream_start without newStep, multiple Codex tools and text chunks can accumulate under the same assistant longer than intended.

Relevant files:

• packages/heterogeneous-agents/src/adapters/codex.ts
• src/store/chat/slices/aiChat/actions/heterogeneousAgentExecutor.ts

5. Check Tool Persistence Invariants

Read persistToolBatch and persistToolResult before changing UI code.

persistToolBatch

The expected order is:

1. Pre-register assistant tools[] 2. Create role: 'tool' messages 3. Backfill result_msg_id onto assistant tools[]

If tool rows are created before assistant tools[] are registered, orphan tool messages are likely.

persistToolResult

tool_result must resolve the tool row through toolMsgIdByCallId.

Warning signs:

• tool_result for unknown toolCallId
• tool rows with empty content forever
• missing result_msg_id

For Claude Code, remember that tool results originate from raw type: 'user' events.

Main vs subagent scope

• Main-agent tool state is per-step.
• toolMsgIdByCallId is global across main and subagent scopes.
• Subagent chunks must not be forwarded into the main gateway handler.

If subagent events leak to the main handler, the main bubble can inherit the wrong tools[] and content.

6. Focused Tests

Run the smallest useful test set first.

bunx vitest run --silent='passed-only' 'packages/heterogeneous-agents/src/adapters/codex.test.ts'
bunx vitest run --silent='passed-only' 'packages/heterogeneous-agents/src/adapters/claudeCode.test.ts'
bunx vitest run --silent='passed-only' 'src/store/chat/slices/aiChat/actions/__tests__/heterogeneousAgentExecutor.test.ts'

Especially useful places:

• packages/heterogeneous-agents/src/adapters/codex.test.ts
• packages/heterogeneous-agents/src/adapters/claudeCode.test.ts
• src/store/chat/slices/aiChat/actions/__tests__/heterogeneousAgentExecutor.test.ts

Claude Code-specific assertions worth adding when fixing bugs:

• same message.id does not emit newStep
• changed message.id does emit stream_end plus stream_start { newStep: true }
• partial text/thinking is emitted once
• tool_result from user events reaches the right tool row
• subagent chunks carry subagent.parentToolCallId
• TodoWrite result synthesizes pluginState.todos

When the bug comes from a real trace, distill it into the closest existing test file instead of relying on manual UI-only repros.

7. Repro-To-Fix Workflow

1. Capture a raw trace and save it under .heerogeneous-tracing/. 2. Confirm whether the bug appears in raw events, adapted events, or persistence. 3. Add or update the narrowest failing test near the broken layer. 4. Fix the smallest layer that can explain the symptom. 5. Re-run focused tests. 6. Only then do an Electron smoke test with the agent-testing skill if UI confirmation is still needed.

Do not start with a broad Electron repro if a raw trace or adapter test can prove the fault zone faster.