Multi Agent System Engineer

Multi-Agent System Engineer

When to Use

• Designing multi-agent topology: supervisor, hierarchical, peer-to-peer, or blackboard
• Decomposing work across specialized agents with routing, delegation, and merge rules
• Defining inter-agent message schemas, handoff payloads, and protocol boundaries
• Partitioning vs sharing state, scratchpads, artifacts, and consensus across agents
• Building fan-out/fan-in, DAG workflows, and synchronization barriers in agent graphs
• Resolving conflicts when agents disagree or duplicate work
• Engineering fault tolerance: retries, partial failure, compensation, and saga-style recovery
• Setting system-level budgets: tokens, latency, parallelism, and cost per workflow run
• Observability across agent traces, correlation IDs, and multi-step workflow debugging
• Testing and simulating multi-agent flows before production
• Deploying multi-agent runtimes on queues, durable workflows, and scaled workers

When NOT to Use

• Single-agent loop, tools, MCP, checkpoints, and one runtime only → agentic-ai-developer
• Foundation model training, fine-tuning, classical ML pipelines → ai-engineer, ai-researcher
• AI ops cadence, vendor contracts, rollout governance without system design → ai-lead-ops
• Internal developer platform, golden paths, portals—no agent orchestration → platform-engineer
• Cross-team milestones, RAID, program status without agent architecture → technical-program-manager
• Corporate AI policy, risk tiering, model cards without system build → ai-risk-governance
• Pre-flight go/no-go or architecture review without implementing topology → build-validator
• Enterprise strategy, portfolio, and org design whiteboard only → enterprise-strategist

Related skills

Core Workflows

1. Frame the multi-agent system

1. Define the end-to-end job, success metric, and SLA (latency, cost, quality) 2. List agents by role (planner, executor, critic, specialist)—not by model name 3. Choose topology and justify: supervisor, hierarchical, P2P, blackboard, or hybrid 4. Map trust boundaries: which agent may call which tools and external systems 5. Set system budgets: max parallel agents, tokens per run, wall time, dollars per task

See `references/multi_agent_system_engineer_scope.md` for scope, deliverables, and boundaries vs `agentic-ai-developer`.

2. Topology, roles, and routing

ingress → router/supervisor → {workers} → reducer/merger → egress

Checklist:

• [ ] Each agent has one primary responsibility and explicit inputs/outputs
• [ ] Routing rules are deterministic where safety matters; LLM routing elsewhere is logged
• [ ] Fan-out has a matching fan-in with merge semantics (vote, concat, structured reduce)
• [ ] Dangerous tools are centralized or gated—not duplicated on every worker

See `references/agent_roles_topology_and_routing.md` for topology patterns and routing tables.

3. Protocols and messaging

• Define message envelope: correlation_id, from, to, intent, payload, artifacts, constraints
• Version schemas; reject unknown versions at boundaries
• Prefer structured payloads over free-text handoffs for machine agents
• Document idempotency keys for retried messages

See `references/inter_agent_protocols_and_messaging.md` for handoff contracts and A2A-style patterns.

4. State, coordination, and consensus

• Classify state: ephemeral scratchpad, workflow state, shared blackboard, durable store
• Partition tenant and thread keys on every read/write
• Use barriers or quorum when parallel agents must align before the next phase
• Resolve conflicts with explicit policy: supervisor wins, vote, or escalate to human

See `references/shared_state_coordination_and_consensus.md` for blackboard vs partitioned models.

5. Fault tolerance, observability, and testing

• Retry at message and workflow level with caps; distinguish transient vs terminal errors
• Compensate or mark partial success; never leave workflows stuck without timeout
• Trace: one workflow_run_id spanning all agent spans; redact secrets in cross-agent logs
• Test: unit agents, pairwise handoffs, full DAG golden paths, chaos on one worker

See `references/fault_tolerance_observability_and_testing.md` for test matrices and SLOs.

6. Deployment, cost, and governance

• Short synchronous graphs for interactive UX; queue or durable engine for long DAGs
• Scale workers horizontally; pin graph version and agent config per deployment
• Attribute cost per agent step; alert on budget burn rate
• Gate releases on multi-agent regression suite and policy checks

See `references/deployment_cost_and_governance.md` for queue vs durable workflow tradeoffs.

When to load references

Framework pointers (optional)

Use framework docs for API specifics; this skill stays pattern-first:

Do not duplicate full framework tutorials—encode the system contracts (topology, messages, state, failure) in the stack the team chose.

Routing vs agentic-ai-developer

Agent roles, topology, and routing

Table of contents

1. Agent roles 2. Topology patterns 3. Routing strategies 4. Fan-out and fan-in 5. DAG workflows 6. Anti-patterns

Agent roles

Define roles by responsibility, not model SKU:

Rules:

• One primary role per agent instance in a given workflow version
• Specialists should not also route—separation reduces prompt injection surface
• Critics should not have write tools to production systems unless explicitly required

Topology patterns

Supervisor (hub-and-spoke)

        ┌─────────────┐
        │ Supervisor  │
        └──────┬──────┘
     ┌─────────┼─────────┐
     ▼         ▼         ▼
  Worker A  Worker B  Worker C

• Use when: Clear delegation, centralized policy, need single point for HITL
• Risks: Supervisor bottleneck, single point of failure—replicate supervisor logic in code where possible

Hierarchical

 CEO agent
    ├── Manager A → workers
    └── Manager B → workers

• Use when: Large task trees, org-like decomposition, staged planning
• Risks: Deep trees burn tokens; cap depth and require managers to emit structured sub-plans only

Peer-to-peer (P2P)

 Agent A ←→ Agent B ←→ Agent C

• Use when: Negotiation, debate, iterative refinement between equals
• Risks: Non-termination—set max rounds and explicit stop tokens

Blackboard

 Agents read/write shared structured board

• Use when: Multiple specialists contribute to one evolving artifact (report, plan)
• Risks: Write conflicts—use optimistic locking, sections, or supervisor merge

Hybrid

Common production pattern: router → parallel specialists → merger → critic → egress.

Routing strategies

Routing checklist:

• [ ] Log routing decision with reason code (not only model prose)
• [ ] Fallback path when no agent accepts (human or safe default)
• [ ] Prevent circular routing (A→B→A) with visit counters or DAG enforcement
• [ ] Sticky routing when continuity matters (same specialist for a thread)

Fan-out and fan-in

Fan-out: duplicate or split work across N workers.

Fan-in: merge N results.

Required parameters:

• N max parallelism
• Per-worker timeout
• Fan-in timeout (wait for k-of-n or all-n)
• Policy for partial fan-in (proceed with subset vs fail)

DAG workflows

Represent multi-agent work as a directed acyclic graph (cycles only via explicit iteration node with cap):

        [plan]
           │
     ┌─────┴─────┐
     ▼           ▼
 [research]   [research]
     │           │
     └─────┬─────┘
           ▼
        [merge]
           ▼
        [draft]
           ▼
        [critic]──fail──► [revise] (loop max 3)
           │ pass
           ▼
        [publish]

Node metadata:

• agent_role, input_schema, output_schema
• retry_policy, timeout, compensation (optional)
• conditional_edges (e.g., critic pass/fail)

Conditional branches: encode as explicit edges, not implicit prompt luck.

Anti-patterns

• Agent soup — many undifferentiated agents with overlapping tools
• Chatty P2P without cap — unbounded debate loops
• Fan-out without fan-in — orphaned partial results and billing leaks
• Supervisor does all work — workers become decorative
• Routing only in natural language — no structured route for audits
• Per-agent secrets — duplicate credentials across workers

Deployment, cost, and governance

Table of contents

1. Runtime deployment modes 2. Queues and durable workflows 3. Scaling and isolation 4. Cost and token budgets 5. Governance and release 6. Framework mapping (high level)

Runtime deployment modes

Rule: match orchestration engine to workflow lifetime, not individual LLM call latency.

Queues and durable workflows

Message queue pattern

API → enqueue(workflow_start) → workers consume node tasks → state store

Properties:

• Visibility timeout > p99 node duration
• Dead-letter queue (DLQ) for poison messages
• Poison handling: alert, quarantine, do not infinite redrive

Durable workflow pattern

Engine persists workflow state and replays from last checkpoint on crash.

Use when:

• Human gates may wait hours
• Fan-out with many branches
• Legal/audit need of execution history

Design:

• Activities (side effects) separated from orchestration logic
• Activities must be idempotent
• Version workflow definitions; pin running instances to version

Comparison

Scaling and isolation

Multi-tenancy: enforce tenant_id on every enqueue and state read; never share blackboards across tenants.

Blue/green: deploy new graph_version alongside old; route fraction of traffic for canary.

Cost and token budgets

Budget hierarchy

org_budget
  └── workflow_run_budget
        └── per_node_budget
              └── per_agent_call_budget

Enforcement points:

• Router refuses expensive subgraph if remaining budget low
• Supervisor stops fan-out when projected cost exceeds cap
• Hard kill switch when global org burn rate spikes

Attribution

Record per span: model, tokens, tool API cost, wall time.

Roll up to: cost per successful workflow, cost per agent role, cost per tenant.

Optimization (system level)

Coordinate with ai-context-engineer for per-call packing; this skill owns fleet-level caps.

Governance and release

Configuration registry

Version and audit:

• Graph topology definition
• Agent role → prompt + model + tool allowlist
• Routing rules and feature flags
• Message schema versions

Release process

1. Contract tests + golden DAG pass in CI 2. Canary on % traffic with compare metrics 3. Full rollout with rollback pin to previous graph_version 4. Post-release: cost and success rate review at 24h

Policy alignment

Engage ai-risk-governance for:

• Required HITL gates by risk tier
• Data residency and cross-border agent calls
• Logging retention for inter-agent messages
• Prohibited tool combinations across roles

Access control

• Which teams may publish new agent cards or graph versions
• Separation: operators scale workers; developers change graphs; security approves tool allowlists

Framework mapping (high level)

Framework-agnostic contracts first; map concepts as follows:

Do not let framework defaults replace explicit:

• Handoff schemas
• Merge policies
• Failure matrices
• Budget enforcement

Implement those in your orchestration layer or shared library regardless of framework.

Single-agent depth: once topology is set, use agentic-ai-developer to implement each node's loop, tools, and eval harness inside the graph.

Fault tolerance, observability, and testing

Table of contents

1. Failure taxonomy 2. Retry and compensation 3. Partial failure in fan-out 4. Observability 5. Testing strategy 6. SLOs and alerting

Failure taxonomy

Classify per DAG node in the failure matrix—not only per HTTP request.

Retry and compensation

Retry layers

Backoff: exponential with jitter; max attempts per node in config, not buried in prompts.

Compensation (saga-style)

For nodes with irreversible side effects, define compensating actions:

Not all steps are compensatable—mark pivot points where workflow must halt and human intervenes.

Partial failure in fan-out

Document user-visible behavior: does the user see partial research or an error?

Straggler handling: cancel slow branches after T seconds; include cancelled in merge metadata.

Observability

Trace hierarchy

workflow_run (correlation_id)
  ├── span: router
  ├── span: fan-out
  │     ├── span: worker-1
  │     └── span: worker-2
  ├── span: merge
  └── span: critic

Required attributes:

• workflow_run_id, graph_version, tenant_id
• agent_role, agent_instance_id, model (if applicable)
• input_tokens, output_tokens, cost_usd (estimated)
• tool_calls count, error_code
• payload_version on handoffs

Redaction: secrets, PII, and raw tool payloads off by default in production; enable debug tier for support.

Dashboards

Debug replay

Store: graph version, inputs hash, artifact refs, routing decisions, model versions.

Replay modes: dry-run (no side effects), single-node re-execute, full (dangerous in prod).

Testing strategy

Test pyramid for multi-agent systems

Simulation

• Stub agents: deterministic responses from fixtures
• Property checks: fan-in always receives ≤ N messages
• Policy tests: critic must reject forbidden tool plans

Golden DAG paths

Maintain versioned scenarios:

scenario: crm_research_v3
graph_version: "2025.04.1"
steps:
  - expect_route: specialist_pool
  - expect_fan_out: 3
  - expect_artifact: comparison_table.v1
  - max_cost_usd: 2.50
  - max_latency_sec: 120

Run on CI for every graph or prompt change affecting routing.

Regression gates

Block release if:

• Success rate drops > X% vs baseline on golden set
• Cost per task increases > Y% without approval
• New schema breaks contract tests

SLOs and alerting

Alerts:

• Spike in supervisor retries (routing broken)
• Merge timeouts (fan-out sizing wrong)
• Cost burn anomaly per tenant
• HITL queue age > threshold

Pair with ai-lead-ops for incident response; this skill defines what to measure.

Inter-agent protocols and messaging

Table of contents

1. Message envelope 2. Handoff payload 3. A2A-style patterns (conceptual) 4. Versioning and compatibility 5. Delivery semantics 6. Security and trust

Message envelope

Use a consistent envelope for every inter-agent message (in-process or via queue):

{
  "envelope_version": "1.0",
  "message_id": "uuid",
  "correlation_id": "workflow-run-uuid",
  "causation_id": "parent-message-uuid",
  "timestamp": "ISO-8601",
  "from": { "agent_id": "planner-1", "role": "planner" },
  "to": { "agent_id": "executor-3", "role": "executor" },
  "intent": "assign_work",
  "payload_version": "2.1",
  "payload": { },
  "artifacts": [ { "uri": "...", "type": "application/json", "hash": "..." } ],
  "constraints": {
    "deadline": "ISO-8601",
    "token_budget": 8000,
    "must_not": ["call_delete_tool"]
  }
}

Field rules:

• correlation_id is stable for the user-visible job across all agents
• causation_id links to the message that triggered this one (trace tree)
• Large blobs go in artifacts, not inline in payload
• constraints are machine-enforced where possible, not suggestions

Handoff payload

Minimum handoff content when transferring responsibility:

Bad handoff: "Continue the previous conversation" with full chat log pasted.

Good handoff: Structured work order referencing artifact URIs and schema version.

Example work order

{
  "work_order_id": "wo-42",
  "goal": "Produce vendor comparison table for 3 shortlisted CRMs",
  "inputs": {
    "requirements_artifact": "s3://bucket/reqs.v2.json",
    "shortlist": ["vendor_a", "vendor_b", "vendor_c"]
  },
  "output_schema": "crm_comparison_table.v1",
  "acceptance_criteria": [
    "All rows cite source URL",
    "No pricing without retrieved_at date"
  ]
}

A2A-style patterns (conceptual)

Agent-to-agent (A2A) ecosystems emphasize discoverable agents with advertised capabilities. Adapt conceptually without binding to one vendor spec:

Mapping to internal systems:

• Agent card → service registry or config manifest in repo
• Task delegation → queue message with work_order schema
• Streaming → event bus or webhook callbacks on correlation_id

Interoperability checklist:

• [ ] Schemas published in registry with owner team
• [ ] Auth between agents uses short-lived tokens scoped to workflow
• [ ] Timeouts and cancellation propagated on delegation chain

Versioning and compatibility

Migration: support N and N-1 payload versions during rollout; log deprecation warnings.

Delivery semantics

Idempotency key: hash(correlation_id + intent + payload_hash + target_agent).

Duplicate handling:

• Executor checks idempotency store before side effects
• Merger uses deterministic reduce for same inputs

Security and trust

• Treat all inter-agent payloads as potentially influenced by untrusted upstream agents or tools
• Validate against JSON Schema before acting
• Never pass raw credentials in messages—reference secret handles
• Sign or MAC messages crossing process boundaries when threat model requires
• Log message metadata; redact payload bodies in production traces by default

Prompt injection across agents: a compromised worker can poison handoffs. Mitigations:

• Schema validation and allowlisted fields
• Critic/supervisor with no tool access reviews handoffs for policy
• Separate context namespaces per agent (no full transcript sharing)

Multi-agent system engineer scope

Table of contents

1. Mission 2. In scope 3. Out of scope 4. Boundary vs agentic-ai-developer 5. Typical deliverables 6. Quality bar 7. Handoffs to peer roles

Mission

Engineer multi-agent systems as coherent distributed applications: multiple LLM-backed (or hybrid) agents coordinated by explicit topology, protocols, and shared runtime policies. Own system-level concerns—who talks to whom, how work is split and merged, how state and failures propagate—not the implementation details of a single ReAct loop in isolation.

In scope

Out of scope

Boundary vs agentic-ai-developer

Rule of thumb: If removing one agent leaves no "system" (only a loop), use agentic-ai-developer. If agents are peers in a coordinated fleet, use this skill.

Typical deliverables

1. System context diagram — agents, external systems, data stores, trust boundaries 2. Topology ADR — chosen pattern, alternatives rejected, scaling implications 3. Role catalog — responsibility, inputs, outputs, allowed tools, SLAs 4. Routing specification — rules, fallbacks, human escalation paths 5. Message schema registry — versioned handoff types with examples 6. State partition map — what is shared, per-thread, or immutable artifact 7. Failure and compensation matrix — per node: retry, skip, compensate, abort DAG 8. Observability contract — required trace fields, dashboards, alert thresholds 9. Test plan — unit agent, edge, full golden DAG, load and chaos scenarios 10. Deployment topology — sync API vs queue vs durable engine, scaling knobs

Quality bar

Before calling a multi-agent system production-ready:

• [ ] Every agent role has a single accountable owner in the org (team or service)
• [ ] Handoff schemas are versioned; unknown versions fail closed at boundaries
• [ ] Fan-out always defines fan-in merge semantics and timeout for stragglers
• [ ] Workflow runs are idempotent or keyed; duplicate delivery does not double-charge
• [ ] One correlation ID ties all agent spans for a user-visible task
• [ ] System budgets enforced (parallelism, tokens, wall time, cost)
• [ ] Conflict policy documented when agents disagree
• [ ] Regression suite covers full DAG and critical partial-failure paths
• [ ] Kill switch disables graph version or individual agent without orphaning state

Handoffs to peer roles

Shared state, coordination, and consensus

Table of contents

1. State categories 2. Partitioning model 3. Blackboard pattern 4. Synchronization 5. Consensus and conflict resolution 6. Human-in-the-loop at system level

State categories

Design principle: default partitioned state; share only through schemas and artifact refs.

Partitioning model

Keys should include at minimum:

tenant_id / org_id
user_id or principal_id
thread_id or workflow_run_id
agent_instance_id (optional)

Rules:

• Workers read only slices of state for their work order
• Supervisor holds routing and global budget counters
• No agent reads another agent's scratchpad unless explicitly passed in handoff

Checkpointing (system view):

• Checkpoint after each DAG node completion, not only after each LLM call
• Resume reloads workflow state + artifact index; re-invoke only failed nodes when safe

Blackboard pattern

Structured shared surface agents read/write:

{
  "board_version": 3,
  "sections": {
    "requirements": { "owner": "planner", "content_ref": "..." },
    "findings": { "entries": [ { "agent": "research-2", "claim": "...", "evidence_ref": "..." } ] },
    "open_risks": [ { "id": "r1", "severity": "high", "status": "open" } ]
  },
  "locks": { "findings": "research-1" }
}

Concurrency:

• Section-level locks or optimistic concurrency with board_version
• Append-only logs for findings to reduce write conflicts
• Supervisor compacts board periodically into summary artifact

Synchronization

Barriers

Pause DAG until condition met:

Timeouts

• Barrier timeout triggers policy: fail workflow, proceed with partial, or escalate
• Document default: fail closed for regulated domains; best-effort partial for research

Clocks and ordering

• Use logical workflow step numbers, not wall clock, for ordering events
• If using event bus, partition by correlation_id for per-workflow ordering

Consensus and conflict resolution

When agents disagree (facts, plan, priority):

Conflict record (audit):

{
  "conflict_id": "c-9",
  "topic": "vendor_pricing",
  "positions": [
    { "agent": "research-1", "claim": "...", "evidence_ref": "..." },
    { "agent": "research-2", "claim": "...", "evidence_ref": "..." }
  ],
  "resolution": "supervisor_chose",
  "resolved_by": "supervisor-1",
  "rationale": "research-2 source more recent"
}

Human-in-the-loop at system level

System-level HITL differs from single-tool approval:

Timeouts: stalled human gates should default deny or cancel workflow per risk tier—document in ai-risk-governance alignment.

Queue fairness: prioritize by SLA tier; expose correlation id and condensed context for reviewers.