Create Integration

create-integration

When to use

Use this skill when tasks include:

• creating a new integration package from scratch
• scaffolding data streams and applying post-scaffold edits
• understanding package topology, file placement, and manifest patterns
• running the end-to-end build workflow (scaffold → data collection setup → pipeline → system tests → review)
• questions about package structure, layout, or manifest.yml shape

IMPORTANT: Loading references

This skill has four reference files. Load the appropriate one(s) based on your task:

When creating a full integration (end-to-end): → MUST read `references/create-workflow.md` fully before starting. This contains the complete orchestration workflow, all phases, subagent delegation instructions, and guardrails.

When adding data streams to an existing package: → MUST read `references/add-datastream-workflow.md` fully before starting. This covers verifying the package, scaffolding streams, and the CEL → pipeline → system-test sequence.

When scaffolding a package or data stream, or applying post-scaffold edits: → Read references/scaffold-commands.md for the scaffold commands, post-scaffold checklist, and common pitfalls.

When reviewing or understanding package topology and file layout: → Read references/package-layout.md for canonical trees, manifest patterns, and review checklists for both integration and input packages.

What to provide when creating an integration

Include any combination of the following:

Example invocations

Create a new "acme_firewall" integration for Acme Firewall appliance.
  API docs: https://docs.acme.com/api/v2/events
  Auth: Bearer token header. Pagination: offset-based with total_count.
  @samples/acme_events.json. Single data stream "event" using cel input.

New syslog integration "my_appliance" with tcp,udp inputs.
  @notes/research-brief.md. Two streams: "log" (syslog) and "traffic" (syslog).

What to provide when adding data streams to an existing package

Use @-mentions for files/folders and paste links inline.

Example invocations

Add "audit" stream to @packages/acme_firewall using cel input.
  API endpoint: /api/v2/audit_logs
  Pagination: timestamp cursor.
  @samples/acme_audit.json

Add "traffic" and "threat" streams to acme_firewall.
  Both use tcp,udp inputs (syslog).
  @samples/traffic.log @samples/threat.log

Subagents overview

Do not load CEL, pipeline, ECS, or field-mapping skills yourself. Delegate to subagents that load their own domain skills.

All specialised work is delegated to the platform's generic / general-purpose subagent (Cursor: generalPurpose Task agent; Claude Code: general-purpose Task agent; or the equivalent on other platforms). Each task prompt must *point the subagent at the relevant `-subagent-guidance.md file by path** and instruct it to read that file (plus the skill SKILL.md it lists in "First steps") end-to-end before doing any other work. **Do NOT read the guidance file yourself or paste its contents into the task prompt** — that doubles its context cost. Pass only the path plus the task-specific context. The subagent will load the manual itself in its own fresh context. Full dispatch rules and per-step detail live in references/create-workflow.md and references/add-datastream-workflow.md`.

For cloud storage inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub): skip data collection setup and system tests. The scaffold provides a usable template; trim vars to match needs. See references/create-workflow.md for details.

References

• references/create-workflow.md — full phases 1–8 for creating a new integration, subagent instructions, guardrails, data anonymization
• references/add-datastream-workflow.md — phases 1–4 for adding data streams to an existing package, CEL/pipeline/system-test sequence
• references/scaffold-commands.md — scaffold commands, post-scaffold edits, base-fields.yml format
• references/package-layout.md — integration and input package topology, manifest patterns

Add Data Stream — Workflow

This reference covers the end-to-end workflow for adding one or more data streams to an existing Elastic integration package. Read this fully before starting.

Dispatch convention (read once, applies to every subagent step below)

All specialised work in this workflow is delegated to the platform's generic / general-purpose subagent (Cursor: generalPurpose Task agent; Claude Code: general-purpose Task agent; or the equivalent on other platforms). Do not invoke a named specialised subagent.

Every subagent task prompt must:

1. Begin with an instruction to read the subagent's operating manual. Point the subagent at the relevant *-subagent-guidance.md file by path and tell it to read that file (plus the skill SKILL.md it points at in its "First steps" section) end-to-end before doing any other work. Do NOT read the guidance file yourself or paste/embed its content into the task prompt — that doubles the context cost. The subagent must load the manual itself in its own fresh context. The guidance file contains the skill-load sequence, workflow, scope boundaries, and reporting contract. 2. Provide all context the subagent needs (it cannot see your conversation): package path, data stream path, sample data, API docs / payloads, research brief, authoritative requirement files, requirements, existing package state (especially package-level vars in the root manifest.yml), API credentials when supplied.

CRITICAL: Only run ONE subagent at a time. Process data streams sequentially — never launch multiple builder subagents (CEL, data-collection setup, pipeline, system test) in parallel. Complete all work for one data stream before starting the next.

Builder / reviewer manuals (pass these by path, do not embed)

Phase 1: Parse context and verify package

1. Extract from the user message: target package, stream name(s), stream type, input type(s), and any constraints. 2. Read any @-mentioned files. Fetch any documentation or API URLs provided inline. 3. Verify the target package exists at packages/<package_name>/ and read its root manifest.yml to understand existing structure (existing data streams, policy template inputs, shared vars). 4. If package name is ambiguous or missing, ask before proceeding. 5. Default stream type to logs unless explicitly specified as metrics.

Phase 2: Scaffold the data stream

For each requested stream:

1. Verify you are inside the package directory:

cd packages/<package_name>

2. Verify _dev/build/build.yml exists with a current ECS reference. If missing, create it as described in references/scaffold-commands.md (post-scaffold step 1) or the ecs-field-mappings skill.

3. Run the data-stream scaffold:

elastic-package create data-stream --name <stream_name> --type <logs|metrics> --inputs <input_types>

Important: every file produced by elastic-package create data-stream is placeholder scaffolding only. For cel inputs this is mandatory: generated CEL/template/manifest content is never production-ready and must be replaced with real implementation logic. For non-CEL inputs, generated manifests/templates may include useful defaults but still must be treated as placeholders and implemented against the actual source requirements.

4. Update data_stream/<stream>/manifest.yml: set correct title and description, review vars. 5. For CEL inputs: strip the verbose generic scaffold vars (the subagent will configure the template properly), and ensure only vars referenced by cel.yml.hbs remain. 6. Wire package-level config if needed: add new package-level vars (shared auth, URL) to root manifest.yml under policy_templates[].inputs[].

7. Validate the scaffold:

elastic-package check

Treat both package-level and data stream manifest.yml files as placeholders after scaffold generation.

Phase 3: Delegate specialized work per data stream

CRITICAL: Process one data stream at a time. Complete all steps for one stream before starting the next.

Step 1: Data collection setup (input-type dependent)

CEL inputs

Dispatch a subagent per the Dispatch convention above, pointing it at cel-programs/references/builder-subagent-guidance.md as its operating manual.

The task prompt must include (in addition to the read-the-manual directive):

1. Package and data stream paths. 2. API endpoint details (URL, auth method, pagination pattern, response structure). 3. Sample data or research brief findings. 4. Existing package-level vars in the root manifest.yml (so the subagent reuses shared auth/URL vars rather than redefining them at the stream level). 5. At least one representative API request and response payload (sanitized). 6. Links to authoritative requirement files. 7. API credentials (if the user provided tokens, API keys, OAuth client ID/secret): pass these so the subagent can also test against the real API. Mock-first development remains the primary path. 8. Path to the research results folder (if any) — the subagent will look for a test-api.py script there to validate the mock against documented API behaviour. 9. Any stream-specific constraints.

The subagent will: set up the system test mock first (docker-compose + elastic/stream config + test-default-config.yml), start the mock locally, run any research test-api.py against it, verify the mock completeness gate (2+ pages + terminal page + round-2 cursor resume + regression guard), then develop and validate the CEL program with mito incrementally (skeleton → error handling → events → pagination → cursor). Only after mito passes does it write cel.yml.hbs, run celfmt -s -agent, configure manifest vars, and define initial field mappings.

The CEL program builder does NOT: create pipeline test fixtures, touch the ingest pipeline or fields/ecs.yml, modify sample_event.json, run system tests, or implement document deduplication logic.

Wait for the subagent to complete before proceeding to Step 2.

TCP, UDP, HTTP endpoint, logfile, Kafka, Pub/Sub inputs

Dispatch a subagent per the Dispatch convention above, pointing it at integration-testing/references/builder-setup-subagent-guidance.md as its operating manual.

The task prompt must include (in addition to the read-the-manual directive):

1. Package and data stream paths. 2. Input type(s) for the data stream. 3. Sample log data or event payloads. 4. Existing package-level vars in the root manifest.yml (so the subagent reuses shared config rather than re-defining shared auth/URL vars at the stream level). 5. Log format details (JSON, syslog, CEF, key-value, etc.). 6. Any stream-specific constraints (ports, auth, TLS requirements).

The subagent will: set up _dev/deploy/docker/docker-compose.yml, create sample log files, configure the agent stream template, write system test configs, and clean up scaffold manifest vars. It examines 2-3 existing integrations of the same input type in the official repo for patterns.

The setup subagent does NOT: build ingest pipelines, run system tests (that's a separate system-test invocation in Step 3), or handle CEL programs.

Wait for the subagent to complete before proceeding to Step 2.

Cloud storage inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub)

Skip the data collection setup step. The scaffold provides a usable agent stream template — review and trim vars to match the integration's needs. System tests will be skipped for this data stream (see Step 3). Proceed directly to Step 2.

Step 2: Ingest pipeline

Dispatch a subagent per the Dispatch convention above, pointing it at ingest-pipelines/references/builder-subagent-guidance.md as its operating manual.

The task prompt must include (in addition to the read-the-manual directive):

1. Package and data stream paths. 2. For CEL streams: tell it the data structure is already known from the CEL builder output and the system test mock data it produced. Point it to the mock API response files so it understands the data format. 3. For non-CEL streams: provide sample log data and log format details (JSON, syslog, CEF, key-value, etc.). Tell it whether the input is CEL or not, so it knows whether to include CEL-only opening processors. 4. Representative request/response payloads or raw-event fixtures. 5. Links to authoritative requirement files. 6. Expected ECS categorization if known.

The subagent will: design and implement the ingest pipeline, define field mappings, create pipeline test fixtures, run elastic-package test pipeline --generate, and verify the generated expected output.

The pipeline builder does NOT: run system tests or modify sample_event.json.

Wait for the pipeline builder to complete before proceeding to Step 3.

Step 3: System test (input-type dependent)

Running full system tests in the orchestrator thread burns context. Do not execute `elastic-package test system` yourself.

CEL, TCP, UDP, HTTP endpoint, logfile, Kafka, Pub/Sub inputs

Ensure data_stream/<stream>/_dev/test/system/test-*-config.yml includes `wait_for_data_timeout: 1m` before running. The system-test subagent will add it if missing.

Dispatch a subagent per the Dispatch convention above, pointing it at integration-testing/references/builder-system-test-subagent-guidance.md as its operating manual, to run the system test.

The task prompt must include (in addition to the read-the-manual directive):

1. Clarify this is a system test run (not a data-collection setup invocation). 2. Absolute package path, data stream name, input type. 3. Confirmation that the Elastic stack is up. 4. Instruction to cd packages/<package_name>/, run elastic-package build, then `elastic-package test system --data-streams <stream> --generate`. The --generate flag is required — it produces sample_event.json from the first indexed document. 5. Ask for a concise report: pass/fail per test config, error excerpts, whether sample_event.json was generated, and any issues classified by domain (pipeline / CEL / mock API / docker-compose / sample logs) that need orchestrator intervention.

If the system test fails, fix straightforward issues yourself or re-dispatch a subagent (per the Dispatch convention) based on the domain-classified report:

• Pipeline / fields / pipeline-test issues → point the subagent at ingest-pipelines/references/builder-subagent-guidance.md
• CEL program or mock API issues → point the subagent at cel-programs/references/builder-subagent-guidance.md
• Docker-compose / sample log / template / non-CEL test config issues → point the subagent at integration-testing/references/builder-setup-subagent-guidance.md

Never create `sample_event.json` manually.

Wait for this subagent to finish before moving to the next data stream.

Cloud storage inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub)

Skip system tests. These inputs require cloud infrastructure that cannot be reliably emulated in Docker. See integration-testing skill → references/system-testing-cloud-skip.md.

• Focus on pipeline tests for coverage.
• sample_event.json must be created through alternative means (construct from pipeline test expected output, or run a temporary local stack session with real cloud credentials).
• Note in the final report: "System tests skipped for <stream> — no docker-based mock available for <input> inputs."

Repeat for each data stream

Move to the next data stream and repeat Steps 1–3. Do not start a new data stream until the current one is complete.

Phase 4: Validate and report

After all data streams are complete (including system tests):

1. Run the full check sequence from the package directory:

elastic-package format
elastic-package lint
elastic-package check

2. Fix any minor issues (manifest wiring, formatting). For significant pipeline errors, re-dispatch a subagent (per the Dispatch convention) pointing it at ingest-pipelines/references/builder-subagent-guidance.md with the specific issues to fix. For CEL errors, re-dispatch pointing the subagent at cel-programs/references/builder-subagent-guidance.md.

3. Optionally dispatch a subagent (per the Dispatch convention) pointing it at review-integration/references/reviewer-subagent-guidance.md for a quality check of the new stream(s). The task prompt must additionally pass: (a) which tests have already passed (so the reviewer does not re-run them); (b) explicit request to verify manifest/template parity (no unused vars in package-level or data stream manifest.yml not consumed by *.yml.hbs); (c) any focus areas specific to the newly added stream(s).

4. If the reviewer reports CEL-related issues:

• Formatting-only issues (indentation, style): run celfmt -s yourself:

     cd packages/<package_name>/data_stream/<stream>/agent/stream
     celfmt -s -agent -i cel.yml.hbs -ocel.yml.hbs

• Logic issues (error handling, cursor management, pagination): re-dispatch a subagent (per the Dispatch convention) pointing it at cel-programs/references/builder-subagent-guidance.md with the specific issues to fix.

5. Report back with:

• Files created and modified (with paths)
• Input type and pipeline/CEL architecture chosen
• How the stream fits into the existing package structure
• Decisions made and rationale
• System test results per data stream
• TODO items requiring user input
• Next steps

Data anonymization

All data committed to the repository must be fully anonymized. No real production data, customer data, or identifiable information may appear in any committed file — including sample events, test fixtures, mock API responses, documentation examples, configuration defaults, and manifest placeholder values.

Replace every identifying value with a synthetic example value of the same format before committing. This applies to all fixtures, mock responses, sample events, documentation, and default manifest values (use https://api.example.com, not real URLs).

Ensure subagents receive this instruction: all fixture data, mock API responses, and sample events they produce must use anonymized values. Refer to the anonymize-logs skill for the full anonymization policy and placeholder conventions.

Guardrails

• Always use elastic-package create data-stream for scaffolding. Never fabricate stream directories manually.
• Treat all scaffold output as placeholders only. A passing scaffold validation does not mean the data stream implementation is complete.
• Treat package-level and data stream manifest.yml as placeholders until aligned with implemented templates and requirements.
• Verify root `manifest.yml` sets `format_version: "3.4.2"` and `conditions.kibana.version: "^8.19.0 || ^9.1.0"`. If the existing package has different values, update them. These settings belong only in the root manifest, not in data stream manifests.
• For CEL streams, remove all unused manifest vars (package-level and data stream-level). If a var is not used in cel.yml.hbs, remove it.
• Run from inside the target package directory (packages/<name>/).
• Run elastic-package build before any system test whenever package files changed.
• Verify the package exists before attempting scaffold.
• Do not duplicate package-level vars that already exist in root manifest.yml.
• Do not create or modify sample_event.json manually. It is only generated by elastic-package test system.
• Do not create *-expected.json manually. It is only generated by elastic-package test pipeline --generate.
• For CEL inputs, strip unused scaffold vars.
• Choose --inputs based on the product's data delivery method. Allowed values: aws-cloudwatch, aws-s3, azure-blob-storage, azure-eventhub, cel, entity-analytics, etw, filestream, gcp-pubsub, gcs, http_endpoint, journald, netflow, redis, tcp, udp, winlog.
• Do not load domain-specific skills (CEL, pipelines, ECS, field mappings) into your own context. Delegate to the subagents that already have that knowledge.
• Never include `data_stream.dataset` in `cel.yml.hbs` or as a manifest var for integration packages (type: integration). The framework routes documents automatically.

Create Integration — Full Workflow

This reference covers the complete end-to-end workflow for creating a new Elastic integration package. Read this fully before starting creation work.

Dispatch convention (read once, applies to every subagent step below)

All specialised work in this workflow is delegated to the platform's generic / general-purpose subagent (Cursor: generalPurpose Task agent; Claude Code: general-purpose Task agent; or the equivalent on other platforms). Do not invoke a named specialised subagent.

Every subagent task prompt must:

1. Begin with an instruction to read the subagent's operating manual. Point the subagent at the relevant *-subagent-guidance.md file by path and tell it to read that file (plus the skill SKILL.md it points at in its "First steps" section) end-to-end before doing any other work. Do NOT read the guidance file yourself or paste/embed its content into the task prompt — that doubles the context cost. The subagent must load the manual itself in its own fresh context. The guidance file contains the skill-load sequence, workflow, scope boundaries, and reporting contract. 2. Provide all context the subagent needs (it cannot see your conversation): package path, data stream path, sample data, API docs / payloads, research brief, authoritative requirement files, requirements, existing package state, API credentials when supplied.

CRITICAL: Only run ONE subagent at a time. Process data streams sequentially — never launch multiple builder subagents (CEL, data-collection setup, pipeline, system test) in parallel. Complete all work for one data stream before starting the next.

Builder / reviewer manuals (pass these by path, do not embed)

Phase 1: Parse context

1. Extract from the user message: package name, product description, input type(s), data stream name(s), auth method, pagination pattern, and any constraints. 2. Read any @-mentioned files (research briefs, sample data). Fetch any documentation or API URLs provided inline. 3. If critical information is missing (package name or input type), ask before proceeding. 4. Default to package type integration unless explicitly told otherwise. 5. If the product/vendor needs research and no brief is provided, hand off to the /research-integration skill (or instruct the user to invoke it) to investigate documentation, API details, and sample payloads before proceeding. That skill orchestrates its own research subagents — do not launch a named deep-research subagent yourself.

Phase 2: Scaffold the package

1. Verify you are in the repository root (check for packages/ directory). 2. Run the package scaffold and apply all post-scaffold steps per references/scaffold-commands.md (scaffold command, _dev/build/build.yml creation, manifest edits, initial validation).

Mandatory manifest version settings — after scaffolding, verify the root manifest.yml has these exact values (the scaffold may generate different defaults):

• format_version: "3.4.2"
• conditions.kibana.version: "^8.19.0 || ^9.1.0"

3. Start the Elastic stack (needed for system tests later):

elastic-package stack up -d -v

This runs in detached mode. Do not wait for it to finish — continue while the stack boots. If the stack is already running, this is a no-op.

Phase 3: Scaffold data streams

For each requested data stream, scaffold and apply post-scaffold edits per references/scaffold-commands.md.

Phase 4: Delegate specialized work per data stream

CRITICAL: Process one data stream at a time. Complete all steps for one stream before starting the next.

For each data stream, follow this sequence. The steps vary by input type.

Step 1: Data collection setup (input-type dependent)

CEL inputs

Dispatch a subagent per the Dispatch convention above, pointing it at cel-programs/references/builder-subagent-guidance.md as its operating manual.

The task prompt must include (in addition to the read-the-manual directive):

1. Package and data stream paths. 2. API endpoint details (URL, auth method, pagination pattern, response structure). 3. Sample data or research brief findings. 4. At least one representative API request and response payload (sanitized). 5. Links to authoritative requirement files. 6. API credentials (if the user provided them): pass these so the subagent can additionally test against the real API alongside mock-first development. 7. Any stream-specific constraints. 8. Path to the research results folder (if any) — the subagent will look for a test-api.py script there to validate the mock against documented API behaviour.

The subagent will: set up the system test mock first (docker-compose + elastic/stream config + test-default-config.yml), start the mock locally, run any research test-api.py against it, verify the mock completeness gate (2+ pages + terminal page + round-2 cursor resume + regression guard), then develop and validate the CEL program with mito incrementally (skeleton → error handling → events → pagination → cursor). Only after mito passes does it write cel.yml.hbs, run celfmt -s -agent, configure manifest vars, and define initial field mappings.

The CEL program builder does NOT: create pipeline test fixtures, touch the ingest pipeline or fields/ecs.yml, modify sample_event.json, run system tests, or implement document deduplication logic.

Wait for the subagent to complete before proceeding to Step 2.

TCP, UDP, HTTP endpoint, logfile, Kafka, Pub/Sub inputs

Dispatch a subagent per the Dispatch convention above, pointing it at integration-testing/references/builder-setup-subagent-guidance.md as its operating manual.

The task prompt must include (in addition to the read-the-manual directive):

1. Package and data stream paths. 2. Input type(s) for the data stream. 3. Sample log data or event payloads (paste inline or reference file paths). 4. Log format details (JSON, syslog, CEF, key-value, etc.). 5. Package-level vars already defined in the root manifest.yml (shared auth, base URL) so the subagent can reuse them. 6. Any stream-specific constraints (ports, auth, TLS requirements).

The subagent will: set up _dev/deploy/docker/docker-compose.yml with the appropriate service pattern, create sample log files in _dev/deploy/docker/sample_logs/, configure the agent stream template, write system test config files, and clean up scaffold manifest vars. It examines 2-3 existing integrations of the same input type in the official elastic/integrations repo for manifest var conventions and template structure.

The setup subagent does NOT: build ingest pipelines, run system tests (that's a separate system-test invocation in Step 3), or handle CEL programs.

Wait for the subagent to complete before proceeding to Step 2.

Cloud storage inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub)

These inputs do not have a standard docker-based system test pattern. Skip the data collection setup step:

1. The scaffold provides a usable agent stream template — review and trim vars to match the integration's needs. Consider moving shared credentials (access keys, connection strings) to the policy template level. For `aws-s3` inputs: remove the ssl configuration section from the data stream manifest.yml — the scaffold adds SSL vars to all input types, but aws-s3 does not use them (S3 connectivity is handled through the AWS SDK, not direct TLS socket configuration). 2. Note that system tests will be skipped for this data stream (see Step 3). 3. Proceed directly to Step 2 (ingest pipeline).

Step 2: Ingest pipeline

Dispatch a subagent per the Dispatch convention above, pointing it at ingest-pipelines/references/builder-subagent-guidance.md as its operating manual.

The task prompt must include (in addition to the read-the-manual directive):

1. Package and data stream paths. 2. For CEL streams: tell it the data structure is already known from the CEL builder output and system test mock data. Point it to the mock API response files. Require the pipeline to follow CEL-only opening processors, ecs.version: 9.3.0, full on_failure baseline, JSE00001 rename/remove, parsing from event.original, and rename over set when mapping into ECS. 3. For non-CEL streams: provide sample log data and log format details (JSON, syslog, CEF, key-value, etc.). Tell it whether the input is CEL or not, so it knows whether to include CEL-only opening processors. 4. Representative request/response payloads or raw-event fixtures. 5. Links to authoritative requirement files. 6. Expected ECS categorization if known.

The subagent will: design and implement the ingest pipeline, define field mappings, create pipeline test fixtures, run elastic-package test pipeline --generate, and verify the generated expected output.

The pipeline builder does NOT: run system tests or modify sample_event.json.

Wait for the pipeline builder to complete before proceeding to Step 3.

Step 3: System test (input-type dependent)

Running full system tests in the orchestrator thread burns context. Do not execute `elastic-package test system` yourself.

CEL, TCP, UDP, HTTP endpoint, logfile, Kafka, Pub/Sub inputs

System test configs must set `wait_for_data_timeout: 1m`. The CEL builder or the data-collection setup subagent should add this when creating the file; if missing, the system-test subagent will add it before running.

Dispatch a subagent per the Dispatch convention above, pointing it at integration-testing/references/builder-system-test-subagent-guidance.md as its operating manual, to run the system test.

The task prompt must include (in addition to the read-the-manual directive):

1. Clarify this is a system test run (not a data-collection setup invocation). 2. Absolute package path, data stream name, input type. 3. Confirmation that the Elastic stack is up (elastic-package stack up -d -v already issued). 4. Instruction to cd packages/<package_name>/, run elastic-package build, then `elastic-package test system --data-streams <stream> --generate`. The --generate flag is required — it produces sample_event.json from the first indexed document. Without it, sample_event.json will not be created and a separate run will be needed later. 5. Ask for a concise report: pass/fail per test config, error excerpts, whether sample_event.json was generated, and any issues classified by domain (pipeline / CEL / mock API / docker-compose / sample logs) that need orchestrator intervention.

If the system test fails, fix straightforward issues yourself or re-dispatch a subagent (per the Dispatch convention) based on the domain-classified report:

• Pipeline / fields / pipeline-test issues → point the subagent at ingest-pipelines/references/builder-subagent-guidance.md
• CEL program or mock API issues → point the subagent at cel-programs/references/builder-subagent-guidance.md
• Docker-compose / sample log / template / non-CEL test config issues → point the subagent at integration-testing/references/builder-setup-subagent-guidance.md

Never create `sample_event.json` manually.

Wait for this subagent to finish before moving to the next data stream or phase.

Cloud storage inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub)

Skip system tests. These inputs require cloud infrastructure that cannot be reliably emulated in Docker. See integration-testing skill → references/system-testing-cloud-skip.md.

• Focus on pipeline tests for coverage.
• sample_event.json must be created through alternative means (construct from pipeline test expected output, or run a temporary local stack session with real cloud credentials).
• Note in the final report: "System tests skipped for <stream> — no docker-based mock available for <input> inputs."

Repeat for each data stream

Move to the next data stream and repeat Steps 1–3.

Phase 5: Validate

After all data streams are complete, run the full check sequence yourself:

elastic-package format
elastic-package lint
elastic-package check

Fix any minor issues (manifest typos, formatting, changelog). For significant pipeline or CEL errors, re-delegate to the appropriate subagent.

Phase 6: Review

Dispatch a subagent per the Dispatch convention above, pointing it at review-integration/references/reviewer-subagent-guidance.md as its operating manual, to run the review.

The task prompt must include (in addition to the read-the-manual directive):

1. Package path. 2. The original requirements / research brief. 3. What was built (data streams, input types, architecture decisions). 4. Which tests have already passed: list pipeline tests and system tests that succeeded so the reviewer does not re-run them. 5. Explicit request to verify manifest/template parity: no unused vars in package-level or data stream manifest.yml not consumed by corresponding *.yml.hbs templates.

The reviewer returns a severity-ranked list of issues with domain tags, formatted per review-integration/references/review-output-template.md.

Phase 7: Fix from review

• Minor issues (manifest fields, changelog, documentation, field file typos): fix directly.
• Pipeline issues: re-dispatch a subagent (per the Dispatch convention) pointing it at ingest-pipelines/references/builder-subagent-guidance.md with the specific issues to fix.
• CEL issues:
• Formatting-only (indentation, style): run celfmt -s yourself:

    cd packages/<package_name>/data_stream/<stream>/agent/stream
    celfmt -s -agent -i cel.yml.hbs -ocel.yml.hbs

• Logic issues (error handling gaps, cursor problems, pagination bugs): re-dispatch a subagent (per the Dispatch convention) pointing it at cel-programs/references/builder-subagent-guidance.md with the specific issues to fix.

After fixes, run elastic-package check again.

Phase 8: Report

Report back with:

• Files created (with paths)
• Input type and pipeline/CEL architecture chosen
• Decisions made and rationale
• Review results (pass/fail, any remaining items)
• TODO items that need user input
• Next steps

Data anonymization

All data committed to the repository must be fully anonymized. No real production data, customer data, or identifiable information may appear in any committed file — including sample events, test fixtures, mock API responses, documentation examples, and manifest placeholder values.

Replace every identifying value with a synthetic example value of the same format before committing:

• Pipeline test fixtures and expected output
• System test mock API responses and sample logs
• sample_event.json (regenerate from anonymized test data)
• Documentation examples in README templates
• Default values in manifest vars (use https://api.example.com, not real URLs)

Ensure subagents receive this instruction: all fixture data, mock API responses, and sample events must use anonymized values. Refer to the anonymize-logs skill for the full anonymization policy and placeholder conventions.

Guardrails

• Always use elastic-package create for scaffolding. Never fabricate scaffold files manually.
• Treat all scaffold output as placeholders. A passing scaffold validation does not mean the integration logic is implemented.
• Treat manifest.yml as a placeholder until aligned with implemented templates and requirements.
• Root `manifest.yml` must set `format_version: "3.4.2"` and `conditions.kibana.version: "^8.19.0 || ^9.1.0"`. The scaffold may generate different values — always override. These settings belong only in the root manifest, not in data stream manifests.
• For CEL streams, remove all unused manifest vars. If a var is not used in cel.yml.hbs, remove it.
• Run from the correct directory: packages/ for package creation, packages/<name>/ for data-stream creation.
• Run elastic-package build before any system test whenever package files changed.
• Always create `_dev/build/build.yml` immediately after scaffolding the package, before creating data streams. Required for ECS field resolution.
• Do not leave default placeholder values in manifest.yml (title, description, owner).
• Do not create or modify sample_event.json manually. Only generated by elastic-package test system.
• Do not run `elastic-package test system` in the orchestrator thread — delegate per the Dispatch convention pointing the subagent at integration-testing/references/builder-system-test-subagent-guidance.md.
• Do not develop CEL programs or mock APIs in the orchestrator thread — delegate per the Dispatch convention pointing the subagent at cel-programs/references/builder-subagent-guidance.md.
• Do not create *-expected.json manually. Only generated by elastic-package test pipeline --generate.
• Do not uncomment {{ event "stream" }} in the doc template until sample_event.json exists.
• For CEL inputs, strip unused scaffold vars rather than leaving the verbose generic scaffold.
• Never include `data_stream.dataset` in `cel.yml.hbs` or as a manifest var for integration packages (type: integration). The framework routes documents automatically. Only input-type packages (type: input) use this field. Setting it in an integration package overrides routing and causes "0 hits" in system tests.
• Set version to 0.1.0 for new integrations.
• Do not load domain-specific skills (CEL, pipelines, ECS, field mappings) into your own context. Delegate to subagents.

Package Layout Reference

Use this reference for understanding package topology, required files, and manifest patterns.

Package types

---

Integration package

Canonical tree

<package>/
  manifest.yml                    # required root metadata
  changelog.yml                   # required changelog entries
  docs/
    README.md                     # generated docs output (required directory)
  img/                            # icons/screenshots
  data_stream/
    <stream_name>/
      manifest.yml                # required data stream metadata
      agent/
        stream/
          *.yml.hbs               # stream-level policy template
      fields/
        *.yml                     # required fields directory
      elasticsearch/
        ingest_pipeline/
          *.yml                   # default and optional sub-pipelines
      sample_event.json           # optional; required if doc template uses {{ event }}
      _dev/
        test/
          pipeline/               # ingest pipeline test fixtures
          system/                 # system test scenarios
      routing_rules.yml           # optional dataset routing
      lifecycle.yml               # optional data stream lifecycle
  kibana/                         # optional dashboards/searches/ml modules
  _dev/
    build/
      build.yml                   # required: ECS dependency (git@v9.3.0)
      docs/README.md              # docs template source
    deploy/
      docker/                     # local containerized test environment
      tf/                         # Terraform cloud test infrastructure
  validation.yml                  # optional package-level validation config

Root manifest.yml pattern

format_version: "3.4.2"
name: <package_name>
title: <Package Title>
version: "0.1.0"
description: <What the integration collects>
type: integration
categories:
  - <category>
conditions:
  kibana:
    version: "^8.19.0 || ^9.1.0"
policy_templates:
  - name: <package_name>
    title: <Template Title>
    description: <Template description>
    inputs:
      - type: <input_type>
        title: <Input title>
        description: <Input description>
owner:
  github: <org_or_team>
  type: elastic

Notes:

• Keep name stable; package identity and folder name alignment matter.
• Integration templates generally define inputs: as a list.
• Each input entry maps to one or more stream templates under data streams.

Data stream manifest.yml pattern

title: <Stream Title>
type: logs
streams:
  - input: <input_type>
    title: <Stream title>
    description: <Stream description>
    template_path: <stream>.yml.hbs
    vars:
      - name: <var_name>
        type: <var_type>
        required: true

Field files

Common split in data_stream/<stream>/fields/:

• base-fields.yml — routing constants (data_stream.*, event.module, event.dataset, @timestamp)
• ecs.yml — all ECS fields the pipeline sets (name + external: ecs only)
• fields.yml — stream-specific custom (non-ECS) fields
• agent.yml — Elastic Agent runtime metadata (cloud.*, host.containerized, input.type, log.offset)
• beats.yml — Filebeat/Beats-specific fields when the input emits them (input.type, log.offset, log.flags)

See ecs-field-mappings skill for the canonical formats of each file.

Ingest pipelines

• data_stream/<stream>/elasticsearch/ingest_pipeline/default.yml — entry pipeline
• Optional named sub-pipelines called via pipeline processor
• For complex integrations: category pipelines, object/domain pipelines, helper pipelines

Tests

• data_stream/<stream>/_dev/test/pipeline/ — test-*.log or test-*.json, optional *-config.yml, *-expected.json
• data_stream/<stream>/_dev/test/system/ — test-default-config.yml and similar; each must include wait_for_data_timeout: 1m

sample_event.json

Optional per spec but required when the doc template uses {{ event "<stream>" }}. Never create manually — generated by:

elastic-package test system --data-streams <stream> --generate

If system tests are not yet available, remove or leave commented the {{ event "stream" }} directive in the doc build template.

Build configuration

_dev/build/build.yml is required for all integration packages. Not generated by scaffold — create manually immediately after scaffold creation:

dependencies:
  ecs:
    reference: "git@v9.3.0"

Without this file, tests report ECS fields as undefined.

Multi-stream patterns

For complex integrations with routing:

• One source stream ingests raw records
• routing_rules.yml splits records into stream-specific datasets
• Target streams keep manifest.yml + fields/ and may not have direct input templates

# routing_rules.yml pattern
- source_dataset: <package_name>.event
  rules:
    - target_dataset: <package_name>.<target_a>
      if: <condition_a>
    - target_dataset: <package_name>.<target_b>
      if: <condition_b>

Structure review checklist

• Root required files exist and metadata is coherent
• Policy template inputs map cleanly to data stream templates
• Each data stream has manifest.yml and fields/
• Ingest pipeline files are placed under stream elasticsearch/ingest_pipeline/
• Tests live under stream _dev/test/ and not mixed into runtime folders
• _dev/build/build.yml exists with current ECS reference

---

Input package

Canonical tree

<package>/
  manifest.yml                    # required root metadata, type: input
  changelog.yml                   # required changelog entries
  docs/
    README.md                     # package docs
  img/                            # icon and optional screenshots
  agent/
    input/
      *.yml.hbs                   # input-level template(s)
  fields/
    *.yml                         # root-level field definitions
  _dev/
    build/
      build.yml                   # build-time dependencies/options
      docs/README.md              # docs template source
    deploy/                       # optional docker/tf helpers
    test/                         # optional tests for package behavior
  kibana/                         # optional tags/dashboard/search assets
  validation.yml                  # optional validation rules
  sample_event.json               # optional sample event at package root

Primary differences from integration packages

• No data_stream/ hierarchy in most input packages
• Input templates live at agent/input/ (not data_stream/<name>/agent/stream/)
• Fields are typically package-level under root fields/
• Policy templates use a direct input: key rather than an inputs: list

Root manifest.yml shape for input packages

format_version: "3.4.2"
name: <package_name>
title: <title>
description: <summary>
type: input
version: "x.y.z"
conditions:
  kibana:
    version: "^8.19.0 || ^9.1.0"
policy_templates:
  - name: <template_name>
    type: logs
    title: <template_title>
    description: <template_description>
    input: <input_name>
    template_path: <template_file>.yml.hbs
    vars:
      - name: <var_name>
        type: <var_type>

agent/input/*.yml.hbs

• Emits final input policy fragment
• Includes input vars (hosts, paths, tags, SSL/auth options)

Fields

Same file roles as integration packages but scoped to root fields/ (not per-stream).

Review checklist for input packages

• Root manifest.yml has type: input
• policy_templates use direct input and valid template_path
• Input templates are under agent/input/
• Fields are coherent at root fields/ and follow naming conventions
• No accidental data_stream/ structure

Scaffold Commands and Post-Scaffold Edits

Use this reference for scaffolding packages and data streams and applying the required post-scaffold edits.

Preconditions

Before running scaffold commands, verify: 1. elastic-package is available (elastic-package --help) 2. you are in the correct directory (see working directory rules below) 3. package/data stream name is finalized (renames later are noisy)

Working directory rules

• create package: run from the packages/ directory
• create data-stream: run from inside the target package directory (packages/<name>/)

Create a package

cd packages
elastic-package create package --name <package_name> --type integration

Flags:

• --name: package name (required)
• --type: one of integration, input, content (required; use integration for most new integrations)

Quick validation after scaffold:

cd <package_name>
elastic-package check

What create package generates

• manifest.yml — package metadata, policy_templates, owner
• changelog.yml — initial version entry
• docs/README.md — user-facing documentation (generated from build template)
• _dev/build/docs/README.md — doc build template
• img/ — sample screenshot and logo assets
• validation.yml

The scaffold does NOT generate `_dev/build/build.yml`. Create this manually immediately after scaffold (see post-scaffold edits below).

Create a data stream

cd packages/<package_name>
elastic-package create data-stream --name <stream_name> --type logs --inputs <input1>,<input2>

Flags:

• --name: data stream name (required)
• --type: logs or metrics (required)
• --inputs: comma-separated list of input types (required when type is logs)

Choose --inputs based on how the product sends data:

• syslog-sending appliances (firewalls, network devices): tcp,udp
• cloud services with object storage: aws-s3, azure-blob-storage, gcs
• API-based services: cel, http_endpoint
• local log files: filestream
• Windows event sources: winlog

Allowed input values: aws-cloudwatch, aws-s3, azure-blob-storage, azure-eventhub, cel, entity-analytics, etw, filestream, gcp-pubsub, gcs, http_endpoint, journald, netflow, redis, tcp, udp, winlog

What create data-stream generates

• data_stream/<name>/manifest.yml — data stream config with input streams and vars
• data_stream/<name>/agent/stream/<input>.yml.hbs — one agent stream template per input type
• data_stream/<name>/fields/base-fields.yml — minimum required field definitions
• data_stream/<name>/elasticsearch/ingest_pipeline/default.yml — starter ingest pipeline

Validate after creation:

elastic-package check

Post-scaffold edits

Apply these immediately after scaffold creation.

1. Create _dev/build/build.yml (required — not generated by scaffold)

This file configures the ECS dependency for field resolution. Without it, pipeline and system tests fail with "field X is undefined" for all ECS fields.

dependencies:
  ecs:
    reference: "git@v9.3.0"

Create this before creating any data streams.

2. Update root manifest.yml

• Set title, description, version (use 0.1.0 for new integrations)
• Set categories (one or two relevant values)
• Set owner (github team/org)
• Set format_version: "3.4.2". The scaffold may generate a different version — always override to 3.4.2.
• Set conditions.kibana.version: "^8.19.0 || ^9.1.0"

3. Update changelog.yml

Align version and description with the actual change.

4. Edit _dev/build/docs/README.md

Fill in the comment-guided sections with integration-specific content while preserving the scaffold structure and static sections. Do not replace the entire template with a hand-written README.

The {{ event "stream" }} directive requires sample_event.json to exist at build time. If system tests are not yet set up, remove or leave this directive commented; add it back once sample_event.json is generated.

5. Replace fields/base-fields.yml

The scaffold-generated output varies. Replace it with the standard format, substituting the actual package and stream names. All six fields are ECS fields — use external: ecs on each entry so descriptions are inherited from ECS. Override type and value only where the data stream needs a constant_keyword with a fixed value:

- name: data_stream.type
  external: ecs
- name: data_stream.dataset
  external: ecs
- name: data_stream.namespace
  external: ecs
- name: event.module
  external: ecs
  type: constant_keyword
  value: <package_name>
- name: event.dataset
  external: ecs
  type: constant_keyword
  value: <package_name>.<stream_name>
- name: '@timestamp'
  external: ecs

6. Create fields/beats.yml

The scaffold does not generate fields/beats.yml. Create it in the same fields/ directory alongside base-fields.yml, ecs.yml, and fields.yml with exactly this content:

- name: input.type
  type: keyword
  description: Type of filebeat input.
- name: log.offset
  type: long
  description: Log offset.

This file is required for every data stream regardless of input type (CEL, syslog, filestream, etc.).

7. Update data stream manifest.yml

Set title, review generated stream vars and input configuration. Apply the tz_offset rule if applicable.

`tz_offset` var rule: include only when syslog messages from that source lack a timezone. Never add it to CEL, HTTP endpoint, or other non-syslog streams.

- name: tz_offset
  type: text
  title: Timezone Offset
  description: >-
    IANA timezone name or UTC offset (e.g. `America/New_York` or `-05:00`) to
    use when the syslog messages do not include a timezone. Defaults to `Local`.
  required: true
  show_user: true
  default: Local

For syslog (tcp/udp) streams, always retain ssl and processors vars regardless of other simplification.

8. Edit agent stream templates (*.yml.hbs)

Adjust vars and configuration for the target product. For cel input data streams, do not edit the template directly here — delegate to a subagent (see the Dispatch convention in references/create-workflow.md or references/add-datastream-workflow.md) pointing it at cel-programs/references/builder-subagent-guidance.md as its operating manual. That subagent owns the mock-first / mito-incremental workflow, cel.yml.hbs authoring, and manifest var cleanup for CEL streams.

CEL scaffold notes: the cel input scaffold generates an extremely verbose generic manifest and template (300+ lines). The CEL builder subagent strips unused vars and replaces the scaffolded program with an integration-specific one — leave the verbose scaffold in place until then rather than editing it manually here.

`aws-s3` scaffold notes: the scaffold generates the template file as aws_s3.yml.hbs (with underscore), but the convention across existing integrations is aws-s3.yml.hbs (with hyphen). Rename the file after scaffolding to match the convention. The scaffold also generates ~120 lines of manifest vars and a complex template covering S3/SQS toggle, credential placement, and many optional settings. Review and trim to match the integration's actual needs, and consider moving credentials to the policy template level if shared across data streams. Remove the `ssl` configuration section from the data stream manifest.yml — the scaffold adds SSL vars to all input types, but aws-s3 does not use them (S3 connectivity is handled through the AWS SDK, not direct TLS socket configuration). Keeping unused ssl vars adds UI clutter with no effect.

9. Run validation

elastic-package check

Fix any issues before moving to mappings, pipeline, and testing work.

The doc build template

The _dev/build/docs/README.md template produces the final docs/README.md during build. It contains:

• Structured sections (Overview, Compatibility, Deployment, Troubleshooting, Reference, etc.)
• Go template comments ({{/* ... */}}) with guidance on what to fill in
• Template directives like {{ fields "stream" }}, {{ event "stream" }}, {{ inputDocs }}, {{ ilm }}, {{ transform }}

Preserve the static scaffold structure — do not replace the entire template. Fill in the comment-guided sections with integration-specific content.

sample_event.json and *-expected.json files must never be created manually:

• sample_event.json is generated by elastic-package test system --data-streams <stream> --generate
• *-expected.json is generated by elastic-package test pipeline --generate

10. Verify ecs.version in scaffolded pipeline

The scaffold may generate default.yml with an older ecs.version (e.g., 8.17.0). The correct version is `9.3.0`, matching the _dev/build/build.yml reference (dependencies.ecs.reference: "git@v9.3.0"). Update the set ecs.version processor in the scaffolded pipeline before handing off to the pipeline builder (see ingest-pipelines/references/builder-subagent-guidance.md).

Common pitfalls

• Forgetting to create `_dev/build/build.yml` — not generated by scaffold; required for ECS field resolution
• Forgetting to create `fields/beats.yml` — not generated by scaffold; required for input.type and log.offset field declarations
• Stale `ecs.version` in scaffolded pipeline — scaffold may generate 8.17.0 or similar; always update to 9.3.0
• `aws-s3` template filename — scaffold generates aws_s3.yml.hbs (underscore) but convention is aws-s3.yml.hbs (hyphen); rename after scaffolding
• `aws-s3` unused SSL vars — scaffold adds ssl vars to all input types; remove them for aws-s3 since S3 uses the AWS SDK, not direct TLS sockets
• Running create package outside packages/
• Running create data-stream outside the target package directory
• Keeping scaffold placeholder values for title/description/owner
• Accepting default version 0.0.1 — use 0.1.0
• Broad category selection instead of one or two relevant categories
• Replacing the entire doc template instead of editing within its structure
• Manually creating sample_event.json or *-expected.json
• Uncommenting {{ event "stream" }} before sample_event.json exists (causes build failure)
• Leaving the verbose CEL scaffold unchanged — strip unused vars and replace placeholder CEL program