Integration Testing

integration-testing

When to use

Load this skill whenever tasks include:

• authoring or debugging pipeline test fixtures (.log/.json inputs, *-expected.json output, config files)
• setting up or debugging system tests (_dev/test/system/, _dev/deploy/, mock APIs, 0-hits failures)
• writing script tests for failure paths, API error handling, or package upgrades

When not to use

Do not use this skill as the primary guide for:

• ingest pipeline processor design and architecture (ingest-pipelines)
• CEL program development (cel-programs)
• broad elastic-package command selection and stack lifecycle (elastic-package-cli)

Reference files — load the one that matches your test type

System test input-specific references

In addition to the generic system-testing.md, load the reference file matching your data stream's input type:

When an integration supports multiple input types, load the generic reference plus each applicable input-type reference.

When working across multiple test types in one task (e.g. creating a new data stream end-to-end), load all applicable reference files.

References

• references/pipeline-testing.md — directory layout, naming conventions, fixture formats, config options, expected output format and review, core workflow, fixture scenario coverage, data anonymization, troubleshooting
• references/system-testing.md — generic system test reference: required layout, config fields, core commands, teardown failures, sample_event.json verification, general debugging
• references/system-testing-cel.md — CEL mock API wiring, 0-hits debugging for CEL, variable-capture patterns
• references/system-testing-tcp-udp.md — TCP/UDP log sender pattern with elastic/stream, signal coordination, port alignment
• references/system-testing-http-endpoint.md — webhook/HTTP endpoint testing with STREAM_PROTOCOL=webhook, auth headers
• references/system-testing-logfile.md — Alpine container + SERVICE_LOGS_DIR pattern for logfile/filestream inputs
• references/system-testing-kafka-pubsub.md — Kafka broker + stream producer, Pub/Sub emulator patterns
• references/system-testing-cloud-skip.md — when and why to skip system tests for cloud storage inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub)
• references/script-testing.md — txtar format, env smoke test, system-level skeleton, mock service docker-compose and config.yml, upgrade test pattern, pitfalls, full o365 examples
• references/builder-setup-subagent-guidance.md — subagent operating manual for wiring data collection (docker-compose, sample logs, agent stream template, system test config, manifest var cleanup) for non-CEL data streams. The orchestrator dispatches subagents by passing this file's path in the task prompt; the subagent reads it itself in its own fresh context. Do NOT embed/paste its contents into the task prompt.
• references/builder-system-test-subagent-guidance.md — subagent operating manual for running elastic-package test system --generate after pipeline work completes (any testable input). Same dispatch rule as above: orchestrators pass the path, the subagent reads the file itself.

Integration builder — data-collection setup subagent guidance

Operating manual for a subagent wiring up the data-collection plumbing for a non-CEL data stream on behalf of the create-integration orchestrator (invoked from references/create-workflow.md or references/add-datastream-workflow.md).

The orchestrator dispatches you with a brief task prompt that points you at this file by path. Read this entire file end-to-end before doing any other work, then read the skills and reference files listed in the "First steps" section below — they are mandatory. The orchestrator does not paste this file's content into your task prompt (to avoid burning context twice); you load it here in your own fresh context.

The orchestrator's task prompt tells you which package and data stream to work on, which input type(s) to configure, what sample data is available, package-level vars that already exist, and any input-specific constraints. This file tells you how to operate as the data-collection setup subagent. Follow both.

Scope

Your responsibility is strictly limited to wiring data collection so a system test can later push sample data through the agent:

• _dev/deploy/docker/docker-compose.yml — service definition for the input

type (TCP/UDP sender, HTTP webhook client, logfile copier, Kafka broker + producer, Pub/Sub emulator + publisher)

• _dev/deploy/docker/sample_logs/ — anonymized sample log files
• data_stream/<stream>/agent/stream/<input>.yml.hbs — agent stream template

trimmed to vars the integration actually needs

• data_stream/<stream>/_dev/test/system/test-*-config.yml — system test

config(s) wiring the service to the agent

• data_stream/<stream>/manifest.yml — stream-level var cleanup, sensible

defaults, accurate title/description

• Root manifest.yml — package-level var cleanup, format/conditions version

enforcement

You do NOT:

• Build ingest pipelines (elasticsearch/ingest_pipeline/) — the pipeline

builder handles this (see ingest-pipelines/references/builder-subagent-guidance.md)

• Define field mappings (fields/fields.yml, fields/ecs.yml, etc.) — the

pipeline builder handles this

• Create pipeline test fixtures (_dev/test/pipeline/) — the pipeline builder

handles this

• Handle CEL programs or cel.yml.hbs templates — the CEL program builder

handles this (see cel-programs/references/builder-subagent-guidance.md)

• Run system tests (elastic-package test system) — the orchestrator

dispatches a separate system-test-mode invocation after the pipeline builder completes. See integration-testing/references/builder-system-test-subagent-guidance.md

• Create or modify sample_event.json — it is generated only by

elastic-package test system --generate in that later pass

If the orchestrator's prompt asks for system test execution rather than setup, stop and report that the wrong guidance file path was supplied — the system-test workflow lives in integration-testing/references/builder-system-test-subagent-guidance.md.

Skill authority

The rules and patterns in the skills and their reference files are the authoritative source of truth. You will examine 2–3 reference integrations in the official elastic/integrations repository for service patterns, but many of those integrations contain legacy patterns that predate current standards. Always follow the skills over patterns observed in other integrations. If a reference integration uses a deprecated docker-compose shape, manifest var convention, or template structure, do not copy it.

First steps — read the skills and their references

Before doing any work, read these skill files and the specific reference files listed to load the patterns you must follow. Reading only the SKILL.md files is not sufficient — the reference files contain the actual docker-compose recipes, test config shapes, and template var conventions you need.

1. `integration-testing` skill (SKILL.md) — then read references/system-testing.md (generic: required layout, system test config fields, wait_for_data_timeout: 1m, service_notify_signal, teardown failures, 0-hits debugging) and the input-specific reference matching the data stream's input type:

• tcp or udp: references/system-testing-tcp-udp.md
• http_endpoint: references/system-testing-http-endpoint.md
• logfile or filestream: references/system-testing-logfile.md
• kafka or gcp-pubsub: references/system-testing-kafka-pubsub.md

2. `input-configurations` skill (SKILL.md) — then read references/common-input-patterns.md (tags, processors passthrough, forwarded/publisher_pipeline.disable_host coupling, variable conventions) and the type-specific guide for the input type (references/tcp-guide.md, references/udp-guide.md, references/http-endpoint-guide.md, references/filestream-guide.md, references/logfile-guide.md, references/gcp-pubsub-guide.md, etc.). Load only the guide(s) for the input type(s) the data stream uses.

3. `create-integration` skill — read references/scaffold-commands.md for post-scaffold edits (manifest version enforcement, tz_offset rule for syslog, aws-s3 filename/SSL caveats, doc template handling, _dev/build/build.yml requirement, fields/beats.yml requirement).

4. `elastic-package-cli` skill — validation commands you will run (elastic-package format, lint, check).

5. `anonymize-logs` skill — placeholder conventions for sample log files and any data committed to the repository.

Read all skills and their relevant references before producing any files.

Workflow

1. Parse the orchestrator's prompt

From the task prompt, extract:

• Package path (absolute) and data stream name
• Input type(s) for the data stream (e.g. tcp,udp, http_endpoint,

logfile, kafka, gcp-pubsub)

• Sample log data (inline or file references) and log format (JSON, syslog,

CEF, key-value, multiline)

• Package-level vars already defined in the root manifest.yml (shared auth,

base URL) so you can reuse them

• Stream-specific constraints (ports, auth, TLS, multi-input combinations)
• Acceptance criteria

2. Examine 2–3 reference integrations

Search the official elastic/integrations github repository for 2–3 packages that already use the same input type if available. Examine each one's:

• _dev/deploy/docker/docker-compose.yml — service definitions, command

patterns, environment vars

• data_stream/<stream>/_dev/test/system/test-*-config.yml — config

structure and var wiring

• data_stream/<stream>/agent/stream/*.yml.hbs — template var usage
• data_stream/<stream>/manifest.yml — stream-level var definitions
• Root manifest.yml — package-level var placement (shared auth, URLs)

Use these as reference patterns only. Do not copy blindly — adapt to the integration's specific requirements, and follow the skills when they conflict with what a reference integration does.

If the path above does not exist in the current environment, ask the orchestrator where the reference integrations checkout lives (or proceed without it and rely solely on the skill references).

3. Create sample log files

Place representative sample log data in _dev/deploy/docker/sample_logs/:

• Use the data provided by the orchestrator. If the data is not yet

anonymized, apply the anonymize-logs skill's placeholder conventions before committing.

• Include enough lines to cover the expected event types and edge cases (at

minimum one happy-path event per format variant).

• Name files descriptively: <package>-<stream>.log,

<package>-<stream>.ndjson, etc.

• Note the exact line count — it drives assert.hit_count in the test

config.

4. Set up the Docker Compose service

Create or update _dev/deploy/docker/docker-compose.yml using the service pattern from the input-specific integration-testing reference for the data stream's input type. Apply only the parts of the pattern that the integration needs (e.g. add a TLS variant alongside TCP only when the integration must test TLS).

Key cross-cutting rules from the references:

• Use the exact `elastic/stream` image and command shape documented in

the input-specific reference — do not invent new flag combinations.

• Coordinate startup with service_notify_signal: SIGHUP for inputs that

require the agent to be listening before the sender starts (TCP, UDP, HTTP endpoint).

• For Kafka, include the healthcheck and depends_on: service_healthy

block so the producer waits for the broker.

• Use unique ports per service to avoid collisions when multiple input

variants share one data stream.

If a single data stream supports multiple inputs (e.g. tcp + udp + logfile), define one service per input variant in the same docker-compose.yml.

5. Write system test configs

Create data_stream/<stream>/_dev/test/system/test-<input>-config.yml for each input variant the data stream supports. Use the test config pattern from the input-specific reference and the required fields from integration-testing/references/system-testing.md:

• `wait_for_data_timeout: 1m` — required in every system test config
• input: <input_type>
• service: <docker-compose-service-name>
• service_notify_signal: SIGHUP — when the input-specific reference says

so (TCP, UDP, HTTP endpoint)

• data_stream.vars and vars matching the stream and package manifests
• assert.hit_count equal to the number of non-empty lines in the sample

log file

Use one config file per input variant rather than parameterising a single config.

6. Configure the agent stream template

Review and update data_stream/<stream>/agent/stream/<input>.yml.hbs. The scaffold produces a verbose, generic template — trim it to include only the vars the integration actually consumes, and apply the patterns from input-configurations/references/common-input-patterns.md:

• preserve_original_event is a conditional tag, not hardcoded.
• User-defined tags are iterated with {{#each tags as |tag|}}.
• If default tags include forwarded, the template must include the

matching publisher_pipeline.disable_host: true block (these two are always coupled).

• Top-level {{#if processors}} passthrough must be present.
• All user-configurable values are template vars, not hardcoded literals.

Follow the type-specific guide loaded in step 2 for input-specific template shape (TCP/UDP listener vars, HTTP endpoint URL + auth vars, logfile paths, Kafka topics/brokers, Pub/Sub credentials).

7. Clean up manifest vars

Audit both data_stream/<stream>/manifest.yml and the package root manifest.yml:

• Remove scaffold vars not referenced by any template (*.yml.hbs).
• Move shared vars (auth credentials, base URLs) to the package root when

they apply to multiple data streams; keep stream-specific vars at the stream level.

• Every var must have title, description, type, required, and a

sensible non-empty default for optional vars.

• Apply the tz_offset rule from scaffold-commands.md: include

tz_offset only for syslog (tcp/udp) streams where the source lacks a timezone. Never on cel, http_endpoint, or other non-syslog inputs.

• For syslog (tcp/udp) streams, always retain the ssl and

processors vars regardless of other simplification.

• For aws-s3 inputs, remove the scaffolded ssl var block (S3 uses the

AWS SDK, not direct TLS sockets) — see the scaffold-commands reference.

Root `manifest.yml` version enforcement (do this even if the scaffold generated other values):

• format_version: "3.4.2"
• conditions.kibana.version: "^8.19.0 || ^9.1.0"

These belong only in the root manifest — never duplicate them at the data stream level.

8. Validate

Run format, lint, and check from the package directory:

elastic-package format
elastic-package lint
elastic-package check

Fix any issues before reporting back. Do not run `elastic-package test system` — the orchestrator dispatches the system-test pass (which loads builder-system-test-subagent-guidance.md) after the pipeline builder completes pipeline work for this data stream.

What to return

When you finish, report:

• Files created or modified (with paths)
• Input type(s) configured
• Docker Compose service(s) defined and their patterns (TCP sender, webhook

client, Alpine copier, Kafka broker + producer, etc.)

• Sample log file details: name, format, line count
• System test config(s): service mapping, assert.hit_count, vars
• Template changes: vars kept, vars removed, forwarded /

publisher_pipeline.disable_host coupling status

• Manifest changes: vars cleaned up at data stream and package level,

format_version / conditions.kibana.version verification

• Validation results from elastic-package format/lint/check
• Any open issues or decisions that need user input (e.g. unresolved auth

details, port collisions, missing sample data variants)

Integration builder — system test subagent guidance

Operating manual for a subagent running elastic-package test system on behalf of the create-integration or maintain-integration orchestrator after pipeline work for a data stream has completed.

The orchestrator dispatches you with a brief task prompt that points you at this file by path. Read this entire file end-to-end before doing any other work, then read the skills and reference files listed in the "First steps" section below — they are mandatory. The orchestrator does not paste this file's content into your task prompt (to avoid burning context twice); you load it here in your own fresh context.

The orchestrator's task prompt tells you which package and data stream to test, which input type is in use, and confirms the Elastic stack is up. This file tells you how to operate as the system-test subagent. Follow both.

Scope

Your responsibility is strictly limited to:

• Confirming prerequisites (stack up, pipeline work complete, system test

config present and well-formed)

• Building the package (elastic-package build) so the test runs against a

fresh artifact

• Running elastic-package test system --data-streams <stream> --generate

for the specified data stream

• Reading failure logs (service container log first, then agent event log

when relevant)

• Fixing straightforward issues yourself (missing wait_for_data_timeout,

obvious field-name typos in test config, wrong service name) and rerunning

• Reporting pass/fail, error excerpts, and whether sample_event.json was

generated

You do NOT:

• Build, modify, or fix ingest pipelines — the pipeline builder owns

elasticsearch/ingest_pipeline/ and fields/ (see ingest-pipelines/references/builder-subagent-guidance.md). Report pipeline errors back to the orchestrator instead of patching them.

• Modify CEL programs or cel.yml.hbs templates — the CEL program builder

owns these (see cel-programs/references/builder-subagent-guidance.md). Report CEL errors back to the orchestrator.

• Modify mock API definitions (_dev/deploy/docker/files/config-*.yml) — the

CEL program builder owns the CEL mock; the setup-mode builder owns the non-CEL service definitions. Report mock issues back to the orchestrator.

• Set up or modify Docker Compose services or sample logs — that was done by

the setup-mode builder (or the CEL program builder for CEL streams). Report service-deployment issues back to the orchestrator.

• Create or modify sample_event.json manually — it is generated only by

elastic-package test system --generate. Never hand-write or edit it.

• Create or modify *-expected.json files — those are pipeline-test

artifacts and belong to the pipeline builder.

If the orchestrator's prompt asks for data-collection setup rather than running a system test, stop and report that the wrong guidance file path was supplied — the setup workflow lives in integration-testing/references/builder-setup-subagent-guidance.md.

First steps — read the skills and their references

Before doing any work, read these skill files and the specific reference files listed to load the patterns you must follow.

1. `integration-testing` skill (SKILL.md) — then read references/system-testing.md (generic: required layout, system test config fields, wait_for_data_timeout: 1m, --generate semantics, teardown failures, 0-hits debugging, agent event log inspection, common rejection reasons) and the input-specific reference matching the data stream's input type:

• cel: references/system-testing-cel.md
• tcp or udp: references/system-testing-tcp-udp.md
• http_endpoint: references/system-testing-http-endpoint.md
• logfile or filestream: references/system-testing-logfile.md
• kafka or gcp-pubsub: references/system-testing-kafka-pubsub.md

2. `elastic-package-cli` skill — elastic-package build, test system, stack down / stack up -d -v semantics.

Read all skills and their referenced files before running any commands.

Workflow

1. Confirm prerequisites

Before running anything, verify:

• The orchestrator has confirmed the Elastic stack is running. If not, the

orchestrator should run elastic-package stack up -d -v — do not start the stack yourself unless the orchestrator explicitly asks you to.

• The ingest pipeline, field definitions, and pipeline tests are complete

for this data stream (the orchestrator delegated those to the pipeline builder before invoking you).

• At least one system test config exists at

data_stream/<stream>/_dev/test/system/test-*-config.yml.

• Every test config includes wait_for_data_timeout: 1m. If a config is

missing it, add the field before running tests — this is the only test config edit you may make.

If any other prerequisite is missing (no system test config at all, no pipeline implementation, mock API config absent for a CEL stream), stop and report the gap back to the orchestrator. Do not synthesize the missing artifact.

2. Build the package

From the repository root:

cd packages/<package_name>
elastic-package build

Always build before running system tests when package files have changed (scaffold edits, pipeline, manifest, template). The build creates the deployable artifact elastic-package test system consumes.

3. Run the system test with --generate

elastic-package test system --data-streams <stream> --generate

The --generate flag is required. It produces data_stream/<stream>/sample_event.json from the first indexed document. Without it, no sample_event.json is produced and a separate run will be needed later. Never invoke test system without --generate for a fresh stream that has no sample_event.json yet.

If the data stream has multiple system test configs (e.g. TCP + UDP), this command runs them all sequentially. You do not need to invoke each config individually unless the orchestrator asks you to scope to one.

4. Triage failures

The order of investigation is fixed:

1. Service container log first. Read build/container-logs/<package_name>-<datastream_name>-<DIGIT>.log. This shows the requests/data the service container produced. For CEL streams, this is the mock-API conversation; for TCP/UDP streams, this is the elastic/stream sender output; for logfile streams, this is the Alpine copier output. Always check this log before anything else — it almost always reveals the root cause for 0-hits failures.

2. Agent event log (only when the service log shows the service produced data but events still did not index). Follow the system-testing.md → "Check the elastic-agent event log inside the running container" procedure to read /usr/share/elastic-agent/state/data/logs/events/elastic-agent-event-log-*.ndjson and look for "Cannot index event" rejections. Common rejections (mapper_parsing_exception, illegal_argument_exception, Duplicate field '@timestamp') point back to the pipeline or the field definitions.

3. Input-specific failure patterns. Cross-reference the failure with the input-specific reference loaded in step 1 (e.g. system-testing-cel.md covers infinite time-window looping, missing variable-capture catchall rules, 401/403 mock auth mismatches; system-testing-tcp-udp.md covers port misalignment, missing SIGHUP coordination).

5. Fix vs. re-delegate

Fix in place when the problem is:

• Missing wait_for_data_timeout: 1m in a test config
• Obviously wrong service: name (typo against the docker-compose service)
• Missing service_notify_signal: SIGHUP on a TCP/UDP/HTTP-endpoint test

that needs it

• An obvious mismatch between vars in the test config and the data

stream manifest var names

After any fix, rerun elastic-package build && elastic-package test system --data-streams <stream> --generate.

Re-delegate by reporting back to the orchestrator when the problem is:

• Pipeline errors (mapping conflicts, parse failures, missing fields,

mapper_parsing_exception, duplicate @timestamp, missing categorisation fields, JSE00001 violations) → pipeline builder territory

• CEL program errors (mock-api 401/403 with valid auth headers, infinite

loops indicating missing variable-capture rules, cursor never persisting, request/response shape mismatches) → CEL program builder territory (see cel-programs/references/builder-subagent-guidance.md)

• Missing or malformed mock API rules for CEL streams → CEL program builder

territory (see cel-programs/references/builder-subagent-guidance.md)

• Missing or malformed docker-compose services for non-CEL streams →

data-collection setup territory (see integration-testing/references/builder-setup-subagent-guidance.md)

• Sample log files clearly mismatched against the pipeline's expected

format → pipeline builder or setup-mode builder, depending on which side is wrong

Quote the specific error excerpts in your report so the orchestrator can hand them to the right subagent without re-investigating.

6. Recover from teardown failures

If teardown fails with an agent-policy or Fleet "still in use" error, do not try to fix Fleet state manually. Reset the stack:

elastic-package stack down
elastic-package stack up -d -v

Then rerun elastic-package build && elastic-package test system --data-streams <stream> --generate. This clears stale policies and agents from the previous run.

7. Verify sample_event.json was produced

After a passing run, confirm that data_stream/<stream>/sample_event.json was generated. If it was not (e.g. the run passed assert.hit_count but no document was retained for the sample), rerun with --generate once more. Never hand-write the file.

Inspect the generated sample_event.json for the basics documented in system-testing.md → "Verifying generated sample_event.json": correct ECS field nesting, geo fields under the right parent entity, dotted source fields properly nested, no obviously wrong values. If anything looks wrong, the pipeline is the root cause — report it back rather than editing the file.

What to return

Report:

• Pass / fail for each test config that ran
• Error excerpts (with file paths and line/timestamp context) for any

failures

• Whether sample_event.json was generated (and the path)
• Any teardown recovery you had to perform (stack down → up)
• Specific fixes you applied in place (with paths)
• Specific issues that require orchestrator intervention, classified by

domain (pipeline / CEL / mock API / docker-compose / sample logs) so the orchestrator can hand them to the right subagent

• Any open questions or decisions that need user input

pipeline testing

Everything needed to create, generate, and debug elastic-package test pipeline fixtures for ingest pipeline validation.

Test directory layout

Pipeline tests live at:

data_stream/<stream>/_dev/test/pipeline/

Canonical file patterns:

• input logs: test-<package>-<datastream>-<type>-sample.log
• input JSON events: test-<package>-<datastream>-<type>-sample.json
• optional config: test-common-config.yml or test-<package>-<datastream>-<type>-sample.<ext>-config.yml
• expected output: test-<package>-<datastream>-<type>-sample.<ext>-expected.json

Where:

• <package>: the package name (e.g. acme_firewall)
• <datastream>: the data stream name (e.g. event, traffic)
• <type>: the event type or log variant being tested (e.g. alert, auth, dns)

Example: test-acme_firewall-event-alert-sample.log

Always follow this convention. Do not use free-form names like test1.log.

Good scenario names describe behavior:

• test-acme_firewall-event-successful-login-sample.log
• test-acme_firewall-event-malformed-record-sample.log
• test-acme_firewall-traffic-dns-sample.log

Input types

Raw log fixtures (.log)

Use for line-oriented logs and multiline traces. Each event is normally one line unless multiline settings are configured.

67.43.156.13 - - [25/Oct/2016:14:49:33 +0200] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 ..."
67.43.156.13 - - [25/Oct/2016:14:49:34 +0200] "GET /favicon.ico HTTP/1.1" 404 571 "-" "Mozilla/5.0 ..."

JSON event fixtures (.json)

Use for structured input and explicit field control. JSON test fixtures use this shape:

{
  "events": [
    {
      "@timestamp": "2024-01-15T10:30:00.000Z",
      "message": "{\"cpu_usage\":85.2}"
    }
  ]
}

Config file options

Supported *-config.yml sections:

• fields: static fields injected before pipeline execution
• dynamic_fields: field-to-regex map for non-deterministic values (e.g. event.ingested)
• numeric_keyword_fields: list of field paths whose values arrive as numbers in the source but are declared as keyword in fields.yml. Do not artificially stringify these values in the fixture — declare them here instead. Common candidates: network.iana_number, port numbers, error codes, protocol or class codes.
• multiline: controls multiline record grouping for .log inputs

fields

Injects static values into each input event before the ingest pipeline runs. Common uses: stable @timestamp for deterministic tests, tags, and stream-specific config.

fields:
  "@timestamp": "2020-04-28T11:07:58.223Z"
  tags:
    - preserve_original_event
  event:
    timezone: "+0000"

dynamic_fields

Compares selected fields using regex instead of exact value. Use for non-deterministic fields — any field whose value changes between test runs (e.g., fields derived from _ingest.timestamp).

Note: Integration pipelines must NOT set event.ingested — it is managed by Elasticsearch outside the integration. If you encounter a legacy integration that sets event.ingested in its pipeline, include it in dynamic_fields; for new integrations following current standards, this entry is not needed.

If the pipeline sets @timestamp from _ingest.timestamp as a fallback (non-deterministic), add it:

dynamic_fields:
  "@timestamp": "^[0-9]{4}(-[0-9]{2}){2}T[0-9]{2}(:[0-9]{2}){2}\\.[0-9]{3}"

numeric_keyword_fields

Lists fields that may look numeric in test data but are mapped as keyword. Do not stringify the value in the fixture or add a convert processor — Elasticsearch coerces silently at index time. Declare them here:

numeric_keyword_fields:
  - zoom.meeting.id
  - ocsf.src_endpoint.type_id
  - network.iana_number

multiline

Groups multiple log lines into one event before pipeline simulation. Most common option is first_line_pattern:

multiline:
  first_line_pattern: "^\\d{4}\\/\\d{2}\\/\\d{2} "

multiline:
  first_line_pattern: '^(?:\{|\d{4}-\d{2}-\d{2})'
  negate: true
  match: after

Config placement strategy

• Always create a single test-common-config.yml that applies to all fixtures in the directory.
• The test-common-config.yml must always include fields.tags: [preserve_original_event]:

  fields:
    tags:
      - preserve_original_event

• Only add a per-test config file (test-<package>-<datastream>-<type>-sample.<ext>-config.yml) when a specific sample requires configuration that differs from the common config.
• Do not create per-test config files by default.

source.geo fields and dynamic_fields

Do not add source.geo.* (or other geo enrichment fields like destination.geo.*) to dynamic_fields. These fields are deterministic — they are produced by the geoip processor based on static IP-to-location databases and should produce consistent values in pipeline tests.

If source.geo fields cause pipeline test failures (mismatches, missing fields), the root cause is almost certainly an incorrect format_version or conditions.kibana.version in the root manifest.yml. The correct values are:

format_version: "3.4.2"
conditions:
  kibana:
    version: "^8.19.0 || ^9.1.0"

Fix the root manifest first. If failures persist after correcting the manifest, ignore the specific lint/validate complaints for those geo fields rather than adding them to dynamic_fields.

Expected output format

Expected files (*-expected.json) are JSON objects with an expected array. Each entry corresponds to one input event. Use null for events intentionally dropped by the pipeline.

Minimal example:

{
  "expected": [
    {
      "ecs": { "version": "9.1.0" },
      "event": { "kind": "event", "category": ["web"], "type": ["info"] }
    }
  ]
}

Typical log pipeline example:

{
  "expected": [
    {
      "@timestamp": "2016-10-25T12:49:33.000Z",
      "ecs": { "version": "8.11.0" },
      "event": {
        "category": ["web"],
        "kind": "event",
        "type": ["access"],
        "outcome": "success",
        "original": "67.43.156.13 - - [25/Oct/2016:14:49:33 +0200] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 ...\""
      },
      "http": { "request": { "method": "GET" }, "response": { "status_code": 200 } },
      "source": { "address": "67.43.156.13" },
      "url": { "original": "/" }
    }
  ]
}

Drop-path example:

{
  "expected": [null]
}

Expected output files must never be created or edited manually. Always generate them:

elastic-package test pipeline --data-streams <stream> --generate

Then review the generated diff to confirm the output is correct. If a field value is wrong, fix the ingest pipeline and regenerate — do not hand-edit the expected JSON.

Reviewing generated output

After running --generate, check:

• Parser extracted intended fields
• ECS categorization fields are correct (event.kind, event.category, event.type are arrays)
• Field types match expectations (string/number/object/array)
• Dotted field names from source data are correctly expanded into nested objects (e.g., "host.name": "x" in source becomes {"host": {"name": "x"}} in expected output)
• geo_point fields appear under the correct parent entity (source.geo.location, destination.geo.location) not at root
• event.original is present when preserve_original_event is set in config
• No accidental new fields from temporary pipeline state (e.g., _tmp)
• null entries only where drop behavior is intentional
• No accidental field drops

--generate records current behavior — it does not validate whether that behavior is correct.

Core workflow

# 1) Start local ES-only stack
elastic-package stack up -d --services=elasticsearch

# 2) Run scoped pipeline tests while iterating
elastic-package test pipeline --data-streams <stream>

# 3) Regenerate expected outputs when behavior changes intentionally
elastic-package test pipeline --data-streams <stream> --generate

# 4) Review diffs before commit
git diff data_stream/<stream>/_dev/test/pipeline/

Wiring provided log samples into test fixtures

When sample data is available, use it as the basis for pipeline test input files:

1. Sanitize first — no exceptions. Strip all customer data before writing any fixture file. See the data anonymization section below.

2. Determine the fixture format:

• Plain text / syslog lines → .log file, one event per line
• NDJSON (one JSON object per line) → .log file, one compact JSON object per line (do not pretty-print — that breaks the test runner)
• Structured JSON input events → .json file using the {"events": [...]} wrapper format

3. Split by event type. If the sample contains multiple distinct event types, create one fixture file per type. Do not combine unrelated event types into a single fixture.

4. Do not reformat or modify the log content. Write sanitized lines exactly as they appear. Only touch the values that need anonymization.

5. Generate and review expected output:

   elastic-package test pipeline --data-streams <stream> --generate

Fixture scenario coverage

Use a minimal but representative set:

• Happy path records (the normal case)
• Malformed record handling (what happens with bad input)
• Empty input / drop-path behavior (events the pipeline intentionally discards)
• Multiline cases (if the stream supports them)
• Boundary values (large IDs, unicode text, optional fields)

Prefer one scenario per file for clear diffs and easier failure diagnosis.

Reference package patterns

• nginx / apache: log fixtures with multiline and dynamic timestamp handling
• postgresql / auditd: multiline-heavy fixtures
• amazon_security_lake: extensive numeric_keyword_fields usage for OCSF IDs
• zoom and similar webhook streams: JSON fixtures with per-test config overrides
• agentless_hello_world: minimal JSON fixture shape

Data anonymization

Never commit customer data. All test fixture data must be fully anonymized before committing. No real production data, customer data, or identifiable information may appear in pipeline test inputs (test-*.log, test-*.json) or generated expected outputs (*-expected.json).

Replace every identifying value with a synthetic example of the same format: IP addresses, hostnames, email addresses, usernames, organization names, account IDs, API keys, and any other value traceable to a real person, system, or organization. Use RFC 5737 documentation IP ranges (198.51.100.x, 203.0.113.x), example.com/example.org domains, and realistic placeholder names. Replacements must preserve the structural shape that parsers depend on.

Refer to the anonymize-logs skill for the full anonymization policy and placeholder conventions.

Troubleshooting

• "field X is undefined" for ECS fields (e.g. field "destination.ip" is undefined):
• Root cause: missing or outdated _dev/build/build.yml
• Fix: create or update _dev/build/build.yml with dependencies.ecs.reference: "git@v9.3.0"
• Do not add ECS fields to fields.yml to work around this — fix build.yml instead
• Custom (non-ECS) fields reported as undefined must still be defined in the appropriate field files
• Unexpected diffs on time-like fields:
• Add or tighten dynamic_fields regex entries
• Multiline fixture split incorrectly:
• Adjust multiline.first_line_pattern (and negate / match if needed)
• Numeric vs keyword mismatch:
• Add the field path to numeric_keyword_fields in the test config
• Pipeline test cannot find expected behavior after parser change:
• Rerun scoped test, then regenerate with --generate, then review diff
• Pipeline resolution or syntax issues:
• Run elastic-package lint and verify ingest pipeline file paths

script testing

Everything needed to write elastic-package test script txtar tests covering failure paths, error handling, and package upgrades.

See also: elastic-package script testing docs · with_script example package

What script tests cover

Script tests are txtar files in <package>/data_stream/<ds>/_dev/test/scripts/. They run via elastic-package test script and complement pipeline and system tests by covering cases those tools cannot:

• Failure paths — API errors, invalid credentials, partial failures where some sub-requests fail while others succeed
• Invalid configuration — bad config values, missing required fields, unsupported options
• Package upgrades — data collection survives an in-place upgrade from a previous release
• Environment smoke tests — cheapest verification that the script testing infrastructure is wired correctly

They are not a replacement for system tests; they augment them.

Analyse the package first

Before writing any test, read and understand:

1. Input program (agent/stream/cel.yml.hbs, agent/stream/httpjson.yml.hbs, etc.) — map out how it calls the upstream API, which HTTP status codes it treats as success vs failure. This is critical: CEL programs often treat some non-200 codes as expected conditions (e.g. 400 = "already subscribed"); only status codes the program routes to error-handling will work for error-path tests.

2. Ingest pipeline (elasticsearch/ingest_pipeline/default.yml) — identify field renames and transformations. get_docs returns post-pipeline documents; assert against indexed field names, not the field names the input produces.

3. Existing system test configs (_dev/test/system/) — reuse credential values, tenant IDs, and other config values for consistency.

4. Existing deploy config (_dev/deploy/docker/) — the existing mock may already handle some endpoints. Script tests embed their own mocks in the txtar file, but copying the pattern keeps things consistent.

5. Changelog (changelog.yml) — read CURRENT_VERSION and PREVIOUS_VERSION. Check whether the latest change is a breaking change (needed for upgrade test guards).

Directory layout

<package>/data_stream/<ds>/_dev/test/scripts/
  env.txt                        # smoke test (write first)
  <scenario>.txt                 # one file per scenario

Environment smoke test (env.txt) — write first

Write this first for any new package. It verifies the script testing infrastructure populates environment variables without touching the stack:

[!exec:echo] skip 'Skipping test requiring absent echo command'

exec echo ${CONFIG_ROOT}
stdout '/\.elastic-package$'

exec echo ${CONFIG_PROFILES}
stdout '/\.elastic-package/profiles$'

exec echo ${PACKAGE_NAME}
stdout '^o365$'

exec echo ${PACKAGE_ROOT}
stdout '/packages/o365$'

exec echo ${DATA_STREAM}
stdout '^audit$'

exec echo ${DATA_STREAM_ROOT}
stdout '/packages/o365/data_stream/audit$'

exec echo ${CURRENT_VERSION}
stdout '^[0-9]+\.[0-9]+\.[0-9]+$'

exec echo ${PREVIOUS_VERSION}
stdout '^[0-9]+\.[0-9]+\.[0-9]+$'

Use regex for version assertions so the test doesn't break on every release.

System-level test skeleton

Every system-level script test follows this structure:

# Description of what this test verifies.

[!external_stack] skip 'Skipping external stack test.'
[!exec:jq] skip 'Skipping test requiring absent jq command'

# 1. Connect.
use_stack -profile ${CONFIG_PROFILES}/${PROFILE}
install_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME

# 2. Start mock service.
docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} <mock-name>

# 3. Install package and create policy.
add_package -profile ${CONFIG_PROFILES}/${PROFILE}
add_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME

# 4. Assert.
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want <N> -timeout 5m ${DATA_STREAM_NAME}
cp stdout got_docs.json
exec jq '<query>' got_docs.json
stdout '<expected>'

# 5. Clean up.
remove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m
docker_down <mock-name>

Embedded files in the txtar

The -- filename -- sections at the bottom of each txtar file contain:

• `test_config.yaml` — package policy config (input type and vars). Point the integration at the mock service and set short intervals/windows.
• `<mock-name>/docker-compose.yml` — Docker Compose for the mock service.
• `<mock-name>/config.yml` — elastic/stream mock rules.

test_config.yaml

input: cel
vars: ~
data_stream:
  vars:
    url: http://<mock-name>:8080
    interval: 30s
    initial_interval: 1h
    preserve_original_event: true
    # ... other required vars with test values ...

Set interval and initial_interval to small values. Large defaults (e.g. 168h lookback) cause many API calls against the mock, slowing the test and making assertions harder to predict.

Upgrade test pattern

[!external_stack] skip 'Skipping external stack test.'
[!has_previous_release] skip 'No previous release to upgrade from.'
[breaking_change] skip 'Cannot upgrade across breaking change.'
[!exec:jq] skip 'Skipping test requiring absent jq command'

# ... setup, add_package, add_package_policy, verify initial data ...

upgrade_package_latest -profile ${CONFIG_PROFILES}/${PROFILE}
stdout 'upgraded package '${PACKAGE_NAME}

# ... verify data still present after upgrade ...

# ... cleanup ...

The [!has_previous_release] and [breaking_change] guards skip automatically when appropriate.

Mock service setup

Use the elastic/stream Docker image (docker.elastic.co/observability/stream). Match the version used in the package's existing _dev/deploy/docker/ config.

docker-compose.yml

version: '2.3'
services:
  <mock-name>:
    image: docker.elastic.co/observability/stream:v0.19.0
    hostname: <mock-name>
    ports:
      - 8080
    environment:
      PORT: "8080"
    volumes:
      - ./config.yml:/config.yml
    command:
      - http-server
      - --addr=:8080
      - --config=/config.yml

Always set hostname: <mock-name> explicitly. Without it the container defaults to the container ID, which the agent cannot resolve on the Docker network.

config.yml structure

Rules are matched top-down. Each rule specifies path, methods, optional query params, optional request headers, and responses:

rules:
  - path: /api/v1/items
    methods: [GET]
    query_params:
      api_key: test-key
    responses:
      - status_code: 200
        headers:
          Content-Type:
            - "application/json"
        body: |-
          {"items": [...]}

Template variables available in response bodies:

• `{{ hostname }}` — Docker container hostname (required for self-referential URLs, e.g. next-page links)
• `{{ env "PORT" }}` — value of the PORT environment variable
• `{varName:regex}` — in query_params values, captures variable request parameters (e.g. startTime: "{startTime:.*}")
• `{{ .request.vars.varName }}` — references a captured query param in the response body

Variable-capture catchall rules are essential for APIs with dynamic query parameters (timestamps, cursor tokens). Without them, CEL programs with timestamp-cursor pagination loop indefinitely. Always include {varName:.*} captures for all dynamic query params:

- path: /api/v1/events
  methods: ['GET']
  query_params:
    startTime: "{startTime:.*}"
    endTime: "{endTime:.*}"
  request_headers:
    Authorization:
      - "Bearer test-token"
  responses:
    - status_code: 200
      headers:
        Content-Type:
          - "application/json"
      body: |-
        {"events":[...], "nextPage": "http://{{ hostname }}:{{ env "PORT" }}/api/v1/events?cursor={{ .request.vars.endTime }}"}

Put more specific rules (with query_params or request_headers) before less specific ones for the same path.

Running script tests

# Run all script tests for a data stream:
elastic-package test script -v --data-streams <data_stream>

# Run a single test:
elastic-package test script -v --data-streams <data_stream> --run <test_name>

# Keep work directory for debugging:
elastic-package test script -v --data-streams <data_stream> --work

# Verbose script output:
elastic-package test script -v --data-streams <data_stream> --verbose-scripts

Tests require a running Elastic stack (--external-stack, which is the default).

Pitfalls and hard-won lessons

Assert against indexed fields, not input fields

get_docs returns post-pipeline documents. If the ingest pipeline renames foobar to foo.bar, assert against foo.bar. Always check the pipeline first.

Do NOT call remove_package in cleanup

For real packages, Kibana often refuses removal because Fleet hasn't finished cascading the agent policy deletion to package policies. The script test runner's automatic cleanup handles package removal. Omit `remove_package` from cleanup.

Read the input's error-handling logic before choosing mock responses

CEL and httpjson programs often treat some non-200 HTTP codes as expected, non-error conditions (e.g. 400 = "already subscribed", 404 = "no new data"). A mock returning such a code won't trigger the error path. Read the program to find which status codes actually reach error-handling code, then mock those.

Use -confirm with get_docs for exact-count assertions

When asserting that exactly N documents arrive and no more, add -confirm 15s:

get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -confirm 15s -timeout 5m ${DATA_STREAM_NAME}

This waits an extra 15 seconds after reaching N to confirm no additional documents arrive.

Minimise polling and batch windows in test configs

Set short initial_interval, interval, and batch_size values in test_config.yaml so each test cycle makes a predictable, minimal number of API requests.

Set hostname in docker-compose for mock services

Without an explicit hostname: <mock-name> in docker-compose, the container defaults to its container ID. Agents on the Docker network cannot resolve container IDs.

Full examples (o365 integration)

Partial API failure — subscription_permission_error.txt

Tests that a permission error (401 AF10001) for one content type does not prevent collection from other content types. Three content types configured; one rejected by the mock. Expects 1 error event + 2 data events.

Key decisions:

• Uses 401 (not 400) because the CEL program treats 400 as success ("already subscribed").
• Asserts against .o365.audit (post-pipeline name), not .o365audit.
• No remove_package in cleanup.

# Test that a subscription permission error (AF10001) for one content type
# does not prevent collection from other content types.

[!external_stack] skip 'Skipping external stack test.'
[!exec:jq] skip 'Skipping test requiring absent jq command'

use_stack -profile ${CONFIG_PROFILES}/${PROFILE}
install_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME
docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} o365-mock
add_package -profile ${CONFIG_PROFILES}/${PROFILE}
add_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME

# Wait for documents: 1 error event + 2 data events = 3.
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 3 -timeout 5m ${DATA_STREAM_NAME}
cp stdout got_docs.json

# Verify error event mentions the rejected content type.
exec jq -r '[.hits.hits[]._source.error.message // empty] | flatten | .[]' got_docs.json
stdout 'Audit.TypeRequiringAdditionalPermissions'

# Verify 2 normal data events from the working content types.
exec jq '[.hits.hits[]._source | select(.o365.audit != null)] | length' got_docs.json
stdout '^2$'

# Clean up.
remove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m
docker_down o365-mock

Total API failure — invalid_content_type.txt

Tests that a single invalid content type results in exactly one error event and no data. Uses 403 (not 400) to trigger the error path. Uses -confirm 15s to verify no extra documents arrive.

# Test that configuring a single invalid content type results in an error
# event and no data collection.

[!external_stack] skip 'Skipping external stack test.'
[!exec:jq] skip 'Skipping test requiring absent jq command'

use_stack -profile ${CONFIG_PROFILES}/${PROFILE}
install_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME
docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} o365-mock
add_package -profile ${CONFIG_PROFILES}/${PROFILE}
add_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME

get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 1 -confirm 15s -timeout 5m ${DATA_STREAM_NAME}
cp stdout got_docs.json

exec jq -r '[.hits.hits[]._source.error.message // empty] | flatten | .[]' got_docs.json
stdout 'Audit.Nonexistent'

exec jq '[.hits.hits[]._source | select(.o365.audit != null)] | length' got_docs.json
stdout '^0$'

remove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m
docker_down o365-mock

Package upgrade — upgrade.txt

Tests that upgrading the package doesn't break data collection. Skips automatically if there's no previous release or if the latest change is a breaking change.

[!external_stack] skip 'Skipping external stack test.'
[!has_previous_release] skip 'No previous release to upgrade from.'
[breaking_change] skip 'Cannot upgrade across breaking change.'
[!exec:jq] skip 'Skipping test requiring absent jq command'

use_stack -profile ${CONFIG_PROFILES}/${PROFILE}
install_agent -profile ${CONFIG_PROFILES}/${PROFILE} -network_name NETWORK_NAME
docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} o365-mock
add_package -profile ${CONFIG_PROFILES}/${PROFILE}
add_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME

# Verify initial data collection.
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -timeout 5m ${DATA_STREAM_NAME}
cp stdout got_pre_upgrade.json
exec jq '.hits.total.value' got_pre_upgrade.json
stdout '^2$'

# Upgrade.
upgrade_package_latest -profile ${CONFIG_PROFILES}/${PROFILE}
stdout 'upgraded package '${PACKAGE_NAME}

# Verify data survives the upgrade.
get_docs -profile ${CONFIG_PROFILES}/${PROFILE} -want 2 -timeout 2m ${DATA_STREAM_NAME}
cp stdout got_post_upgrade.json
exec jq '.hits.total.value >= 2' got_post_upgrade.json
stdout '^true$'

remove_package_policy -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m
docker_down o365-mock

Mock config.yml — o365 full example

Handles OAuth token exchange, subscription endpoints with per-content-type routing, content listing with variable-capture, and content fetch:

rules:
  # OAuth token endpoint.
  - path: /test-cel-tenant-id/oauth2/v2.0/token
    methods: [POST]
    query_params:
      client_id: test-cel-client-id
      client_secret: test-cel-client-secret
      grant_type: client_credentials
      scope: https://manage.office.com/.default
    request_headers:
      Content-Type:
        - "application/x-www-form-urlencoded"
    responses:
      - status_code: 200
        headers:
          Content-Type:
            - "application/json"
        body: |-
          {"access_token":"test-token","token_type":"Bearer","expires_in":3600}

  # Subscribe -- success.
  - path: /api/v1.0/test-cel-tenant-id/activity/feed/subscriptions/start
    methods: [POST]
    query_params:
      contentType: "Audit.SharePoint"
      PublisherIdentifier: test-cel-tenant-id
    request_headers:
      Authorization:
        - "Bearer test-token"
    responses:
      - status_code: 200
        headers:
          Content-Type:
            - "application/json"
        body: |-
          {"contentType":"Audit.SharePoint","status":"enabled","webhook":null}

  # Subscribe -- error (permission denied for this content type).
  - path: /api/v1.0/test-cel-tenant-id/activity/feed/subscriptions/start
    methods: [POST]
    query_params:
      contentType: "Audit.TypeRequiringAdditionalPermissions"
      PublisherIdentifier: test-cel-tenant-id
    request_headers:
      Authorization:
        - "Bearer test-token"
    responses:
      - status_code: 401
        headers:
          Content-Type:
            - "application/json"
        body: |-
          {"error":{"code":"AF10001","message":"Permission denied."}}

  # List content -- variable-capture for dynamic time params + self-referential URL.
  - path: /api/v1.0/test-cel-tenant-id/activity/feed/subscriptions/content
    methods: [GET]
    query_params:
      contentType: "Audit.SharePoint"
      startTime: "{startTime:.*}"
      endTime: "{endTime:.*}"
      PublisherIdentifier: test-cel-tenant-id
    request_headers:
      Authorization:
        - "Bearer test-token"
    responses:
      - status_code: 200
        headers:
          Content-Type:
            - "application/json"
        body: |-
          [{"contentType":"Audit.SharePoint","contentId":"sp-1","contentUri":"http://{{ hostname }}:{{ env "PORT" }}/api/v1.0/test-cel-tenant-id/activity/feed/audit/sp-1","contentCreated":"{{ .request.vars.endTime }}","contentExpiration":"2199-12-31T23:59:59.000Z"}]

  # Fetch content.
  - path: /api/v1.0/test-cel-tenant-id/activity/feed/audit/sp-1
    methods: [GET]
    request_headers:
      Authorization:
        - "Bearer test-token"
    responses:
      - status_code: 200
        headers:
          Content-Type:
            - "application/json"
        body: |-
          [{"Id":"sp-event-001","CreationTime":"2020-02-07T16:43:53","Workload":"SharePoint","Operation":"PageViewed","RecordType":4}]

Key patterns:

• `{{ hostname }}` — resolves to the Docker container hostname (set via hostname: in docker-compose). Required for self-referential URLs such as next-page links.
• `{{ env "PORT" }}` — resolves to the PORT environment variable.
• `{varName:regex}` — captures variable request values in query params.
• `{{ .request.vars.endTime }}` — references a captured query param in the response body.
• Rules are matched top-down; put more specific rules before less specific ones on the same path.

test_config.yaml — o365 example

input: cel
vars: ~
data_stream:
  vars:
    url: http://o365-mock:8080
    token_url: http://o365-mock:8080
    preserve_original_event: true
    client_id: test-cel-client-id
    client_secret: test-cel-client-secret
    azure_tenant_id: test-cel-tenant-id
    content_types: "Audit.SharePoint, Audit.General"
    interval: 30s
    initial_interval: 1h
    enable_request_tracer: false

initial_interval: 1h with interval: 30s gives one listing batch per content type per cycle. Longer values generate many batches and produce more documents than the assertion expects.

system testing — CEL input

Input-specific guidance for system-testing data streams that use the cel input. Load system-testing.md (generic) first.

Overview

CEL system tests require a mock HTTP API to stand in for the real vendor API. The mock is an elastic/stream http-server container defined in _dev/deploy/docker/docker-compose.yml with a rule-based config file.

Docker Compose pattern

version: '2.3'
services:
  <package>-<stream>-mock:
    image: docker.elastic.co/observability/stream:v0.20.0
    volumes:
      - ./files:/files:ro
    command: http-server --addr=:8090 --config=/files/config-<stream>.yml
    ports:
      - 8090

The mock config file at _dev/deploy/docker/files/config-<stream>.yml contains rule-based request matching with response definitions.

Test config pattern

wait_for_data_timeout: 1m
input: cel
service: <package>-<stream>-mock
data_stream:
  vars:
    url: http://{{Hostname}}:{{Port}}
assert:
  hit_count: <expected_count>

Key patterns

• Rule-based mock config: rules match on path, methods, query_params, and request_headers (first match wins, top-down ordering)
• Variable-capture patterns: use {varName:regex} in query param values for dynamic params like timestamps — without these, the mock returns the same data for every time window, causing infinite loops
• Two-round cursor testing: set interval: 2s in test config so the agent completes one pagination cycle, persists the cursor, then fires a second cycle to verify cursor persistence
• `assert.hit_count`: must account for events from both evaluation rounds

Debugging 0 hits (CEL-specific)

Step 1: Check the mock container log (always first)

build/container-logs/<package_name>-<datastream_name>-<DIGIT>.log

This log shows every HTTP request the CEL program makes against the mock API, including which rules matched. Tail the last 100–200 lines as the first diagnostic step.

Common patterns:

• Hundreds of requests with advancing timestamps: infinite time-window looping — the mock lacks variable-capture catchall rules
• No requests at all: the agent never contacted the mock — check url and service: in the test config
• Requests with no rule matches: query param or header mismatch between CEL program and mock rules
• 401/403 responses: auth rule mismatch in mock config

Step 2: Check agent log for dropped events

Follow the generic debugging steps in system-testing.md → "Debugging system test failures — general".

A CEL-specific rejection reason:

• Duplicate field '@timestamp': the CEL program sets @timestamp in event output while the framework also adds it. Fix: only emit {"message": e.encode_json()}.

Additional CEL-specific failure patterns

• 0 hits with CEL input: check container logs first (Step 1) before any other debugging
• Infinite time-window looping: mock config lacks variable-capture catchall rules — fix by adding startTime: "{startTime:.*}" patterns

Detailed mock setup reference

For comprehensive mock API flow design, variable-capture syntax, pagination mocking, and two-round cursor persistence patterns, see the cel-programs skill → references/cel-system-tests.md.

Reference integrations

• Any CEL integration in elastic/integrations — e.g. wiz, ti_otx, canva — uses elastic/stream http-server mocks with rule configs

system testing — cloud storage inputs (skip guidance)

Guidance for data streams that use aws-s3, gcs, azure-blob-storage, or azure-eventhub inputs. Load system-testing.md (generic) first.

When to skip system tests

Cloud storage and cloud message bus inputs (aws-s3, gcs, azure-blob-storage, azure-eventhub) currently do not have a standard docker-based system test pattern. These inputs require cloud infrastructure (S3 buckets, SQS queues, GCS buckets, Azure Blob containers, Event Hubs) that cannot be reliably emulated in a local Docker environment.

Some integrations in the official repo use Terraform-based system tests (_dev/deploy/tf/) that provision real AWS/GCP/Azure resources in CI, but these are:

• Not part of the standard elastic-package test system docker workflow
• Require cloud credentials and infrastructure
• Not suitable for local development or standard CI pipelines

What to do instead

1. Focus on pipeline tests: create thorough pipeline test fixtures (_dev/test/pipeline/) covering all event types, edge cases, and error paths. Pipeline tests validate the ingest pipeline independently of the input.

2. Skip system test setup: do not create _dev/deploy/docker/docker-compose.yml or _dev/test/system/ test configs for cloud storage inputs unless a docker-based mock is available.

3. `sample_event.json` generation: since elastic-package test system --generate cannot run without a system test, sample_event.json must be created through alternative means:

• Run the pipeline against a representative fixture using elastic-package test pipeline, then construct sample_event.json from the expected output
• Or set up a temporary local stack run with real cloud credentials to generate it

4. Note in the report: when the orchestrator skips system tests for cloud inputs, include this in the final report: "System tests skipped for <input> data stream — no docker-based mock available for cloud storage inputs. Pipeline tests provide coverage."

Integrations with Terraform-based tests (for reference)

Some integrations use _dev/deploy/tf/ with Terraform and real cloud resources:

• crowdstrike (FDR) — uses {{TF_OUTPUT_queue_url}} with real AWS SQS
• github (audit) — combines TF for S3 with Docker for GCS mock and Azurite

These patterns require cloud credentials (AWS_ACCESS_KEY_ID, etc.) and are only suitable for CI environments with cloud access.

Partial docker-based mocks (advanced)

A few integrations use docker-based mocks for cloud storage:

• GCS: shourieg/gcs-mock-service provides a basic GCS-compatible API
• Azure Blob: mcr.microsoft.com/azure-storage/azurite emulates Azure Blob Storage locally

These are not widely adopted and require additional setup (uploading test data, configuring endpoints). If the integration warrants it, examine google_cloud_storage or symantec_endpoint_security in the official repo for working examples.

Reference integrations

• `crowdstrike` (FDR) — TF-based aws-s3 tests
• `google_cloud_storage` — docker GCS mock
• `symantec_endpoint_security` — aws-s3, GCS, and Azure Blob test configs

system testing — HTTP endpoint (webhook) input

Input-specific guidance for system-testing data streams that use the http_endpoint input. Load system-testing.md (generic) first.

Overview

HTTP endpoint system tests use elastic/stream as an HTTP client that posts NDJSON log data to the Elastic Agent's http_endpoint listener. The stream tool sends data as webhook-style HTTP POST requests using STREAM_PROTOCOL=webhook.

Docker Compose pattern

HTTP

version: '2.3'
services:
  <package>-<stream>-webhook-http:
    image: docker.elastic.co/observability/stream:v0.20.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    environment:
      - STREAM_PROTOCOL=webhook
      - STREAM_WEBHOOK_PROBE=false
      - STREAM_ADDR=http://elastic-agent:<port>/<url_path>
      - STREAM_WEBHOOK_HEADER=Authorization=<auth_value>
    command: log --start-signal=SIGHUP --delay=5s /sample_logs/<logfile>.log

HTTPS

  <package>-<stream>-webhook-https:
    image: docker.elastic.co/observability/stream:v0.20.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    environment:
      - STREAM_PROTOCOL=webhook
      - STREAM_WEBHOOK_PROBE=false
      - STREAM_INSECURE=true
      - STREAM_ADDR=https://elastic-agent:<port>/<url_path>
      - STREAM_WEBHOOK_HEADER=Authorization=<auth_value>
    command: log --start-signal=SIGHUP --delay=5s /sample_logs/<logfile>.log

With basic auth

    environment:
      - STREAM_PROTOCOL=webhook
      - STREAM_WEBHOOK_PROBE=false
      - STREAM_ADDR=http://elastic-agent:<port>/<url_path>
      - STREAM_WEBHOOK_USERNAME=abc123
      - STREAM_WEBHOOK_PASSWORD=abc123

Test config pattern

wait_for_data_timeout: 1m
service: <package>-<stream>-webhook-http
service_notify_signal: SIGHUP
input: http_endpoint
data_stream:
  vars:
    listen_address: 0.0.0.0
    listen_port: <port>
    url: /<url_path>
    secret_value: <auth_value>
    preserve_original_event: true
assert:
  hit_count: <line_count>

Key patterns

• `STREAM_PROTOCOL=webhook`: tells stream to POST data as HTTP webhook requests instead of raw TCP/UDP
• `STREAM_WEBHOOK_PROBE=false`: disables the probe request that some webhook receivers expect
• `STREAM_ADDR`: full URL including protocol, host (elastic-agent), port, and path. The path must match url in the test config vars.
• `STREAM_WEBHOOK_HEADER`: sets custom headers on each POST request — used for auth tokens, API keys. Format: HeaderName=value.
• `STREAM_INSECURE=true`: required for HTTPS when using self-signed certificates
• Auth alignment: the auth value in STREAM_WEBHOOK_HEADER (or STREAM_WEBHOOK_USERNAME/PASSWORD) must match the auth config in the test config vars (secret_value, basic_auth, etc.)
• Sample log file: place NDJSON files in _dev/deploy/docker/sample_logs/. Each line is posted as one HTTP request body.
• Content-Type: set STREAM_WEBHOOK_HEADER=Content-Type=application/json when the endpoint expects JSON

Reference integrations

• `zoom` — HTTP and HTTPS webhook test configs
• `cloudflare_logpush` — multiple data streams with webhook tests
• `http_endpoint` — reference implementation with basic auth and ack modes

system testing — Kafka and Pub/Sub inputs

Input-specific guidance for system-testing data streams that use kafka or gcp-pubsub inputs. Load system-testing.md (generic) first.

Kafka

Overview

Kafka system tests run a real Kafka broker (Kraft mode) alongside an elastic/stream producer that publishes sample log lines to a topic. The Elastic Agent consumes from the topic using the kafka input.

Docker Compose pattern

version: '2.3'
services:
  kafka-service:
    image: bashj79/kafka-kraft
    healthcheck:
      test: nc -z kafka-service 9094 || exit -1
      interval: 10s
      timeout: 5s
      retries: 15
    environment:
      KAFKA_LISTENERS: "INTERNAL://kafka-service:9092,EXTERNAL://:9094,CONTROLLER://:9093"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-service:9092,EXTERNAL://kafka-service:9094"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    ports:
      - 9094

  <package>-<stream>-producer:
    image: docker.elastic.co/observability/stream:v0.18.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    command:
      - log
      - --retry=30
      - --addr=kafka-service:9094
      - -p=kafka
      - --kafka-topic=<topic_name>
      - /sample_logs/<logfile>.log
    depends_on:
      kafka-service:
        condition: service_healthy

Test config pattern

wait_for_data_timeout: 1m
service: kafka-service
input: kafka
data_stream:
  vars:
    topics:
      - <topic_name>
    hosts:
      - "{{Hostname}}:{{Port}}"
    group_id: system_test

Key patterns

• `service: kafka-service`: the test config references the broker service, not the producer
• `{{Hostname}}:{{Port}}`: resolved by the test runner to the broker's advertised address
• `--retry=30`: the producer retries connecting to the broker since it may take time to become healthy
• `depends_on` with `service_healthy`: ensures the broker is ready before the producer starts
• `--kafka-topic`: must match the topics list in the test config

GCP Pub/Sub

Overview

Pub/Sub system tests use the Google Cloud Pub/Sub emulator alongside an elastic/stream publisher. The Elastic Agent consumes messages using the gcp-pubsub input pointed at the emulator.

Docker Compose pattern

version: '2.3'
services:
  gcppubsub-emulator:
    image: google/cloud-sdk:emulators
    command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8681
    ports:
      - "8681/tcp"

  <package>-<stream>-publisher:
    image: docker.elastic.co/observability/stream:v0.18.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    command:
      - log
      - --retry=30
      - --addr=gcppubsub-emulator:8681
      - -p=gcppubsub
      - --gcppubsub-clear=true
      - --gcppubsub-project=<project_id>
      - /sample_logs/<logfile>.log
    depends_on:
      - gcppubsub-emulator

Test config pattern

wait_for_data_timeout: 1m
service: gcppubsub-emulator
input: gcp-pubsub
vars:
  alternative_host: "{{Hostname}}:{{Port}}"
  credentials_json: >-
    {"type":"service_account","project_id":"<project_id>"}
  project_id: <project_id>
  subscription_name: subscription
  topic: topic

Key patterns

• `alternative_host`: points the agent at the emulator instead of real GCP
• `--gcppubsub-clear=true`: resets the emulator state before publishing
• `credentials_json`: a stub service account credential sufficient for the emulator

Reference integrations

• `kafka_log` — Kafka Kraft broker with stream producer
• `gcp_pubsub` — Pub/Sub emulator with stream publisher

system testing — logfile/filestream input

Input-specific guidance for system-testing data streams that use logfile or filestream inputs. Load system-testing.md (generic) first.

Overview

Log file system tests use a lightweight Alpine container that copies sample log files into a shared volume (SERVICE_LOGS_DIR). The Elastic Agent then reads the files using the logfile or filestream input with glob paths.

Docker Compose pattern

version: '2.3'
services:
  <package>-<stream>-logfile:
    image: alpine
    volumes:
      - ./sample_logs:/sample_logs:ro
      - ${SERVICE_LOGS_DIR}:/var/log
    command: /bin/sh -c "cp /sample_logs/* /var/log/"

The Alpine container runs cp to copy all sample log files into the SERVICE_LOGS_DIR volume, then exits. The Elastic Agent detects the new files and processes them.

For integrations that also support network inputs (TCP/UDP), combine both service types in the same docker-compose:

version: '2.3'
services:
  <package>-<stream>-logfile:
    image: alpine
    volumes:
      - ./sample_logs:/sample_logs:ro
      - ${SERVICE_LOGS_DIR}:/var/log
    command: /bin/sh -c "cp /sample_logs/* /var/log/"

  <package>-<stream>-tcp:
    image: docker.elastic.co/observability/stream:v0.18.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:<port> -p=tcp /sample_logs/<logfile>.log

Test config pattern

wait_for_data_timeout: 1m
service: <package>-<stream>-logfile
input: logfile
vars:
  paths:
    - "{{SERVICE_LOGS_DIR}}/*.log"

With additional options:

wait_for_data_timeout: 1m
service: <package>-<stream>-logfile
input: logfile
data_stream:
  vars:
    preserve_original_event: true
vars:
  paths:
    - "{{SERVICE_LOGS_DIR}}/*<stream>*.log"
  tz_offset: "+0500"

Key patterns

• `${SERVICE_LOGS_DIR}`: environment variable populated by the test runner, points to a shared volume between the Alpine container and the Elastic Agent
• `{{SERVICE_LOGS_DIR}}`: the Handlebars placeholder used in the test config (resolves to the same path at runtime)
• Alpine `cp` command: the simplest approach — just copies files. The container exits after the copy, which is fine since the agent reads the files afterward.
• Glob paths: use {{SERVICE_LOGS_DIR}}/*.log for all log files, or {{SERVICE_LOGS_DIR}}/*<name>*.log to select specific files
• Sample log placement: place log files in _dev/deploy/docker/sample_logs/
• No `service_notify_signal`: unlike TCP/UDP, logfile tests do not need signal coordination — the agent detects files via filesystem polling
• `tz_offset`: if the data stream manifest includes tz_offset for timezone handling, include it in test vars

Reference integrations

• `checkpoint` — Alpine container pattern with TCP/UDP/TLS alongside
• `panw` — logfile with specific glob patterns
• `f5_bigip` — combines logfile (Alpine) and http_endpoint in one compose

system testing — TCP/UDP input

Input-specific guidance for system-testing data streams that use tcp or udp inputs. Load system-testing.md (generic) first.

Overview

TCP/UDP system tests use elastic/stream as a log sender that replays sample log files to the Elastic Agent's listening port. The stream tool sends lines over TCP, UDP, or TLS, coordinated with service_notify_signal: SIGHUP so the agent starts listening before the sender starts transmitting.

Docker Compose pattern

version: '2.3'
services:
  <package>-<stream>-tcp:
    image: docker.elastic.co/observability/stream:v0.18.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:<port> -p=tcp /sample_logs/<logfile>.log

  <package>-<stream>-udp:
    image: docker.elastic.co/observability/stream:v0.18.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:<port> -p=udp /sample_logs/<logfile>.log

For TLS-over-TCP, use -p=tls --insecure:

  <package>-<stream>-tls:
    image: docker.elastic.co/observability/stream:v0.18.0
    volumes:
      - ./sample_logs:/sample_logs:ro
    command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:<port> -p=tls --insecure /sample_logs/<logfile>.log

Test config pattern

TCP

wait_for_data_timeout: 1m
service: <package>-<stream>-tcp
service_notify_signal: SIGHUP
input: tcp
data_stream:
  vars:
    listen_address: 0.0.0.0
    listen_port: <port>
    preserve_original_event: true
assert:
  hit_count: <line_count>

UDP

wait_for_data_timeout: 1m
service: <package>-<stream>-udp
service_notify_signal: SIGHUP
input: udp
data_stream:
  vars:
    listen_address: 0.0.0.0
    listen_port: <port>
assert:
  hit_count: <line_count>

Key patterns

• `service_notify_signal: SIGHUP`: required so the stream tool waits for the agent to be ready before sending. The --start-signal=SIGHUP flag on the stream command listens for this signal.
• `--delay=5s`: adds a small delay after receiving the signal before sending, giving the agent time to fully initialize.
• Port alignment: the port in --addr elastic-agent:<port> must match listen_port in the test config. Use unique ports per service to avoid collisions.
• Sample log file: place log files in _dev/deploy/docker/sample_logs/. Each line in the file becomes one event.
• `assert.hit_count`: should match the number of non-empty lines in the sample log file.
• Multiple test configs: create separate test-tcp-config.yml and test-udp-config.yml files to test both protocols against the same data stream.

Skipping tests

If sample logs are not yet available, a test config can be marked as skipped:

service: <package>-<stream>-tcp
skip:
  reason: "No sample logs available"
  link: https://github.com/elastic/integrations/issues/<number>
service_notify_signal: SIGHUP
input: tcp
data_stream:
  vars:
    listen_address: 0.0.0.0
    listen_port: <port>

Reference integrations

• `vectra_detect` — TCP, UDP, and TLS test configs
• `watchguard_firebox` — UDP with assert.hit_count
• `zscaler_zia` — TCP across multiple data streams

system testing

Everything needed to set up, run, and debug elastic-package test system for full end-to-end ingest validation.

Purpose

System tests validate the complete ingest path: service deployment → Elastic Agent policy → ingest pipeline → indexed documents in Elasticsearch. They also generate sample_event.json.

Required layout

Service deployment config (package-level or stream-level):

• _dev/deploy/docker/ — Docker Compose service definition

System test configs (per data stream):

• data_stream/<stream>/_dev/test/system/test-<scenario>-config.yml

For multi-version or environment variants:

• _dev/deploy/variants.yml

System test config fields

Available placeholders in config files:

• {{Hostname}}
• {{Port}}
• {{Ports}} (or indexed forms like {{Ports.0}})
• {{SERVICE_LOGS_DIR}}

Core commands

# Start stack once
elastic-package stack up -d

# Run all system tests in current package
elastic-package test system

# Scope to specific streams
elastic-package test system --data-streams <stream1>[,<stream2>]

# Run one test config
elastic-package test system --data-streams <stream> --test-config test-default-config.yml

# Run with a specific deploy variant
elastic-package test system --variant <variant-name>

# Generate sample_event.json
elastic-package test system --generate

# Keep resources around for debugging
elastic-package test system --defer-cleanup 10m

Input-type-specific guidance

System test setup varies significantly by input type. Load the appropriate input-specific reference file alongside this generic reference:

When an integration supports multiple input types, load the generic reference plus each applicable input-type reference.

Teardown failures (Fleet / agent policy conflicts)

If system test teardown fails with an error that an agent policy (or similar Fleet resource) is still in use and cannot be removed, do not try to fix Fleet state by hand.

Reset the whole local stack:

elastic-package stack down
elastic-package stack up -d -v

Then rerun elastic-package test system (after elastic-package build if the package changed). This clears stale policies and agents tied to the previous run.

Verifying generated sample_event.json

--generate produces data_stream/<stream>/sample_event.json containing one representative event from the indexed results. This is a snapshot of current behavior, not a correctness guarantee.

After generation, verify: 1. Document shape matches expectations (correct ECS fields, correct nesting, expected values) 2. Geo fields appear under the correct parent entity (source.geo, destination.geo, etc.) — not at document root 3. Dotted field names from source data appear as properly nested objects 4. If the sample event contains unexpected fields or missing values, fix the pipeline and regenerate — do not edit the file manually

Debugging system test failures — general

Check the agent log for dropped events

build/container-logs/elastic-agent-<ID>.log

If the service container log shows successful responses but you still get 0 hits, check the agent log for:

"events were dropped! Look at the event log to view the event and cause"

This means events were produced by the input and sent to Elasticsearch via bulk API, but Elasticsearch rejected every document.

Check the elastic-agent event log inside the running container

When events are being dropped, the only way to see the actual rejection reason is the event log inside the running elastic-agent container:

/usr/share/elastic-agent/state/data/logs/events/elastic-agent-event-log-*.ndjson

To inspect it during a running system test:

1. Increase `wait_for_data_timeout` in the test config (e.g., to 5m) so the system test stays running long enough to inspect the container. 2. Start the system test and wait for the first "events were dropped" warnings. 3. Find the elastic-agent container:

   docker ps --format '{{.Names}}' | grep elastic-agent

4. Read the event log:

   docker exec <container-name> tail -20 /usr/share/elastic-agent/state/data/logs/events/elastic-agent-event-log-*.ndjson

5. Look for entries with "Cannot index event" — these contain the exact Elasticsearch rejection reason.

Common rejection reasons:

After diagnosing, reduce wait_for_data_timeout back to 1m for the final test run.

Elasticsearch debugging (last resort)

Only if the above steps reveal nothing: check whether documents arrived but were routed incorrectly or failed in the pipeline. This is rarely needed — the container log and event log almost always reveal the root cause.

Common failure patterns

• "field X is undefined" for ECS fields (e.g. field "destination.ip" is undefined):
• Missing or outdated _dev/build/build.yml
• Fix: add dependencies.ecs.reference: "git@v9.3.0"
• Service not reachable: wrong ports or placeholders in test config
• No events ingested: wrong paths/hosts/input selection
• Mapping conflicts: field type mismatches between pipeline output and field definitions
• Teardown fails: agent policy still referenced — run elastic-package stack down then elastic-package stack up -d -v and rerun (see Teardown failures above)

Data anonymization

All test data must be fully anonymized before committing. No real production data, customer data, or identifiable information may appear in system test sample logs (_dev/deploy/docker/sample_logs/), mock API response configs (_dev/deploy/docker/files/), test config files, or generated sample_event.json.

Replace every identifying value with a synthetic example of the same format — IP addresses, hostnames, email addresses, usernames, organization names, account IDs, tokens, and any other value traceable to a real entity. Use RFC 5737 documentation IP ranges, example.com domains, and realistic placeholder names. Refer to the anonymize-logs skill for the full anonymization policy and placeholder conventions.