Package Spec

package-spec

Skill authority

The rules and patterns defined in this skill and its reference files are the authoritative source of truth. When examining existing integrations in the elastic/integrations repository for reference, you may encounter patterns that conflict with what is specified here -- many integrations contain legacy patterns that predate current standards. Always follow this skill over patterns observed in other integrations. If a reference integration uses a deprecated or prohibited pattern, do not copy it.

When to use

Use this skill when tasks include:

• building or reviewing manifest.yml at root or data stream level
• adding or validating changelog.yml entries
• selecting the correct change type and semantic version bump
• configuring policy templates, inputs, and variable declarations
• debugging elastic-package lint or elastic-package check errors on manifests or changelogs
• reviewing variable scoping across package, policy template, input, and data stream levels
• validating Handlebars template variables against manifest declarations
• configuring routing rules and their required manifest flags
• determining which format_version is needed for a package's features

When NOT to use

• Package scaffolding and directory layout (create-integration)
• Ingest pipeline design (ingest-pipelines)
• Field mapping and ECS compliance (ecs-field-mappings)
• CEL programs (cel-programs)
• Transform configuration (see review-integration skill's references/transform-guide.md)

Handoff

For package directory layout and required files, see create-integration -> references/package-layout.md. For elastic-package CLI commands and troubleshooting, see elastic-package-cli.

---

format_version

The format_version field in manifest.yml declares which elastic/package-spec version the package conforms to. The current standard for new packages is "3.4.2".

Use the minimum version that supports the features the package actually uses, not the latest available spec version. Bumping without needing new features:

• forces users to run a newer Kibana than necessary
• breaks backward compatibility for no reason
• makes it harder to determine which features the package depends on

Only bump when the package uses a feature introduced in a newer spec version:

See references/format-version-features.md for the full feature-to-version table including recent spec additions (3.6.0+), and references/manifest-rules.md for the review procedure.

---

conditions.kibana.version

The current standard constraint is "^8.19.0 || ^9.1.0". This is set in the root manifest.yml only -- data stream manifests must NOT set their own conditions.

When an integration uses features that require a newer agent (e.g., CEL functions introduced in v9.3.0), the constraint must be adjusted accordingly. For systematic version verification of CEL features, see the review-integration skill's version check references.

---

Variable scoping

Fleet variables exist at four levels:

1. Package level -- manifest.yml top-level vars: 2. Policy template level -- manifest.yml under policy_templates[].vars: 3. Input level -- manifest.yml under policy_templates[].inputs[].vars: 4. Data stream level -- data_stream/*/manifest.yml under streams[].vars:

A variable declared in an inner scope must not reuse the name of a variable in an outer scope. This is variable shadowing and is rejected by elastic-package validation.

See references/manifest-rules.md -> Variable shadowing for full rules, examples, and common patterns.

---

Manifest rules (brief)

• Every Handlebars `{{var}}` must be declared in a manifest -- undeclared variables silently resolve to empty strings. Handlebars helpers ({{#if}}, {{#each}}, {{#unless}}, {{#contains}}) and built-in variables ({{data_stream.type}}, {{data_stream.dataset}}, {{data_stream.namespace}}, {{output}}) are exempt.

• Routing rules require dynamic flags -- when a data stream uses routing_rules.yml, the data stream manifest must declare elasticsearch.dynamic_dataset: true and elasticsearch.dynamic_namespace: true.

• Use proper YAML nesting, not dotted keys -- elasticsearch.dynamic_dataset as a literal key name creates a single flat key, not a nested object. Use nested elasticsearch: -> dynamic_dataset: structure.

See references/manifest-rules.md for complete rules, correct/incorrect examples, and the review checklist.

---

Changelog schema

changelog.yml is a version-grouped array; newer versions go on top:

- version: "1.2.0"
  changes:
    - description: Added example parsing for edge-case payloads.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/12345

Each entry requires description, type, and link. Valid types: enhancement, bugfix, breaking-change.

Version bump rules

• patch (x.y.Z): bug fixes and low-risk fixes
• minor (x.Y.z): new content -- new data streams, new fields, new features
• major (X.y.z): breaking changes -- field type changes or removals on existing integrations, ECS mapping conflicts, required config/auth changes that break existing policies, data stream restructuring, default behavior changes that alter collected or normalized data

Adding changelog entries

Edit changelog.yml directly, or use elastic-package changelog add (see elastic-package-cli skill for command flags and --next patch|minor|major usage).

Common changelog pitfalls

• Adding the entry under the wrong version or not at the top
• Missing link field -- elastic-package lint validates that the PR/issue number is a positive integer and rejects pull/0; use a real PR number or pull/99999 as a development placeholder and replace before merge
• Bumping manifest/package version inconsistently with changelog intent

See references/changelog-patterns.md for detailed patterns, breaking-change checklist, and CI examples.

---

Upstream: elastic/package-spec

The elastic/package-spec repository is the upstream authority for package structure, manifest schema, and validation rules. The spec/changelog.yml in that repo documents which features were added in each spec version.

Key points from the package-spec versioning model:

• Packages must specify format_version in root manifest.yml
• A package at format_version: x.y.z must be valid against specs in the range [x.y.z, X.0.0) where X = x + 1
• Patch versions may add stricter validations (e.g., 3.6.0 added pipeline tag and on_failure validation)
• Minor versions add new feature support
• Major versions are reserved for significant format changes

See references/format-version-features.md for the curated feature-to-version table.

Reference files

changelog-patterns

This reference captures practical changelog.yml patterns used in this repository and the package spec constraints enforced by elastic-package lint.

Schema baseline

Source: docs/extend/changelog-spec.md (from the elastic/integrations upstream repo)

spec:
  type: array
  items:
    type: object
    additionalProperties: false
    properties:
      version:
        $ref: "./manifest.spec.yml#/definitions/version"
      changes:
        type: array
        items:
          type: object
          additionalProperties: false
          properties:
            description:
              type: string
            type:
              type: string
              enum:
                - "breaking-change"
                - "bugfix"
                - "enhancement"
            link:
              type: string
          required:
            - description
            - type
            - link
    required:
      - version
      - changes

Operational rules:

• changelog.yml is required in each package
• top-level is a list of version sections
• each version section contains one or more changes
• only enhancement, bugfix, and breaking-change are valid type values

Entry ordering and grouping

Repository convention:

# newer versions go on top

Patterns:

• one section per package version
• all changes for the same package version are grouped under that section
• prepend new versions at the top to preserve descending order

Representative entry patterns

Enhancement

From packages/nginx/changelog.yml (elastic/integrations):

- version: "2.3.0"
  changes:
    - description: Use links panel in Dashboards.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/14380

Bugfix

From packages/nginx/changelog.yml (elastic/integrations):

- version: "2.3.2"
  changes:
    - description: Remove unused agent files.
      type: bugfix
      link: https://github.com/elastic/integrations/pull/14995

Breaking change

From packages/apache/changelog.yml (elastic/integrations):

- version: "3.0.0"
  changes:
    - description: Remove third-party pipeline for previously removed 'third-party REST API' input.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/16133

Multi-change release section

Use this pattern when several changes ship in the same version:

- version: "1.18.0"
  changes:
    - description: Prepare package for serverless.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/9818
    - description: Remove duplicated and ambiguous field definitions.
      type: bugfix
      link: https://github.com/elastic/integrations/pull/9818

Multi-line description

From packages/wiz/changelog.yml (elastic/integrations):

- version: "4.0.0"
  changes:
    - description: |
        As `sourceRule` is deprecated by the Wiz Get Issue API, this version removes the deprecated `source_rule` field from the issue data stream.
        Previous versions added the new `source_rules` field to the issue data stream.
        Users should update their custom-user artifacts if they are using the deprecated `source_rule` field to use the new `source_rules` field.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/16892

Link conventions

Preferred links:

• PR URL: https://github.com/elastic/integrations/pull/<number>
• issue URL: https://github.com/elastic/integrations/issues/<number>

Guidelines:

• always provide a resolvable URL
• prefer the PR that introduces the change when available
• use issue links only when a PR link is not the right source of detail

Versioning guidance

Apply semver aligned with user impact:

• patch (x.y.Z): backward-compatible bug fixes
• minor (x.Y.z): backward-compatible enhancements
• major (X.y.z): breaking changes

When selecting breaking-change, check for:

• field type changes (mapping conflicts for existing data)
• field removals or renames
• ECS field mapping collisions or incompatible remapping
• required config/auth changes that break existing policies
• data stream split/merge/restructure changes
• default behavior changes that alter collected/normalized output

CLI workflows

elastic-package changelog add can create entries in the expected format.

Add entry for next patch/minor/major:

# run inside packages/<package_name>/
elastic-package changelog add \
  --type bugfix \
  --description "Fix parser for empty status field" \
  --link "https://github.com/elastic/integrations/pull/12345" \
  --next patch

Add entry for explicit version:

elastic-package changelog add \
  --type enhancement \
  --description "Add support for additional API region" \
  --link "https://github.com/elastic/integrations/pull/12345" \
  --version 1.14.0

Validate after changelog updates:

elastic-package lint
# or full sequence:
elastic-package check

CI and automation pattern

Automation in .github/workflows/docs-edit-automation.yml (elastic/integrations) uses:

elastic-package changelog add \
  --type "$CHANGE_TYPE" \
  --description "$CHANGELOG_DESC" \
  --link "https://github.com/${GITHUB_REPOSITORY}/pull/$PR_NUMBER" \
  --next "$VERSION_BUMP"

This pattern is useful for scripted, repeatable changelog updates where the change type and version bump can be inferred from workflow context.

Package spec format_version features

Curated table of features by format_version, sourced from the elastic/package-spec repository's spec/changelog.yml. Use this to determine the minimum format_version a package needs based on the features it uses.

Feature-to-version table

Core features (pre-3.4.2)

Current standard: 3.4.2

This is the standard format_version for new integrations. All features up to 3.4.2 are available.

Recent additions (3.5.0+)

These features require bumping beyond the current standard. Only use them if the package specifically needs the feature.

Breaking changes at 3.6.0

Spec version 3.6.0 introduced stricter validations that are breaking changes for packages that were previously valid:

1. Pipeline tag validations: every processor in an ingest pipeline must have a tag field. Packages at 3.6.0+ that lack tags will fail elastic-package check. 2. Pipeline on_failure validations: the pipeline-level on_failure block must follow the expected structure. Missing or malformed on_failure blocks fail validation.

Packages at the current standard 3.4.2 are NOT subject to these stricter validations. Bumping to 3.6.0+ adds these requirements. Only bump if the package needs a 3.6.0+ feature AND the pipeline already complies with tag and on_failure rules (which they should, per the ingest-pipelines skill).

Upstream source

The authoritative source is spec/changelog.yml in the elastic/package-spec repository. When new spec versions are released, update this table by reviewing that changelog for features relevant to integration developers.

The spec versioning model:

• A package at format_version: x.y.z must be valid against specs in [x.y.z, X.0.0) where X = x + 1
• Patch versions (x.y.Z) may add stricter validations
• Minor versions (x.Y.z) add new feature support
• Major versions (X.y.z) are reserved for significant format changes

Manifest Rules Reference

Comprehensive rules for manifest.yml validation in Elastic integration packages. Covers format_version selection, variable scoping, Handlebars template declarations, routing rules, and YAML structure.

---

Variable shadowing

The rule

Fleet variables can be declared at four levels:

1. Package level -- manifest.yml top-level vars: 2. Policy template level -- manifest.yml under policy_templates[].vars: 3. Input level -- manifest.yml under policy_templates[].inputs[].vars: 4. Data stream level -- data_stream/*/manifest.yml under streams[].vars:

A variable declared in an inner scope MUST NOT reuse the name of a variable in an outer scope. This is variable shadowing and is rejected by elastic-package validation.

Why it is rejected

When a variable name appears at multiple levels, Fleet cannot determine which value the user intended. The inner declaration shadows the outer one, leading to:

• confusing UI where the same variable appears twice
• ambiguous Handlebars template resolution
• validation failures in elastic-package check

Correct vs incorrect

# WRONG -- "api_key" declared at package level AND input level
# manifest.yml
vars:
  - name: api_key          # Package-level
    type: password
    title: API Key
    required: true

policy_templates:
  - name: events
    inputs:
      - type: cel
        vars:
          - name: api_key  # Shadows package-level "api_key"
            type: password
            title: API Key

# CORRECT -- unique names at each scope
# manifest.yml
vars:
  - name: api_key
    type: password
    title: API Key
    required: true

policy_templates:
  - name: events
    inputs:
      - type: cel
        vars:
          - name: initial_interval  # Different name, no shadow
            type: text
            title: Initial Interval

Common shadowing patterns

How to detect and fix

1. List all variable names at each scope level across the root manifest.yml and all data_stream/*/manifest.yml files. 2. Check that no name appears in both an outer and inner scope. 3. If shadowing is found, rename the inner variable to be more specific (e.g., events_api_key instead of api_key).

---

Handlebars variable declarations

The rule

Every Handlebars variable referenced in agent input templates (*.yml.hbs files) MUST be declared as a variable in the corresponding manifest. Undeclared variables silently resolve to empty strings, causing broken configs at runtime.

Variables can be declared at any manifest scope:

• Package-level: manifest.yml -> vars:
• Policy template: manifest.yml -> policy_templates[].vars:
• Input-level: manifest.yml -> policy_templates[].inputs[].vars:
• Data stream: data_stream/*/manifest.yml -> streams[].vars:

How to check

1. Scan the .yml.hbs file for all {{variable_name}} references (excluding Handlebars helpers and built-in variables). 2. For each variable, verify it exists in one of the manifest scopes above. 3. Flag any variable that is not declared anywhere.

Correct vs incorrect

# CORRECT -- all template variables are declared
# data_stream/events/agent/stream/cel.yml.hbs
interval: {{interval}}
resource.url: {{url}}/api/v1/events
{{#if proxy_url}}
resource.proxy_url: {{proxy_url}}
{{/if}}
{{#if ssl}}
resource.ssl: {{ssl}}
{{/if}}

# data_stream/events/manifest.yml -- variables declared
streams:
  - input: cel
    vars:
      - name: interval
        type: text
        title: Polling Interval
        default: 5m
      - name: url
        type: text
        title: API URL
        required: true
      - name: proxy_url
        type: text
        title: Proxy URL
      - name: ssl
        type: yaml
        title: SSL Configuration

# WRONG -- "api_version" used in template but not declared
# data_stream/events/agent/stream/cel.yml.hbs
resource.url: {{url}}/api/{{api_version}}/events
#                         ^^^^^^^^^^^^^^^^ NOT in any manifest!

Built-in variables

Some variables are provided by Fleet automatically and do NOT need declaration in manifests:

• {{data_stream.type}} -- logs, metrics, etc.
• {{data_stream.dataset}} -- the data stream dataset name
• {{data_stream.namespace}} -- the namespace
• {{output}} -- output configuration

Handlebars helpers (not variables)

These are control-flow helpers, not variable references. Do not flag them:

• {{#if var}}...{{/if}}
• {{#unless var}}...{{/unless}}
• {{#each items as |item|}}...{{/each}}
• {{#contains "value" array}}...{{/contains}}

Common mistakes

1. Copy-paste from another integration -- template references a variable from the source integration that was never added to this package's manifest. 2. Renamed variable -- manifest variable was renamed but the template still uses the old name. 3. Wrong scope -- variable declared at package level but template expects it at data stream level. This works at runtime, but may confuse users when reviewing the code.

---

Routing rules configuration

The rule

When a data stream uses routing_rules.yml to route documents to different backing indices based on field values, the data stream manifest must declare both:

elasticsearch:
  dynamic_dataset: true
  dynamic_namespace: true

Without these flags, Elastic Agent does not have write permissions to the dynamically-named data streams that routing rules create. Documents are rejected with permission errors at index time.

Why both flags are required

• dynamic_dataset: true -- allows the data stream name's dataset component to vary based on routing rules (e.g., logs-mypackage.alerts vs logs-mypackage.events).
• dynamic_namespace: true -- allows the namespace component to vary (e.g., routing to a different namespace per tenant).

Even if only one dimension varies, both flags are typically needed because Fleet's permission model grants write access based on the full {type}-{dataset}-{namespace} triple.

Correct vs incorrect

# WRONG -- routing_rules.yml exists but manifest lacks dynamic flags
# data_stream/events/manifest.yml
title: Events
type: logs
streams:
  - input: cel
    title: Events

# data_stream/events/routing_rules.yml exists

# CORRECT -- dynamic flags declared alongside routing_rules
# data_stream/events/manifest.yml
title: Events
type: logs
elasticsearch:
  dynamic_dataset: true
  dynamic_namespace: true
streams:
  - input: cel
    title: Events

# data_stream/events/routing_rules.yml
- source_dataset: events
  rules:
    - target_dataset: alerts
      if: ctx.severity == "critical"
  default_dataset: events

Routing rules file structure

# data_stream/*/routing_rules.yml
- source_dataset: <original_dataset>
  rules:
    - target_dataset: <new_dataset>
      if: <painless condition>
      namespace: <optional namespace override>
  default_dataset: <fallback_dataset>

Checklist

1. Does the data stream have a routing_rules.yml file? 2. If yes, does the data stream manifest declare dynamic_dataset: true? 3. Does it also declare dynamic_namespace: true? 4. Are both flags under a properly nested elasticsearch: key (not dot-notation)? 5. Are the routing rule conditions valid Painless expressions? 6. Is there a default_dataset fallback for unmatched documents?

---

YAML structure

The rule

YAML keys must use proper dictionary nesting, not dot-separated flat keys. While some YAML parsers treat elasticsearch.dynamic_dataset as a literal key name, Fleet and elastic-package expect nested structure.

Dot-notation keys create a single key whose name contains a literal dot, rather than a nested object. This can cause validation failures or silent misinterpretation.

Correct vs incorrect

# WRONG -- dot in key name, creates a single key "elasticsearch.dynamic_dataset"
elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true

# CORRECT -- proper YAML nesting
elasticsearch:
  dynamic_dataset: true
  dynamic_namespace: true

# WRONG
elasticsearch.source_mode: synthetic
elasticsearch.index_mode: time_series

# CORRECT
elasticsearch:
  source_mode: synthetic
  index_mode: time_series

Where this applies

This rule applies to all manifest files in the package:

• manifest.yml (top-level package manifest)
• data_stream/*/manifest.yml (data stream manifests)
• any YAML file validated by elastic-package

The most common occurrence is the elasticsearch section in data stream manifests, where dynamic_dataset, dynamic_namespace, source_mode, and index_mode must be nested under an elasticsearch: parent key.

---

format_version selection

The rule

The format_version field in manifest.yml declares which package spec version the package conforms to. Always use the minimum version that supports the features the package actually uses, not the latest available spec version.

The current standard is "3.4.2".

Why minimum matters

Using the latest spec version without needing its features:

• forces users to run a newer Kibana than necessary
• breaks backward compatibility for no reason
• makes it harder to determine which features the package actually depends on

Feature-to-version mapping

Only bump when the package uses a feature introduced in a newer spec version:

When bumping is justified

A format_version bump is justified only when the PR also introduces a feature that requires the higher version. Review questions:

1. What format_version is declared? 2. Does the package use any feature that requires this version? 3. Could a lower version work? 4. If the PR bumps format_version, does it also introduce a feature that requires the bump?

---

Review checklist

• [ ] format_version matches package needs -- HIGH if wrong
• [ ] No variable shadowing across scopes -- HIGH
• [ ] Every {{var}} in templates declared in manifest -- HIGH
• [ ] Routing rules have dynamic_dataset + dynamic_namespace -- HIGH if routing_rules.yml present but flags missing
• [ ] YAML uses nested structure, not dotted keys -- MEDIUM
• [ ] All variables have title, description, type, required -- MEDIUM
• [ ] Defaults present for optional variables -- LOW