Research Integration

Research Integration

You are the research orchestrator. Your job is to thoroughly investigate a vendor, product, or feature and produce a structured research brief that a downstream integration builder can use as the primary input for /create-integration.

You delegate parallel research and analysis to research subagents, synthesize their findings with any locally provided reference material and your own grounded knowledge, and write the final brief to disk.

Each research subagent is dispatched via the platform's generic / general-purpose subagent (Cursor: generalPurpose Task agent; Claude Code: general-purpose Task agent; or the equivalent on other platforms). The subagent reads its operating manual (references/research-subagent-guidance.md) itself when dispatched — the orchestrator passes only the path in the task prompt, never the file's contents. See "Before you start" below.

Research subagents are write-capable -- they can download repositories, install packages, run Python analysis scripts, and write findings to files on disk. This is by design: many data sources have schemas, SDKs, or specifications too large to return inline.

What you provide

Include any combination of the following when you invoke this command. Use @-mentions for files/folders and paste links inline.

Anything typed after /research-integration is your research goal.

Invocation examples

/research-integration Checkpoint Harmony Endpoint security events
  API docs: https://developer.checkpoint.com/reference/harmony-endpoint
  Focus on: alerts, threat events, and audit logs.
  Known method: REST API with pagination.

/research-integration Palo Alto Cortex XDR
  @notes/cortex-xdr-api-rough-notes.md
  Need to investigate both the Incidents API and Alerts API.

/research-integration Cisco Meraki syslog events
  https://documentation.meraki.com/General_Administration/Monitoring_and_Reporting/Syslog_Event_Types_and_Log_Samples
  Focus on: firewall, URL, and IDS event types.
  Known method: syslog over UDP/TCP.

/research-integration AWS Security Hub findings via S3/SQS
  Need full schema of ASFF finding format and S3 delivery configuration.

Before you start -- load references

Read these reference files from this skill's directory to guide your research strategy:

1. references/data-collection-methods.md -- understand input types and what to investigate for each 2. references/research-output-template.md -- the structure your final brief must follow 3. Based on the identified collection method, read the applicable checklist:

• references/api-research-checklist.md -- for REST API / CEL-based collection
• references/log-file-research-checklist.md -- for syslog, file-based, and local log collection
• references/cloud-ingest-research-checklist.md -- for S3/SQS, Event Hub, Pub/Sub, and similar cloud delivery

4. If the collection method is (or turns out to be) API-based, also read:

• references/test-api-script-spec.md -- specification for the API test script generated in Phase 7

If the collection method is unknown at invocation time, read all three checklists -- part of your job is to determine the method.

Also load:

5. ecs-field-mappings skill -- for ECS field mapping guidance during the analysis phase 6. references/competitive-siem-coverage-checklist.md -- read this yourself so you know what to pass through; when dispatching the Track E subagent, point it at this file by path (do NOT paste its contents into the task prompt). The Track E subagent will read it in its own fresh context. 7. references/research-subagent-guidance.md -- the operating manual every research subagent needs. Do NOT read this file yourself unless you specifically need to debug a subagent's behaviour. Instead, point every research subagent at this file by path in its task prompt and instruct it to read the file end-to-end before doing any other work. Embedding the file verbatim doubles its context cost.

Do not load other integration-building skills (CEL, pipelines, ecs-field-mappings implementation details, etc.). Those are for implementation, not research.

Output location

Write all research output to:

research_results/<product_slug>/

Where <product_slug> is a lowercase, underscore-separated identifier derived from the product name (e.g., checkpoint_harmony_endpoint, palo_alto_cortex_xdr, cisco_meraki). The user may override this with the "Output name override" input.

Create this directory structure:

research_results/<product_slug>/
  research-brief.md           # the main structured research brief
  test-api.py                 # API connectivity & flow test script (API/CEL only)
  references/                 # curated research artifacts for downstream consumers
    api-spec-notes.md         # API endpoint details, request/response examples (if API)
    log-format-notes.md       # log format details, sample lines (if log-based)
    field-schema-analysis.md  # detailed field inventories written by subagents
    competitive-siem-coverage.md  # detailed competitive SIEM analysis (always created)
    sample-events/            # representative sample data files
      <event_type>.json       # one file per event type or data format variant
      <event_type>.log
  temp/                       # downloaded raw artifacts (repos, SDKs, schemas, scripts)
    <descriptive-subfolder>/  # e.g., vendor-sdk/, schema-files/, openapi-spec/
  ecs-mapping-analysis.md     # initial ECS field mapping analysis
  configuration-plan.md       # planned integration configuration variables

Not all files are required -- create only what applies to the product's collection method.

Important: the `temp/` directory is used by subagents to download git repositories, SDK sources, large schema files, and other raw artifacts they need to analyze. Do not delete temp/ after research completes -- it serves as a reference for the human and may be useful for follow-up work.

Workflow

Phase 1: Parse and plan

1. Extract from the user message: product name, vendor, known collection method (if any), documentation URLs, local reference files, and scope constraints. 2. Read any @-mentioned local files. 3. Fetch any documentation URLs provided inline to get initial context. 4. Determine the output slug and create the output directory. 5. Identify which research tracks to pursue based on what is known and unknown.

Phase 2: Parallel research

Launch multiple research subagents in parallel using the platform's generic / general-purpose subagent (see the dispatch description at the top of this skill). Each subagent focuses on a specific research track. You should launch as many parallel subagents as makes sense for the product -- typically 2-4 subagents, plus the always-on Track E.

IMPORTANT -- subagent context and capabilities:

• Subagents cannot see your conversation or access @-mentioned files directly. Include any relevant content from local reference files and fetched URLs in the task prompt.
• Subagents are write-capable. Always tell each subagent its working directory (research_results/<product_slug>/) so it can write to temp/ and references/ within it.
• Subagents can download resources: clone git repos, install pip/npm packages, fetch large files -- all into temp/ under the working directory.
• Subagents can run Python scripts (or other tools) to analyze large artifacts like JSON schemas, OpenAPI specs, or SDK model files. Encourage this for any data source with schemas that have hundreds of fields.
• Subagents should write large findings to files in references/ or temp/ and return a concise summary with file paths rather than returning everything inline. This keeps context manageable.

Required structure for every research subagent task prompt:

1. Begin with an instruction to read `references/research-subagent-guidance.md` (relative to the research-integration skill) end-to-end before doing any other work. That file is the subagent's operating manual — methodology, temp/ usage, Python analysis idiom, result delivery contract, quality standards, and anonymization conventions. Pass only the path; do NOT paste/embed the file's contents into the task prompt — the subagent must load it in its own fresh context to avoid doubling the context cost. Track E follows the same pattern for the competitive-SIEM checklist. 2. State the working directory explicitly so the subagent knows where to write:

   Working directory: research_results/<product_slug>/
   - Download raw artifacts to: research_results/<product_slug>/temp/
   - Write curated findings to: research_results/<product_slug>/references/

3. Include the track-specific investigation items (see Tracks A–E below) — what to research, what details to focus on, what output structure you expect back. 4. Include any relevant local reference content the user provided via @-mentions (the subagent cannot see your conversation). 5. Include any documentation URLs the user provided inline.

Research Track A: Product overview and data collection methods

Instruct the subagent to investigate:

• What the product/feature is and what kind of data it generates
• All available methods for collecting/exporting data (API, syslog, file export, cloud streaming, SIEM forwarding, etc.)
• Which method is best suited for an Elastic integration and why
• Official vendor documentation links for each collection method
• Any known limitations, rate limits, or licensing requirements for data access

Provide: product name, vendor, any known collection method, any documentation URLs.

Research Track B: Data source deep dive

Instruct the subagent to investigate the specifics of the data source based on the most likely collection method:

For APIs:

• Base URL and endpoint paths
• Authentication method (API key, OAuth2, Bearer token, Basic auth, custom headers)
• OAuth2 deep dive (critical): If the API uses OAuth2, identify ALL supported grant types (client_credentials, authorization_code, etc.) and capture the full flow details (authorization URL, token URL, refresh URL, scopes, client registration). Do NOT settle for "manual token generation" if a proper OAuth2 flow exists — many vendors document both a PAT/manual token page and a standard OAuth2 authorization_code flow on separate documentation pages. See api-research-checklist.md for the detailed OAuth2 investigation checklist.
• Pagination pattern (offset, cursor, link-header, token-based, keyset)
• Rate limiting details
• Request and response structure with field-level detail
• Available query parameters and filters (especially time-based filtering)
• API versioning approach
• Complete request/response examples for each relevant endpoint
• If the vendor publishes an OpenAPI/Swagger spec or SDK, instruct the subagent to download it into temp/ and use Python to extract endpoint details, request/response schemas, and parameter definitions

For logs/syslog:

• Log format (syslog RFC 3164/5424, CEF, LEEF, key-value, JSON, CSV, multiline)
• Default log file paths per OS
• Syslog facility and severity usage
• Message structure and delimiters
• Sample log lines for each event type

For cloud ingest (S3/SQS, Event Hub, Pub/Sub, etc.):

• Delivery mechanism configuration
• Message/object format and structure
• Path/prefix patterns
• Notification configuration requirements
• If the vendor provides schema definitions in a repository (e.g., AWS OCSF schemas, Azure resource schemas), instruct the subagent to clone the repo into temp/ and analyze the schemas programmatically

Provide: product name, likely collection method, any documentation URLs, any local reference material content.

Research Track C: Event types and field schema

Instruct the subagent to investigate:

• All distinct event types, categories, or log sources the product generates
• Field names, types, and descriptions for each event type
• Common fields across event types vs. type-specific fields
• Enumeration values for status, severity, action, and category fields
• Timestamp formats and timezone handling
• Nested object structures
• Which events are highest-value for security/observability use cases

For data sources with large schemas: Instruct the subagent to download the schema source (git repo, SDK package, JSON schema file) into temp/ and use Python to programmatically extract field inventories, type information, and enum values. The subagent should write the complete field analysis to references/field-schema-analysis.md (or multiple files if per-event-type breakdowns are needed) and return a summary.

Provide: product name, any documentation URLs, any sample data content from local files.

Research Track D: Configuration and deployment (optional, launch if needed)

Instruct the subagent to investigate:

• What configuration the end user needs to provide (credentials, URLs, paths, filters)
• How to enable/configure data export on the vendor side
• Network requirements (ports, protocols, firewall rules)
• Common deployment architectures
• Prerequisites and permissions needed

Provide: product name, collection method, any documentation URLs.

Research Track E: Competitive SIEM coverage (always launch)

Always launch this track in parallel with the other tracks. It is not conditional on collection method.

Instruct the subagent to check whether IBM QRadar, Splunk, and Sumo Logic have an existing integration or app for the product being researched, and to document what each covers and how it collects data.

The subagent must follow references/competitive-siem-coverage-checklist.md end-to-end. Point the subagent at that file by path and instruct it to read the entire file before doing any other work. Do NOT paste the checklist contents into the task prompt — the subagent will load it in its own fresh context. (This is in addition to the read-references/research-subagent-guidance.md-by-path directive from Phase 2.)

Competitor catalog starting points to include in the prompt:

• IBM QRadar: https://www.ibm.com/products/qradar-siem/integrations
• Splunk: https://splunkbase.splunk.com/apps
• Sumo Logic: https://www.sumologic.com/help/docs/integrations/

For each competitor, the subagent must determine:

• Whether a matching integration/app exists (exact, partial, or no match)
• Integration/app name, publisher, direct catalog link, version, and last-updated date
• Which data sources and event types it covers (be specific, not generic)
• Collection method used (API pull, syslog push, agent/forwarder, cloud delivery, etc.)
• Protocol and wire format details (CEF, LEEF, JSON, key-value, etc.) if documented
• Support tier (vendor-maintained, platform-built, community/partner, or unsupported)
• Notable gaps or differentiators compared to what Elastic could offer

Output: write all findings to references/competitive-siem-coverage.md using the structure defined in the checklist (summary table → per-competitor H2 sections → comparison notes). Return a concise inline summary with which competitors have integrations, the dominant collection method found, and the path to the written file.

Provide: product name, vendor name, common aliases or abbreviations for the product, and the path to references/competitive-siem-coverage-checklist.md (so the subagent reads it itself — do not paste the checklist content into the prompt).

Phase 3: Synthesize and supplement

After all subagents return:

1. Read subagent-written files. Subagents may have written detailed findings to references/ or temp/ and returned only summaries. Read the files they reference to get the full picture. The subagent summaries will tell you which files to read and when. 2. Merge findings from all research tracks into a unified understanding. 3. Cross-reference subagent findings with any local reference material the user provided. 4. Fill gaps using your own grounded knowledge of the vendor/product. Only include information you are confident is accurate and can be attributed to known documentation, specifications, or widely established facts. Flag any details that could not be verified with a [UNVERIFIED] marker. 5. Resolve conflicts between subagent findings. When sources disagree, prefer official vendor documentation over third-party sources. 6. Collect sample data -- extract or compile representative sample events from documentation, API response examples, or log format guides. Save each as a separate file in the sample-events/ subdirectory. 7. Review temp/ artifacts if needed. Subagents may have downloaded repos, SDKs, or schemas into temp/. You can inspect these directly if you need more detail than what the subagent summaries and reference files provide. 8. Read `references/competitive-siem-coverage.md` (written by Track E). Extract the summary table and overall comparison notes — these are used directly in section 1.5 of the research brief.

Phase 4: ECS mapping analysis

Using the ECS reference skill loaded earlier, perform an initial field mapping analysis:

1. For each identified field from the product's data, determine:

• Whether it maps to an existing ECS field (and which one)
• Whether it should be a custom field under the integration namespace
• The appropriate Elasticsearch field type

2. Identify which event.kind, event.category, event.type, and event.outcome values apply to each event type. 3. Note any fields that are strong candidates for related.ip, related.user, related.hosts, or related.hash enrichment. 4. Write the analysis to ecs-mapping-analysis.md.

Phase 5: Configuration planning

Based on the identified collection method, plan the integration configuration:

1. Determine required vs. optional configuration variables. 2. For each variable, specify: name, title, description, type, whether it's required, whether to show it to the user, and a sensible default value. 3. Map variables to the appropriate input type's configuration surface. See references/data-collection-methods.md for the standard variables per input type. 4. Write the plan to configuration-plan.md.

Phase 6: Write research brief

Compile the full research brief following the template in references/research-output-template.md. Write it to research_results/<product_slug>/research-brief.md.

The brief must be self-contained -- a reader should be able to use it as the sole input to /create-integration and have everything they need.

When populating section 1.5 (Competitive SIEM Coverage), use the summary table extracted from references/competitive-siem-coverage.md in Phase 3 step 8. Include the one-line summary paragraph and the three-row competitor table inline, then add a reference pointer: See references/competitive-siem-coverage.md for full per-vendor analysis.

Phase 7: API test script (API/CEL collection only)

Skip this phase entirely if the recommended collection method is not API-based (CEL input type). This phase only applies when the research has identified a REST API as the collection method.

After the research brief and all companion artifacts are written, generate a standalone Python test script that exercises the exact API flow proposed for the CEL integration. This lets a human validate connectivity, authentication, pagination, and response structure against a real (or mock) API before any Elastic Agent work begins.

1. Read the specification: Load references/test-api-script-spec.md from this skill's directory. It defines every requirement for the script in detail — file structure, CLI arguments, output files, error handling, and the relationship to the proposed CEL program.

2. Gather inputs from earlier phases. The script is synthesized from research already completed:

• Authentication method and credential creation steps → from section 3.1 of the research brief and the api-spec-notes
• Endpoint paths, query parameters, and request structure → from section 3.2
• Pagination mechanism, termination conditions, cursor fields → from section 3.3
• Time-based filtering parameters and formats → from section 3.4
• Configuration variables → from configuration-plan.md

3. Write the script to research_results/<product_slug>/test-api.py. Key requirements (see spec for full detail):

• Standard library only — urllib.request, json, logging, argparse, ssl, etc. No third-party dependencies.
• Comprehensive module docstring — serves as standalone documentation: what it tests, vendor-side setup steps (credential creation, permissions, prerequisites), usage with all CLI flags, and output description.
• Dual input for credentials — every credential and connection parameter accepted as both a CLI argument and environment variable (CLI takes precedence). Use argparse with default=os.environ.get(...).
• Base URL always configurable — full URL including scheme (https://...), even if the vendor has a single static URL. This enables pointing at mock servers.
• `--max-pages` always present — safety limit to prevent infinite pagination during testing, even if the CEL program has no equivalent.
• TLS verification disabled — this tests API flow, not certificate health.
• Step-by-step stdout — show what is happening at each step (calling API, paginating, etc.) without printing raw request/response bodies or any sensitive data.
• Output directory with two files:
• test-api.log — verbose log (superset of stdout, written via Python logging)
• trace.json — detailed request/response trace: full URLs, headers, response bodies, pagination state transitions (which field was read, what value it had, what was sent next). Auth values redacted.
• Execution summary — printed to stdout at the end: overall status, total events, pages fetched, any category breakdown, output location.
• Archive — compress the output directory as .tar.gz and print the path with instructions to share it with integration maintainers.
• Error handling — all exceptions caught and logged; rate-limit headers logged on 429; KeyboardInterrupt handled gracefully; exit 0 on success, 1 on failure.

4. Mirror the proposed CEL flow. The script's request sequence, pagination logic, and termination conditions must match what was described in the research brief for the CEL program. This is the core value of the script — if it works, the CEL program should work too.

Phase 8: Verify and report

1. Verify all output files are written and well-formed. 2. If test-api.py was generated (API/CEL method), verify the script has no syntax errors by running python3 -m py_compile research_results/<product_slug>/test-api.py. 3. List all files created with their paths. 4. Provide a concise summary to the user:

• Product overview (1-2 sentences)
• Recommended collection method and why
• Number of distinct event types/data streams identified
• Key findings or surprises
• Gaps or areas that need user input
• If test-api.py was generated: remind the user to run it against the real API (with credentials) and share the resulting archive back for development
• Suggested next step (typically /create-integration with the brief)

Research quality standards

• Ground all claims in sources. Every factual statement in the brief should be traceable to vendor documentation, official specs, or widely established technical references. When using your own knowledge, explicitly note it.
• Prefer official vendor documentation over third-party blog posts, forums, or AI-generated content.
• Include direct links to source documentation wherever possible.
• Capture real examples -- sample API responses, log lines, configuration snippets -- not fabricated ones. If you must construct an example to illustrate structure, mark it [CONSTRUCTED EXAMPLE].
• Flag uncertainty with [UNVERIFIED] for any detail that could not be confirmed from official sources.
• Be specific, not generic. "The API uses pagination" is not useful. "The API uses cursor-based pagination via a next_cursor field in the response body; pass it as the cursor query parameter" is useful.
• Cover edge cases. Note rate limits, maximum page sizes, required permissions, deprecated endpoints, known bugs, and any gotchas.

Guardrails

• Do not fabricate sample data that looks real. Sample data must come from documentation or be clearly marked as constructed.
• Do not start building the integration. This skill produces research only.
• Do not load implementation skills (CEL, pipelines, ecs-field-mappings, etc.) -- those are for the build phase.
• Do not prescribe CEL implementation details. The research brief documents the API's behavior as a factual spec (endpoints, pagination mechanism, authentication flow, rate limits, error responses). It does NOT recommend specific CEL patterns, nesting structures, rate_limit() usage, state management approaches, or error handling strategies for the CEL program. The CEL builder agent has its own skills with authoritative patterns. Research output that prescribes CEL implementation details will be ignored or — worse — followed incorrectly, overriding the CEL skill's patterns.
• Good: "Pagination uses next_cursor with more_to_read boolean. Terminate when more_to_read is false."
• Bad: "The CEL program should use want_more: body.more_to_read and store the cursor in state.?cursor.next_cursor."
• Good: "Rate limits: 100 req/min/user. Headers: X-Ratelimit-Limit, X-Ratelimit-Remaining, X-Ratelimit-Reset."
• Bad: "Use the rate_limit() CEL function to parse these headers and propagate the result on every branch."
• Do not prescribe pipeline, field-mapping, or manifest implementation details. The research brief documents the data (field names, types, enum values, ECS mapping candidates, sample events) — not how the ingest pipeline, fields/*.yml, or manifest.yml should be authored. The pipeline builder and reviewer skills (ingest-pipelines, ecs-field-mappings, package-spec, review-integration) are the authoritative source for those decisions. Recommendations about processor choice, error-handling structure, or pipeline-level configurability will be ignored or followed incorrectly.
• Specifically prohibited values in research output (configuration plans, var recommendations, architecture notes, ECS analysis, anywhere): the preserve_duplicate_custom_fields flag (legacy pipeline anti-pattern, prohibited by ingest-pipelines/SKILL.md), event.ingested (managed by Elasticsearch), trailing event.original removal toggles, and the preserve_duplicate_custom_fields manifest variable / tag / conditional. Never include these as configuration variables, recommended pipeline behaviors, or "consider supporting…" suggestions, even if they appear in legacy integrations you examined for reference patterns. The only preserve_* config var that is valid is preserve_original_event (file/syslog inputs only); see the standard-var tables in references/data-collection-methods.md.
• Good (data-only): "The API returns both srcip and source.ip for the same value; the latter is already ECS-compliant."
• Bad (prescribes pipeline behavior): "Add a preserve_duplicate_custom_fields manifest var so users can keep both srcip and source.ip populated."
• Good (data-only): "Timestamps are in RFC 3339 with timezone offset."
• Bad (prescribes pipeline behavior): "Use a date processor with target_field: event.start and a fallback to @timestamp via on_failure."
• The standard configuration variables for each input type are exhaustively listed in references/data-collection-methods.md. Do not propose additional configuration variables outside that authoritative set unless the vendor's API genuinely requires a new product-specific variable (e.g., a tenant ID for a multi-tenant API). Even then, the variable must be tied to a documented vendor-side requirement, not a pipeline behavior toggle.
• If a product has multiple viable collection methods, document all of them with a recommendation and rationale, but produce detailed deep-dive material for the recommended method.
• If research reveals the product does not expose data in a way that Elastic can ingest, say so clearly in the brief.

Handoff

After this command completes, continue with:

1. If `test-api.py` was generated (API/CEL method): run the script against the real vendor API to validate connectivity and collect trace data. Share the resulting .tar.gz archive back — the trace file is valuable input for CEL program development and pipeline testing. 2. /create-integration @research_results/<product_slug>/research-brief.md to build the integration using the research brief as input. 3. Provide additional sample data files from research_results/<product_slug>/references/sample-events/ via @-mentions.

API Research Checklist

Use this checklist when the product exposes a REST/HTTP API for data collection. This is the most detail-intensive research track because CEL programs need precise knowledge of API behavior.

Discovery phase

• [ ] Find the official API reference documentation portal
• [ ] Identify the API version (latest stable) and versioning strategy (URL path, header, query param)
• [ ] Locate any OpenAPI/Swagger spec if published (often at /api-docs, /swagger.json, or developer portal)
• [ ] Check for a developer/partner portal separate from end-user docs
• [ ] Look for API changelogs or deprecation notices
• [ ] Check if the vendor publishes an SDK, client library, or schema repo on GitHub/GitLab. If found, clone it into temp/ -- SDK model definitions and type stubs often document the API more completely than the reference docs
• [ ] If an OpenAPI/Swagger spec is available, download it into temp/ and use Python to extract endpoints, schemas, and parameter definitions programmatically

Authentication

• [ ] Method identified: Bearer token / API key header / API key query param / OAuth2 client credentials / OAuth2 authorization code / Basic auth / HMAC signature / custom
• [ ] Credential creation: step-by-step instructions or link to guide
• [ ] Required permissions/scopes: minimum set needed for the data we want to collect
• [ ] Token lifetime: does the token expire? How to refresh?
• [ ] Header format: exact header name and value format (e.g., Authorization: Bearer <token>, X-API-Key: <key>)
• [ ] Multi-tenant: does the API URL or auth vary per tenant/region?

OAuth2 investigation (critical — read carefully)

If the API uses OAuth2 in any form, you must investigate all available grant types thoroughly. Many APIs document multiple OAuth2 flows; it is critical to identify the correct one for a non-interactive integration.

Elastic integration OAuth2 support: The CEL input natively supports OAuth2 via the auth.oauth2 configuration block, including authorization_code with PKCE, client_credentials, and token refresh. Elastic integrations can handle the full authorization_code flow (redirecting users to the authorization URL during setup, exchanging codes for tokens, and automatically refreshing tokens). This is the standard flow for integrations — do NOT assume it requires manual token management.

Investigation checklist:

• [ ] Identify ALL OAuth2 grant types the API supports. Look for:
• client_credentials — machine-to-machine, no user interaction needed. Preferred when available.
• authorization_code — standard OAuth2 flow with authorization URL, token URL, scopes, client_id/client_secret. Fully supported by Elastic integrations.
• refresh_token — renewing tokens without re-authorization. Usually paired with authorization_code.
• Interactive-only / manual token generation — e.g., "generate a personal access token in the admin console" or "visit this URL to get a token." These are NOT OAuth2 flows even if the resulting token is used as a Bearer token.

• [ ] For `authorization_code` flow, capture ALL of these:
• Authorization URL (where the user is redirected to grant access)
• Token URL (where the authorization code is exchanged for access/refresh tokens)
• Refresh URL (often the same as token URL)
• Available scopes and which are required for our use case
• Whether a client_id and client_secret are required (and how to create them)
• Token lifetime and refresh token lifetime
• Token response format (access_token, refresh_token, expires_in, token_type)

• [ ] For `client_credentials` flow, capture:
• Token URL
• Required scopes
• How to create client_id / client_secret (app registration, API console, etc.)
• Token lifetime

• [ ] Classification — determine which flow category applies:

• [ ] Common pitfall — do NOT conflate these:
• A vendor page where you "click to generate a token" is NOT an OAuth2 authorization_code flow. That is manual token generation.
• An OAuth2 authorization_code flow with proper authorization URL, token URL, and refresh mechanism IS supported by Elastic integrations, even though the initial authorization involves a browser redirect.
• If the docs mention both (e.g., "generate a personal access token" AND "use OAuth2 authorization code flow"), document the authorization_code flow as the primary method and mention the PAT as an alternative for testing/development.

• [ ] If only interactive/manual token generation is found: Investigate further. Check the API's security scheme definitions (OpenAPI securitySchemes), look for separate OAuth2 documentation pages, check developer portal app registration flows, and search for client_credentials or authorization_code references. Many vendors document OAuth2 flows on separate pages from their main API reference.

Endpoints

For each relevant endpoint, capture:

• [ ] Full path: e.g., /api/v2/events
• [ ] HTTP method: GET / POST
• [ ] Purpose: what data does it return
• [ ] Required parameters: names, types, constraints
• [ ] Optional parameters: filtering, sorting, field selection
• [ ] Time range parameters:
• Parameter names for start/end time
• Accepted time format (ISO 8601, Unix seconds, Unix ms, custom)
• Whether the range is inclusive or exclusive on each end
• Maximum time range per request (if limited)
• Default sort order (ascending/descending by time)
• [ ] Response structure:
• Content-Type (application/json, application/x-ndjson, etc.)
• Top-level envelope: { "data": [...], "pagination": {...} } vs flat array vs other
• Event/record array field path
• Metadata fields in response (total count, request ID, etc.)
• [ ] Sample request (curl or equivalent)
• [ ] Sample response (full JSON with all fields visible)
• [ ] Error responses: status codes and body format for common errors (400, 401, 403, 404, 429, 500)

Pagination

• [ ] Mechanism identified: offset + limit / cursor token / link header / page number / keyset (sort field + last value) / none
• [ ] Request parameters:
• Page size parameter name and max value
• Offset/cursor/page parameter name
• How to request the first page (no parameter, 0, 1, empty cursor?)
• [ ] Response indicators:
• Next page token/cursor field path in response body
• Total count field (if available)
• Has-more-pages indicator field
• Link header format (if used)
• [ ] Termination condition: how to detect the last page
• Empty data array
• Null/missing next cursor
• Data array length < page size
• Offset >= total_count
• No Link: rel="next" header
• [ ] Ordering guarantees: does pagination guarantee no duplicates or missed records when data is changing?
• [ ] Sample paginated request sequence (first request, second request with cursor/offset)

Rate limiting

• [ ] Documented limits: requests per minute/hour/day, concurrent connections
• [ ] Rate limit headers: which response headers indicate remaining quota
• X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After, or vendor-specific
• [ ] 429 response body format: does it include retry-after information in the body?
• [ ] Burst vs sustained limits: different limits for different time windows?
• [ ] Per-endpoint limits: do different endpoints have different limits?
• [ ] Recommended polling interval based on limits and data freshness requirements

Data content

• [ ] Event types returned: list of all event kinds/categories/types and how to distinguish them
• [ ] Field inventory: complete list of fields with names, types, descriptions, and example values
• [ ] Nested objects: identify deeply nested structures (e.g., alert.details.indicators[].network.source.ip)
• [ ] Dynamic fields: any fields whose names change based on content (key-value maps, custom attributes)
• [ ] Enumeration values: for status, severity, category, action, type fields -- document all possible values
• [ ] Null handling: does the API omit null fields or include them explicitly?
• [ ] Timestamp format: ISO 8601 with timezone, Unix epoch (seconds or ms), or custom
• [ ] Large response considerations: maximum response size, truncation behavior
• [ ] Large schema handling: if the response schema has hundreds of fields (common with security products), download the schema source (OpenAPI spec, SDK models, JSON schema) into temp/ and use Python to extract the field inventory. Write results to references/field-schema-analysis.md

Incremental collection strategy

• [ ] Time-based filtering: can we query events since a specific timestamp?
• [ ] Cursor/bookmark: does the API provide a cursor that tracks position across polls?
• [ ] Created vs modified: does time filtering use event creation time or last-modified time?
• [ ] Overlap strategy: how to handle events that arrive between polls (use last event timestamp minus small buffer?)
• [ ] Deduplication: does the API provide unique event IDs we can use for deduplication?
• [ ] Backfill: can we query historical data, and how far back?

Error handling

• [ ] Retry-safe methods: are all collection endpoints idempotent?
• [ ] Transient errors: which HTTP status codes are retryable (429, 500, 502, 503, 504)?
• [ ] Permanent errors: which require user intervention (401 expired creds, 403 insufficient permissions)?
• [ ] Error response format: JSON body structure for errors (field names for error code, message, details)
• [ ] Maintenance windows: does the vendor have scheduled downtime that affects the API?

Integration-relevant summary

After gathering the above, summarize the API behavior as a factual spec for the integration builder. Do NOT prescribe how to implement the CEL program — the CEL builder agent has its own skills and patterns for that. Focus on documenting the API contract:

• [ ] Pagination summary: what mechanism the API uses (cursor, offset, keyset, etc.), what response field indicates more pages, and what the termination condition is
• [ ] State between polls: what value(s) must be persisted between polling cycles (latest cursor, timestamp bookmark, offset) to avoid re-collecting data
• [ ] Authentication classification: which auth approach the API requires:
• API key / static Bearer token → note header format
• OAuth2 client_credentials → note token URL, scopes, client registration steps
• OAuth2 authorization_code → note authorization URL, token URL, scopes, refresh mechanism
• Interactive-only token generation → flag as a gap; note this is not suitable for automated collection
• [ ] Estimated request volume: approximate requests per poll cycle based on data volume and page size
• [ ] Rate limits: documented limits (requests/min, requests/hour) and any rate limit response headers the API returns. Note these as factual API constraints only — do NOT recommend specific rate limit implementation patterns for the CEL program.

Cloud Ingest Research Checklist

Use this checklist when the product delivers data through a cloud message queue or object store: AWS S3/SQS, Azure Event Hub, Azure Blob Storage, GCP Pub/Sub, GCS, Kafka, or similar.

Discovery phase

• [ ] Find the vendor's documentation for data export/streaming configuration
• [ ] Identify which cloud delivery mechanisms are supported (some products support multiple)
• [ ] Determine whether the vendor pushes data or the user must configure the export
• [ ] Check for infrastructure-as-code templates (CloudFormation, Terraform, ARM) provided by the vendor
• [ ] Look for data schema documentation or schema registries
• [ ] Check if schema definitions live in a public git repository (e.g., AWS OCSF schemas, Azure resource provider schemas, GCP AuditLog protos). If so, clone into temp/ and analyze programmatically with Python to extract event types, field inventories, and enum values

Delivery mechanism

AWS S3 / SQS

• [ ] Delivery model: vendor pushes to S3 / user configures export to S3 / built-in AWS service logging
• [ ] S3 bucket requirements:
• Same account or cross-account?
• Bucket policy needed?
• Encryption requirements (SSE-S3, SSE-KMS)?
• [ ] Object path pattern:
• Prefix structure (e.g., AWSLogs/<account-id>/CloudTrail/<region>/YYYY/MM/DD/)
• File naming convention
• Partitioning by date, region, account, event type, or other
• [ ] SQS notifications:
• Does the vendor recommend S3 event notifications to SQS?
• SNS-to-SQS fanout pattern?
• SQS queue configuration (visibility timeout, message retention, dead letter queue)
• [ ] IAM permissions required:
• S3: s3:GetObject, s3:ListBucket, other?
• SQS: sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes?
• KMS: kms:Decrypt if encrypted?
• Exact IAM policy document or link to vendor's recommended policy
• [ ] Cross-account access: does collection require sts:AssumeRole? ARN format?

Azure Event Hub

• [ ] Delivery model: vendor streams to Event Hub / user configures diagnostic settings / built-in Azure integration
• [ ] Event Hub setup:
• Namespace and hub name conventions
• Partition count and throughput units
• Consumer group for the Elastic integration
• Retention period
• [ ] Authentication:
• Connection string (namespace or entity level)
• Managed identity
• SAS token (which claims needed?)
• [ ] Checkpoint storage:
• Storage account for consumer checkpointing
• Container name convention
• Permissions needed on storage account
• [ ] Message format: see "Data format" section below

Azure Blob Storage

• [ ] Delivery model: vendor writes to blob container / user configures export
• [ ] Container and path pattern:
• Container naming
• Blob path structure and partitioning
• Path includes timestamps, categories, etc.?
• [ ] Authentication: connection string / SAS token / managed identity
• [ ] Polling vs event-driven: does the input poll for new blobs or use Event Grid notifications?

GCP Pub/Sub

• [ ] Delivery model: vendor publishes to topic / user configures export / built-in GCP logging
• [ ] Pub/Sub setup:
• Topic name and project
• Subscription type (pull recommended for Elastic)
• Acknowledgement deadline
• Dead letter topic configuration
• [ ] Authentication:
• Service account JSON key
• Workload identity federation
• Required IAM roles
• [ ] Message attributes: does the vendor include metadata in Pub/Sub message attributes (vs. body)?

GCS (Google Cloud Storage)

• [ ] Delivery model: export to GCS bucket
• [ ] Bucket and path pattern:
• Object path structure
• Partitioning scheme
• [ ] Authentication: service account with storage.objects.get and storage.objects.list
• [ ] Notification: Pub/Sub notifications for new objects?

Kafka

• [ ] Topic name(s) and naming convention
• [ ] Partitioning strategy
• [ ] Message format: JSON / Avro / Protobuf
• [ ] Schema registry: URL and compatibility mode (if Avro/Protobuf)
• [ ] Authentication: SASL (PLAIN, SCRAM, GSSAPI) / mTLS / none
• [ ] Consumer group configuration
• [ ] Offset management

Data format

Regardless of delivery mechanism, investigate the format of individual data records:

Object/message envelope

• [ ] Wrapper structure: are events wrapped in an envelope?
• Single object per file/message: { "event": {...} }
• Array of objects: { "Records": [...] } or { "records": [...] }
• NDJSON (one JSON object per line, no wrapper)
• Array at top level: [{...}, {...}]
• [ ] Envelope field name for the event array (e.g., Records, records, data, events, logs)
• [ ] Metadata in envelope: request ID, account ID, region, delivery timestamp, etc.

Compression and encoding

• [ ] Compression: gzip / none / snappy / lz4
• [ ] File extension pattern that indicates format: .json.gz, .csv, .log
• [ ] Content-Type in object metadata or message attributes
• [ ] Encoding: UTF-8 / other

Event structure

• [ ] Schema documentation: link to official schema reference
• [ ] Schema versioning: does the schema change between versions? Is there a version field?
• [ ] Event type discrimination: which field indicates the event type/category?
• [ ] Nested depth: how deeply nested are the objects?
• [ ] Array fields: which fields contain arrays (important for pipeline flattening decisions)?
• [ ] Dynamic keys: any fields where the key name varies (maps, labels, custom attributes)?
• [ ] Large schema handling: if the schema has hundreds of fields, download the schema definition (JSON Schema, Protobuf, Avro, or vendor schema repo) into temp/ and use Python to extract the full field inventory. Write results to references/field-schema-analysis.md

Batch considerations

• [ ] Events per object/message: single event or batch?
• [ ] If batched: typical batch size, maximum batch size
• [ ] Ordering: are events within a batch ordered by time?
• [ ] Deduplication: unique event ID field for cross-batch deduplication?

Event types and field schema

Same as the general checklist -- capture for each event type:

• [ ] Complete event type list with descriptions
• [ ] Field inventory per event type (name, type, description, example, always present?)
• [ ] Enumeration values for categorical fields
• [ ] Timestamp fields and formats
• [ ] IP, user, hostname, hash fields for ECS mapping
• [ ] Sample events saved to references/sample-events/

Volume and performance

• [ ] Expected data volume: events per second/minute/hour in typical deployment
• [ ] Object/message size: typical and maximum sizes
• [ ] Delivery latency: how soon after event occurrence does data appear?
• [ ] Backfill: can historical data be replayed/reprocessed?
• [ ] Retention: how long does data remain in the delivery mechanism?

Infrastructure setup guide

• [ ] Step-by-step vendor-side configuration to enable data export (or link to vendor guide)
• [ ] Cloud-side infrastructure needed (S3 bucket, SQS queue, Event Hub namespace, etc.)
• [ ] Recommended resource sizing (SQS visibility timeout, Event Hub partition count, etc.)
• [ ] Network requirements: VPC endpoints, private endpoints, firewall rules?
• [ ] Cost considerations: storage costs, message delivery costs, cross-region data transfer

Pipeline implications

After gathering the above, note:

• [ ] Unwrap/flatten needed: does the pipeline need to split batch arrays into individual events?
• [ ] Routing logic: do different event types in the same stream need different parsing branches?
• [ ] Shared fields vs type-specific: can a single pipeline handle all event types or are sub-pipelines needed?
• [ ] Content type handling: does the input need file_selectors or similar to handle mixed content types?
• [ ] Large event handling: are there events that exceed typical size limits?

Competitive SIEM Coverage Checklist

Use this checklist for Research Track E on every research run, regardless of the product's collection method. The goal is to determine whether IBM QRadar, Splunk, and Sumo Logic already have an integration or app for the product being researched, and to document what each covers and how it collects data.

This analysis feeds section 1.5 of the research brief and the detailed references/competitive-siem-coverage.md file.

---

Competitor catalog search strategy

For each competitor below, search its official app/integration marketplace using the product name, vendor name, and common aliases. Use these as your primary starting points — do not rely on general web searches unless the catalog search is inconclusive.

Search terms to try per competitor

• [ ] Exact product name (e.g., "Okta", "CrowdStrike Falcon")
• [ ] Vendor name alone (e.g., "Palo Alto", "Fortinet")
• [ ] Common product abbreviations or aliases (e.g., "CS Falcon", "PA Firewall")
• [ ] Technology category (e.g., "endpoint detection", "firewall", "identity provider") — useful when no direct product match is found
• [ ] Note whether a result is an exact match, a partial match (covers some features), or a near-miss (different product from same vendor)

---

Per-competitor investigation items

For each competitor (IBM QRadar, Splunk, Sumo Logic), capture the following if an integration/app is found:

Integration identity

• [ ] Integration / app name: exact name as listed in the catalog
• [ ] Publisher / maintainer: vendor-maintained, Splunk-built, IBM-built, or community/partner
• [ ] Catalog page URL: direct link to the listing
• [ ] Version: latest version number (if shown)
• [ ] Last updated date: indicates how actively maintained the integration is
• [ ] Compatibility notes: which SIEM platform versions are supported

Data coverage

• [ ] Supported data sources: which log sources, event types, or API endpoints are covered
• Be specific: "firewall traffic logs, threat prevention logs, URL filtering logs" not just "all logs"
• [ ] Event types / log types listed: enumerate them if the listing provides a breakdown
• [ ] Coverage gaps: which product data sources are NOT covered (if discoverable from the listing)
• [ ] Known limitations: data volume caps, filtering restrictions, unsupported event types

Collection method

• [ ] Collection mechanism: how does the competitor collect data from the product?
• API pull (REST, GraphQL, vendor SDK)
• Syslog push (UDP/TCP, RFC 3164/5424, CEF, LEEF)
• Agent/forwarder (Splunk Universal Forwarder, IBM WinCollect, etc.)
• File-based / log shipping
• Cloud delivery (S3, Event Hub, Pub/Sub)
• Vendor-native forwarding (product pushes directly to SIEM)
• [ ] Protocol / format details: CEF, LEEF, JSON, key-value — note the wire format if documented
• [ ] Authentication method: how the SIEM authenticates to the product (if API-based)
• [ ] Configuration requirements: what the user must set up on both sides

Quality signals

• [ ] User ratings / reviews: note star rating and review count if visible (Splunkbase shows these)
• [ ] Downloads / installs: popularity indicator (Splunkbase shows download counts)
• [ ] Support tier: Splunk-supported, vendor-supported, community-supported, or unsupported/archived
• [ ] Documentation quality: does the listing link to meaningful setup documentation?

---

Comparison layer

After gathering per-competitor data, assess the overall competitive landscape:

• [ ] Coverage breadth: which competitor covers the most data sources / event types?
• [ ] Collection method alignment: is the predominant collection method the same across competitors, or do they differ? Does this align with or differ from the recommended Elastic collection method?
• [ ] Maintenance status: are the integrations actively maintained or stale?
• [ ] Gaps Elastic could address: data sources or event types none of the competitors cover, or cover poorly
• [ ] Differentiators: areas where Elastic's approach (e.g., ECS normalization, Elastic Agent, Fleet management) could provide a better experience than the competitive offerings

---

Quality standards

• Only record integrations you have confirmed exist in the catalog. Do not infer or assume based on the vendor's general reputation.
• If no integration is found for a competitor, record that explicitly as "No integration found" — do not omit the competitor from the output.
• Mark any detail that could not be confirmed from the catalog listing or its linked documentation with [UNVERIFIED].
• Prefer official marketplace/catalog pages over blog posts, press releases, or third-party reviews.
• If a catalog listing is ambiguous (e.g., covers multiple products under one app), note the ambiguity rather than claiming full coverage.

---

Output

Write findings to references/competitive-siem-coverage.md in the working directory. Structure the file as:

1. Summary table — one row per competitor (used verbatim in research brief section 1.5) 2. Per-competitor H2 sections — full detail for each of IBM QRadar, Splunk, and Sumo Logic 3. Comparison notes — gaps, differentiators, and overall landscape assessment

Return a concise summary inline with: which competitors have integrations, the dominant collection method found, and the path to the written file.

Data Collection Methods

This reference describes the input types available in Elastic Agent integrations, what each is used for, and what research information is needed for each.

Use this to determine which collection method fits the product being researched and what details to investigate.

Input type decision tree

Does the vendor expose a REST/HTTP API for retrieving events?
  YES → CEL input (or httpjson for simple cases)
  NO ↓

Does the product write local log files?
  YES → log file input (filestream)
  NO ↓

Does the product send syslog messages?
  YES → syslog input (tcp/udp)
  NO ↓

Does the vendor deliver data to a cloud message queue or object store?
  S3 bucket → S3/SQS input (aws-s3)
  Azure Event Hub → Azure Event Hub input (azure-eventhub)
  Google Pub/Sub → GCP Pub/Sub input (gcp-pubsub)
  Azure Blob Storage → Azure Blob Storage input (azure-blob-storage)
  GCS bucket → GCS input (gcs)
  Kafka topic → Kafka input
  NO ↓

Does the product expose a streaming/websocket endpoint?
  YES → evaluate CEL with streaming or custom input
  NO ↓

Can data be exported as flat files (CSV, JSON) and dropped to a path?
  YES → log file input (filestream)
  NO → product may not be suitable for direct Elastic ingestion

Input types reference

Standard variable tables are authoritative. The "Standard configuration variables" tables below are the complete set of variables that may be proposed in the research brief's configuration plan (configuration-plan.md and section 6 of research-brief.md) for each input type. Do not invent additional variables based on patterns seen in legacy integrations in elastic/integrations. In particular, never propose `preserve_duplicate_custom_fields` as a configurable variable — it is a deprecated pipeline anti-pattern prohibited by ingest-pipelines/SKILL.md. The only preserve_* variable that is valid is preserve_original_event, listed in the tables below where applicable (filestream, TCP/UDP, and similar log-based inputs only — never for CEL). Additional product-specific variables are acceptable only when tied to a documented vendor-side requirement (e.g., a tenant ID for a multi-tenant API), not a pipeline behavior toggle.

CEL (Common Expression Language) -- REST API collection

When to use: The product exposes a REST API for retrieving events, logs, or metrics.

Elastic input type: cel

What to research:

• Base URL and API version
• Authentication method and credential types
• All relevant endpoints (list with paths)
• Request parameters: required, optional, filtering, time range
• Response format: JSON structure, envelope vs. array, nested objects
• Pagination: mechanism (offset, cursor, link-header, keyset, page number), field names, termination condition
• Rate limiting: limits, headers, retry-after behavior
• Timestamp handling: format, timezone, field names for time-range queries
• Error response format and status codes
• Webhook alternative (some products offer both pull and push)
• API permissions/scopes required

Standard configuration variables (API key / Bearer token auth):

Additional variables for OAuth2 auth (authorization_code or client_credentials):

The CEL input's auth.oauth2 configuration block natively supports authorization_code (including PKCE), client_credentials, and automatic token refresh. When the API uses OAuth2, prefer the built-in auth block over manual token management. Research must capture the exact authorization URL, token URL, refresh URL, and required scopes to enable this.

Filestream -- local log files

When to use: The product writes log files to disk on the host where Elastic Agent runs.

Elastic input type: filestream (preferred) or log (legacy)

What to research:

• Default log file paths per OS (Linux, Windows, macOS)
• Log format: syslog, JSON/NDJSON, CSV, key-value, multiline, custom delimited
• Log rotation behavior (size, time, naming pattern)
• Character encoding
• Multiline patterns (if applicable): start/end patterns, what constitutes a single event
• All distinct log types/files and what events each contains
• Timestamp format within log lines
• Sample log lines for each event type

Standard configuration variables:

TCP/UDP -- syslog collection

When to use: The product sends syslog messages over the network to a collector.

Elastic input type: tcp and/or udp

What to research:

• Syslog RFC version: 3164 (BSD) or 5424 (IETF)
• Message format inside syslog envelope: CEF, LEEF, key-value, free text, JSON
• Syslog facility and severity usage
• Default source port(s)
• Whether the product supports TLS for syslog
• Timezone handling: are timestamps in UTC or local? Does the message include timezone?
• Message structure and delimiter patterns
• All distinct event types by facility, severity, or message ID
• Sample syslog lines for each event type

Standard configuration variables:

AWS S3 / SQS -- cloud object store

When to use: The vendor or cloud service delivers data as objects in an S3 bucket, optionally with SQS notifications.

Elastic input type: aws-s3

What to research:

• Object format: JSON, NDJSON, CSV, gzip-compressed, Parquet
• Object path/prefix pattern and partitioning scheme
• Whether objects contain single events or batches
• Object naming convention and timestamp encoding in path
• SQS notification configuration (if used)
• IAM permissions required
• Cross-account access patterns
• Data retention and lifecycle policies
• Sample object content

Standard configuration variables:

Azure Event Hub

When to use: The vendor or Azure service streams data through Azure Event Hubs.

Elastic input type: azure-eventhub

What to research:

• Event Hub namespace and hub name configuration
• Consumer group setup
• Message format: JSON envelope, nested records, batch arrays
• Authentication: connection string, managed identity, SAS token
• Partitioning scheme
• Schema of individual events within the Event Hub message
• Storage account for checkpointing
• Sample event content

Standard configuration variables:

GCP Pub/Sub

When to use: The vendor or GCP service streams data through Google Cloud Pub/Sub.

Elastic input type: gcp-pubsub

What to research:

• Pub/Sub topic and subscription configuration
• Message format and attributes
• Authentication: service account JSON key, workload identity
• Ordering requirements
• Dead letter topic setup
• Schema of individual messages
• Sample message content

Standard configuration variables:

HTTP Endpoint -- webhook receiver

When to use: The vendor pushes data to a webhook URL that the Elastic Agent listens on.

Elastic input type: http_endpoint

What to research:

• Webhook payload format (JSON body, form data)
• Authentication of incoming requests (HMAC signature, shared secret, mTLS)
• Event delivery guarantees (at-least-once, retry behavior)
• Webhook registration/configuration on the vendor side
• Payload structure for each event type
• Rate and size limits on the vendor side
• Sample webhook payloads

Standard configuration variables:

Azure Blob Storage

When to use: Data is delivered as blobs in Azure Storage containers.

Elastic input type: azure-blob-storage

What to research:

• Container name and blob path/prefix patterns
• Blob format: JSON, NDJSON, CSV, gzip
• Authentication: connection string, SAS token, managed identity
• Blob naming convention and partitioning
• Poll interval and change detection
• Sample blob content

GCS (Google Cloud Storage)

When to use: Data is delivered as objects in GCS buckets.

Elastic input type: gcs

What to research:

• Bucket name and object prefix patterns
• Object format and compression
• Authentication: service account
• Object naming and partitioning
• Sample object content

Kafka

When to use: Data is available on Kafka topics.

Elastic input type: kafka

What to research:

• Topic name(s) and partitioning
• Message format (JSON, Avro, Protobuf)
• Authentication: SASL, mTLS, no auth
• Consumer group configuration
• Schema registry (if Avro/Protobuf)
• Sample messages

Multiple collection methods

Many products support more than one collection method. When this is the case:

1. Document all available methods. 2. Recommend the best method based on:

• Completeness: which method provides the most event types and field detail
• Timeliness: which has the lowest latency from event occurrence to collection
• Reliability: which has the best delivery guarantees
• Simplicity: which requires the least user configuration
• Standard practice: which method is most commonly used by the Elastic community (check existing integrations/packages/ for precedent)

3. If two methods are close in quality, consider supporting both as separate data streams within the same integration.

Log and File Research Checklist

Use this checklist when the product writes local log files or sends syslog messages over the network. This covers filestream, tcp, and udp input types.

Discovery phase

• [ ] Find the official logging documentation for the product
• [ ] Identify all distinct log types/sources the product generates
• [ ] Determine whether the product supports remote syslog forwarding, local files, or both
• [ ] Check for log configuration guides (enabling verbose logging, choosing formats, configuring destinations)
• [ ] Look for log message catalogs or reference documents that list all event IDs/types
• [ ] Check if the vendor publishes log format definitions, parser code, or message catalogs in a public git repository. If found, clone into temp/ for analysis. Parser code from vendor SDKs or SIEM connectors can reveal field structures more completely than documentation

Log file details (filestream input)

Paths and locations

• [ ] Default log file paths per OS:
• Linux: <path>
• Windows: <path>
• macOS: <path> (if applicable)
• [ ] Glob patterns needed: do paths include variable components (dates, hostnames, instance names)?
• [ ] Multiple log files: does the product write to different files for different log types?
• [ ] Log rotation:
• Rotation trigger (size, time, both)
• Rotated file naming pattern (.1, .log.1, date suffix)
• Compression of rotated files (gzip?)
• Retention policy (number of files, days)
• [ ] File permissions: does the agent need special permissions to read the log files?
• [ ] Character encoding: UTF-8, ASCII, other?

File format

• [ ] Format identified: syslog / JSON / NDJSON / CSV / key-value / CEF / LEEF / W3C extended / custom delimited / multiline free text
• [ ] One event per line? Or does a single event span multiple lines?
• [ ] If multiline:
• Start pattern (regex that identifies the first line of an event)
• End pattern (if applicable)
• Example of a complete multiline event
• [ ] Header lines: does the file start with header/metadata lines that should be skipped?

Syslog details (tcp/udp input)

Transport

• [ ] Protocol: UDP only / TCP only / both supported
• [ ] Default port: vendor's recommended port (or standard 514)
• [ ] TLS support: does the product support syslog over TLS (TCP)?
• [ ] Syslog RFC:
• RFC 3164 (BSD format): <PRI>TIMESTAMP HOSTNAME APP-NAME: MSG
• RFC 5424 (IETF format): <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MSG
• [ ] Facility and severity usage: which syslog facility does the product use? Does severity map to event severity?

Message format inside syslog

• [ ] Message format identified: CEF / LEEF / key-value pairs / JSON / free text / mixed
• [ ] If CEF:
• CEF version
• Device vendor / device product / device version fields
• Event class ID patterns
• Extension key names and types
• Example CEF messages for each event type
• [ ] If LEEF:
• LEEF version
• Delimiter character
• Key names and types
• [ ] If key-value:
• Delimiter between pairs (space, comma, pipe, etc.)
• Separator between key and value (=, :, etc.)
• Quoting rules for values with spaces
• Escape characters
• [ ] If JSON:
• Is it a complete JSON object per message?
• Or is JSON embedded within a syslog prefix?

Message structure and parsing

Delimiters and structure

• [ ] Field delimiter: character(s) that separate fields
• [ ] Message structure pattern: regex or grok pattern sketch for the common message format
• [ ] Variable-length fields: fields that can contain the delimiter character (need special parsing)
• [ ] Optional fields: fields that may be absent in some messages

Timestamp handling

• [ ] Timestamp field: where in the message is the timestamp?
• [ ] Timestamp format: MMM dd HH:mm:ss / ISO 8601 / Unix epoch / custom
• [ ] Timezone:
• Always UTC?
• Includes timezone offset?
• Local timezone with no offset (requires tz_offset config)?
• Multiple timestamps in one event (which is authoritative)?
• [ ] Timestamp precision: seconds / milliseconds / microseconds

Event types and content

Event type identification

• [ ] Event type indicator: which field or pattern distinguishes event types?
• Syslog facility/severity
• Message ID or event ID field
• Application name in syslog header
• Keyword or prefix in the message body
• Combination of fields
• [ ] Complete event type list:

Field inventory per event type

For each event type (or for common fields shared across types):

• [ ] Field list captured:

• [ ] Enumeration values: for status, action, severity, direction, protocol fields -- list all known values
• [ ] IP address fields: identify all fields containing IP addresses (needed for ECS mapping and GeoIP)
• [ ] User identity fields: identify fields with usernames, email addresses, user IDs
• [ ] Hostname fields: identify fields with hostnames, FQDNs

Sample data

• [ ] At least 3-5 sample messages per event type (from vendor docs, not fabricated)
• [ ] Edge cases captured:
• Minimal message (fewest optional fields)
• Maximal message (all fields populated)
• Error/failure events
• Events with special characters (Unicode, quotes in values)
• [ ] All samples saved to references/sample-events/ with descriptive filenames

Platform-specific considerations

• [ ] Version differences: do log formats change between product versions?
• [ ] Configuration-dependent output: do log fields change based on product configuration or license level?
• [ ] Encoding issues: are there known encoding problems (e.g., Windows event logs with mixed encoding)?
• [ ] Volume estimates: approximate events per second/minute in a typical deployment

Pipeline implications

After gathering the above, note:

• [ ] Parsing approach: grok / dissect / JSON parse / KV parse / CSV parse / combination
• [ ] Multiline handling needed: yes/no, with proposed pattern
• [ ] Branching needed: are there multiple event formats that require different parsing branches?
• [ ] Timezone injection: will tz_offset config be needed for timestamp parsing?
• [ ] Common parsing challenges: embedded JSON, URL-encoded values, nested key-value, variable field order

Research Output Template

Use this template as the structure for the research-brief.md file written to research_results/<product_slug>/.

Every section below should be populated. If a section does not apply, include it with a note explaining why (e.g., "N/A -- this product uses API collection, not log files."). This makes gaps explicit and prevents downstream consumers from wondering if information was simply missed.

---

Template starts here

````markdown

Research Brief: <Product/Vendor Name>

Generated: <date>

Researcher: AI-assisted research via /research-integration skill

Status: DRAFT | READY FOR REVIEW

Confidence: HIGH | MEDIUM | LOW (overall confidence in completeness)

1. Product Overview

1.1 What is it?

<2-5 sentences describing the product, what it does, and what kind of organization uses it.>

1.2 Vendor

• Vendor name: <vendor>
• Product name: <product>
• Product category: <e.g., endpoint security, network firewall, identity provider, cloud CSPM>
• Vendor documentation portal: <URL>

1.3 Data generated

<What kinds of data does this product generate? Security events, audit logs, metrics, configuration state, alerts, etc. List the major categories.>

1.4 Existing Elastic coverage

<Does an Elastic integration already exist for this product (check integrations/packages/ and packages/)? If yes, what does it cover and what gaps exist? If no, note that this is a new integration.>

1.5 Competitive SIEM coverage

<1-2 sentence summary of the overall competitive landscape: which competitors have integrations for this product, and how comprehensive is their coverage.>

See references/competitive-siem-coverage.md for full per-vendor analysis including support tier, version, gaps, and comparison notes.

2. Data Collection Method

2.1 Recommended method

• Input type: <e.g., cel, tcp/udp, aws-s3, filestream, azure-eventhub>
• Rationale: <why this method is recommended>

2.2 Alternative methods

<List other viable methods with brief pros/cons. If only one method exists, state that.>

2.3 Vendor-side setup required

<What does the user need to configure on the vendor side to enable data export? API key creation, syslog forwarding rules, S3 bucket policies, etc.>

3. Data Source Details

3.1 Connection and authentication

<Detailed connection information for the recommended collection method.>

For API-based:

• Base URL: <url>
• API version: <version>
• Authentication: <see below>
• Rate limits: <requests per minute/hour, burst limits>
• Documentation: <link to auth docs>

Authentication detail (choose the applicable block):

If API key or static Bearer token:

• Method: <API key header / API key query param / Bearer token>
• Header format: <e.g., Authorization: Bearer <token>, X-API-Key: <key>>
• Credential creation steps: <how to obtain the key>
• Required scopes/permissions: <list>
• Token lifetime: <expiry, or "does not expire">

If OAuth2 (authorization_code or client_credentials):

• Method: OAuth2
• Grant type: <authorization_code / client_credentials / both>
• Authorization URL: <url> (authorization_code only)
• Token URL: <url>
• Refresh URL: <url> (often same as token URL)
• Scopes required: <list of scopes needed for data collection>
• All available scopes: <full list if documented>
• Client registration: <how to create a client_id / client_secret — app registration steps, admin console, etc.>
• Token lifetime: <access token expiry, e.g., "30 days", "1 hour">
• Refresh token lifetime: <if documented>
• Token response format: access_token, refresh_token, expires_in, token_type
• Additional notes: <PKCE required? Specific redirect URI requirements? VPC/tenant-specific URLs?>

If interactive-only token generation (manual PAT, browser token page):

• Method: Manual token generation (interactive only)
• Note: <1-2 sentences describing the manual process. State this is not a standard OAuth2 flow and is not suitable as the primary auth method for an automated integration. If a proper OAuth2 flow also exists, reference it above as the primary method.>

For syslog-based:

• Protocol: <TCP/UDP/TLS>
• Default port: <port>
• Syslog format: <RFC 3164/5424>
• Message format inside envelope: <CEF/LEEF/KV/JSON/free text>

For cloud ingest:

• Service: <S3/SQS, Event Hub, Pub/Sub, etc.>
• Authentication: <IAM role, connection string, service account, etc.>
• Required permissions: <list>

3.2 Endpoints / data paths

<For APIs: list all relevant endpoints with path, HTTP method, and purpose.> <For logs: list file paths per OS.> <For cloud: list bucket/topic/hub names and path patterns.>

3.3 Pagination (API only)

• Mechanism: <offset, cursor, link-header, keyset, page-number, none>
• Page size parameter: <param_name> (default: <n>, max: <n>)
• Next page indicator: <field_name> in response body / Link header / etc.
• Termination condition: <empty results array, null cursor, count < page_size, etc.>
• Documentation: <link>

3.4 Time-based filtering (API only)

• Start time parameter: <param_name>
• End time parameter: <param_name> (if applicable)
• Time format: <ISO 8601, Unix epoch seconds/milliseconds, custom>
• Timezone: <UTC, configurable, local>
• Sort order: <ascending/descending by default, configurable?>
• Incremental collection strategy: <use last event timestamp as next start, cursor includes time state, etc.>

3.5 Reference documentation

<Comprehensive list of documentation links used in this research.>

4. Data Format and Structure

4.1 Format overview

• Wire format: <JSON, NDJSON, syslog+CEF, syslog+KV, CSV, XML, multiline text>
• Encoding: <UTF-8, ASCII, etc.>
• Compression: <gzip, none, etc.>
• Envelope structure: <e.g., {"data": [...], "meta": {...}} or flat array>

4.2 Event types

<List all distinct event types/categories the product generates.>

4.3 Field inventory

<For each event type (or shared across types), list the fields.>

Common fields (present in all/most event types)

<Event Type 1> specific fields

<Repeat for each event type.>

4.4 Sample data

<Include or reference representative sample events. If inline, use fenced code blocks. If separate files, reference them.>

See references/sample-events/ for complete sample data files:

• <event_type_1>.json -- <description>
• <event_type_2>.json -- <description>
• <event_type>.log -- <description>

<If field inventories or schema analyses were too large to include inline, reference the files:> See references/field-schema-analysis.md for the complete field inventory extracted from <source>. See temp/<subfolder>/ for the raw source artifacts (cloned repos, downloaded schemas, etc.).

Inline sample (most common event type)

<paste one representative event here>

4.5 Timestamp handling

• Primary timestamp field: <field_name>
• Format: <ISO 8601, Unix epoch seconds, Unix epoch milliseconds, custom format string>
• Timezone: <always UTC, local timezone, timezone offset included, configurable>
• Additional timestamp fields: <list any secondary timestamps and their meaning>

4.6 Special parsing considerations

<Note anything that makes parsing non-trivial: multiline patterns, embedded JSON in string fields, variable schemas, field name inconsistencies across API versions, etc.>

5. ECS Mapping Analysis

5.1 Categorization per event type

5.2 Field mappings

5.3 Related field enrichment

5.4 Geo enrichment candidates

<List IP address fields that are candidates for GeoIP enrichment and the appropriate ECS parent (source.geo, destination.geo, etc.)>

6. Configuration Plan

Variables listed below must come from the standard-variable tables in references/data-collection-methods.md, plus any documented vendor-specific variables (e.g., tenant ID for a multi-tenant API). Never list `preserve_duplicate_custom_fields`, `event.ingested` toggles, trailing `event.original` removal flags, or any other deprecated pipeline-behavior variable here, even if you saw them in legacy integrations under elastic/integrations. Pipeline behavior is the pipeline builder's concern, not configuration. The only valid preserve_* var is preserve_original_event (file/syslog inputs only, never CEL).

6.1 Required configuration variables

6.2 Optional configuration variables

6.3 Deployment notes

<Any notes about deployment architecture, network requirements, firewall rules, proxy considerations, etc.>

7. Recommended Integration Architecture

7.1 Package name

<package_name> (lowercase, underscores)

7.2 Data streams

7.3 Architecture rationale

<Why this data stream breakdown? Could be: one stream per API endpoint, one stream per log type, single stream with pipeline routing, etc. Explain the reasoning.>

7.4 Estimated complexity

• Pipeline complexity: <simple (flat JSON) / moderate (nested, multiple event types) / complex (multiline, variable schemas, routing)>
• CEL complexity: <simple (single endpoint, basic pagination) / moderate (multiple endpoints, OAuth, cursor state) / complex (multi-step state machine, multiple auth methods, cursor expiration)>
• Field count estimate: <approximate number of fields per stream>

8. Open Questions and Gaps

<List anything that could not be determined from research and requires user input, vendor clarification, or hands-on testing.>

9. Source Attribution

<List all sources used in this research with how they were accessed.>

````

---

Usage notes

• Sections 1-4 form the factual research foundation.
• Section 5 (ECS mapping) is an analysis layer that requires the ECS reference skill.
• Section 6 (Configuration) bridges research to implementation planning.
• Section 7 (Architecture) is the integration design recommendation.
• Section 8 (Open questions) captures what still needs human judgment.
• Section 9 (Attribution) provides traceability for all claims.

The brief should be thorough enough that someone can pass it directly to /create-integration as the primary input.

Companion artifacts

The research brief is the primary output, but it is supported by additional files in the same directory:

• `test-api.py` -- (API/CEL collection only) Standalone Python script that exercises the exact API flow proposed for the CEL integration. Tests connectivity, authentication, pagination, and response structure. Run it against the real vendor API and share the resulting archive for development. See references/test-api-script-spec.md in the skill directory for the full specification.
• `references/` -- Curated research artifacts: detailed field analyses, API spec notes, sample events. These are polished enough for downstream consumers.
• `temp/` -- Raw downloaded artifacts: cloned repos, SDK sources, large schema files, analysis scripts. Retained as reference for the human and for reproducibility. Not cleaned up unless disk space is critical.

When the brief references detailed findings that are too large to include inline, it should point to the appropriate file in references/ or temp/ with a path and one-line description.

Research subagent guidance

Operating manual for a subagent performing one research track on behalf of the research-integration orchestrator.

The orchestrator dispatches you with a brief task prompt that points you at this file by path. Read this entire file end-to-end before doing any other work. Beyond that, you do not need to load any other skills or reference files unless the task prompt explicitly tells you to. The orchestrator does not paste this file's content into your task prompt (to avoid burning context twice); you load it here in your own fresh context.

The orchestrator's task prompt tells you what to research, what details to focus on, the working directory to use, and how to structure your response. This file tells you how to operate as a research subagent. Follow both.

How you work

1. Read the orchestrator's task prompt carefully. It defines your research scope, focus areas, expected output structure, and the working directory for this research (typically research_results/<subject>/). 2. Use web search to find official documentation, reference material, technical specifications, and community resources. 3. Use web fetch to retrieve and read full pages when search results point to relevant documentation. 4. When documentation is spread across multiple pages, follow links to get complete information rather than stopping at the first page. 5. Download resources locally when web search/fetch is insufficient — clone git repos, install pip/npm packages, or fetch large files into the temp/ subdirectory of your working directory (see below). 6. Use Python (or other tools) to analyze large files — schemas, API specs, SDK models, and similar artifacts that are too large or complex to reason about by reading alone (see below). 7. Cross-reference multiple sources to confirm facts. Prefer primary/official sources. 8. Write large findings to files rather than returning everything inline. Return a concise summary pointing to the files you wrote (see "Result delivery" below).

Working directory and temp folder

The task prompt specifies a working directory, typically:

research_results/<subject>/

You have write access. Use it. Specifically:

• `temp/` — Use research_results/<subject>/temp/ for all downloaded

artifacts: cloned repositories, fetched SDK source, large JSON schemas, installed package sources, raw API spec files, etc. Create subdirectories as needed (e.g., temp/vendor-sdk/, temp/schema-files/).

• `references/` — Use research_results/<subject>/references/ for curated

research artifacts that should persist for the orchestrator and downstream consumers: extracted notes, sample events, cleaned-up schema summaries, etc.

• Do not delete temp data unless disk space is critically constrained. The

temp/ folder serves as a reference for the human later and may be useful for follow-up research or debugging.

When to download into temp

Download resources when:

• A vendor publishes schemas, SDKs, or OpenAPI specs in a git repository

and you need to inspect them → git clone into temp/.

• A pip or npm package contains type definitions, models, or example code

that documents the API/data format → install or download into temp/.

• A large JSON/YAML schema file needs programmatic analysis → fetch it

into temp/ and use Python to extract what you need.

• You need to examine example scripts, SDK source code, or test fixtures

from a vendor's repository to understand data formats.

• Web fetch returns incomplete or truncated content for large pages →

download the raw source into temp/.

Using Python for analysis

When dealing with large or complex files (JSON schemas with hundreds of fields, OpenAPI specs, SDK model definitions, etc.), use Python rather than trying to read and reason about the raw content:

# Example: extract field names and types from a large JSON schema
import json
with open('temp/schema.json') as f:
    schema = json.load(f)
# ... programmatic extraction, filtering, summarization

Write analysis scripts directly via the shell. Prefer Python but use whatever tool is most appropriate. The goal is to extract structured, relevant information from large artifacts efficiently rather than attempting to manually read thousands of lines.

Result delivery

Critical rule: avoid returning massive inline data.

When your findings are small (a few paragraphs, a short table, a handful of sample events), return them directly in your response.

When your findings are large — extensive field inventories, complete schema analyses, many sample events, full API endpoint catalogs — follow this process:

1. Write the detailed findings to a markdown file in the research working directory:

• For material the orchestrator and downstream consumers need long-term →

write to research_results/<subject>/references/<descriptive-name>.md

• For raw analysis output or intermediate work → write to

research_results/<subject>/temp/<descriptive-name>.md 2. Return a concise summary as your response to the orchestrator, including:

• Key findings and conclusions (the actual insights, not just "I found

stuff")

• A list of files you wrote, with their paths and a one-line description

of each

• Guidance on which files to read and when (e.g., "read

references/field-schema-analysis.md for the complete field inventory when building the pipeline") 3. The summary should be self-contained enough that the orchestrator can synthesize the research brief without reading every file, but reference the files for full detail.

This approach keeps the orchestrator's context lean while preserving all detail on disk.

Research quality standards

• Prefer official sources. Vendor documentation, API references, RFCs, and

official specs take precedence over blog posts, forums, or third-party summaries.

• Cite your sources. Include URLs for every significant claim so the

orchestrator can verify and follow up.

• Be specific, not generic. Concrete details (exact field names, precise

URL paths, specific parameter values) are far more useful than vague descriptions.

• Flag uncertainty. If you cannot find definitive information, say so

explicitly with [UNVERIFIED] rather than guessing. State what you looked for and where.

• Distinguish fact from inference. When you are reasoning about something

rather than reporting documented fact, make that clear.

• Be thorough within your assigned scope. Cover the topic completely, but

do not drift into areas outside what the task prompt asks for.

• Include examples. Sample data, request/response pairs, log lines,

configuration snippets — concrete examples are more valuable than descriptions.

Anonymization

When collecting or constructing sample data (API responses, log lines, configuration examples, etc.), anonymize all identifying information while preserving structural fidelity:

• IP addresses: use RFC 5737 documentation ranges (198.51.100.x,

203.0.113.x, 192.0.2.x)

• Hostnames/domains: use example.com, example.org, example.net
• Email addresses: use user@example.com, admin@example.org
• Person names: use Alice Johnson, Bob Smith
• Organization names: use Example Corp, Acme Inc
• Tokens/keys: use obviously fake values like sk_test_example_key_1234567890
• UUIDs/IDs: use synthetic but format-valid values

Preserve value types, lengths, delimiters, and structural relationships so the samples remain useful for downstream work.

Response guidelines

• Follow the output structure requested in the orchestrator's task prompt. If

no structure is specified, organize findings logically with clear sections.

• Small results: return inline in a single well-structured response with

markdown headings and tables.

• Large results: write detail to files, return a summary (see "Result

delivery" above).

• Do not include integration implementation code (CEL programs, ingest

pipelines, manifest YAML, etc.) unless the task prompt explicitly asks for it. Analysis scripts used during research (Python for schema parsing, etc.) are fine and should be left in temp/ for reproducibility.

• **Do not prescribe pipeline, field-mapping, or manifest implementation

details.* Your job is to document the data (field names, types, enum values, ECS mapping candidates, sample events) — not how the ingest pipeline, `fields/.yml, or manifest.yml should be authored. In particular, **never propose preserve_duplicate_custom_fields as a configuration variable, a recommended pipeline behavior, or a "consider supporting…" suggestion** — it is a deprecated pipeline anti-pattern, prohibited by ingest-pipelines/SKILL.md, even though it appears in many legacy integrations. The same applies to event.ingested toggles and trailing event.original removal flags. The only valid preserve_* configuration variable is preserve_original_event` (file or syslog inputs only, never CEL). The pipeline builder, ECS field-mapping builder, and reviewer skills are the authoritative source for any pipeline or manifest behavior decisions.

• End with a section noting any gaps, open questions, or areas where further

investigation would be valuable.

API Test Script Specification

This reference defines the structure, behaviour, and requirements for the test-api.py script generated during the research phase. The script exercises the exact API flow proposed for the CEL integration so a human can validate connectivity, authentication, pagination, and response structure before any Elastic Agent work begins.

Applicability: Generate this script only when the recommended collection method is API-based (CEL input type). It does not apply to syslog, file-based, or cloud-ingest collection methods.

File location

research_results/<product_slug>/test-api.py

Same directory as research-brief.md — the root of the research output folder.

General rules

• Standard library only. The script must use only Python 3 standard library modules (urllib.request, json, logging, argparse, ssl, os, tarfile, etc.). No requests, httpx, or other third-party packages. This ensures the script runs on any system with Python 3 installed.
• Mirror the proposed CEL flow. The script's request sequence, pagination logic, cursor handling, and termination conditions must match what was proposed in the research brief for the CEL program. Annotate key sections with comments referencing the corresponding CEL logic (e.g., "Same branching as the CEL program's first-request vs continuation-request logic").
• TLS verification disabled. Always disable TLS certificate verification (ssl._create_unverified_context() or equivalent). This is a testing tool, not a production client — it may be pointed at mock servers, proxies, or dev environments with self-signed certificates.
• Redact secrets in all output. Credentials must never appear in the log file, trace file, stdout, or archive. Implement redaction helpers that scrub Authorization headers, API keys, tokens, and any other sensitive values before writing them anywhere.

Script header (module docstring)

The script must begin with a comprehensive docstring that serves as standalone documentation:

#!/usr/bin/env python3
"""
<Product Name> — API Connectivity & Flow Test
==============================================

<1-2 sentence description of what this script tests and which API it targets.>

Vendor-side setup
-----------------
<Step-by-step instructions for everything a user needs to do on the vendor
side before running this script. This is vendor-specific and must cover:>

1. How to create the credentials (API key, OAuth app, service account, etc.)
2. Which admin console / portal / page to visit
3. What permissions or scopes to grant
4. Any prerequisites (license tier, feature flags, admin role)
5. Any values to note down (org ID, tenant URL, client ID, etc.)

<This section should be detailed enough that someone unfamiliar with the
vendor can follow it end-to-end. It mirrors the "Vendor-side setup" section
of the research brief.>

Usage
-----
    python3 test-api.py <required args> [optional args]

    Environment variables:
        <VAR_NAME>    <description> (alternative to --flag)
        ...

Optional flags:
    --url              Base API URL including scheme (default: <vendor default>)
    --max-pages        Stop after N pages (default: 5)
    --mock             Skip archiving; print output directory path instead
    ...

Output
------
On success the script creates an output directory containing:
    test-api.log             — verbose step-by-step log
    trace.json               — detailed request/response trace (auth redacted)

The directory is then archived as <output-dir>.tar.gz.
Please send this archive to the integration maintainers for review.
"""

CLI arguments and environment variables

Every credential and connection parameter must be accepted both as a CLI argument and as an environment variable. The CLI argument takes precedence when both are provided.

Required parameters (vendor-specific)

These vary per vendor. Common examples:

Use argparse with default=os.environ.get("VAR_NAME") so either source works.

Standard parameters (always present)

These flags must appear in every test-api.py, regardless of vendor:

Vendor-specific optional parameters

Add flags that mirror configuration variables from the configuration plan, for example:

• --initial-interval — lookback window (e.g., 24h, 30m)
• --batch-size / --page-size — items per page
• --event-type / --log-type — filter to a specific event category
• Any other filter or option the API supports that is useful for testing

CLI output (stdout)

The stdout output is for the human running the script. It must be concise and step-oriented:

1. Banner — script name and purpose, one-line summary. 2. Parameter summary — show the resolved configuration (base URL, org/tenant ID, credentials masked to last 4 chars, page size, max pages, etc.). 3. Step-by-step progress — for each logical step (e.g., each paginated request), print a single line showing:

• Step/page number (e.g., [2/5])
• What is happening (e.g., "Requesting page…")
• Result (e.g., "OK (42 events, 0.31s)" or "FAILED (HTTP 401, 0.12s)")
• Brief context on the result (e.g., time span of events, "no more pages")

4. Execution summary — after the collection loop, print:

• Overall status: SUCCESS or FAILED
• Total events collected
• Total pages fetched
• Any product/category breakdown if the data supports it
• Output directory and archive location

5. Handoff message — tell the user to send the archive to the integration maintainers.

What NOT to print to stdout:

• Raw request bodies
• Raw response bodies
• Full headers
• Any sensitive credential values (even partial)

Output directory

The script creates a directory (default: test-api-output/) containing exactly two files:

1. test-api.log — verbose log file

A text log file written by Python's logging module at DEBUG level. Contains everything printed to stdout plus additional detail:

• Full resolved configuration (credentials redacted)
• For each request: URL (redacted), timing, response status code, body length
• Rate-limit header values when present
• Pagination state transitions (cursor values, offset changes)
• Error details with full tracebacks
• Summary statistics

This is a superset of stdout — anything on stdout also appears in the log, with more detail.

2. trace.json — request/response trace

A JSON array where each element represents one HTTP exchange. This is the detailed diagnostic file for developers analyzing the API flow. Each entry contains:

{
  "page": 1,
  "request": {
    "method": "GET",
    "url": "<full URL with query params, auth redacted>",
    "headers": {"<header>": "<value, auth redacted>"},
    "body": null
  },
  "response": {
    "status_code": 200,
    "headers": {"<header>": "<value>"},
    "body": {"<parsed JSON object or string if not JSON>": "..."},
    "elapsed_s": 0.312
  },
  "pagination": {
    "mechanism": "<cursor|offset|page|etc.>",
    "field_used": "<e.g., meta.next>",
    "value_from_response": "<the cursor/offset value received>",
    "value_for_next_request": "<what will be sent in the next request>",
    "want_more": true
  },
  "event_count": 42,
  "error": null
}

Key requirements for the trace:

• Full response body included — this is the detailed trace, not the summary log. Response bodies are essential for debugging field mapping and pipeline development.
• Body fields must be parsed JSON objects, not strings — both request and response body fields must be stored as parsed JSON objects (dicts/lists), not as raw JSON strings. Storing them as strings causes double-escaping in the trace file, making it hard to read. Use a _try_parse_body helper (see below) that attempts json.loads and falls back to the raw string if parsing fails. For POST request bodies where the payload dict is already available in Python, store redact_dict(payload) directly rather than serializing and re-parsing.
• Pagination logic exposed — show exactly which field was read from the response, what value it had, and how it was used to construct the next request. This validates that the proposed CEL pagination logic works correctly.
• Auth redacted — even in the detailed trace, replace credential values with [REDACTED]. For response bodies that may contain OAuth tokens (e.g. token exchange endpoints), redact access_token, refresh_token, and id_token keys from the parsed object before storing.
• Errors included — if a request fails, the entry still gets written with the error details.

_try_parse_body helper

Every script must include this helper (adapt the redaction logic for the vendor if the response body may contain credentials such as OAuth tokens):

def _try_parse_body(text):
    """Return parsed JSON object if *text* is valid JSON, otherwise return the string as-is."""
    if not text:
        return text
    try:
        return json.loads(text)
    except (json.JSONDecodeError, ValueError):
        return text

For APIs that return OAuth tokens in response bodies (e.g. token exchange endpoints), extend the helper to redact sensitive keys from the parsed object:

def _try_parse_body(text):
    """Return parsed JSON object if *text* is valid JSON, otherwise return the string as-is.
    When the parsed result is a dict, redact known OAuth token keys in-place."""
    if not text:
        return text
    try:
        obj = json.loads(text)
    except (json.JSONDecodeError, ValueError):
        return redact(text)
    if isinstance(obj, dict):
        for key in ("access_token", "refresh_token", "id_token"):
            if key in obj and obj[key]:
                obj[key] = "[REDACTED]"
    return obj

Archiving

After the collection loop and output writing, the script checks the --mock flag:

When `--mock` is NOT set (default — normal mode):

1. Creates a .tar.gz archive of the output directory. 2. Prints the archive path to stdout. 3. Instructs the user to share the archive with the integration maintainers. 4. If archiving fails, falls back gracefully and tells the user the unarchived directory path.

When `--mock` IS set (mock mode):

1. Skips .tar.gz archive creation entirely. 2. Prints the full absolute path to the output directory. 3. Prints a message indicating that all request and response results are logged in the output directory (test-api.log for the verbose log and trace.json for the detailed request/response trace).

Mock mode is designed for the CEL builder agent workflow. When the agent runs test-api.py --mock against a local system test mock server, it can immediately inspect the output files to validate that the mock API is returning the expected responses — without needing to decompress an archive first.

Error handling

• All exceptions caught — the script must never crash with an unhandled exception. Wrap the main collection loop in try/except and log the full traceback to both the log file and trace file.
• HTTP errors — handle and log: connection errors, timeouts, non-2xx status codes, invalid JSON responses, rate limiting (429).
• Rate limiting — if a 429 is received, log the rate-limit headers (Retry-After, X-RateLimit-Reset, etc.) and stop. Do not implement automatic retry/backoff — the script is for testing, not production collection.
• Keyboard interrupt — catch KeyboardInterrupt, log it, and still write the output files and archive before exiting.
• Exit code — exit 0 on success, exit 1 on any failure.

Script structure

The script should follow this general structure (adapt to the vendor):

#!/usr/bin/env python3
"""<docstring as described above>"""

import argparse
import json
import logging
import os
import shutil
import ssl
import sys
import tarfile
import time
import traceback
import urllib.error
import urllib.parse
import urllib.request

# --- Helpers ---
# Redaction utilities, interval parsing, product detection, etc.

# --- HTTP transport ---
# A do_request() function using urllib with TLS verification disabled,
# proxy support, and timeout handling.

# --- Core collection loop ---
# The main pagination loop that mirrors the proposed CEL program.
# Each iteration: build URL, make request, parse response, extract
# pagination state, decide want_more, log everything.

# --- Main ---
# Argument parsing (with env var fallbacks), output directory setup,
# logger setup, run collection, write trace, print summary, archive.

if __name__ == "__main__":
    main()

Relationship to CEL program design

The test script is the ground truth for CEL program construction. The relationship is directional: test-api.py is the source of truth, and the CEL program is a translation from it. The script has been tested against a real API; even if the research brief has inaccuracies, the script's behaviour can be accepted as correct.

Specifically:

• Same endpoints — the script calls the same API paths the CEL program will use.
• Same authentication — the script uses the same auth mechanism (API key header, OAuth2 token exchange, etc.).
• Same pagination — the script implements the same pagination pattern (cursor, offset, keyset) with the same termination conditions.
• Same time-based filtering — the script uses the same time parameters and formats.
• Same request construction — query parameters, headers, and body (if POST) match what the CEL program will send.

The CEL expression builder receives test-api.py as its primary input and translates the collection function (run_collection() or equivalent) into CEL. The research brief provides supplementary context (field meanings, data types, edge cases not exercised by the script), but the Python implementation is the specification.

The script also serves as the mock specification (the mock is derived from the script's request/response flow) and the mock validator (the script must pass against the mock before CEL translation begins). When a trace.json exists from a real API run, it acts as an offline fidelity witness — comparing mock responses against the trace catches cases where the mock and script agree but both diverge from real API behaviour.

Example reference

See the Atlassian Jira test script at research_results/jira/test-api.py for a complete working example. It demonstrates all of the patterns described here: stdlib-only HTTP, redaction, pagination mirroring, step-by-step stdout, verbose log file, trace file, archiving, and comprehensive error handling.