Cx Data Pipeline

Data Pipeline Skill

Use this skill when configuring how Coralogix processes, enriches, and transforms data. It covers parsing rules (extract structured fields from raw logs), enrichments (add context from lookup tables), Events2Metrics (derive metrics from log/span events), and recording rules (precompute PromQL expressions).

---

CLI Commands

Key flags:

• All create/update operations use --from-file <path> (or - for stdin)
• All commands support -o json for structured output and -p <profile> for profile selection
• cx parsing-rules update and cx recording-rules update require both --from-file and the rule group ID
• cx enrichments custom search requires --id <table-id> and --query <text>
• cx parsing-rules bulk-delete requires --ids <id1> <id2> ...

---

Working with JSON Payloads

These commands use complex JSON structures. Always template from an existing resource to avoid format errors:

# 1. Get an existing resource as a template
cx parsing-rules get <rule-group-id> -o json > template.json

# 2. Modify the template (change fields, remove the ID for create operations)

# 3. Create or update
cx parsing-rules create --from-file template.json
cx parsing-rules update --from-file template.json <rule-group-id>

This pattern applies to all create/update operations across all 4 commands. It prevents payload format errors that are the #1 cause of failed attempts.

---

Parsing Rules Workflow

1. List Existing Rules

cx parsing-rules list -o json
cx parsing-rules list -o json | jq '[.[] | {id, name, enabled, rule_count: (.rules | length)}]'

2. Get a Template

cx parsing-rules get <existing-rule-group-id> -o json > rule-template.json

3. Create New Rule Group

Edit the template for your new service, then:

cx parsing-rules create --from-file rule-template.json

4. Verify Parsing

Query recent logs to confirm fields are extracted (load cx-telemetry-querying for log querying):

cx logs 'source logs | filter $d.subsystem == "my-service" | limit 10' -o json

5. Check Usage Limits

cx parsing-rules usage-limits -o json

---

Enrichment Workflow

1. List Enrichment Rules

cx enrichments list -o json
cx enrichments settings -o json
cx enrichments limit -o json

2. Create Custom Enrichment Table (if needed)

cx enrichments custom list -o json
cx enrichments custom create --from-file table-definition.json

table-definition.json must use the v5 JSON shape (inline file content, not multipart file=@...):

{
  "name": "IP Lookup",
  "description": "Maps IPs to locations",
  "file": {
    "textual": "ip,city\n1.2.3.4,London",
    "extension": "csv",
    "name": "lookup.csv",
    "size": 24
  }
}

For updates, include customEnrichmentId (number) plus the same fields.

3. Add Enrichment Rules

cx enrichments add --from-file enrichment-rules.json

enrichment-rules.json must use requestEnrichments (not enrichments from list output). Each enrichmentType is an object, not a string:

{
  "requestEnrichments": [
    {
      "fieldName": "sourceIPs",
      "enrichmentType": { "geoIp": { "withAsn": true } }
    }
  ]
}

Other types: {"aws": {"resourceType": "ec2"}}, {"suspiciousIp": {}}, {"customEnrichment": {"id": 1}}.

4. Search Custom Table Data

cx enrichments custom search --id <table-id> --query "search term"

5. Verify Enriched Fields

Query logs on hot storage (FrequentSearch tier) to confirm enriched fields appear. Avoid querying archive for verification - ingestion delays can cause false negatives.

cx logs 'source logs | filter $d.enriched_field != null | limit 5' -o json

---

Events2Metrics Workflow

E2M derives Prometheus metrics from log/span events. See [`references/e2m-schemas.md`](references/e2m-schemas.md) for the full JSON wire format, enums, and cardinality rules.

How E2M is computed (read this first)

E2M aggregates events as they stream through the real-time ingestion pipeline into metric series (~1-min resolution). It is forward-only — metrics start from the moment the E2M is created; there is no backfill.

All ingested data flows through the pipeline; a TCO policy routes each stream into a tier, and the tier decides what's possible:

The axis is tier / processing level — NOT "Frequent Search vs archive" (Medium is archive and E2M works on it). Do not tell users to "point E2M at archive instead of Frequent Search" — that is incorrect.

1. Design the metric

Choose logs2metrics vs spans2metrics, the source field(s) + aggregations, and labels (with cardinality in mind — see references/e2m-schemas.md). To scope the E2M to a dataset, set the optional dataSource field to "<dataspace>/<dataset>"; this requires the account feature e2m_dataset_source_enabled (otherwise the API rejects it with "dataSource is not enabled for this company"). Omit it for the standard logs/spans stream.

2. Size it: check limits & cardinality

cx e2m limits -o json              # account E2M count limit + used
cx e2m labels-cardinality -o json  # see caveat below

The labels-cardinality endpoint is a draft forecast — given proposed labels + query it returns the per-day distinct-permutation count over the last 7 days, so you can size a design before creating it. But `cx e2m labels-cardinality` currently takes no arguments, so it sends no draft and returns an empty list (a CLI gap — it can't forecast yet). Until that's wired up, forecast via the UI or estimate permutations manually (product of distinct label values) and set permutationsLimit. Never use high-cardinality fields (IDs, raw URLs, IPs) as labels. Note the forecast only sees Frequent-Search (High-tier) data.

3. Template from an existing definition

Only cx e2m get returns the full payload ({"e2m": {...}}); list prints a summary. Extract .e2m and drop read-only fields:

cx e2m get <existing-e2m-id> -o json | jq '.e2m | del(.id, .permutations, .createTime, .updateTime, .metricName)' > e2m.json

4. Create the E2M

cx e2m create --from-file e2m.json

5. Verify the metric

Confirm series are being produced (load cx-telemetry-querying for metrics querying):

cx metrics search --name "<targetBaseMetricName>"
cx metrics query "<target_metric_name>" --time now

Troubleshooting: E2M produces no metric series

1. Check the source data's TCO tier — if it's routed to Low/compliance (or blocked), E2M cannot run. Fix with a TCO change (cx tco list / cx-cost-optimization), not an E2M change. 2. Verify the query matches streaming data — run the E2M's lucene filter as a live cx logs/cx spans query and confirm it returns recent results. Note cx logs queries Frequent-Search (High-tier) by default; for a Medium-tier (archive) source add --tier archive, since the data won't appear in a default Frequent-Search query even though E2M still produces series. 3. Remember it's forward-only — no series exist for data ingested before the E2M was created.

Cost optimization: convert High-tier logs to metrics

When the aggregated/metric view is what the customer most cares about, convert High-tier logs → metrics, then downgrade the raw logs High → Medium. Medium still supports E2M/alerts/dashboards and costs less (S3 archive, no hot storage) — you keep cheap, detailed metrics while dropping expensive Frequent-Search retention.

1. Find high-volume High-tier sources: cx usage summary / cx tco list (see cx-cost-optimization). 2. Confirm which fields drive dashboards/alerts (see cx-telemetry-querying). 3. Build + verify the E2M first (steps above). 4. Then change the TCO policy to move the raw logs High → Medium. Keep data on High or Medium (both support E2M); do not drop it to Low/compliance if metrics or alerts are still needed.

---

Recording Rules Workflow

1. List Existing Recording Rules

cx recording-rules list -o json
cx recording-rules list -o json | jq '[.[] | {id, name, rules: [.rules[]?.record]}]'

2. Get a Template

cx recording-rules get <existing-id> -o json > recording-rule-template.json

3. Create Recording Rule Group

cx recording-rules create --from-file recording-rule-group.json

4. Verify with PromQL

Confirm the precomputed metric is available (load cx-telemetry-querying for metrics querying):

cx metrics query "new_precomputed_metric" --time now

---

Key Principles

• Always template from existing - cx <command> get <id> -o json > template.json before any create
• Verify after create - query logs/metrics to confirm the pipeline change took effect
• Use `-o json` - all payload inspection and creation should use JSON output
• Check limits first - cx parsing-rules usage-limits and cx e2m limits before creating to avoid hitting caps
• Bulk operations - use cx parsing-rules bulk-delete --ids for cleanup, not individual deletes

---

Additional Resources

Reference Files

• [`references/e2m-schemas.md`](references/e2m-schemas.md) - Complete Events2Metrics JSON wire format: type/aggType enum values, logsQuery/spansQuery filters, metric labels & fields, the TCO-tier compute model, cardinality/permutations sizing, and gotchas

---

Related Skills

• `cx-telemetry-querying` - discover what data is available before configuring pipeline, and verify parsing results, enriched fields, and E2M metric series via log/metrics queries
• `cx-cost-optimization` - find high-volume High-tier sources worth converting to metrics, and move the raw logs High→Medium (TCO) after the E2M is verified

Events2Metrics (E2M) Schema Reference

Complete JSON schema reference for Coralogix Events2Metrics definitions, using the actual REST API wire format that cx e2m reads and writes. Use this when constructing payloads for cx e2m create / cx e2m update.

Tip: The most reliable way to build an E2M is to fetch an existing one with cx e2m get <id> -o json, strip the read-only fields, modify, and pipe it into cx e2m create --from-file -. Note that cx e2m list and cx e2m create print a simplified view (id/name/type/metric_name) — only `cx e2m get` returns the full definition you can template from.

How E2M is computed (read this first)

E2M is not a stored query — it aggregates events as they stream through the real-time ingestion pipeline into metric series (~1-minute resolution). It is forward-only: metrics start accumulating from the moment the E2M is created; there is no backfill.

• All ingested data flows through the ingestion pipeline; a TCO policy routes each stream into a tier.
• E2M works on data in High tier (Frequent Search / hot storage) and Medium tier (lands in S3 archive, but still processed by the pipeline).
• E2M does NOT work on Low / compliance tier (compliance storage only, no aggregation features) or blocked data.
• The axis is tier / processing level — not "Frequent Search vs archive" (Medium is archive and E2M works on it).

If an E2M produces zero metric series, the usual cause is that its source data is routed to Low/compliance (or the query matches nothing) — the fix is a TCO change, not an E2M change. See the SKILL workflow's troubleshooting section.

---

Source types

The type field + query oneof select what events are aggregated:

E2M_TYPE_UNSPECIFIED (0) is never used in a real definition.

Datasets (`dataSource`): an optional top-level dataSource string in "<dataspace>/<dataset>" format (e.g. "my_dataspace/my_dataset") scopes the E2M to a specific dataset instead of the standard logs/spans stream. It is supported via the API/CLI — add it to the create body. If omitted, the E2M defaults to the standard logs/spans stream for the chosen type.

Requires an account feature flag. dataSource is gated behind e2m_dataset_source_enabled per company. If the feature isn't enabled, the API rejects the definition with "dataSource is not enabled for this company". If you hit that error, the account needs the feature turned on — it's not a payload problem.

{ "type": "E2M_TYPE_LOGS2METRICS", "dataSource": "my_dataspace/my_dataset", "logsQuery": { ... } }

---

Top-level structure

cx e2m create and cx e2m update post the bare definition object (the HTTP body maps to the e2m field). A cx e2m get response wraps the same object as {"e2m": { ... }}, so when templating, extract .e2m.

Creating a new E2M — drop id (and other read-only fields); the server assigns a fresh id:

cx e2m get <id> -o json | jq '.e2m | del(.id, .permutations, .createTime, .updateTime, .metricName)' > e2m.json
cx e2m create --from-file e2m.json

Updating an existing E2M — cx e2m update is a PUT with no id in the path, so the body must keep `id` (it's the only identifier the API uses). Drop only the server-derived fields:

cx e2m get <id> -o json | jq '.e2m | del(.permutations, .createTime, .updateTime, .metricName)' > e2m.json
# edit e2m.json (keep "id"), then:
cx e2m update --from-file e2m.json

Create payload (E2MCreateParams)

{
  "name": "service_catalog_latency",
  "description": "avg + sum latency for catalog service",
  "permutationsLimit": 30000,
  "type": "E2M_TYPE_LOGS2METRICS",
  "logsQuery": {
    "lucene": "coralogix.metadata.applicationName:catalog",
    "applicationnameFilters": [],
    "subsystemnameFilters": [],
    "severityFilters": ["SEVERITY_ERROR", "SEVERITY_CRITICAL"]
  },
  "metricLabels": [
    { "targetLabel": "app",       "sourceField": "coralogix.metadata.applicationName" },
    { "targetLabel": "subsystem", "sourceField": "coralogix.metadata.subsystemName" }
  ],
  "metricFields": [
    {
      "targetBaseMetricName": "latency_ms",
      "sourceField": "log_obj.latency_ms",
      "aggregations": [
        { "enabled": true, "aggType": "AGG_TYPE_COUNT", "targetMetricName": "latency_ms_count" },
        { "enabled": true, "aggType": "AGG_TYPE_AVG",   "targetMetricName": "latency_ms_avg" },
        { "enabled": true, "aggType": "AGG_TYPE_SUM",   "targetMetricName": "latency_ms_sum" }
      ]
    }
  ]
}

Field reference

`id` — UUID. Omit on `create` (server assigns it); required in the body on `update` (the PUT has no path id). Present in get responses.

Server-derived fields (present in get responses; drop from create/update bodies): permutations ({limit, hasExceededLimit}), createTime, updateTime, metricName, isInternal.

logsQuery

Severity enum values: SEVERITY_DEBUG, SEVERITY_VERBOSE, SEVERITY_INFO, SEVERITY_WARNING, SEVERITY_ERROR, SEVERITY_CRITICAL.

spansQuery

metricLabels[]

{ "targetLabel": "app", "sourceField": "coralogix.metadata.applicationName" }

• targetLabel — the output Prometheus label name. Pattern ^[\w/-]+$.
• sourceField — the event field to read the value from (e.g. coralogix.metadata.applicationName, log_obj.region).

metricFields[] and aggregations[]

{
  "targetBaseMetricName": "latency_ms",
  "sourceField": "log_obj.latency_ms",
  "aggregations": [
    { "enabled": true, "aggType": "AGG_TYPE_HISTOGRAM", "targetMetricName": "latency_ms_histogram",
      "histogram": { "buckets": [10, 50, 100, 500, 1000] } },
    { "enabled": true, "aggType": "AGG_TYPE_SAMPLES",   "targetMetricName": "latency_ms_max",
      "samples": { "sampleType": "SAMPLE_TYPE_MAX" } }
  ]
}

• targetBaseMetricName — base name; each aggregation gets its own targetMetricName. Pattern ^[\w/-]+$.
• sourceField — numeric event field to aggregate. (For AGG_TYPE_COUNT the value is irrelevant — it counts matching events.)

`aggType` values:

Each aggregation: enabled (bool), aggType (string enum), targetMetricName (string), plus the agg_metadata oneof (histogram or samples) only for those two types.

---

Cardinality & Limits

Permutations = the product of the number of distinct values across all `metricLabels`. A label on app (20 values) × subsystem (15) × status_code (40) = 12,000 permutations. Add one high-cardinality label and it explodes.

• Never use high-cardinality fields as labels: user IDs, request/trace IDs, session IDs, raw URLs, full paths, IPs, timestamps. These multiply permutations without bound and will breach the limit.
• Keep labels to bounded, low-cardinality dimensions (app, subsystem, region, status class, severity, endpoint group).
• permutationsLimit caps the design; if exceeded, the read-back permutations.hasExceededLimit is true and new permutations stop being produced.

Checking limits and forecasting cardinality:

cx e2m limits -o json              # account E2M count limit + used
cx e2m labels-cardinality -o json  # see caveat below

• cx e2m limits reports the E2M count limit and how many are used (the underlying API also tracks labels/permutations/metrics caps).
• The labels-cardinality endpoint is a draft forecast tool. Given a proposed query + metricLabels, the backend runs a cardinality aggregation over the last 7 days of the company's logs/spans and returns the distinct label-permutation count per day for exactly that draft design — so you can size a design before creating it. With no labels supplied it returns an empty result.
• CLI limitation: cx e2m labels-cardinality currently takes no arguments, so it sends no draft and returns an empty list — it does not expose the forecast yet (it would need --metric-labels/--query). Until then, forecast a draft via the Coralogix UI (the create flow calls this endpoint with your labels+query), or estimate manually (below) as a rough check.
• The forecast only sees Frequent-Search (High-tier) data — it queries the hot *_newlogs* index. For a Medium-tier (archive) source it will undercount, so treat manual estimates as the floor.

Manual estimate (rough fallback) — sizing a 2-label design:

• Labels: app (~12 distinct), severity (6 distinct).
• Estimated permutations = 12 × 6 = 72 — comfortably under any limit.
• Adding endpoint (~300 distinct) → 12 × 6 × 300 = 21,600. Closer to limits; prefer grouping endpoints into a bounded route field, or drop the label and filter in lucene instead.

---

Gotchas

• Enums are strings, not integers. Use "type": "E2M_TYPE_LOGS2METRICS" and "aggType": "AGG_TYPE_AVG" — not 1 / 4. (Some internal/legacy exports use integers and a $case discriminator; the cx CLI uses the OpenAPI string form.)
• Template from `get`, not `list`. cx e2m list / create print a simplified summary; only cx e2m get <id> -o json returns the full definition ({"e2m": {...}}).
• Create/update body is the bare object, not wrapped in {"e2m": ...}. Extract .e2m from a get response first.
• `update` (replace) requires `id` in the body — the PUT has no path id, so dropping id makes the update fail to find the rule. create must omit id.
• Simple aggregations (MIN/MAX/COUNT/AVG/SUM) need no `agg_metadata` — only histogram and samples carry metadata. (The proto models the empty case as a none marker; in JSON, just omit the metadata.)
• Max 10 metric fields per E2M.
• `targetMetricName` must be unique within the definition; collisions silently clobber series.
• Forward-only. Creating an E2M never backfills historical data — only events ingested after creation produce series.
• Verify with metrics, not logs: cx metrics search --name "<targetBaseMetricName>" then a PromQL query.