
Opentelemetry Collector
- 21 installs
- 4 repo stars
- Updated June 9, 2026
- coralogix/cx-skills
Helps with ai & agent building tasks.
About
opentelemetry-collector is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted development.
- opentelemetry-collector
- AI & Agent Building
- AI-coding skill
Opentelemetry Collector by the numbers
- 21 all-time installs (skills.sh)
- Ranked #10,289 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Jul 29, 2026 (Skillselion catalog sync)
npx skills add https://github.com/coralogix/cx-skills --skill opentelemetry-collectorAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 21 |
|---|---|
| repo stars | ★ 4 |
| Last updated | June 9, 2026 |
| Repository | coralogix/cx-skills ↗ |
What it does
Helps with ai & agent building tasks.
Files
coralogix-exporter-config
{
"context": "Evaluating a Coralogix support response for this user question:\n\nMy coralogix exporter config has endpoint: https://ingress.eu2.coralogix.com and I've set private_key: $CORALOGIX_PRIVATE_KEY but nothing is being sent to Coralogix. What's wrong?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-domain",
"description": "The response contains \"domain\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-eu2-coralogix-com",
"description": "The response contains \"eu2.coralogix.com\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "env-coralogix-private-key",
"description": "The response matches the pattern: \\$\\{env:CORALOGIX_PRIVATE_KEY\\}|\\$\\{env:",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-recommends-swapping-domain-to-a-coralogix",
"description": "Identifies that the coralogix exporter uses `domain:` (a bare hostname) rather than `endpoint:` with a URL, AND that the private_key must use the bracketed `${env:CORALOGIX_PRIVATE_KEY}` form. A response that recommends swapping `domain:` to a Coralogix UI hostname (`team.coralogix.com`, `coralogix.in`, `coralogixsg.com`, etc.) is a FAIL \u2014 the UI host is not the data-ingestion host.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
coralogix-exporter-domain-env-var-syntax
You are a Coralogix support expert. A user has asked the following question:
---
My coralogix exporter config has endpoint: https://ingress.eu2.coralogix.com and I've set private_key: $CORALOGIX_PRIVATE_KEY but nothing is being sent to Coralogix. What's wrong?
---
coralogix-exporter-config
{
"context": "Evaluating a Coralogix support response for this user question:\n\nGive me a minimum coralogix exporter block using the standard send-your-data env var. Respond with just the YAML, in a ```yaml fenced block.",
"type": "weighted_checklist",
"checklist": [
{
"name": "yaml-domain-bare-hostname",
"description": "The response matches the pattern: (?ms)```yaml.*domain:\\s*[\"']?[a-z0-9.<>_-]*coralogix\\.[a-z]+[\"']?.*```",
"max_score": 3,
"category": "INTENT"
},
{
"name": "env-coralogix-private-key",
"description": "The response matches the pattern: \\$\\{env:CORALOGIX_PRIVATE_KEY\\}",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-endpoint-field-in-yaml",
"description": "The response does not match the pattern: (?ms)```yaml.*endpoint:.*```",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
coralogix-exporter-minimum-yaml-output
You are a Coralogix support expert. A user has asked the following question:
---
Give me a minimum coralogix exporter block using the standard send-your-data env var. Respond with just the YAML, in a ```yaml fenced block.
---
helm-chart-selection
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe're deploying the coralogix/opentelemetry-coralogix Helm chart (the otel-agent chart) for a new EKS cluster. Any reason not to use it?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-deprecated",
"description": "The response contains \"deprecated\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-otel-integration",
"description": "The response contains \"otel-integration\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-reassures-the-user-the-deprecated-chart-is",
"description": "Says clearly that `opentelemetry-coralogix` / `otel-agent` is deprecated and that new deployments should use the `otel-integration` chart. A response that reassures the user the deprecated chart is fine, still supported, safe to use, or a good choice for a new deployment is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
deprecated-otel-agent-chart-redirect
You are a Coralogix support expert. A user has asked the following question:
---
We're deploying the coralogix/opentelemetry-coralogix Helm chart (the otel-agent chart) for a new EKS cluster. Any reason not to use it?
---
ecs-ec2-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nOn our ECS EC2 Linux daemonset, every log row is being attributed to the OTel collector's own container instead of the source app container. The config has resourcedetection with detectors including \"ecs\". What should we do?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-ecs",
"description": "The response contains \"ecs\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "remove-disable-don",
"description": "The response matches the pattern: (?i)(remove|disable|don't include|omit).*ecs|ecs.*(remove|detector)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-keeps-the-ecs-detector-enabled-on-the-daem",
"description": "Recommends removing the `ecs` detector from `resourcedetection.detectors` on the daemonset, because it stamps the collector's own container ID onto every log record. Bonus credit for pointing at the `ecsattributes/container-logs` CDOT processor for proper per-container attribution. A response that keeps the `ecs` detector enabled on the daemonset is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
ecs-ec2-daemonset-ecs-detector-remove
You are a Coralogix support expert. A user has asked the following question:
---
On our ECS EC2 Linux daemonset, every log row is being attributed to the OTel collector's own container instead of the source app container. The config has resourcedetection with detectors including "ecs". What should we do?
---
ecs-ec2-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe've deployed the ECS EC2 CDOT daemonset. Our applications in other tasks are trying to send OTLP to localhost:4317 but nothing is being received. What are we missing?",
"type": "weighted_checklist",
"checklist": [
{
"name": "node-ip-host",
"description": "The response matches the pattern: (?i)(node[ -]ip|host[ -]ip|instance[ -]ip|169\\.254\\.169\\.254|ec2 metadata|private ip)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "host-network-hostnetwork",
"description": "The response matches the pattern: (?i)(host[ -]?network|hostNetwork|not localhost|network namespace|different container)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-that-ecs-tasks-each-get-their-own-ne",
"description": "Explains that ECS tasks each get their own network namespace, so an app in one task cannot reach the daemonset over `localhost` \u2014 apps must target the EC2 host IP (obtainable from the instance metadata service at 169.254.169.254, or the task-level environment), and the CDOT daemonset must run with `networkMode: host` / `hostNetwork`.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
ecs-ec2-daemonset-localhost-network
You are a Coralogix support expert. A user has asked the following question:
---
We've deployed the ECS EC2 CDOT daemonset. Our applications in other tasks are trying to send OTLP to localhost:4317 but nothing is being received. What are we missing?
---
ecs-fargate-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nOur default ECS Fargate sidecar has detectors [env, system] in its resourcedetection config \u2014 we copied this from our ECS EC2 daemonset. Should we add the ecs detector too or leave it out?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-ecs",
"description": "The response contains \"ecs\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "add-enable-include",
"description": "The response matches the pattern: (?i)(add|enable|include|is correct|works|should)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "sidecar-fargate-shares",
"description": "The response matches the pattern: (?i)(sidecar|fargate|shares? (the )?task|awsvpc)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "says-the-ecs-detector-should-be-enabled-on-a",
"description": "Says the `ecs` detector SHOULD be enabled on a Fargate sidecar \u2014 unlike the EC2 daemonset, the sidecar shares the task's awsvpc network namespace and the detector correctly attributes data to the right task, not to the collector. The \"remove ecs detector\" rule from ECS-EC2 daemonset mode is EC2-specific and inapplicable to Fargate sidecar.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
ecs-fargate-sidecar-ecs-detector-enable
You are a Coralogix support expert. A user has asked the following question:
---
Our default ECS Fargate sidecar has detectors [env, system] in its resourcedetection config — we copied this from our ECS EC2 daemonset. Should we add the ecs detector too or leave it out?
---
ecsattributes-windows-fallback
{
"context": "Evaluating a Coralogix support response for this user question:\n\nOur ECS EC2 Windows daemon collector is on an older CDOT build and cannot reach the Docker engine, but we still need the host-related resource attributes for Infra Explorer. What should we do?",
"type": "weighted_checklist",
"checklist": [
{
"name": "windows-ecsattributes-fallback",
"description": "The response matches the pattern: (?i)(Windows|Docker|ecsattributes|task metadata|Infra Explorer)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "upgrade-v0-5-11-or-newer",
"description": "The response matches the pattern: (?i)(v0[.]5[.]11|0[.]5[.]11|upgrade|newer|current CDOT)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-fallback-to-ecs-task-metadata",
"description": "Explains that `ecsattributesprocessor` falls back to ECS task metadata when Docker is unavailable on Windows, so host-related resource attributes can still populate for Infra Explorer. Mentions that older builds need an upgrade rather than claiming the Windows path cannot work without Docker.",
"max_score": 2,
"category": "RUBRIC"
},
{
"name": "no-claims-windows-needs-docker",
"description": "A response that says Windows ECS must have Docker, that host metadata cannot be recovered on Windows, or that the user should just remove the ECS attributes processor is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
ecs-windows-ecsattributes-fallback
You are a Coralogix support expert. A user has asked the following question:
---
Our ECS EC2 Windows daemon collector is on an older CDOT build and cannot reach the Docker engine, but we still need the host-related resource attributes for Infra Explorer. What should we do?
---
eks-fargate-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe deployed the otel-eks-fargate integration from a copy of the manifest we saved about a year ago. Pod metrics and related-data visualizations in APM are empty even though the collector is running. What should we check before debugging the collector?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-k8sattributes",
"description": "The response contains \"k8sattributes\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "re-pull-latest",
"description": "The response matches the pattern: (?i)(re-?pull|latest|newer|current|update|upstream)",
"max_score": 3,
"category": "INTENT"
}
]
}
eks-fargate-stale-manifest-update
You are a Coralogix support expert. A user has asked the following question:
---
We deployed the otel-eks-fargate integration from a copy of the manifest we saved about a year ago. Pod metrics and related-data visualizations in APM are empty even though the collector is running. What should we check before debugging the collector?
---
coralogix-exporter-config
{
"context": "Evaluating a Coralogix support response for this user question:\n\nexporters:\n coralogix:\n domain: \"$CORALOGIX_DOMAIN\"\n private_key: \"$PRIVATE_KEY\"\n\nThis was working on otel-contrib v0.70 but since upgrading to v0.95 nothing exports. What changed?",
"type": "weighted_checklist",
"checklist": [
{
"name": "env-z",
"description": "The response matches the pattern: \\$\\{env:|\\$\\{[A-Z_]+\\}",
"max_score": 3,
"category": "INTENT"
},
{
"name": "bracket-brace-curly",
"description": "The response matches the pattern: (?i)(bracket|brace|curly|\\$\\{)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-that-recent-collector-coralogix-expo",
"description": "Explains that recent collector / coralogix-exporter versions (\u2265 v0.76) stopped expanding the unbracketed `$VAR` form silently \u2014 the bracketed `${env:VAR}` form is now required. Points at both `domain:` and `private_key:` needing the fix.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
env-var-bracket-syntax-upgrade
You are a Coralogix support expert. A user has asked the following question:
---
exporters: coralogix: domain: "$CORALOGIX_DOMAIN" private_key: "$PRIVATE_KEY"
This was working on otel-contrib v0.70 but since upgrading to v0.95 nothing exports. What changed?
---
coralogix-exporter-config
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe set up the Coralogix Linux host integration with a filelog receiver. To route logs from different files to different Coralogix subsystems, we added an operator that sets cx.application.name and cx.subsystem.name as resource attributes on each log record. The operators are definitely running \u2014 we can see the attributes in the debug exporter output. But all logs arrive in Coralogix under the global application_name and subsystem_name values hardcoded in the exporter block. The per-file attributes are completely ignored. What's wrong?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-application-name-attributes",
"description": "The response contains \"application_name_attributes\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-subsystem-name-attributes",
"description": "The response contains \"subsystem_name_attributes\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-says-to-rename-the-operator-output-keys-or",
"description": "Explains that the coralogix exporter only does dynamic resource-attribute routing when application_name_attributes and subsystem_name_attributes are explicitly listed in the exporter config \u2014 and that a hardcoded application_name value takes precedence over the attribute lookup unless the attributes list is configured. The fix is to add application_name_attributes: [cx.application.name] (and the subsystem equivalent) to the exporter block. A response that says to rename the operator output keys, or that focuses only on \"set the resource attribute\" without mentioning these two exporter fields, is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
filelog-resource-attr-cx-routing-ignored
You are a Coralogix support expert. A user has asked the following question:
---
We set up the Coralogix Linux host integration with a filelog receiver. To route logs from different files to different Coralogix subsystems, we added an operator that sets cx.application.name and cx.subsystem.name as resource attributes on each log record. The operators are definitely running — we can see the attributes in the debug exporter output. But all logs arrive in Coralogix under the global application_name and subsystem_name values hardcoded in the exporter block. The per-file attributes are completely ignored. What's wrong?
---
fleet-management
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe're running otel-integration with the Fleet Management supervisor enabled. I helm-upgraded with new processor settings in values.yaml and my changes don't seem to have taken effect. What am I missing?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-supervisor",
"description": "The response contains \"supervisor\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "override-ing-overriden",
"description": "The response matches the pattern: (?i)(override(s|ing|n)?|overriden|precedence|beats|wins|takes over|takes precedence|UI (config|settings)|fleet (manager|management).*config|remote[- ]?config|Fleet Manager)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-that-with-presets-fleetmanagement-su",
"description": "Explains that with `presets.fleetManagement.supervisor.enabled: true` the supervisor owns the runtime config \u2014 Fleet Manager UI settings take precedence over values.yaml, and helm-upgrading the chart has no effect on processor config. Recommends either editing the config in the Fleet Manager UI or disabling the supervisor to let Helm values take effect again.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
fleet-manager-supervisor-precedence
You are a Coralogix support expert. A user has asked the following question:
---
We're running otel-integration with the Fleet Management supervisor enabled. I helm-upgraded with new processor settings in values.yaml and my changes don't seem to have taken effect. What am I missing?
---
spanmetrics-placement
{
"context": "Evaluating a Coralogix support response for this user question:\n\nIn our otel-integration setup with a central gateway, APM shows traces but no error rates and no p99 latencies. The agent's traces pipeline is: [memory_limiter, k8sattributes, resourcedetection, batch] \u2192 loadbalancing to gateway. The gateway's traces pipeline is: [memory_limiter, k8sattributes, spanmetrics, tail_sampling, batch]. Walk me through what to change and why.",
"type": "weighted_checklist",
"checklist": [
{
"name": "no-recommend-running-spanmetrics-on-both-agen",
"description": "The response MUST meet every one of these points to be CORRECT: 1. Identifies that `spanmetrics` on the gateway only sees sampled\n traces \u2014 so APM metrics undercount errors and latencies.\n2. Recommends moving `spanmetrics` to the AGENT pipeline, upstream of\n any sampling (either as a connector on the agent, or emitting into\n a dedicated spanmetrics metrics pipeline exported direct to\n Coralogix from the agent).\n3. Mentions that the `transactions` / `groupbytrace/transactions`\n processor must run BEFORE `spanmetrics` so the cgx.transaction\n dimensions are populated on the metrics.\n4. Does NOT recommend increasing the sample rate as a workaround. 5. Does NOT recommend running spanmetrics on both agent AND gateway\n (that causes double-counting).",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
gateway-spanmetrics-tail-sampling-walkthrough
You are a Coralogix support expert. A user has asked the following question:
---
In our otel-integration setup with a central gateway, APM shows traces but no error rates and no p99 latencies. The agent's traces pipeline is: [memory_limiter, k8sattributes, resourcedetection, batch] → loadbalancing to gateway. The gateway's traces pipeline is: [memory_limiter, k8sattributes, spanmetrics, tail_sampling, batch]. Walk me through what to change and why.
---
gke-autopilot-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nHelm install of otel-integration fails with \"admission webhook warden-validating.common-webhooks.networking.gke.io denied the request\" and \"denied by autogke-no-write-mode-hostpath\". We're on GKE Autopilot. How do we deploy?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-autopilot",
"description": "The response contains \"autopilot\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "storecheckpoints-hostpath-write",
"description": "The response matches the pattern: (?i)(storeCheckpoints|hostPath|write mode)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-says-the-cluster-is-misconfigured-or-that",
"description": "Explains that GKE Autopilot's Warden admission webhook blocks writable hostPath mounts, and points the user at the Autopilot-specific values file or at disabling the offending writable hostPath option (e.g. `storeCheckpoints`). A response that says the cluster is misconfigured or that the user should switch away from Autopilot is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
gke-autopilot-warden-hostpath
You are a Coralogix support expert. A user has asked the following question:
---
Helm install of otel-integration fails with "admission webhook warden-validating.common-webhooks.networking.gke.io denied the request" and "denied by autogke-no-write-mode-hostpath". We're on GKE Autopilot. How do we deploy?
---
coralogix-exporter-config
{
"context": "Evaluating a Coralogix support response for this user question:\n\nAfter upgrading opentelemetry-collector-contrib from 0.140 to 0.144, our collector fails to start with: \"Error: exporters::coralogix: profiles signal is not supported with HTTP protocol, use gRPC protocol (default) instead\". We have protocol: http set in our coralogix exporter config, but we are not sending profiles \u2014 only logs, metrics, and traces. The same config worked fine on 0.140. What changed and how do we fix it?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-profiles",
"description": "The response contains \"profiles\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "grpc-remove-protocol",
"description": "The response matches the pattern: (?i)(grpc|remove.*protocol|protocol.*grpc|drop.*http)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-says-to-add-a-profiles-pipeline-or-that-sa",
"description": "Explains that v0.144 added profiles support to the coralogix exporter, and the exporter now validates HTTP compatibility for ALL signals at startup \u2014 not just signals in active pipelines \u2014 so protocol: http fails even with no profiles pipeline. The fix is to remove protocol: http (gRPC is the default and the only supported transport for profiles). A response that says to add a profiles pipeline, or that says to downgrade the collector version, is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
http-protocol-profiles-signal-upgrade
You are a Coralogix support expert. A user has asked the following question:
---
After upgrading opentelemetry-collector-contrib from 0.140 to 0.144, our collector fails to start with: "Error: exporters::coralogix: profiles signal is not supported with HTTP protocol, use gRPC protocol (default) instead". We have protocol: http set in our coralogix exporter config, but we are not sending profiles — only logs, metrics, and traces. The same config worked fine on 0.140. What changed and how do we fix it?
---
infra-explorer-setup
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe want Infrastructure Explorer to populate for our K8s cluster. We've enabled kubernetesResources preset and we see logs flowing through the standard coralogix exporter, but the Resource Catalog stays empty. What pieces are we missing?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-x-coralogix-ingress",
"description": "The response contains \"x-coralogix-ingress\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-metadata-as-otlp-logs-v1",
"description": "The response contains \"metadata-as-otlp-logs/v1\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "coralogix-resource-catalog-separate",
"description": "The response matches the pattern: (?i)(coralogix/resource_catalog|separate exporter|dedicated exporter)",
"max_score": 3,
"category": "INTENT"
}
]
}
infra-explorer-resource-catalog-exporter
You are a Coralogix support expert. A user has asked the following question:
---
We want Infrastructure Explorer to populate for our K8s cluster. We've enabled kubernetesResources preset and we see logs flowing through the standard coralogix exporter, but the Resource Catalog stays empty. What pieces are we missing?
---
linux-standalone-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe deployed the Coralogix Linux host integration through the UI wizard. Logs and metrics are flowing but in APM our Node.js and Python services don't have language icons like we see in other environments. What's happening?",
"type": "weighted_checklist",
"checklist": [
{
"name": "telemetry-sdk",
"description": "The response matches the pattern: telemetry\\.sdk",
"max_score": 3,
"category": "INTENT"
},
{
"name": "transform-stripped-missing",
"description": "The response matches the pattern: (?i)(transform|stripped|missing|removed)",
"max_score": 3,
"category": "INTENT"
}
]
}
linux-wizard-telemetry-sdk-stripped
You are a Coralogix support expert. A user has asked the following question:
---
We deployed the Coralogix Linux host integration through the UI wizard. Logs and metrics are flowing but in APM our Node.js and Python services don't have language icons like we see in other environments. What's happening?
---
scope-deflection
{
"context": "Evaluating a Coralogix support response for this user question:\n\nNo traces are reaching Coralogix. I've added a transform processor with set(attributes[\"forwarded\"], true) to make sure at least something goes through, but it hasn't changed anything. Should I try a different OTTL statement?",
"type": "weighted_checklist",
"checklist": [
{
"name": "receiver-exporter-connectivity",
"description": "The response matches the pattern: (?i)(receiver|exporter|connectivity|DNS|domain|egress|endpoint|private_key|credentials|network)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-suggests-another-ottl-statement-as-the-fix",
"description": "Says clearly that \"no data reaching Coralogix\" is almost never an OTTL problem \u2014 it's upstream (receiver/exporter connectivity, DNS, TLS, proxy, private key, region/domain). Must tell the user to stop trying transform/OTTL statements and check the exporter and network path instead. A response that suggests another OTTL statement as the fix is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
no-traces-not-ottl-deflection
You are a Coralogix support expert. A user has asked the following question:
---
No traces are reaching Coralogix. I've added a transform processor with set(attributes["forwarded"], true) to make sure at least something goes through, but it hasn't changed anything. Should I try a different OTTL statement?
---
opamp-fleet-management
{
"context": "Evaluating a Coralogix support response for this user question:\n\nI'm configuring the OpAMP supervisor for Fleet Management. Our region is eu2. My coralogix exporter is using domain: eu2.coralogix.com but the supervisor keeps getting connection refused. What should the supervisor endpoint be?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-ingress",
"description": "The response contains \"ingress\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "https-ingress-eu2-coralogix-com",
"description": "The response matches the pattern: https://ingress\\.eu2\\.coralogix\\.com",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-opamp",
"description": "The response contains \"opamp\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-that-the-opamp-supervisor-uses-a-dif",
"description": "Explains that the OpAMP supervisor uses a DIFFERENT endpoint from the coralogix data exporter \u2014 it needs the full URL with scheme, `https://ingress.<region>.coralogix.com`, not the bare hostname that goes in the exporter's `domain:` field.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
opamp-supervisor-endpoint-format
You are a Coralogix support expert. A user has asked the following question:
---
I'm configuring the OpAMP supervisor for Fleet Management. Our region is eu2. My coralogix exporter is using domain: eu2.coralogix.com but the supervisor keeps getting connection refused. What should the supervisor endpoint be?
---
pii-redaction
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe do not have `http.route` on spans, so APM operations are based on raw URL-like span names and `url.full`. They include account IDs, merchant IDs, query params, and sometimes tokens. We need to reduce spanmetrics cardinality and avoid sending PII/secrets to Coralogix. Should we use the redaction processor, OTTL, or both, and where should this run?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-redactionprocessor",
"description": "The response matches the pattern: (?i)(redactionprocessor|redaction processor)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-url-sanitizer",
"description": "The response contains \"url_sanitizer\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-sanitize-span-name",
"description": "The response contains \"sanitize_span_name\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "requires-allow-all-keys-or-allowed-keys",
"description": "If the response recommends `redactionprocessor` URL sanitization, it requires `allow_all_keys: true` or a deliberate `allowed_keys` list. A response that proposes `url_sanitizer` without either setting is a FAIL because redactionprocessor is fail-closed and drops unspecified attributes.",
"max_score": 2,
"category": "MUST_NOT"
},
{
"name": "mentions-transform-or-ottl",
"description": "The response matches the pattern: (?i)(OTTL|transform|replace_pattern|replace_all_patterns|SHA256|delete_key)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "run-before-spanmetrics-and-export",
"description": "Says redaction/sanitization must run before `spanmetrics` and before export to Coralogix so high-cardinality or sensitive values are not converted into metrics or shipped raw.",
"max_score": 2,
"category": "RUBRIC"
},
{
"name": "warns-about-over-sanitization",
"description": "Warns that broad URL sanitizers should be validated with representative before/after examples because they can over-sanitize useful path or domain segments; targeted OTTL is better when exact fields or patterns must be preserved.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
redactionprocessor-url-sanitizer
You are a Coralogix support expert. A user has asked the following question:
---
We do not have http.route on spans, so APM operations are based on raw URL-like span names and url.full. They include account IDs, merchant IDs, query params, and sometimes tokens. We need to reduce spanmetrics cardinality and avoid sending PII/secrets to Coralogix. Should we use the redaction processor, OTTL, or both, and where should this run?
---
k8s-topology
{
"context": "Evaluating a Coralogix support response for this user question:\n\nCollector fails to start with: \"cannot start pipelines: failed to start resourcedetection/resource_catalog processor: can't get K8s Instance Metadata; node name is empty\". This is on a node in our daemonset. What's wrong?",
"type": "weighted_checklist",
"checklist": [
{
"name": "cluster-collector-cluster-collector",
"description": "The response matches the pattern: (?i)(cluster[- ]collector|cluster.collector|deployment)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "only-instead-move",
"description": "The response matches the pattern: (?i)(only|instead|move|remove|incorrect|wrong|not (a |on |on the )?daemonset|should (not |run on)|misplac|belongs (on|to))",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-that-resourcedetection-resource-cata",
"description": "Explains that `resourcedetection/resource_catalog` (and Resource Catalog plumbing in general) belongs on the cluster-collector Deployment, not the per-node daemonset \u2014 the daemonset pod can't get node-name / K8s-instance-metadata injection. Recommends moving or removing it from the daemonset configuration.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
resource-catalog-cluster-collector-only
You are a Coralogix support expert. A user has asked the following question:
---
Collector fails to start with: "cannot start pipelines: failed to start resourcedetection/resource_catalog processor: can't get K8s Instance Metadata; node name is empty". This is on a node in our daemonset. What's wrong?
---
ottl-escaping
{
"context": "Evaluating a Coralogix support response for this user question:\n\nOur otel-integration values.yaml has spanNameReplacePattern under opentelemetry-agent.presets.spanMetrics to reduce span_name cardinality:\n\n spanNameReplacePattern:\n - regex: \"cycle-(manager|rpa-manager)\\\\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}_(.+)\"\n replacement: \"cycle-$1.$2\"\n\nhelm upgrade fails with: \"processors::transform/span_name: statement has invalid syntax: 1:28: invalid quoted string ... invalid syntax\". The regex itself works in a standalone regex tester. What's wrong with the values file?",
"type": "weighted_checklist",
"checklist": [
{
"name": "matches-pattern-1",
"description": "The response matches the pattern: \\$\\$1|\\$\\$[0-9]",
"max_score": 3,
"category": "INTENT"
},
{
"name": "single-quote",
"description": "The response matches the pattern: (?i)single[- ]quote",
"max_score": 3,
"category": "INTENT"
},
{
"name": "helm-template",
"description": "The response matches the pattern: (?i)helm template",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-fixes-only-the-yaml-quoting-without-the-es",
"description": "Identifies TWO INDEPENDENT escape layers, not one: (a) YAML parsing \u2014 single-quote the regex string (or use a block scalar) so backslashes aren't consumed by YAML's double-quoted-string parser; and (b) the OpenTelemetry Collector's envprovider \u2014 regex backreferences `$1` / `$2` must be written `$$1` / `$$2` in values.yaml because at collector startup the envprovider expands `$...` references, and `$$` is the literal-`$` escape. Correct attribution matters: Helm's template engine passes `$` through unchanged, so an answer that blames Helm for the `$$` rule is incorrect even if its fix (`$$1`) is right. Recommends running `helm template ... -f values.yaml` before `helm upgrade` to verify the rendered OTTL. A response that fixes only the YAML quoting without the `$$` escape, that blames the regex as syntactically invalid, that attributes the `$$` rule to Helm rather than the collector's envprovider, or that tells the user to rewrite the regex without backreferences, is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
span-name-replace-yaml-envprovider-escape
You are a Coralogix support expert. A user has asked the following question:
---
Our otel-integration values.yaml has spanNameReplacePattern under opentelemetry-agent.presets.spanMetrics to reduce span_name cardinality:
spanNameReplacePattern:
- regex: "cycle-(manager|rpa-manager)\\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}_(.+)"
replacement: "cycle-$1.$2"
helm upgrade fails with: "processors::transform/span_name: statement has invalid syntax: 1:28: invalid quoted string ... invalid syntax". The regex itself works in a standalone regex tester. What's wrong with the values file?
---
spanmetrics-placement
{
"context": "Evaluating a Coralogix support response for this user question:\n\nOur APM dashboards show traces arriving but zero errors and no p99 latencies. The spanmetrics connector is enabled and we see metric series flowing. Our gateway has tail_sampling and spanmetrics both configured on it. What's wrong?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-agent",
"description": "The response contains \"agent\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "before-upstream-of",
"description": "The response matches the pattern: (?i)(before|upstream of|prior to|ahead of).*(sampling|tail)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-recommends-raising-the-sample-rate-samplin",
"description": "Identifies that `spanmetrics` on the gateway only sees the sampled subset of traces (after tail_sampling), so error rates and latency percentiles are computed over a biased sample. Recommends moving `spanmetrics` to the AGENT pipeline, upstream of any sampling. A response that recommends raising the sample rate, sampling 100%, or reworking the sampling policy as the fix is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
spanmetrics-before-tail-sampling-agent
You are a Coralogix support expert. A user has asked the following question:
---
Our APM dashboards show traces arriving but zero errors and no p99 latencies. The spanmetrics connector is enabled and we see metric series flowing. Our gateway has tail_sampling and spanmetrics both configured on it. What's wrong?
---
cardinality-protection
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe are using a custom `spanmetrics` connector instead of the Coralogix Helm preset. Do we automatically get the 100,000 cardinality protection, and what happens when the limit is reached? The team wants to keep `url.full` and `k8s.pod.ip` as dimensions because a dashboard uses them.",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-aggregation-cardinality-limit",
"description": "The response matches the pattern: (aggregation_cardinality_limit|aggregationCardinalityLimit)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-overflow",
"description": "The response matches the pattern: (?i)(overflow|otel_metric_overflow|otel\\.metric\\.overflow)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-url-full",
"description": "The response contains \"url.full\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-k8s-pod-ip",
"description": "The response contains \"k8s.pod.ip\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "custom-connector-needs-explicit-limit",
"description": "Explains that custom `spanmetrics` connectors or older/non-Helm deployments should verify or set the cardinality limit explicitly rather than assuming the Coralogix Helm preset default applies.",
"max_score": 2,
"category": "RUBRIC"
},
{
"name": "limit-is-guardrail-not-label-approval",
"description": "Explains that the limit collapses excess label combinations into an overflow series and protects the pipeline, but it does not make high-cardinality dimensions such as `url.full` or `k8s.pod.ip` good dashboard labels. A response that treats the 100,000 limit as approval to keep those dimensions is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
spanmetrics-cardinality-limit-guardrail
You are a Coralogix support expert. A user has asked the following question:
---
We are using a custom spanmetrics connector instead of the Coralogix Helm preset. Do we automatically get the 100,000 cardinality protection, and what happens when the limit is reached? The team wants to keep url.full and k8s.pod.ip as dimensions because a dashboard uses them.
---
spanmetrics-placement
{
"context": "Evaluating a Coralogix support response for this user question:\n\nI have spanMetrics: enabled: true in our otel-integration Helm values, and the generated ConfigMap shows the spanmetrics connector wired into the traces pipeline. Traces are visible in Coralogix Explore with correct service.name and k8s attributes. But span metrics never appear \u2014 no calls_total, no duration_ms_bucket, and the APM Service Catalog stays empty and shows the onboarding prompt. The otel-integration is deployed with a daemonset agent and a gateway. Where should I look?",
"type": "weighted_checklist",
"checklist": [
{
"name": "agent-daemonset",
"description": "The response matches the pattern: (?i)(agent|daemonset)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-only-rechecks-connector-yaml-or-helm-value",
"description": "Identifies that the spanmetrics connector must run on the agent pipeline (where all traces first arrive), and that if instrumented services bypass the agent and send OTLP directly to the gateway or cluster-collector, spanmetrics never sees those traces and no span metrics are generated. A response that only rechecks connector YAML or Helm values without identifying the agent-placement requirement or the possibility that the app is not routing through the agent is a FAIL.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
spanmetrics-daemonset-gateway-apm-empty
You are a Coralogix support expert. A user has asked the following question:
---
I have spanMetrics: enabled: true in our otel-integration Helm values, and the generated ConfigMap shows the spanmetrics connector wired into the traces pipeline. Traces are visible in Coralogix Explore with correct service.name and k8s attributes. But span metrics never appear — no calls_total, no duration_ms_bucket, and the APM Service Catalog stays empty and shows the onboarding prompt. The otel-integration is deployed with a daemonset agent and a gateway. Where should I look?
---
spanmetrics-placement
{
"context": "Evaluating a Coralogix support response for this user question:\n\nIn otel-integration Helm values, a customer put DB compatibility statements under spanMetrics.dbMetrics.transformStatements. Now db_calls_total has db_namespace populated from db.name, but normal calls_total still has db_system and a blank db_namespace for those database spans. Is this a chart bug, and where should the transform live?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-spanmetrics-transformstatements",
"description": "The response contains \"spanMetrics.transformStatements\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-spanmetrics-dbmetrics-transformstatements",
"description": "The response contains \"spanMetrics.dbMetrics.transformStatements\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-calls-total",
"description": "The response contains \"calls_total\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-db-calls-total",
"description": "The response contains \"db_calls_total\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-db-namespace",
"description": "The response contains \"db.namespace\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-db-name",
"description": "The response contains \"db.name\" (case-sensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "misconfiguration-wrong-config",
"description": "The response matches the pattern: (?i)(misconfiguration|wrong config|wrong placement|not.*chart.*bug)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-that-db-label-compatibility-statemen",
"description": "Explains that DB label compatibility statements that should affect normal Span Metrics belong under top-level spanMetrics.transformStatements so they run before spanmetrics consumes spans, not only under spanMetrics.dbMetrics.transformStatements. It must explain that dbMetrics-only placement can populate db_calls_total while calls_total keeps a blank db_namespace. The answer should treat this as a customer values misconfiguration rather than a chart-default bug, and should populate db.namespace from db.name with endpoint/system fallbacks when db.name is missing. FAIL if it puts the fix only under dbMetrics, after spanmetrics in the metrics pipeline, or says the chart should enable these statements by default as the immediate fix.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
spanmetrics-db-transform-placement
You are a Coralogix support expert. A user has asked the following question:
---
In otel-integration Helm values, a customer put DB compatibility statements under spanMetrics.dbMetrics.transformStatements. Now db_calls_total has db_namespace populated from db.name, but normal calls_total still has db_system and a blank db_namespace for those database spans. Is this a chart bug, and where should the transform live?
---
tail-sampling-placement
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe have tail_sampling configured in our daemonset otel-agent. We're seeing that many of our traces come through incomplete \u2014 spans from some services missing entirely. What's the issue?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-gateway",
"description": "The response contains \"gateway\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "loadbalanc-load-balanc",
"description": "The response matches the pattern: (?i)(loadbalanc|load.balanc)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "trace-id-consistent",
"description": "The response matches the pattern: (?i)(trace[_ ]?id|consistent[_ -]?hash)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "explains-that-tail-sampling-cannot-live-on-da",
"description": "Explains that tail sampling cannot live on daemonset agents because each agent only sees spans from its own node \u2014 a single trace is split across agents and the sampler makes decisions on partial data. Recommends moving tail_sampling to a central gateway tier, fed via a `loadbalancing` exporter that routes by `trace_id` so all spans for a trace land on the same gateway replica.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
tail-sampling-daemonset-gateway-placement
You are a Coralogix support expert. A user has asked the following question:
---
We have tail_sampling configured in our daemonset otel-agent. We're seeing that many of our traces come through incomplete — spans from some services missing entirely. What's the issue?
---
prometheus-target-allocator
{
"context": "Evaluating a Coralogix support response for this user question:\n\nWe enabled opentelemetry-agent.targetAllocator.enabled in the otel-integration chart and have an existing ServiceMonitor selecting our app's /metrics endpoint. The TA pod is running but the metrics aren't reaching Coralogix. The agent collectors don't seem to be scraping the target. How do I figure out whether TA is discovering the ServiceMonitor and distributing it to the collectors?",
"type": "weighted_checklist",
"checklist": [
{
"name": "mentions-port-forward",
"description": "The response contains \"port-forward\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "jobs-scrape-configs",
"description": "The response matches the pattern: /jobs|/scrape_configs",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-stays-on-the-agent-collector-side-receiver",
"description": "Directs the user to port-forward the `coralogix-opentelemetry-targetallocator` service on 8080 and inspect `/jobs` (discovered CRs) and `/scrape_configs` (rendered scrape config) to see what TA thinks it's allocating. Covers at least one of the likely root causes: missing RBAC on the CR API group, a `serviceMonitorSelector` that excludes the CR's namespace/labels, or the CR living in a namespace TA isn't watching. A response that stays on the agent collector side (receivers, exporters, pipeline wiring) without checking TA's own state is a FAIL \u2014 that's treating a TA-integration problem as a generic \"metrics not flowing\" problem.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
target-allocator-debug-port-forward
You are a Coralogix support expert. A user has asked the following question:
---
We enabled opentelemetry-agent.targetAllocator.enabled in the otel-integration chart and have an existing ServiceMonitor selecting our app's /metrics endpoint. The TA pod is running but the metrics aren't reaching Coralogix. The agent collectors don't seem to be scraping the target. How do I figure out whether TA is discovering the ServiceMonitor and distributing it to the collectors?
---
linux-standalone-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nI have a single Ubuntu 22.04 VM. I want to ship its system logs and hostmetrics to Coralogix with the smallest number of manual steps possible. What install method do you recommend?",
"type": "weighted_checklist",
"checklist": [
{
"name": "installer-otel-curl",
"description": "The response matches the pattern: (?i)(installer|otel-installer|curl)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-coralogix-private-key",
"description": "The response contains \"CORALOGIX_PRIVATE_KEY\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
},
{
"name": "mentions-coralogix-domain",
"description": "The response contains \"CORALOGIX_DOMAIN\" (case-insensitive).",
"max_score": 3,
"category": "INTENT"
}
]
}
ubuntu-vm-universal-installer
You are a Coralogix support expert. A user has asked the following question:
---
I have a single Ubuntu 22.04 VM. I want to ship its system logs and hostmetrics to Coralogix with the smallest number of manual steps possible. What install method do you recommend?
---
windows-deployment
{
"context": "Evaluating a Coralogix support response for this user question:\n\nOur Windows OTel collector is at 90% CPU whenever IIS traffic peaks. Pipeline has just otlp receiver, hostmetrics, and the coralogix exporter \u2014 nothing unusual. Before we resize the instance, what should we check?",
"type": "weighted_checklist",
"checklist": [
{
"name": "defender-msmpeng-antivirus",
"description": "The response matches the pattern: (?i)(defender|msmpeng|antivirus)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "filesystem-disk-exclude",
"description": "The response matches the pattern: (?i)(filesystem|disk|i/?o|exclude|exclusion|scan)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "identifies-windows-defender-real-time-scannin",
"description": "Identifies Windows Defender real-time scanning of rotated IIS log files (msmpeng.exe) as the likely culprit, rather than the collector itself, and recommends adding an exclusion for the IIS log path before resizing the instance.",
"max_score": 2,
"category": "RUBRIC"
}
]
}
windows-high-cpu-defender-exclusion
You are a Coralogix support expert. A user has asked the following question:
---
Our Windows OTel collector is at 90% CPU whenever IIS traffic peaks. Pipeline has just otlp receiver, hostmetrics, and the coralogix exporter — nothing unusual. Before we resize the instance, what should we check?
---
opamp-fleet-management
{
"context": "Evaluating a Coralogix support response for this user question:\n\nOur Windows host in K8s is running coralogixrepo/opentelemetry-collector-contrib-windows:0.92.0 (from the otel-integration opentelemetry-agent-windows sub-preset). We want Fleet Management. When we add extensions: [opamp] to the config, the collector refuses to start. What are our options?",
"type": "weighted_checklist",
"checklist": [
{
"name": "bump-upgrade-newer",
"description": "The response matches the pattern: (?i)(bump|upgrade|newer|v?0\\.13[0-9]|older|version|image (tag|pin))",
"max_score": 3,
"category": "INTENT"
},
{
"name": "supervisor-wrapper-binary",
"description": "The response matches the pattern: (?i)(-Supervisor|supervisor (wrapper|binary|service)|opampsupervisor)",
"max_score": 3,
"category": "INTENT"
},
{
"name": "no-claims-windows-doesn-t-support-opamp-at-al",
"description": "Says the pinned 0.92.0 Windows image is too old for the `opamp` extension and recommends either bumping the image to a current Windows build that includes `opamp`, OR using the `-Supervisor` wrapper (opampsupervisor as a separate Windows service). A response that claims Windows doesn't support OpAMP at all is a FAIL \u2014 modern Windows builds do support it.",
"max_score": 2,
"category": "MUST_NOT"
}
]
}
windows-opamp-image-version
You are a Coralogix support expert. A user has asked the following question:
---
Our Windows host in K8s is running coralogixrepo/opentelemetry-collector-contrib-windows:0.92.0 (from the otel-integration opentelemetry-agent-windows sub-preset). We want Fleet Management. When we add extensions: [opamp] to the config, the collector refuses to start. What are our options?
---
Collector component stability
Collector components do not all have the same production readiness. The upstream Collector status is mixed, and each receiver, processor, exporter, connector, and extension documents its own stability per signal in its README. Treat stability as a deployment requirement, not a doc footnote.
Stability levels
| Level | How to answer |
|---|---|
| Development | Do not recommend for production. Use only for exploration or repros. |
| Alpha | Limited, non-critical workloads only. Configuration can change with little notice. |
| Beta | Broader usage is reasonable, but breaking changes can still happen. Read release notes before upgrades. |
| Stable | Production default. Backward compatibility is expected except for exceptional cases such as security fixes. |
| Deprecated | Avoid new deployments. Recommend migration and explain removal risk. |
| Unmaintained | Avoid unless there is no alternative and the user accepts ownership/risk. |
For components that handle multiple signals, check the stability for the exact signal the user is using. A component can be stable for traces and less mature for logs, metrics, or profiles.
Answer pattern
When the user asks whether a component is safe to deploy:
1. Ask or infer the collector version, distribution, component name, signal, and workload criticality. 2. Tell them to check the component README stability table for that exact version/signal. 3. If the component is alpha, say it is not the default recommendation for business-critical production traffic. 4. If they still need it, require a pinned collector version, staging test with representative traffic, rollback plan, and release-note review before upgrades. 5. Offer the nearest stable path when one exists, even if it is less feature-rich.
What validation does not prove
otelcol validate and a successful startup prove that YAML parses and components can be constructed. They do not prove production stability, backward compatibility, safe upgrade behavior, or acceptable operational risk. Do not call a production plan safe only because the config validates.
Common Coralogix-facing examples
- A customer wants a newly released contrib processor for production URL or span-name
sanitization. Check the processor README stability and qualify alpha/beta risk before recommending it broadly.
- A collector image bump pulls in a component with changed config or startup validation.
Read the component release notes and chart compatibility before telling the user to override only image.tag.
- A user points at an OpenTelemetry specification page marked stable, but the language
SDK package or Collector component they need is alpha. The implementation maturity is what matters for the recommendation.
OpenTelemetry Collector: Connectors Configuration
This reference covers the spanmetrics and tail_sampling components. Proper placement in the collector topology is required for APM to calculate correct error rates and latencies.
Topology Rules
- `spanmetrics` connector belongs on the agent. Span metrics must be generated from 100% of spans before any sampling decision. If sampling precedes spanmetrics, APM dashboards undercount error rates and percentiles.
- `tail_sampling` processor belongs on the gateway. Tail sampling requires the full trace in memory to evaluate a policy. Daemonset agents only see their node's spans, so a tail sampler on the agent drops traces partially.
- `transactions` processor must run before `spanmetrics`.
transactionsenriches spans withcgx.transaction.*tags.spanmetricsmust consume those tags to emit per-transaction metric dimensions. - All instrumented services must route spans through the agent daemonset. If an application sends OTLP directly to the gateway or cluster-collector (bypassing the agent), the agent's
spanmetricsconnector never sees those spans. No span metrics are generated, APM Service Catalog stays empty, andcalls_total/duration_ms_bucketmetrics are absent — even though traces appear in Explore. Verify the OTLP exporter endpoint in each service points to the agent (typicallyhttp://<node-ip>:4317) and not directly to the gateway.
Helm Span Metrics DB Label Transforms
If db_calls_total has db_namespace but normal Span Metrics calls_total has a blank db_namespace, check where the Helm DB compatibility transform is configured.
In otel-integration values, DB label compatibility statements that need to affect normal Span Metrics must live under top-level spanMetrics.transformStatements. Those statements run on spans before the spanmetrics connector consumes them, so both calls_total and db_calls_total see the same normalized span attributes.
Do not put this bridge only under spanMetrics.dbMetrics.transformStatements. That DB-metrics-only placement can make db_calls_total look correct while calls_total still has blank db_namespace for the same database spans. Treat that as a values misconfiguration, not as a chart-default bug.
The pre-spanmetrics bridge should populate db.namespace from the first available source: db.name, then endpoint attributes such as server.address, network.peer.name, or net.peer.name, and finally db.system. Use the OTTL skill when the user needs the exact transform statement syntax.
Agent Configuration (spanmetrics)
# AGENT (daemonset)
connectors:
spanmetrics:
dimensions:
- name: http.method
- name: http.status_code
- name: cgx.transaction
- name: cgx.transaction.root
- name: db.system
processors:
transactions: # runs BEFORE spanmetrics
# ... populate cgx.transaction.* tags on spans
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, resourcedetection, transactions, batch]
exporters: [spanmetrics, loadbalancing] # spanmetrics connector fans out to metrics pipeline
metrics/spanmetrics:
receivers: [spanmetrics] # connector-consumed pipeline
processors: [memory_limiter, batch]
exporters: [coralogix] # spanMetrics go direct to CoralogixGateway Configuration (tail_sampling)
To ensure the gateway evaluates complete traces, the agent MUST route spans via the loadbalancing exporter configured with routing_key: traceID.
# GATEWAY (deployment)
processors:
tail_sampling:
decision_wait: 30s
num_traces: 50000
policies:
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
- name: default
type: probabilistic
probabilistic:
sampling_percentage: 10
service:
pipelines:
traces:
receivers: [otlp] # fed by loadbalancing on agents
processors: [memory_limiter, k8sattributes, tail_sampling, batch]
exporters: [coralogix]OpenTelemetry Collector: Exporters Configuration
This reference covers the coralogix, coralogix/resource_catalog, and loadbalancing exporters.
Coralogix Exporter
The coralogix exporter ships telemetry to Coralogix over OTLP. Regional routing is handled by a domain field.
exporters:
coralogix:
domain: "eu2.coralogix.com" # bare hostname for your region; NOT a URL
private_key: "${env:CORALOGIX_PRIVATE_KEY}"
application_name: "my-app"
subsystem_name: "my-service"
timeout: 30sCritical Rules
- Domain is a bare hostname: Set
domain:to the regional data-ingestion hostname (e.g.,eu2.coralogix.com). Never useendpoint:,https://, or a trailing slash. Never use a UI hostname. - Bracketed syntax for env vars: Use
${env:CORALOGIX_PRIVATE_KEY}. The unbracketed$CORALOGIX_PRIVATE_KEYform silently fails to expand in newer versions (≥ v0.76). - Dynamic Routing: Use
application_name_attributesandsubsystem_name_attributesto dynamically route telemetry based on resource attributes, falling back to static strings:
exporters:
coralogix:
domain: "eu2.coralogix.com"
private_key: "${env:CORALOGIX_PRIVATE_KEY}"
application_name_attributes: ["service.namespace", "application"]
subsystem_name_attributes: ["service.name", "k8s.deployment.name"]
application_name: "default-app" # fallback when no derived value
subsystem_name: "default-subsystem"Transport protocol
Do not set protocol: http. Since collector-contrib v0.144, the coralogix exporter validates HTTP compatibility for all signals at startup — including profiles, which require gRPC. The validation runs before pipeline wiring, so protocol: http fails even with no profiles pipeline:
Error: exporters::coralogix: profiles signal is not supported with HTTP protocol,
use gRPC protocol (default) insteadRemove the protocol: field. gRPC is the default and works for all signals.
Back-pressure
The exporter's sending_queue and retry_on_failure blocks provide resilience and back-pressure.
exporters:
coralogix:
domain: "eu2.coralogix.com"
private_key: "${env:CORALOGIX_PRIVATE_KEY}"
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
storage: file_storage # optional: persist to disk for crash-safe buffering
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
timeout: 30sInfrastructure Explorer / Resource Catalog Exporter
Entity metadata requires a dedicated exporter and specific HTTP headers.
exporters:
coralogix/resource_catalog:
domain: "eu2.coralogix.com"
private_key: "${env:CORALOGIX_PRIVATE_KEY}"
application_name: "resource"
subsystem_name: "catalog"
logs:
headers:
x-coralogix-ingress: "metadata-as-otlp-logs/v1"
timeout: 120s- Required Header: Without
x-coralogix-ingress: metadata-as-otlp-logs/v1, entity events reach the logs pipeline but never populate Infrastructure Explorer.
Loadbalancing Exporter
Used on the agent daemonset to route spans to a central gateway deployment based on trace_id. This is required for tail_sampling on the gateway to evaluate complete traces.
exporters:
loadbalancing:
protocol:
otlp:
tls:
insecure: true
resolver:
dns:
hostname: otel-gateway.coralogix.svc.cluster.local
port: 4317
routing_key: traceID- Routing Key:
routing_key: traceIDenforces consistent hashing on the trace ID. Without it, spans distribute round-robin, and the gateway's tail sampler sees incomplete traces, producingsampling_trace_dropped_too_earlymetrics.
OpenTelemetry Collector: Processors Configuration
This reference covers the universal processor chain and critical Kubernetes enrichment processors. Order matters fundamentally.
Universal Processor Chain
Configure pipelines in this order from receiver to exporter:
receivers → memory_limiter → resourcedetection/* → resource/metadata → k8sattributes/ecsattributes → transform/k8s_attributes → [transform/filter …] → batch → exporter1. `memory_limiter` (FIRST): Sheds load when approaching memory limits. Must be first to prevent upstream components from doing wasted work on records that will be dropped. 2. *`resourcedetection/**: Adds host.`, `cloud., k8s.`, `aws. attributes. 3. **resource/metadata**: Injects k8s.cluster.name, cx.otel_integration.name, and deployment markers like cx.agent.type (agent, cluster-collector, gateway). Without this, Coralogix correlation breaks. 4. **k8sattributes or ecsattributes**: Enriches telemetry with orchestrator metadata (e.g., pod names, replica sets). 5. **transform/k8s_attributes**: Derives k8s.deployment.name from k8s.replicaset.name by stripping the hash suffix. Required for proper workload aggregation in APM. 6. **transform / filter**: Custom OTTL rules. 7. **batch` (LAST before exporter)**: Amortizes export cost. Batching prior to enrichment wastes memory and CPU.
k8sattributes: Passthrough vs. Full Extraction
When running both an agent (daemonset) and a gateway (deployment), only one role should perform full extraction to avoid hammering the Kubernetes API server.
- Agent (passthrough): Performs a cheap stamp of
k8s.pod.ipand defers lookup. - Gateway (full extraction): Performs the actual API lookup and enriches the telemetry.
Agent Configuration (Passthrough)
processors:
k8sattributes:
passthrough: true
extract:
metadata: [k8s.namespace.name, k8s.pod.name, k8s.node.name]
filter:
node_from_env_var: K8S_NODE_NAME # required — or the agent scans all nodes
pod_association:
- sources: [{ from: resource_attribute, name: k8s.pod.ip }]
- sources: [{ from: connection }]Gateway Configuration (Full Extraction)
processors:
k8sattributes:
passthrough: false
extract:
metadata:
- k8s.namespace.name
- k8s.pod.name
- k8s.pod.uid
- k8s.deployment.name
- k8s.statefulset.name
- k8s.daemonset.name
- k8s.replicaset.name
- k8s.job.name
- k8s.cronjob.name
- k8s.node.name
- k8s.cluster.uid
- k8s.container.name
labels:
- tag_name: k8s.app.name
key: app.kubernetes.io/name
from: pod
- tag_name: k8s.app.version
key: app.kubernetes.io/version
from: pod
pod_association:
- sources: [{ from: resource_attribute, name: k8s.pod.ip }]
- sources: [{ from: connection }]The Triple-Duplication Anti-Pattern
Applying full extract.metadata on agents, cluster-collectors, and gateways simultaneously leads to:
- High Kubernetes API Server QPS.
- Inflated
otelcol_processor_k8sattributes_pod_tags_add_totalmetrics. - Racing conditions where attributes flap.
Resolution: Set passthrough: true on agents. Perform full extraction on the gateway (or cluster-collector if no gateway exists).
Required Resource Attributes for Coralogix
Coralogix Service Catalog and Infrastructure Explorer require the following attributes:
k8s.cluster.name(becomescx.cluster.name)k8s.node.name+host.nameorhost.idk8s.namespace.namek8s.pod.uid+k8s.pod.namek8s.container.name- Workload identity: One of
k8s.deployment.name,k8s.statefulset.name,k8s.daemonset.name,k8s.job.name,k8s.cronjob.name service.name
Missing workload identity tags mean pods cannot aggregate into parent workloads. Verify workload extraction by querying k8s_pod_phase{k8s_deployment_name=""}.
Data safety and cardinality
Use this reference when the user asks about PII/secrets, URL/span-name sanitization, Span Metrics cardinality, aggregation_cardinality_limit, or expensive APM metrics. The safe answer is layered: prevent bad telemetry at the source, sanitize before metrics are generated, and use limits as guardrails.
Rule of 100 for metric dimensions
If an attribute can have more than about 100 unique values, do not use it as a metric dimension by default. Every distinct label combination creates a new time series, so several individually "moderate" dimensions multiply into very large cardinality.
Good Span Metrics dimensions are bounded and useful for aggregation:
http.methodhttp.response.status_codespan.kindstatus_codehttp.routewhen it is templated, such as/users/{id}/orders/{order_id}- stable service/resource dimensions needed by Coralogix APM
Dangerous dimensions include:
url.full, rawurl.path, or URLs with query stringsuser.id,session.id,request.id,trace_id,span_idcustomer.email, IP addresses, tokens, API keysk8s.pod.ipand broadk8s.pod.nameusage for Span Metrics- raw
db.statement,db.query.text, or SQL with literal values
When a customer says they need one of these for alerts or dashboards, do not just approve it. Offer a lower-cardinality replacement first, then a targeted sanitization plan if they must keep it.
Span Metrics cardinality layers
1. Instrumentation first. Prefer http.route over raw URL paths. Keep IDs and per-request values on traces or logs, not metric labels. 2. Sanitize before `spanmetrics`. Normalize dynamic values in the traces pipeline before the spanmetrics connector consumes spans. For exact OTTL syntax, hand off to the opentelemetry-ottl skill. 3. Trim dimensions. Remove high-cardinality spanmetrics dimensions such as url.full, k8s.pod.ip, broad k8s.pod.name, and unnecessary custom labels. 4. Tune histogram buckets. Excessive buckets multiply duration_ms_bucket series. Keep buckets aligned to the user-facing latency questions. 5. Add guardrails. Use aggregation_cardinality_limit / Helm spanMetrics.aggregationCardinalityLimit to prevent unbounded growth. This is not a substitute for fixing bad dimensions.
In Coralogix Kubernetes Complete Observability, newer chart versions enable a default Span Metrics cardinality limit. For custom collectors, older chart versions, or custom spanmetrics connectors, verify the setting explicitly.
Cardinality limit behavior
The spanmetrics connector's cardinality limit is a system guardrail. When the limit is reached, new label combinations are collapsed into an overflow series rather than continuing to create unique series. That protects the pipeline, but it also means the high-cardinality dimension is no longer useful for precise breakdowns.
Say clearly:
- Raising the limit may restore visibility temporarily, but it increases memory and
backend pressure.
- The durable fix is to remove, normalize, or scope the high-cardinality labels.
- The limit is in-memory collector state; restarts and metric expiration reset tracking.
URL and span-name sanitization
For broad URL-like span names or URL attributes, redactionprocessor can be useful:
processors:
redaction/url_sanitizer:
# Keep existing attributes while only applying URL/span-name sanitization.
# Without allow_all_keys or allowed_keys, redactionprocessor drops keys that
# are not explicitly allowed.
allow_all_keys: true
url_sanitizer:
enabled: true
attributes: ["url.full", "http.url", "url"]
sanitize_span_name: trueUse it when the user wants a general URL sanitizer and accepts component maturity/risk after checking the processor README. Validate representative before/after examples, because broad sanitizers can over-sanitize domains or meaningful path segments. When showing redactionprocessor examples for URL sanitization, include either allow_all_keys: true for pass-through behavior or a deliberate allowed_keys list; otherwise the processor is fail-closed and removes unspecified span/log/datapoint attributes before export.
For targeted customer-specific patterns, use the OTTL transform processor before spanmetrics, for example replacing IDs in url.full, stripping query strings, or normalizing dynamic path segments. Do not put sanitization only in a metrics pipeline after spanmetrics; by then the high-cardinality series have already been created.
PII and secret redaction
Sensitive data should be removed, masked, or hashed before telemetry leaves the collector. The collector skill should identify the pipeline placement and then route exact statement authoring to the OTTL skill.
Common choices:
- Hash stable identifiers such as
user.idwithSHA256when correlation is still
useful but raw values must not leave the environment.
- Mask credentials and Authorization headers with
replace_all_patterns. - Strip query parameters such as
token,api_key,secret, andpassword. - Delete attributes like
user.emailwhen they are not needed for support workflows. - Guard transforms with
whereclauses such asattributes["url.full"] != nilor
IsString(body) so malformed records do not break the pipeline.
Place redaction processors before batch and coralogix, and before any connector that turns spans/logs into metrics. If the question is mostly about exact OTTL syntax, switch to the opentelemetry-ottl skill and use its redaction/cardinality references.
Memory and performance
If you see "the collector is using a lot of memory" or "we keep OOMing" — most of the answer is: (a) memory_limiter is doing its job, (b) the numbers you're watching include kernel page cache that the collector didn't allocate, (c) the real lever is upstream volume or cardinality, not collector sizing.
Contents
memory_limiterbehavior- Go heap vs container RSS and page cache
- Throttling vs OOM
- Chronic
memory_limitercauses - Back-pressure with
sending_queueandfile_storage - Batch processor tuning
- Per-deployment notes
- Key facts
memory_limiter: what it actually does
processors:
memory_limiter:
check_interval: 2s # how often to sample process memory
limit_mib: 512 # hard ceiling — refuse data beyond this
spike_limit_mib: 128 # soft ceiling — start GC aggressively at limit - spike- At
limit_mib, receivers start refusing data.otelcol_processor_memory_limiter_*_rejectedrises, the upstream sees send errors and retries. - Between
limit - spikeandlimit, the processor triggers Go GC on every check. memory_limitermust be the first processor in every pipeline — seeconfig-processors.md. Put it anywhere else and upstream processors do work that gets shed.
memory_limiter is a circuit breaker, not a memory reclamation tool. It protects the collector from OOM at the cost of upstream back-pressure (which callers interpret as a drop or retry). That back-pressure is desirable — it keeps the collector alive during bursts.
Sizing: Go heap vs container RSS
This is the single most common "memory leak" that isn't.
Real user case:
memory_limiter: limit_mib: 512kubectl top pod otel-agent-xxx: 464 MiB- Go runtime (via pprof): HeapAlloc ≈ 49 MB, Go Sys ≈ 117 MB
Delta: ~350 MB of container RSS that the Go runtime didn't allocate. Source: kernel page cache from hostPath /var/log/pods mount, accounted to the container cgroup because the collector reads those files.
Debugging the disconnect
1. Emit self-telemetry (see ops-troubleshooting.md → step 1). Watch process.runtime.go.mem.heap_alloc, process.runtime.go.mem.heap_sys, otelcol_process_memory_rss. 2. If process.runtime.go.mem.heap_alloc is well below limit_mib but kubectl top / process.runtime.uptime + otelcol_process_memory_rss is near it, the delta is kernel page cache. 3. Page cache counts toward cgroup memory limits. memory_limiter samples the Go process — not the cgroup — so it fires on heap_sys, not RSS. If the cgroup OOM-kills the pod, that's the kernel, not the collector.
Fixing the disconnect
- Don't keep scaling `limit_mib` up. That treats the symptom (cache), not the cause.
- Don't raise the pod memory limit alone. It just lets more cache accumulate.
- Do check if
filelogis re-reading from start (unnecessary I/O spike) —start_at: endis usually correct. - Do use
file_storageextension to persist checkpoints — fewer re-reads after restart. - Do consider
filelog'smax_concurrent_filesif the host has many files; high concurrency = more page-cache touching.
Throttling vs OOM (two different things)
| Symptom | Looks like | Actually |
|---|---|---|
| OOM | container killed, pod restart, Exit Code 137 or OOMKilled | kernel killed the collector for exceeding cgroup memory limit |
| Throttling | data arrives intermittently, processor_memory_limiter_*_rejected climbs, but no pod restart | memory_limiter doing its job — shedding load to stay below limit_mib |
Throttling is not a bug. If the user sees throttling and no data loss (retries drained), the collector is handling a burst correctly. If they see throttling and sustained data loss, the upstream rate is genuinely above capacity — look at the source, not the collector.
When chronic memory_limiter firing means something else
memory_limiter shouldn't be firing in steady state. If it is, these are the causes (in order of likelihood):
1. Cardinality explosion in metrics
Each unique combination of resource + datapoint attributes is a series in memory. Common offenders:
- Process IDs, pod UIDs, ephemeral identifiers on resource attributes.
- SDK-specific high-cardinality attributes (
process.command_args,deployment.environmenttypos). - User-added labels on third-party metrics without a deny-list.
Fix: use OTTL to trim (keep_keys, delete_matching_keys, delete_key). See the OTTL skill's references/cardinality.md.
2. Spanmetrics aggregation_cardinality_limit blowup
The spanmetrics connector has its own cardinality ceiling:
connectors:
spanmetrics:
aggregation_cardinality_limit: 100000
dimensions:
- name: http.method
- name: http.status_code
# ...High-cardinality dimensions (e.g. url.path with IDs) blow up the series count. Trim the dimension list to low-cardinality fields or extract the pattern out of the path first (replace_pattern).
3. Tail sampling buffer sizing
Gateway memory ≈ num_traces × avg spans per trace × avg span bytes. If num_traces: 500000 and avg span is 2 KB, you're reserving ~1 GB per gateway replica for trace buffering alone.
Reducing num_traces shrinks the buffer. Reducing decision_wait shrinks the time window but risks dropping late-arriving spans. Scale replicas instead when traces are actually plentiful.
4. Instrumentation doing more than the collector can handle
Real case: Node.js service with TypeORM auto-instrumentation created ~5x the spans of the underlying DB call volume, because the instrumentation traced every ORM-layer hop. Swapping to the narrower MySQL instrumentation dropped the span rate 80% and the collector stopped OOMing.
Fix: look upstream. If a microservice emits 10k traces/sec and the team didn't realize, no amount of collector tuning fixes it. Check with:
otelcol_receiver_accepted_spans{receiver="otlp"}per agent- per-service span counts from the OTel SDK's own metrics (if emitted)
Back-pressure: sending_queue + file_storage
For bursts where the collector is fine but Coralogix ingress is momentarily slow, buffer spans/logs/metrics durably:
extensions:
file_storage:
directory: /var/lib/otelcol/queue
timeout: 10s
exporters:
coralogix:
sending_queue:
enabled: true
num_consumers: 10
queue_size: 10000
storage: file_storage
block_on_overflow: true # apply back-pressure upstream instead of dropping
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
timeout: 30s
service:
extensions: [file_storage]storage: file_storagewrites the queue to disk — survives collector restarts.block_on_overflow: trueback-pressures the pipeline (propagating to receivers) instead of dropping records at the exporter. Combined withsending_queuethis is the right pattern for "no data loss."- Disk path needs write access; with K8s, provision a PVC or an emptyDir (ephemeral — loses queue on pod delete).
Batch processor tuning
batch amortizes export cost. Defaults are usually fine; common adjustments:
processors:
batch:
timeout: 2s # flush at least every 2s
send_batch_size: 1000 # trigger flush at this many records
send_batch_max_size: 2000 # hard cap — no single export beyond this- Too-small batches → exporter overhead per record dominates.
- Too-large batches → export latency, memory held longer, and some Coralogix ingestion paths have a single-request size ceiling (~2 MB default — tune
send_batch_max_sizeunder that). - For logs with huge records, prefer size-based batching (
sizer: bytes) over count-based.
Per-deployment notes
- K8s (otel-integration): Chart defaults are usually sensible. Watch for custom
hostPathmounts (page cache inflation) and per-service memory requests vs actual RSS. - ECS EC2 daemonset: Host-networked, so pod-level memory limits don't apply the same way. Size the EC2 node to accommodate collector+kernel-cache.
- ECS Fargate sidecar: Fargate task memory is the hard ceiling.
limit_mibat ~75% of task memory is safe; leaves headroom for the app. - Windows standalone:
limit_mibas env is fine. Windows counts memory differently — Resource Monitor is more reliable than Task Manager.
Key Facts
- `memory_limiter` is a circuit breaker, not a leak fixer. Firing during bursts is the design.
- `kubectl top` ≠ Go heap. The delta is typically kernel page cache from file mounts; not a leak.
- Throttling is not OOM. Throttling = back-pressure working. OOM = cgroup killed the process.
- Chronic firing = upstream volume or cardinality. Look at receivers, spanmetrics dimensions, instrumentation breadth — before sizing up.
- `sending_queue` + `file_storage` + `block_on_overflow: true` is the no-data-loss back-pressure pattern.
- Batch `send_batch_max_size` under 2 MB for compatibility with some Coralogix ingestion paths.
- Gateway memory ≈ `num_traces × avg spans × avg bytes` — size tail-sampling buffers deliberately.
Troubleshooting: symptom → root cause
Maps recurring user symptoms to the actual root cause and the first action to take. Most issues come down to pipeline placement, exporter config, or an upstream connectivity/IAM problem masquerading as a config issue.
Contents
- Operating rule: confirm this is a collector problem
- Symptom to root-cause table
- Debugging workflow
- Collector self-telemetry metrics
- Debug exporter isolation
- High-signal collector logs
- Scope checks and key facts
Operating rule: is the problem even the OTel Collector?
Before editing pipelines, confirm the symptom actually belongs to the collector. "No data reaching Coralogix" is often a DNS / TLS / proxy / domain / API-key problem — not a pipeline problem. If the user has added a transform processor to "force data through" and it made no difference, the root cause is upstream. Say so clearly before continuing.
Symptom → Root cause table
| Symptom | Most likely root cause | First action |
|---|---|---|
| No data of any kind reaching Coralogix | endpoint: used instead of domain:; full URL used instead of bare hostname; unbracketed $CORALOGIX_PRIVATE_KEY failing to expand in exporter v0.76+; or a UI hostname used as domain: instead of the data-ingestion hostname | Verify domain: is a bare <region>.coralogix.com hostname (see core for the regions and UI-vs-ingestion rules). private_key: "${env:CORALOGIX_PRIVATE_KEY}" (bracketed). Check egress/TLS/proxy to *.coralogix.com:443. |
Collector starts, pipelines load, but exporter logs show rpc error: code = ResourceExhausted or code = PermissionDenied | Rate limit / quota / wrong API key scope | Check ingress quota on the Coralogix side; verify the PRIVATE_KEY is a Send-Your-Data key (see core for key categories). |
| Resource Catalog / Infrastructure Explorer is empty despite K8s deployment running fine | Missing kubernetesResources preset; or using the default coralogix exporter instead of coralogix/resource_catalog; or missing x-coralogix-ingress: metadata-as-otlp-logs/v1 header | Enable opentelemetry-cluster-collector.presets.kubernetesResources.enabled: true. Ensure the dedicated coralogix/resource_catalog exporter has the ingress header. See preset-kubernetes.md. |
| APM shows traces but no error rates / p99 latencies | spanmetrics configured on the gateway (after tail_sampling) instead of the agent — metrics only see sampled spans | Move the spanmetrics connector to the agent, upstream of any sampling. transactions processor must also be upstream of spanmetrics. |
| Traces are incomplete across services — some spans per trace present, others missing | Tail sampling running on daemonset agents (each agent only sees its node's spans); or multiple gateway replicas without consistent-hashing loadbalancer | Put tail_sampling on a central gateway. Feed via loadbalancing exporter with routing by trace_id so all spans for one trace land on the same gateway replica. |
spanMetrics values look double-counted | Same spanmetrics connector running on both agent and gateway — metrics emitted twice | Enable spanmetrics on agent only. Remove any spanmetrics connector from gateway/cluster-collector. |
k8sattributes fires full extraction in 3 collectors, API server at high QPS | Triple-duplication anti-pattern — same full extract.metadata block in daemonset, cluster-collector, gateway | Keep passthrough: true on daemonset agents. Full extraction in one role only (usually gateway). |
Collector refuses to start — cannot start pipelines: failed to start "resourcedetection/resource_catalog" processor: can't get K8s Instance Metadata; node name is empty | Resource Catalog detector running on a pod that is not part of the cluster-collector deployment (e.g. daemonset agent where node-name injection fails) | Keep resourcedetection/resource_catalog on cluster-collector only. Ensure K8S_NODE_NAME env is set from spec.nodeName on the cluster-collector pod spec. |
otelcol_processor_tail_sampling_sampling_trace_dropped_too_early_total climbing | Not enough gateway replicas / num_traces buffer too small / decision_wait too short for the trace duration | Scale gateway replicas. Increase num_traces (trace buffer) and decision_wait (how long to hold a trace before deciding). Check that spans for one trace actually reach the same replica. |
| Duplicate metric series or partial resets | single-writer principle violated — multiple collectors emitting the same metric with different resource attributes | Add resourcedetection (env/ec2/system/host) to the metrics pipeline and deduplicate the resource.attributes set. Each unique series should have exactly one writer. |
Exporter logs INVALID_ARGUMENT on send, but with no upstream body/data error | OTTL problem (transform/filter) — wrong context, missing nil guard | See the OTTL skill's Error Decoder. INVALID_ARGUMENT here is a pipeline processing error before export. |
Exporting failed. Dropping data. with context deadline exceeded | Coralogix ingress slow or client-side backpressure / timeout too aggressive | Increase coralogix.timeout (default 5s) to 30–120s. Add sending_queue.enabled: true + storage: file_storage for durable buffering. Investigate ingress latency metrics side of the house. |
opentelemetry-cluster-collector not reporting cx.agent.type: cluster-collector | Values override broke the resource/metadata processor; or operator wiped the default service.pipelines | Restore default service.pipelines from chart defaults. Prefer extraProcessors hook over wholesale pipeline replacement. See setup-kubernetes.md "Don't override pipelines wholesale." |
| APM gateways consume massive memory, span drops in logs | Gateway sized for steady state; spike from burst traffic + tail_sampling buffer (num_traces) inflation | Size gateway memory for: num_traces × avg spans/trace × avg span size. Reduce decision_wait if traces are short-lived. |
| Windows collector CPU spikes during peak IIS traffic | Not the collector — Windows Defender scanning rotated IIS log files | Open Resource Monitor → Disk. If msmpeng.exe is the top consumer on the IIS log path, exclude that path from Defender real-time scan. See setup-windows-standalone.md. |
| UI-generated Linux host config produces traces without language icons | Host wizard strips telemetry.sdk.* resource attributes | Workaround: transform processor re-adds them on the traces pipeline until the wizard fix lands. See setup-linux-standalone.md. |
memory_limiter fires constantly, memory doesn't seem to drop after GC | Go heap vs kernel page cache disconnect — kubectl top shows container RSS including page cache from hostPath /var/log/pods mounts | Check Go HeapAlloc via the collector's self-telemetry. If Go Sys < RSS significantly, the delta is kernel page cache (not a leak). See ops-memory-performance.md. |
| ECS Fargate sidecar: logs lost when app container crashes | App is essential: true, sidecar is essential: false — ECS kills task including sidecar before buffered logs drain | Add healthCheck to sidecar + dependsOn: [{containerName: otel-collector, condition: HEALTHY}] on the app. Use CDOT image (includes /healthcheck binary). See setup-ecs-fargate.md. |
| ECS EC2 daemonset: every log row attributed to the collector's container | ecs detector in resourcedetection stamps collector's own container ID onto all records | Remove ecs from resourcedetection.detectors on daemonset-mode collectors. Use the ecsattributes/container-logs CDOT processor for per-container attribution. See setup-ecs-ec2.md. |
| Fleet Manager says chart version applied, but user's values.yaml changes don't appear | presets.fleetManagement.supervisor.enabled: true — Fleet Manager UI overrides values.yaml config at runtime | Explain the precedence model: with supervisor enabled, config comes from Fleet Manager, not the Helm values. Edit in the UI or disable supervisor. See preset-fleet-management.md. |
Collector v0.142+ crashes CrashLoopBackOff with older otel-integration chart versions | Upstream collector contains a breaking change not yet absorbed by the chart | Pin the collector image version (chart image.tag) or bump the whole chart to a matching release. Don't mix a new-image-tag override against an old chart. |
Debugging workflow
If you see "it's broken," work through this order before editing pipelines:
1. Enable the collector's self-telemetry
If the collector isn't already emitting its own metrics, turn it on temporarily:
service:
telemetry:
metrics:
level: detailed
readers:
- periodic:
interval: 30000
exporter:
otlp:
protocol: grpc
endpoint: "http://localhost:4317"
logs:
level: infoMetrics to read first:
| Metric | Tells you |
|---|---|
otelcol_receiver_accepted_* vs _refused_* | whether data is entering at all |
otelcol_processor_batch_batch_send_size | batch sizing; near-empty batches mean low volume or upstream drop |
otelcol_processor_memory_limiter_* | back-pressure events |
otelcol_exporter_send_failed_* | export failures by signal |
otelcol_exporter_queue_capacity / _queue_size | sending_queue depth |
otelcol_processor_tail_sampling_sampling_trace_dropped_too_early_total | gateway under-sized / wrong routing |
2. Add a debug exporter, mirror the problematic signal
exporters:
debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [coralogix, debug] # mirrorIf debug sees the data but coralogix doesn't, the problem is the exporter / connectivity. If neither sees it, the problem is upstream (receiver, source app, network).
Only turn verbosity: detailed on in production for short diagnostic windows — it is expensive, especially on Windows.
3. Read the collector logs
Typical high-signal log lines:
Exporting failed. Dropping data.+rpc error: code = X— exporter connectivity / auth / quota. Cross-check againstdomain,private_key, egress.INVALID_ARGUMENT— OTTL problem in transform/filter. Switch to the OTTL skill.can't get K8s Instance Metadata; node name is empty—resourcedetection/resource_catalogon wrong collector type (see table above).cannot start pipelines: failed to start "X" processor— processor on the wrong collector role, or missing env/RBAC.denied by autogke-no-write-mode-hostpath— GKE Autopilot Warden, seesetup-kubernetes.md.
4. Cross-check quickly that it isn't the skill's own scope
- OTTL issues → OTTL skill.
- Fleet Management config precedence →
preset-fleet-management.md. - Memory / page cache →
ops-memory-performance.md.
5. Only now, touch the pipeline
When you do edit pipelines, change one thing at a time and watch the self-telemetry. Batching two changes in a single apply wastes the diagnostic budget.
Key Facts
- Most "no data" problems are connectivity, not pipeline. Check domain/key/egress first.
- `coralogix` domain field is a bare hostname. Not a URL, not
endpoint:. - Enable self-telemetry before editing anything.
otelcol_*metrics are the highest-signal diagnostic. - Mirror suspect signals through a `debug` exporter to isolate where data drops.
- `single-writer` violations cause duplicate metrics and partial resets. Deduplicate resource attributes across collectors.
- If `memory_limiter` fires chronically, look upstream — cardinality/volume — before sizing up the collector.
- APM missing errors = spanmetrics placement. Move to agent, upstream of sampling.
- Incomplete cross-service traces = tail_sampling placement + loadbalancing routing. Central gateway,
consistent_hashingbytrace_id.
Fleet Management: overlap notes
Fleet Management (OpAMP supervisor + remote config via the Coralogix UI) intersects with the OTel Collector configuration in a few places. Only those overlap points are covered here — deep OpAMP protocol internals, supervisor-binary flags, remote-config-rendering precedence, the Coralogix Distribution of OpenTelemetry (CDOT), and the Fleet Manager UI itself are out of scope for this skill.
The three things that matter here
1. Supervisor OpAMP endpoint does use the ingress. prefix — unlike the coralogix exporter
The single most common confusion: the coralogix exporter's domain: field is a bare hostname (eu2.coralogix.com), but the supervisor's OpAMP endpoint is a full URL with the ingress prefix:
# coralogix EXPORTER — bare hostname
exporters:
coralogix:
domain: "eu2.coralogix.com"
# SUPERVISOR OpAMP endpoint — full URL WITH ingress prefix
presets:
fleetManagement:
enabled: true
supervisor:
enabled: true
server:
endpoint: "https://ingress.eu2.coralogix.com/opamp/v1"Both are right in their own place. They look inconsistent because they route through different Coralogix ingest paths. If a user hardcodes https://ingress.<domain> as their coralogix exporter domain: in order to make the supervisor work, they have mixed the two up — separate them.
The chart handles this automatically when .Values.global.domain is set. Direct-to-helm users editing a raw config by hand are the ones who get caught.
2. presets.fleetManagement.supervisor.enabled: true overrides values.yaml at runtime
When the supervisor is enabled, the collector's runtime config comes from the Fleet Manager UI, not from the Helm values.yaml applied at deploy time. A user pushing changes via helm upgrade (especially from ArgoCD) and seeing "why isn't my change taking effect" has hit this.
The precedence is:
Fleet Manager UI config > values.yaml at helm upgrade timeValues.yaml still matters for: chart version, pod resource requests, service account, ingress endpoints, presets themselves. It does not matter for: receiver/processor/exporter configuration, pipeline wiring, filter/transform OTTL.
Diagnostic: if a user says "I redeployed with new values but the config didn't change," check whether supervisor.enabled: true. If so, refer to the Fleet Manager UI.
Opt-out: set supervisor.enabled: false (chart values) and manage config via helm as usual. Users using pure GitOps (ArgoCD with declarative intent) typically prefer this.
3. Windows: image version determines whether in-process opamp works
Both Fleet Management shapes are available on Windows with modern collector builds — verified against the rendered otel-windows-standalone chart (collector v0.130.7), which ships extensions: [opamp] by default.
- In-process extension works when the collector image is ≥ v0.130. Standalone chart defaults here.
- Supervisor wrapper (
-Supervisoron the installer) works regardless of collector version — it runs a separateopampsupervisorservice.
The trap — older image pins:
otel-integration'sopentelemetry-agent-windowssub-preset historically defaults tocoralogixrepo/opentelemetry-collector-contrib-windows:0.92.0, which predates OpAMP on Windows. A user enablingextensions: [opamp]there will see the collector refuse to start.otel-ecs-ec2-windowssimilarly pins an older image in its CloudFormation template.
Practical implication: a Windows user whose extensions: [opamp] fails is either on an old image (fix: bump, or use the supervisor wrapper) or on the K8s Windows sub-preset (same fix). On modern standalone, it just works.
Signals that fleet-management scope is intruding
When any of these appear in a user's issue, refer to the supervisor skill (or this file's section 2):
- "Config keeps reverting to something I didn't set."
- "
helm upgradesays applied, but the pipeline behavior is unchanged." - "We have
opampextension enabled but the collector won't start." (Likely Windows, see section 3.) - "Supervisor endpoint is returning 404."
- "Fleet Manager shows my cluster but no agents."
- "eBPF profiler doesn't connect to Fleet Manager."
- "Changing
X-Coralogix-Distributionheader in the exporter config had no effect." (It's set by the supervisor; edit in Fleet Manager.)
When this skill is the wrong place to look
Questions that need the OpAMP supervisor / Fleet Manager internals rather than the collector-config overlap covered here:
- Deploying with Fleet Management end-to-end from scratch.
- Remote-config precedence details beyond "UI beats values.yaml at runtime" (section 2 above).
otel-supervised-cdotorotel-supervised-ebpf-profilerimage specifics.- Debugging the OpAMP supervisor binary itself (crash loops, auth handshake failures).
Those sit with the Fleet Manager / CDOT documentation, not with this OTel Collector skill.
Key Facts
- Supervisor OpAMP endpoint is `https://ingress.{domain}/opamp/v1`. The coralogix exporter
domain:is a bare hostname. Two different fields, two different shapes — don't mix them. - With `supervisor.enabled: true`, Fleet Manager UI beats values.yaml at runtime. ArgoCD users usually want
supervisor.enabled: false. - Windows supports both in-process `opamp` and the supervisor wrapper — on modern builds. The
otel-windows-standalonechart (collector ≥ v0.130) rendersextensions: [opamp]by default. Older pinned images (notablycontrib-windows:0.92.0used in the K8s Windows sub-preset andotel-ecs-ec2-windows) predate it — there, use the-Supervisorwrapper or bump the image. - Signals of Fleet Management scope = config reverts,
helm upgradehas no effect,X-Coralogix-Distributionchanges ignored. - Deep OpAMP / supervisor / CDOT coverage is out of scope. Only the overlap with collector configuration (endpoint shape, values-vs-UI precedence, Windows image pitfall) lives here.
Coralogix Helm Presets for Kubernetes
This reference covers the Coralogix abstractions built into the otel-integration Helm chart via presets. Do not use full service.pipelines overrides; use presets to enable features safely.
Core Presets
| Preset | Runs on | Purpose | When to enable |
|---|---|---|---|
logsCollection | agent | filelog on pod logs + checkpointing | almost always |
kubeletMetrics | agent | kubeletstats receiver | almost always |
hostEntityEvents | agent | host-level entity events for Infra Explorer | when utilizing Infra Explorer for host context |
kubernetesEvents | cluster-collector | k8sevents receiver | almost always |
kubernetesResources | cluster-collector | k8sobjects scrape + resourcedetection/resource_catalog + dedicated coralogix/resource_catalog pipeline | required for Infrastructure Explorer |
kubernetesApiServerMetrics | cluster-collector | scrape kube-apiserver | for control-plane dashboards |
clusterMetrics | cluster-collector | k8s_cluster receiver (namespace/deploy/replicaset rollups) | almost always |
kubernetesExtraMetrics | agent/cluster-collector | cadvisor + apiserver metrics pre-packaged | instead of rolling your own Prometheus scrape |
Infrastructure Explorer / Resource Catalog
For Infrastructure Explorer to populate, the following prerequisite chain must be complete:
1. `kubernetesResources` preset on the cluster-collector (enabled by default — do not disable it):
opentelemetry-cluster-collector:
presets:
kubernetesResources:
enabled: true # default; shown explicitly to prevent accidental disablement2. Dedicated Exporter with Headers: The preset automatically wires the coralogix/resource_catalog exporter with the x-coralogix-ingress: metadata-as-otlp-logs/v1 header.
3. `hostEntityEvents` Preset on the agent for node-level Infra Explorer entries (enabled by default — do not disable it, except on GKE Autopilot where hostMetrics is unavailable):
opentelemetry-agent:
presets:
hostEntityEvents:
enabled: true # default; must be false on GKE Autopilot (requires hostMetrics)Failure Mode: The resourcedetection/resource_catalog processor crashes on a daemonset with can't get K8s Instance Metadata; node name is empty. Do not enable kubernetesResources on the agent; it must run on the cluster-collector.
Multi-line Logs (Stack Traces)
The logsCollection preset uses a filelog receiver on /var/log/pods. Kubernetes writes logs in CRI format where every line — including individual stack trace lines — is tagged F (final). The standard partial/final recombine never triggers because it only groups P → F sequences.
To recombine multi-line entries (Java stack traces, Python tracebacks), use firstEntryRegex to detect the start of a new logical log entry:
opentelemetry-agent:
presets:
logsCollection:
multiline:
firstEntryRegex: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'See Coralogix multi-line log configuration.
Anti-Pattern: Wholesale Pipeline Overrides
Manual overrides of service.pipelines inside the otel-integration values file strip the default receivers and processors that populate Coralogix dashboards and Infra Explorer.
Symptom: Dashboards populate on one cluster but remain empty on another, despite identical chart versions. Resolution: Remove the custom service.pipelines block. Use presets.<feature>.enabled and extraProcessors / extraReceivers hooks. Wholesale overrides break the resource/metadata processor that sets cx.agent.type, silently breaking chart upgrades and correlation rules.
ECS EC2 (Linux): CDOT daemonset
Coralogix's ECS-on-EC2 integration runs the Coralogix Distribution of OpenTelemetry (CDOT) as an ECS service deployed as a Daemon (one task per EC2 node). Apps in other tasks send OTLP to the host IP of their EC2 node.
- Image:
coralogixrepo/coralogix-otel-collector:<version>— the Coralogix Distribution of OpenTelemetry (CDOT), a fork ofotel/opentelemetry-collector-contribwith extraecsattributes+ log-routing components. - Deployment shape: ECS Service with
schedulingStrategy: DAEMON, host network mode,awsvpcis not used. - Not Fargate. Fargate can't run daemonsets — see
setup-ecs-fargate.md.
Contents
- App-to-collector routing through the EC2 host IP
- Environment variables and task definition shape
- Why to omit the
ecsdetector in daemonset mode - Typical receivers, processors, and pipelines
- IAM requirements
- When to choose ECS EC2 over Fargate
- Operational notes and key facts
App → Collector routing
Because the collector uses host network, it binds to :4317 and :4318 on the node's private IP. Applications need to send OTLP to the node IP, not localhost, unless the app runs in the same network namespace as the collector (rare — bridge-mode tasks still need the host IP).
The common pattern is pulling the instance IP from the AWS metadata service at task startup and passing it as OTEL_EXPORTER_OTLP_ENDPOINT:
INSTANCE_IP=$(curl -sf http://169.254.169.254/latest/meta-data/local-ipv4)
export OTEL_EXPORTER_OTLP_ENDPOINT="http://${INSTANCE_IP}:4317"Bake that into the app container's entrypoint or task-definition command. Don't hardcode localhost — it only works for sidecar-mode (which ECS-EC2 is not).
Environment variables
The CDOT image reads these. They are passed as container environment entries in the task definition, or pulled from AWS Secrets Manager:
| Variable | Purpose | Example |
|---|---|---|
CX_DOMAIN | Coralogix region domain (no URL, no prefix) | eu2.coralogix.com |
CX_PRIVATE_KEY | Send-Your-Data API key | <secret> — pull from Secrets Manager |
CX_APPLICATION_NAME | default application_name for the exporter | my-ecs-app |
OTEL_CONFIG | config YAML path/URL resolved at startup | ssm://arn:aws:ssm:eu-west-1:...:parameter/otel-config, s3://bucket/path/config.yaml, or an inline param-store value |
OTEL_CONFIG accepts the standard OTel collector env-provider syntax. Most users store the config in SSM Parameter Store and reference the parameter ARN.
Minimum task definition (abbreviated)
{
"family": "coralogix-otel-collector",
"networkMode": "host",
"requiresCompatibilities": ["EC2"],
"containerDefinitions": [
{
"name": "otel-collector",
"image": "coralogixrepo/coralogix-otel-collector:v0.5.0",
"essential": true,
"command": ["--config", "env:OTEL_CONFIG"],
"environment": [
{ "name": "CX_DOMAIN", "value": "eu2.coralogix.com" },
{ "name": "CX_APPLICATION_NAME", "value": "my-ecs-app" }
],
"secrets": [
{ "name": "CX_PRIVATE_KEY", "valueFrom": "arn:aws:secretsmanager:...:coralogix-private-key" },
{ "name": "OTEL_CONFIG", "valueFrom": "arn:aws:ssm:...:parameter/otel-config" }
]
}
]
}The ECS Service runs this task definition with schedulingStrategy: DAEMON. The Auto Scaling Group maintains EC2 capacity. CloudWatch Logs captures the collector's own stdout/stderr.
Disable the ecs detector in resourcedetection
The single most common misconfig on this integration.
If you enable the ecs detector in a daemonset-mode collector, resourcedetection pulls the ECS Task Metadata endpoint — but from the collector's own task, not the application task. The collector then stamps its own aws.ecs.container.id onto every log/metric/span, overwriting the real container ID from filelog or the app's OTLP.
processors:
resourcedetection:
detectors: [env, ec2, system] # NOT "ecs" — omit it for daemonset mode
timeout: 2s
override: falseThe ecsattributes/container-logs processor handles per-container attribution correctly from the Docker socket — that's what you want for logs. The ecs detector is appropriate only for sidecar mode, where the collector shares its task with the app.
Receivers and pipelines (typical)
receivers:
filelog/docker:
include: ["/var/lib/docker/containers/*/*.log"]
start_at: end
operators:
- type: json_parser
parse_from: body
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
awsecscontainermetrics: # task-metadata endpoint, container-level metrics
hostmetrics:
collection_interval: 60s
scrapers:
cpu: {}
memory: {}
filesystem: {}
network: {}
processors:
memory_limiter:
check_interval: 2s
limit_mib: 512
ecsattributes/container-logs: # in-house CDOT component — attribute logs to correct container
resourcedetection:
detectors: [env, ec2, system] # no "ecs"
batch:
exporters:
coralogix:
domain: "${env:CX_DOMAIN}"
private_key: "${env:CX_PRIVATE_KEY}"
application_name: "${env:CX_APPLICATION_NAME}"
subsystem_name_attributes: ["docker.container.name"]
service:
pipelines:
logs:
receivers: [filelog/docker]
processors: [memory_limiter, ecsattributes/container-logs, resourcedetection, batch]
exporters: [coralogix]
metrics:
receivers: [awsecscontainermetrics, hostmetrics, otlp]
processors: [memory_limiter, resourcedetection, batch]
exporters: [coralogix]
traces:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, batch]
exporters: [coralogix]IAM
The task execution role needs:
secretsmanager:GetSecretValueon the CX_PRIVATE_KEY secretssm:GetParameter/ssm:GetParameterson the OTEL_CONFIG parameterlogs:CreateLogStream/logs:PutLogEventsif using CloudWatch Logs for the collector's own logs
No AWS service permissions are needed by the collector at runtime for direct-to-Coralogix — the collector doesn't talk to AWS APIs (it only talks to Coralogix). The task role is therefore minimal; users who insist on a broad task role should be pushed back against.
When to choose EC2 over Fargate
| You have… | Recommend |
|---|---|
| existing EC2 ECS cluster, want one collector per node | ECS EC2 daemonset (this file) |
| new workload, want managed capacity, lower ops | ECS Fargate sidecar (setup-ecs-fargate.md) |
| need host-level metrics, hostmetrics receiver, Docker socket access | ECS EC2 — Fargate can't |
| Windows workloads | ECS EC2 Windows (deployment-index callout) |
Gotchas
- Bridge-mode task networking still needs host IP. Even if the app task uses bridge networking, OTLP still has to reach
http://<INSTANCE_IP>:4317.localhostdoes not work across container network namespaces on the same host. - Service/Daemon scheduling strategy matters. Use
schedulingStrategy: DAEMON. Without it, ECS runs one task total, not one per node. - FireLens+fluentbit+Firehose is a different flow — not OTel at all. If a user has that shape, ask why they aren't using this integration; the answer is usually legacy or a feature gap, not a preference.
- Old CloudFormation template marketing language calls CDOT a "Coralogix Daemon." CDOT is a distribution of the OTel Collector — still OpenTelemetry-based. The in-house components are narrow (ecsattributes/container-logs and a log-routing shim). Users asking "is this proprietary?" deserve a precise answer: OTel Contrib + two components, no fork of the core.
Key Facts
- Apps send OTLP to node IP `:4317`, not localhost. Pull the IP from
http://169.254.169.254/latest/meta-data/local-ipv4at startup. - Omit `ecs` from `resourcedetection.detectors` on a daemonset. It stamps the collector's own container ID onto everything.
- Config lives in SSM or S3. Set
OTEL_CONFIGto thessm://ors3://URI,command: ["--config", "env:OTEL_CONFIG"]. - CDOT image is OTel-Contrib + two in-house components (ecsattributes/container-logs and log routing). Not a fork of the core — just a repackage with extras.
- Task role is minimal for direct-to-Coralogix. Execution role needs Secrets/SSM; runtime role needs nothing.
- `ecsattributes/container-logs` (CDOT component) does per-container log attribution. That's what you want for container logs — not the
ecsdetector.