Opentelemetry Semantic Conventions

ai-center-genai

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nWe can see traces from our OpenAI calls in Coralogix, but AI Center shows no LLM rows. A representative span looks like this: name=\"POST /v1/chat/completions gpt-4o-mini\", kind=CLIENT, attributes={http.request.method=\"POST\", server.address=\"api.openai.com\", url.full=\"https://api.openai.com/v1/chat/completions\", llm.model_name=\"gpt-4o-mini\"}. There are no gen_ai.* attributes on the span. What does AI Center actually look for?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-gen-ai-system",
      "description": "The response contains \"gen_ai.system\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-gen-ai-provider-name",
      "description": "The response contains \"gen_ai.provider.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-gen-ai-input-messages",
      "description": "The response contains \"gen_ai.input.messages\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "trace-span",
      "description": "The response matches the pattern: (?i)(trace|span)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-ai-center-detects-genai-spans-f",
      "description": "Explains that AI Center detects GenAI spans from legacy gen_ai.system or newer gen_ai.provider.name / gen_ai.input.messages attributes on trace spans, not from generic HTTP span names alone. It should say logs or metrics alone do not create AI Center rows. FAIL if it says model names in span names are sufficient.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

ai-center-genai

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nA customer routes LLM traffic through LiteLLM. Coralogix traces are present, but AI Center is blank. The proxy span is name=\"POST /chat/completions\", attributes={http.route=\"/chat/completions\", http.request.method=\"POST\", server.address=\"litellm-proxy\", model=\"claude-3-5-sonnet\"}, and it does not have gen_ai.provider.name, gen_ai.system, or gen_ai.input.messages. Should semconv fix this in the collector, or is the proxy/instrumentation missing something?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-gen-ai-provider-name",
      "description": "The response contains \"gen_ai.provider.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-gen-ai-input-messages",
      "description": "The response contains \"gen_ai.input.messages\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-otel-semconv-stability-opt-in",
      "description": "The response contains \"OTEL_SEMCONV_STABILITY_OPT_IN\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "instrument-proxy-custom",
      "description": "The response matches the pattern: (?i)(instrument|proxy|custom)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-the-likely-gap-as-proxy-or-custom",
      "description": "Identifies the likely gap as proxy or custom instrumentation not emitting GenAI semantic-convention span attributes. It must mention checking OTEL_SEMCONV_STABILITY_OPT_IN for newer gen_ai.provider.name / gen_ai.input.messages emission, may mention message-content capture, and should hand SDK/proxy setup to instrumentation owners. FAIL if it claims a collector can infer all required GenAI attributes from generic traces without instrumentation support.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

spanmetrics-cardinality

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nAPM metrics cardinality is huge. Span Metrics dimensions include dimensions: [service.name, span.name, url.full, k8s.pod.name, k8s.pod.ip, http.method]. The service also has routes such as /checkout/{cart_id} available on spans. Which labels should be avoided by default, and what should HTTP grouping use instead?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-url-full",
      "description": "The response contains \"url.full\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-http-route",
      "description": "The response contains \"http.route\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-k8s-pod-name",
      "description": "The response contains \"k8s.pod.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-k8s-pod-ip",
      "description": "The response contains \"k8s.pod.ip\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "cardinality-pod-churn",
      "description": "The response matches the pattern: (?i)(cardinality|pod churn|high[- ]cardinality|unbounded|avoid|not raw|pod IP|pod name churn)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "recommends-avoiding-url-full-k8s-pod-ip-and-b",
      "description": "Recommends avoiding url.full, k8s.pod.ip, and blanket k8s.pod.name dimensions for Span Metrics because of cardinality; HTTP grouping should prefer templated http.route. It may note k8s.pod.name is allowed only for targeted single-writer collision cases. FAIL if it recommends url.full as the standard APM metric dimension.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

instrumentation-identity

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nKubernetes auto-instrumentation injected our pods and traces are arriving, but APM service discovery is empty or shows unknown_service. Which resource attribute is probably missing, and how should it be supplied?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-service-name",
      "description": "The response contains \"service.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-otel-service-name",
      "description": "The response matches the pattern: (OTEL_SERVICE_NAME|OTEL_RESOURCE_ATTRIBUTES)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-resource",
      "description": "The response contains \"resource\" (case-insensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-missing-resource-scope-service-nam",
      "description": "Identifies missing resource-scope service.name as the likely issue and recommends setting OTEL_SERVICE_NAME or equivalent resource attributes during auto-injection. FAIL if it recommends adding service.name only as a span attribute.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

infra-explorer-cloud-platform

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nAfter a customer upgraded their OTel Collector, Azure hosts in Infrastructure Explorer started grouping incorrectly. The raw resource attributes still include cloud.platform, but the value changed from azure_vm to azure.vm and azure_aks to azure.aks. Is this an attribute rename, and where should we fix it?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-cloud-platform",
      "description": "The response contains \"cloud.platform\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-azure-vm-1",
      "description": "The response contains \"azure_vm\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-azure-vm-2",
      "description": "The response contains \"azure.vm\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-azure-aks-1",
      "description": "The response contains \"azure_aks\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-azure-aks-2",
      "description": "The response contains \"azure.aks\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "value-enum",
      "description": "The response matches the pattern: (?i)(value|enum)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "support-accept-handle",
      "description": "The response matches the pattern: (?i)(support|accept|handle|map|match).*(both|old.*new|underscore.*dot|dot.*underscore)|both (values|families|forms)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-this-as-a-cloud-platform-value-enu",
      "description": "Identifies this as a cloud.platform VALUE / enum rename, not an attribute-name rename. Recommends Coralogix-side support for both azure_vm / azure.vm and azure_aks / azure.aks in Infra Explorer / Resource Catalog identity mapping or filters, with a collector resource transform only as a temporary compatibility bridge before Coralogix export / Resource Catalog correlation. FAIL if it tells the customer the attribute is missing or only suggests renaming the key cloud.platform.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

custom-metrics-labels

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nCustom metrics now have both service.instance.id and service_instance_id labels, with different values from app instrumentation and the Target Allocator. Coralogix grouping is split. Is this just duplicate spelling, or a semantic problem?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-service-instance-id-1",
      "description": "The response contains \"service.instance.id\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-service-instance-id-2",
      "description": "The response contains \"service_instance_id\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "same-concept-conflicting",
      "description": "The response matches the pattern: (?i)(same concept|conflicting|align|canonical|split|semantic problem)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "prometheus-normalization",
      "description": "The response matches the pattern: (?i)(normaliz|Prometheus|underscore)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-carrying-service-instance-id-an",
      "description": "Explains that Prometheus/Target Allocator metric paths normalize dotted OTel attribute names into underscore labels, so service_instance_id is the normalized spelling of service.instance.id rather than an unrelated dimension. It should say that carrying both spellings with different values splits metric series identity, and recommend choosing one canonical service or writer identity and aligning/removing/avoiding the duplicate source. FAIL if it omits the normalization relationship, only says to choose a canonical identity, or treats the two labels as unrelated dimensions to keep.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

custom-metrics-temporality

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nA customer sends OTLP delta Sum metrics from many app instances with identical labels. Docs say Coralogix supports delta temporality, but totals are wrong. Should we convert every metric to cumulative, or what else matters?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-delta",
      "description": "The response contains \"delta\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cumulative",
      "description": "The response contains \"cumulative\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "deltatocumulative-convert-cumulative",
      "description": "The response matches the pattern: (?i)(deltatocumulative|convert.*cumulative|cumulative.*convert)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "writer-unique-identity",
      "description": "The response matches the pattern: (?i)(writer[- ]unique|writer identity|service\\.instance\\.id|process|pod|host)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "aggregate-sum",
      "description": "The response matches the pattern: (?i)(aggregate|sum)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "multi-writer-multiple",
      "description": "The response matches the pattern: (?i)(multi[- ]writer|multiple writers|many writers|same labels|identical labels|writer[- ]unique|stable writer)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-delta-support-does-not-remove-m",
      "description": "Explains that delta support does not remove multi-writer identity and aggregation requirements. It should recommend either preserving a stable writer-unique identity when writers should stay separate, or using a real metrics aggregation step that sums the intended low-cardinality streams before converting to cumulative / deltatocumulative when the desired output is one cumulative total. FAIL if it says temporality alone is the root cause, ignores multiple writers with identical labels, or claims groupbyattrs itself aggregates colliding datapoints.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

scope-deflection

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nMy new service needs to report cx.application.name = \"platform\" and cx.subsystem.name = \"ingest\" and I want to set them via an environment variable. How do I do that?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-otel-resource-attributes",
      "description": "The response contains \"OTEL_RESOURCE_ATTRIBUTES\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cx-application-name",
      "description": "The response contains \"cx.application.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cx-subsystem-name",
      "description": "The response contains \"cx.subsystem.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "resource-attribute-scope",
      "description": "The response matches the pattern: (?i)(resource attribute|resource[- ]scope|resource key)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "core",
      "description": "The response matches the pattern: (?i)(\\bcore\\b|OTEL_RESOURCE_ATTRIBUTES)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "gives-correct-resource-scope-guidance-for-set",
      "description": "Gives correct resource-scope guidance for setting cx.application.name / cx.subsystem.name, for example using OTEL_RESOURCE_ATTRIBUTES and configuring Coralogix routing to read those resource keys. It should name core, or core routing, as the responsible owner for detailed Coralogix routing configuration when discussing ownership. PASS for a concise env-var answer that uses OTEL_RESOURCE_ATTRIBUTES at resource scope and does not claim semconv owns detailed routing. FAIL if the response claims cx.* are OpenTelemetry-standard attributes (they are Coralogix proprietary), omits resource scope, claims semconv owns detailed routing, or instructs the user to set them at span / log scope.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

database-monitoring

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nA customer's APM Database Catalog page is empty for their MySQL service even though traces are flowing. They use auto-instrumentation from the OTel Java agent and haven't customised anything. What attributes does Database Monitoring need on each database client span, and which ones is the agent likely producing already?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-db-system",
      "description": "The response contains \"db.system\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-name",
      "description": "The response contains \"db.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-operation",
      "description": "The response contains \"db.operation\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-statement",
      "description": "The response contains \"db.statement\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-net-peer-name",
      "description": "The response contains \"net.peer.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "lists-all-five-coralogix-required-target-attr",
      "description": "Lists all five Coralogix-required target attributes (db.system, db.name, db.operation, db.statement, net.peer.name) and explains that Database Monitoring needs every one. It may separately mention source/newer names the OTel Java agent may emit, such as db.system.name, db.namespace, db.operation.name, db.query.text, and server.address / network.peer.address. PASS if it lists the five target attributes and names those likely agent-produced source names; bridge/mirror wording is helpful but not required unless the response presents the source names as sufficient replacements for the target set. FAIL only if the response omits one of the five target attributes, treats the source names as sufficient replacements without mirroring/bridging, or claims Database Monitoring requires only a different attribute set.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

spanmetrics-dimensions

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nA Helm user put the DB compatibility transform under spanMetrics.dbMetrics.transformStatements. Now db_calls_total has db_namespace populated from db.name, but normal Span Metrics calls_total still has db_system populated and db_namespace blank for database spans. Where should this transform live, and what should the pre-spanmetrics bridge do when db.name and db.namespace are both missing?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-spanmetrics-transformstatements",
      "description": "The response contains \"spanMetrics.transformStatements\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-spanmetrics-dbmetrics-transformstatements",
      "description": "The response contains \"spanMetrics.dbMetrics.transformStatements\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-calls-total",
      "description": "The response matches the pattern: (db_calls_total|DB-specific|database-specific)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-calls-total",
      "description": "The response matches the pattern: (calls_total|normal Span Metrics|general Span Metrics)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-namespace-1",
      "description": "The response contains \"db_namespace\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-namespace-2",
      "description": "The response contains \"db.namespace\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-name",
      "description": "The response contains \"db.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-server-address",
      "description": "The response contains \"server.address\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-network-peer-name",
      "description": "The response contains \"network.peer.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-net-peer-name",
      "description": "The response contains \"net.peer.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-network-peer-address",
      "description": "The response contains \"network.peer.address\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-system",
      "description": "The response contains \"db.system\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "top-level-not",
      "description": "The response matches the pattern: (?i)(top[- ]level|not.*dbMetrics|before spanmetrics|traces pipeline|fallback|first available|chain|when.*missing)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-the-helm-transform-belongs-unde",
      "description": "Explains that the Helm transform belongs under top-level spanMetrics.transformStatements / the traces pipeline before spanmetrics consumes spans, not only under spanMetrics.dbMetrics.transformStatements. It should connect that placement to the symptom where DB-specific metrics get db_namespace while normal/general Span Metrics stay blank. The bridge must populate db.namespace using db.name first when available, and when db.name/db.namespace are missing it should use endpoint/system fallbacks such as server.address, network.peer.name / net.peer.name, network.peer.address, and finally db.system. FAIL if it only maps db.name -> db.namespace and leaves db_namespace blank for endpoint-only database spans, treats this as a chart-default bug instead of misconfiguration, or puts the fix only under dbMetrics or after spanmetrics in the metrics pipeline.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

db-semconv-migration

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nAfter upgrading a customer's OpenTelemetry SQL instrumentation, their spans now have db.system.name, db.namespace, db.operation.name, and db.query.text, plus network.peer.address for the database endpoint, but APM Database Catalog stopped showing databases and spanmetrics has an empty db_system label. Should we tell them to downgrade, or what compatibility bridge is needed for Coralogix?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-db-system-name",
      "description": "The response contains \"db.system.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-system",
      "description": "The response contains \"db.system\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-query-text",
      "description": "The response contains \"db.query.text\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-statement",
      "description": "The response contains \"db.statement\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-network-peer-address",
      "description": "The response contains \"network.peer.address\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-net-peer-name",
      "description": "The response contains \"net.peer.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "bridge-mirror-transform",
      "description": "The response matches the pattern: (?i)(bridge|mirror|transform|compat)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-this-is-a-current-coralogix-com",
      "description": "Explains that this is a current Coralogix compatibility issue: modern/stable DB semconv attributes should be kept but mirrored into the Coralogix-supported names before APM / Span Metrics consume the spans. Must include at least db.system.name -> db.system and db.query.text -> db.statement, and should include db.namespace -> db.name, db.operation.name -> db.operation, and network.peer.address or server.address -> net.peer.name. FAIL if it tells the customer to downgrade the SDK, delete the stable attributes, or claims Coralogix Database Catalog already fully reads only the stable names.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

db-semconv-migration

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nAfter a .NET SqlClient upgrade, database spans use db.system.name=microsoft.sql_server. We bridge db.system.name to db.system, but the UI still expects the old MSSQL value. Is this a name rename or a value compatibility problem?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-db-system-name",
      "description": "The response contains \"db.system.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-db-system",
      "description": "The response contains \"db.system\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-microsoft-sql-server",
      "description": "The response matches the pattern: (microsoft\\.sql_server|mssql|MSSQL|sql_server)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "value-enum-compat",
      "description": "The response matches the pattern: (?i)(value|enum|compat)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-the-value-compatibility-issue-afte",
      "description": "Identifies the value compatibility issue after the name bridge: db.system.name can be mirrored to db.system, but the remaining problem is enum/value compatibility, so the product must accept/map newer values such as microsoft.sql_server or the bridge must map them to the legacy value such as mssql. PASS if it distinguishes the key/name bridge from the enum/value mapping problem. FAIL if it treats the key rename alone as sufficient.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

infra-explorer-host-identity

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nEC2 logs are correlated to hosts, but host.name is the private DNS name. The customer wants the EC2 Name tag in logs and Resource Catalog. Which enrichment and permission should I check?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-host-name",
      "description": "The response contains \"host.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-ec2-describetags",
      "description": "The response contains \"ec2:DescribeTags\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "ec2-name-tag",
      "description": "The response matches the pattern: (?i)(EC2|Name tag|tag)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "resource-detection-resourcedetection",
      "description": "The response matches the pattern: (?i)(resource detection|resourcedetection|resource detector|ec2tags|ec2 detector)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-ec2-name-tag-host-identity-requ",
      "description": "Explains that EC2 Name tag host identity requires EC2 tag/resource detection in the relevant logs/resource catalog path and IAM permission such as ec2:DescribeTags. It should verify host.name is resource-scope. FAIL if it says log parsing alone should change host.name.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

apm-error-tracking

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nAPI Error Tracking has many empty HTTP status codes for Python aiohttp spans. The spans have status ERROR and exception.type events, but no http.response.status_code because the request timed out before a response existed. Is this a normal semconv attribute rename?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-http-response-status-code",
      "description": "The response contains \"http.response.status_code\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-exception-type",
      "description": "The response contains \"exception.type\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "synthetic-workaround-not",
      "description": "The response matches the pattern: (?i)(synthetic|workaround|not standard|custom|fallback signal|not.*rename)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-missing-http-response-status-co",
      "description": "Explains that missing http.response.status_code can happen when no response exists, so this is not a normal OTel attribute rename. It should treat exception.type / span error status as fallback signals; mapping them to a synthetic HTTP status is a Coralogix/product workaround rather than standard OTel semconv. FAIL if it treats exception-to-status mapping as standard OTel semconv.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

resource-attribute-routing

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nCustomer's logs have a \"subsystem\" field at the log record attribute level and the Coralogix exporter's subsystem_name_attributes is reading from resource. I want to write the OTTL transform that copies it up. What should the statement look like exactly, and which Coralogix skill owns implementing it?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "opentelemetry-ottl",
      "description": "The response matches the pattern: (?i)opentelemetry-?ottl",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-resource",
      "description": "The response contains \"resource\" (case-insensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "log-record-level",
      "description": "The response matches the pattern: (?i)(log[- ]record|record[- ]level|attributes)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "diagnoses-the-boundary-correctly-the-value-is",
      "description": "Diagnoses the boundary correctly: the value is at log-record scope and must be copied to resource scope for the exporter to read it. It must name opentelemetry-ottl as the owner for exact OTTL statement implementation. It may give a short conceptual set(...) sketch; do not fail solely because it includes a sketch when it also names opentelemetry-ottl as the owner and copies log-record data up to resource scope. The judge should not grade exact set(...) argument syntax here; exact OTTL syntax belongs to opentelemetry-ottl. FAIL if it treats this as semconv-owned OTTL syntax, copies the value in the wrong direction from resource down to the log record, tells the user to rename the field in their application, or changes exporter configuration as the primary fix.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

resource-attribute-routing

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nWe use the Coralogix exporter with subsystem_name_attributes: [\"log.file.subsystem\"], but every record arrives tagged with the static fallback subsystem instead. The \"log.file.subsystem\" attribute is definitely set in our logs \u2014 debug exporter shows it under the log record attributes. Why isn't the exporter picking it up?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-resource",
      "description": "The response contains \"resource\" (case-insensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "record-span-level",
      "description": "The response matches the pattern: (record|span).*level|log[- ]record|record[- ]level",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-that-subsystem-name-attributes-app",
      "description": "Identifies that subsystem_name_attributes / application_name_attributes on the Coralogix exporter only read RESOURCE-scope attributes. The attribute lives on the log record so the exporter cannot see it. Recommends either setting it at resource scope at the SDK / resource / resourcedetection processor, or copying it up with OTTL. FAIL if it tells the user to rename the field in their application or to change the exporter configuration.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

helm-cardinality

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nWe're using the Coralogix Kubernetes Helm chart at v0.0.202 and did not set spanMetrics.aggregationCardinalityLimit. Is the 100,000 Span Metrics cardinality cap already enabled by default?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-v0-0-203",
      "description": "The response contains \"v0.0.203\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-aggregationcardinalitylimit",
      "description": "The response contains \"aggregationCardinalityLimit\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "v0-0-202-older-chart",
      "description": "The response matches the pattern: (?i)(v0\\.0\\.202|older chart|before v0\\.0\\.203)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "set-explicit-not",
      "description": "The response matches the pattern: (?i)(set .*explicit|not enabled|disabled|0)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-the-coralogix-helm-chart-s-100",
      "description": "Explains that the Coralogix Helm chart's 100,000 default applies starting with chart v0.0.203, so chart v0.0.202 should not be assumed to have overflow protection enabled unless spanMetrics.aggregationCardinalityLimit is set explicitly. FAIL if it says all Coralogix Helm chart versions default to 100,000.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

helm-fleet-placement

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nFleet UI produced JSON with span_metrics.transform_statements, but the rendered Helm ConfigMap doesn't show the DB bridge in the normal spanmetrics traces path. The values file also has spanMetrics.dbMetrics.transformStatements. What should I verify?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-span-metrics-transform-statements",
      "description": "The response contains \"span_metrics.transform_statements\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-spanmetrics-transformstatements",
      "description": "The response contains \"spanMetrics.transformStatements\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-spanmetrics-dbmetrics-transformstatements",
      "description": "The response contains \"spanMetrics.dbMetrics.transformStatements\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "rendered-actual-configmap",
      "description": "The response matches the pattern: (?i)(rendered|actual).*(ConfigMap|collector config|collector)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-fleet-ui-json-helm-values-and-r",
      "description": "Explains that Fleet/UI JSON, Helm values, and rendered collector ConfigMap can differ, and the semconv bridge must render into the top-level spanMetrics.transformStatements / traces path before normal spanmetrics. PASS if it tells the user to verify the rendered ConfigMap and not rely only on the UI JSON or values file. FAIL if it says checking values.yaml is enough.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

scope-deflection

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nI know the DB attribute bridge belongs before spanmetrics, but I need the exact otel-integration Helm values YAML and rendered collector pipeline for spanMetrics.transformStatements. Should semconv provide the values block inline?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-spanmetrics-transformstatements",
      "description": "The response contains \"spanMetrics.transformStatements\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "opentelemetry-collector-skill",
      "description": "The response matches the pattern: (?i)(opentelemetry-?collector|collector skill|collector owner|collector workflow)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "helm-values-pipeline",
      "description": "The response matches the pattern: (?i)(Helm|values|pipeline|ConfigMap)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "says-semconv-can-identify-the-semantic-placem",
      "description": "Says semconv can identify the semantic placement \u2014 before spanmetrics under top-level spanMetrics.transformStatements \u2014 but exact Helm values, rendered ConfigMap, and collector pipeline YAML belong to opentelemetry-collector. FAIL if it answers with full Helm/collector implementation instead of handing off the exact config authoring.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

infra-explorer-hostmetrics

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nInfrastructure Explorer host CPU suddenly shows about 95% usage even though the actual usage is low. The metrics come from the OTel hostmetricsreceiver. I see docs saying system.cpu.time should have cpu.mode and cpu.logical_number, but the receiver output has state and cpu. How should we think about this?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "system-cpu-hostmetricsreceiver-cpu",
      "description": "The response matches the pattern: (?i)(system\\.cpu|hostmetricsreceiver|CPU metrics|CPU usage)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cpu-mode",
      "description": "The response contains \"cpu.mode\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cpu-logical-number",
      "description": "The response contains \"cpu.logical_number\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-state",
      "description": "The response contains \"state\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "hostmetricsreceiver-host-metrics",
      "description": "The response matches the pattern: (?i)(hostmetricsreceiver|host metrics receiver)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-this-as-a-system-cpu-metric-data-p",
      "description": "Identifies this as a system.cpu metric data-point attribute compatibility issue between current OTel system semantic conventions and hostmetricsreceiver output, not a missing host.* resource attribute. It should name the label-family mapping, such as cpu.mode / cpu.logical_number and state / cpu forms (including wait vs iowait where relevant), and explain that the backend/query/transform path must account for the emitted family. FAIL if it says k8sattributes or resource detection is the primary fix.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

http-semconv

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nI'm adding HTTP method as a metric data-point attribute for new OpenTelemetry HTTP server metrics. Should the metric use http.method or http.request.method?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-http-request-method",
      "description": "The response contains \"http.request.method\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-http-method",
      "description": "The response matches the pattern: (http\\.method|legacy|not .*http\\.method)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "stable-current-modern",
      "description": "The response matches the pattern: (?i)(stable|current|modern|new OpenTelemetry|new HTTP server metric)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "recommends-http-request-method-as-the-current",
      "description": "Recommends http.request.method as the current stable OpenTelemetry HTTP method attribute for new native HTTP metric data points. It should treat http.method as legacy for this case and may explain that Coralogix Span Metrics bridges or reads that legacy source internally for compatibility. FAIL if it recommends authoring new HTTP metrics with http.method as the primary/current attribute.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

apm-transactions

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nWe set http.route in a collector transform after export from a Node routing-controllers app, but Coralogix Transactions still grouped by the old span names. How should we think about http.route, span name, and cgx.transaction timing?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-http-route",
      "description": "The response contains \"http.route\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cgx-transaction",
      "description": "The response contains \"cgx.transaction\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "coralogixtransactionsampler-sdk-sampler",
      "description": "The response matches the pattern: (?i)(CoralogixTransactionSampler|SDK/sampler|sampler chain|transaction deriv)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "before-too-late",
      "description": "The response matches the pattern: (?i)(before|too late|ordering|timing)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-route-aware-span-name-http-rout",
      "description": "Explains that route-aware span name / http.route must exist before Coralogix transaction derivation, because collector-side rewrites after the sampler may be too late for cgx.transaction. It may hand sampler/framework setup to opentelemetry-instrumentation. FAIL if it says setting cgx.transaction manually is the fix.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

http-semconv

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nMy HTTP server spans have url.full set on them, my HTTP client spans have url.path and url.query set. APM is showing operation names with full URLs including query strings on the server side. What's wrong?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-url-full",
      "description": "The response contains \"url.full\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "server-inbound",
      "description": "The response matches the pattern: (?i)(server|inbound)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "client-outbound",
      "description": "The response matches the pattern: (?i)(client|outbound)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "calls-out-that-url-full-is-for-client-spans-o",
      "description": "Calls out that url.full is for client spans only (outbound HTTP) and that server spans should use a different URL attribute set (url.path / url.query / url.scheme / http.route \u2014 at least one of these named). Recommends swapping. FAIL only if it does not identify url.full as client-only.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

infra-explorer-k8s

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nCoralogix Infrastructure Explorer shows our hosts but the pod view is empty for our EKS cluster. Traces are flowing fine and the OTel collector is running with hostmetrics + kubeletstats receivers. What resource attributes does Infrastructure Explorer need for pods to show up, and which collector processor populates them?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-k8s-pod-name",
      "description": "The response contains \"k8s.pod.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-k8s-namespace-name",
      "description": "The response contains \"k8s.namespace.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-k8s-cluster-name",
      "description": "The response contains \"k8s.cluster.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-service-name",
      "description": "The response contains \"service.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "k8s-node-name-k8s-container-name",
      "description": "The response matches the pattern: (k8s\\.node\\.name|k8s\\.container\\.name)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "k8s-deployment-name-k8s-statefulset-name-k8s",
      "description": "The response matches the pattern: (k8s\\.deployment\\.name|k8s\\.statefulset\\.name|k8s\\.daemonset\\.name|k8s\\.job\\.name|k8s\\.cronjob\\.name|k8s\\.replicaset\\.name|workload)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-k8sattributes",
      "description": "The response contains \"k8sattributes\" (case-insensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "lists-the-resource-attributes-infrastructure",
      "description": "Lists the resource attributes Infrastructure Explorer needs for the pod view: k8s.cluster.name, k8s.namespace.name, k8s.pod.name, k8s.node.name, k8s.container.name, plus the workload-identity attribute that matches the pod's owner \u2014 any of k8s.deployment.name, k8s.statefulset.name, k8s.daemonset.name, k8s.job.name, k8s.cronjob.name, or k8s.replicaset.name. It must also name service.name as shared service identity. Recommends the k8sattributes processor and notes the attributes must be at resource scope. FAIL if the response insists on k8s.deployment.name as the only correct workload attribute, or if it suggests these attributes belong on the span.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

infra-explorer-ownership

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nI want my services in Infrastructure Explorer filterable by Environment (prod/staging/dev), Service, and Team. Which tags or labels drive these ownership filters?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-deployment-environment-name",
      "description": "The response contains \"deployment.environment.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "cx-service-app-kubernetes-io-service",
      "description": "The response matches the pattern: (cx_service|app\\.kubernetes\\.io/service|CX_SERVICE_NAME)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "cx-team-team",
      "description": "The response matches the pattern: (cx_team|\\bteam\\b)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-that-infrastructure-explorer-reads",
      "description": "Identifies that Infrastructure Explorer reads ownership from specific Kubernetes labels / cloud tags plus the documented deployment.environment.name exception, and names at least one supported key per filter \u2014 for example: Environment from any of `environment` / `cx_env` / `app.kubernetes.io/environment` / `CX_ENV_ID` / `deployment.environment.name`; Service from any of `service` / `cx_service` / `app.kubernetes.io/service` / `CX_SERVICE_NAME` (ideally noting fallback to the workload name when none are set); Team from `team` or `cx_team`. FAIL if the response asserts `service.name` is the Service ownership key (it isn't \u2014 `service.name` is the APM Service Catalog identifier; Infra Explorer ownership uses different keys), or if it makes up a reverse-DNS team attribute as the canonical answer.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

instrumentation-boundary

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nA customer installed the OpenTelemetry Javaagent for Tomcat and asks why they only see HTTP server metrics with http.request.method, http.response.status_code, and http.route, not Tomcat thread pool or JMX runtime metrics. Is this a semantic convention problem?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "http-semconv-server",
      "description": "The response matches the pattern: (?i)(HTTP semconv|HTTP server metrics|http\\.request\\.method|http\\.response\\.status_code|http\\.route)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-jmx",
      "description": "The response contains \"JMX\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-javaagent-http-semantic-convent",
      "description": "Explains that Javaagent HTTP semantic-convention metrics are not Tomcat/JMX runtime metrics, so Tomcat internals require separate JMX collection/setup such as a scraper, receiver, or instrumentation setup. PASS if it says semconv does not create Tomcat/JMX metrics and points to separate Javaagent/instrumentation/JMX setup. FAIL if it says changing HTTP semconv attributes will produce Tomcat runtime metrics.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

infra-explorer-resource-scope

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nAfter enabling reduceResourceAttributes, Datadog receiver spans only sometimes have k8s.pod.name and k8s.namespace.name. Some spans still have processTags with Kubernetes data. How should semconv triage this?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-reduceresourceattributes",
      "description": "The response contains \"reduceResourceAttributes\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-k8s-pod-name",
      "description": "The response contains \"k8s.pod.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-processtags",
      "description": "The response contains \"processTags\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-resource",
      "description": "The response contains \"resource\" (case-insensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "treats-this-as-enrichment-reduction-order-and",
      "description": "Treats this as enrichment/reduction order and scope consistency: Kubernetes metadata must become OTel resource attributes before reduction/export and must be preserved on all relevant signals. PASS if it says processTags are a separate source and do not replace resource-scope k8s.* attributes. FAIL if it says processTags alone are enough for all Coralogix resource correlation.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

logs-serverless-scope

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nFor Lambda logs, the customer wants only cx_metadata.faas_execution and not the rest of the built-in resource metadata. Are these normal log attributes I should filter with semconv, or Coralogix metadata controls?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-cx-metadata-faas-execution",
      "description": "The response contains \"cx_metadata.faas_execution\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cx-logs-metadata-enabled",
      "description": "The response contains \"CX_LOGS_METADATA_ENABLED\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cx-logs-metadata-include-trace-ref",
      "description": "The response contains \"CX_LOGS_METADATA_INCLUDE_TRACE_REF\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-cx-logs-resource-built-in-attributes",
      "description": "The response contains \"CX_LOGS_RESOURCE_BUILT_IN_ATTRIBUTES\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "resource-log-record",
      "description": "The response matches the pattern: (?i)(resource|log record|metadata)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "dashboard-correlation",
      "description": "The response matches the pattern: (?i)(dashboard|correlation)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "explains-that-cx-metadata-faas-execution-is-c",
      "description": "Explains that cx_metadata.faas_execution is Coralogix serverless metadata, not a generic OTel log attribute, and that the relevant controls include CX_LOGS_METADATA_ENABLED, CX_LOGS_METADATA_INCLUDE_TRACE_REF, and CX_LOGS_RESOURCE_BUILT_IN_ATTRIBUTES. It must also warn that removing metadata can affect serverless dashboards or trace/log correlation. FAIL if it treats this as a normal semconv rename.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

metric-cardinality

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nOur metrics backend is overwhelmed \u2014 we're seeing tens of millions of unique time series from a single counter. The labels include user.id and request_path with the full querystring. Where do those attributes belong, and what bounded label can we use for HTTP routes on metrics instead?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-span",
      "description": "The response contains \"span\" (case-insensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-http-route",
      "description": "The response contains \"http.route\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "unbounded-bounded-low",
      "description": "The response matches the pattern: (unbounded|bounded|low[- ]cardinality|high[- ]cardinality|cardinality)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "identifies-that-user-id-and-full-url-paths-ar",
      "description": "Identifies that user.id and full URL paths are high-cardinality and belong on span attributes, not metric attributes. Recommends http.route (templated) on metrics instead of url.path. FAIL if it keeps user.id on metrics or recommends a hashing trick instead of moving to spans.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

scope-deflection

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nProfiling spans? The profiles pipeline has OTel v1.10 proto changes like ProfilesDictionary and profile attribute indices, and app / subsystem metadata is not where logs and traces put it. Should the semconv skill own the full fix?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-profilesdictionary",
      "description": "The response contains \"ProfilesDictionary\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "profiles-profiling",
      "description": "The response matches the pattern: (?i)(profiles|profiling)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "boundary-hand-off",
      "description": "The response matches the pattern: (?i)(boundary|hand ?off|belongs?|owned|owner)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "says-profiling-proto-resource-enrichment-is-a",
      "description": "Says profiling proto/resource enrichment is an associated boundary: semconv can identify resource-vs-profile metadata concerns, but full OTel profiles proto handling and pipeline setup belong to the profiling or collector owner. FAIL if it invents stable Coralogix profile semconv requirements as if they were current trace/span requirements.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

scope-deflection

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nI already know a log field needs to be copied to resource scope. Can you write the exact OTTL set(...) statement and where clause for me, including context selection?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "opentelemetry-ottl",
      "description": "The response matches the pattern: (?i)opentelemetry-?ottl",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "syntax-statement-context",
      "description": "The response matches the pattern: (?i)(syntax|statement|context)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "treats-this-as-a-pure-ottl-syntax-task-and-ex",
      "description": "Treats this as a pure OTTL syntax task and explicitly defers exact statement authoring to opentelemetry-ottl. It may restate that semconv only identifies the required source and destination scopes. PASS if it says to use opentelemetry-ottl to write the exact copy-up statement, set(...) expression, context, or where clause; do not fail repetitive wording if it still clearly defers. Do not require the semconv answer to provide the exact statement. FAIL if semconv writes the full set(...) statement as the final answer or claims OTTL syntax ownership.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

infra-explorer-k8s

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nOur k8sattributes processor only gives k8s.replicaset.name for Deployment-owned pods, and we need a temporary OTTL fallback that derives k8s.deployment.name for Infrastructure Explorer across metrics, traces, and logs. Should semconv write the replace_pattern syntax, or hand that off?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "k8s-replicaset-name-replicaset",
      "description": "The response matches the pattern: (k8s\\.replicaset\\.name|ReplicaSet)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-k8s-deployment-name",
      "description": "The response contains \"k8s.deployment.name\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "opentelemetry-ottl",
      "description": "The response matches the pattern: (?i)opentelemetry-?ottl",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "resource-scope-level",
      "description": "The response matches the pattern: (?i)(resource[- ]scope|resource[- ]level|metrics.*traces.*logs|all three signals)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "diagnoses-the-semantic-shape-the-temporary-re",
      "description": "Diagnoses the semantic shape: the temporary ReplicaSet-to-Deployment fallback must run at resource scope for metrics, traces, and logs, and should not be applied to standalone ReplicaSets. It should defer exact replace_pattern capture-group syntax to opentelemetry-ottl instead of writing the final expression here. FAIL if semconv writes the exact replacement expression as the required answer, runs the transform for only one signal, or treats standalone ReplicaSets as Deployments.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

resource-catalog-lifecycle

{
  "context": "Evaluating a Coralogix support response for this user question:\n\nResource Catalog metadata has resource_ttl, interval, and otel.entity.interval. Resources are disappearing too early after switching collection mode. Which value should semconv treat as the raw collection interval versus the deletion policy?",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "mentions-resource-ttl",
      "description": "The response contains \"resource_ttl\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-interval",
      "description": "The response contains \"interval\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "mentions-otel-entity-interval",
      "description": "The response contains \"otel.entity.interval\" (case-sensitive).",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "ttl-lifecycle-delete",
      "description": "The response matches the pattern: (?i)(TTL|lifecycle|delete|expiration)",
      "max_score": 3,
      "category": "INTENT"
    },
    {
      "name": "distinguishes-raw-collection-interval-from-re",
      "description": "Distinguishes raw collection interval from resource deletion TTL: interval / otel.entity.interval describe collection cadence, while resource_ttl is compatibility/lifecycle metadata and downstream Resource Catalog owners may apply buffering. FAIL if it treats all three fields as equivalent.",
      "max_score": 2,
      "category": "RUBRIC"
    }
  ]
}

scope-deflection