Dt Obs Hosts

Infrastructure Hosts Skill

Monitor and manage host and process infrastructure including CPU, memory, disk, network, and technology inventory.

When to Use This Skill

Use this skill when the user needs to:

• Inventory: "Show me all Linux hosts in AWS us-east-1"
• Monitor: "What hosts have high CPU usage?"
• Troubleshoot: "Which processes are consuming the most memory?"
• Discover: "What databases are running in production?"
• Plan: "Track Kubernetes version distribution for upgrade planning"
• Cost: "Calculate infrastructure costs by cost center"
• Security: "Find all processes listening on port 22"
• Compliance: "Identify hosts running EOL Java versions"
• Quality: "Check data completeness for AWS hosts"
• Optimize: "Find rightsizing candidates based on utilization"

---

Cross-source join required: If the query must combine host data with logs or other

telemetry sources (e.g. "show logs from Linux hosts with their IP addresses") → also read

dt-dql-essentials/references/smartscape-topology-navigation.md before writing the query.

---

Core Concepts

Entities

• HOST - Physical or virtual machines (cloud or on-premise)
• PROCESS - Running processes and process groups
• CONTAINER - Kubernetes containers
• NETWORK_INTERFACE - Host network interfaces
• DISK - Host disk volumes

Metrics Categories

1. Host Metrics - dt.host.cpu.*, dt.host.memory.*, dt.host.disk.*, dt.host.net.* 2. Process Metrics - dt.process.cpu.*, dt.process.memory.*, dt.process.io.*, dt.process.network.* 3. Inventory - OS type, cloud provider, technology stack, versions 4. Cost - dt.cost.costcenter, dt.cost.product 5. Quality - Metadata completeness, version compliance

Alert Thresholds

• CPU/Memory/Disk: 80% warning, 90% critical
• Network: >70% high, >85% saturated
• Disk Latency: >20ms bottleneck
• Network Errors: Drop rate >1%, error rate >0.1%
• Swap: >30% warning, >50% critical

---

Key Workflows

1. Host Discovery and Classification

Discover hosts, classify by OS/cloud, inventory resources.

smartscapeNodes "HOST"
| fieldsAdd os.type, cloud.provider, host.logical.cpu.cores, host.physical.memory
| summarize host_count = count(), by: {os.type, cloud.provider}
| sort host_count desc

OS Types: LINUX, WINDOWS, AIX, SOLARIS, ZOS

→ For cloud-specific attributes, see references/inventory-discovery.md

2. Resource Utilization Monitoring

Monitor CPU, memory, disk, network across hosts.

timeseries {
  cpu = avg(dt.host.cpu.usage),
  memory = avg(dt.host.memory.usage),
  disk = avg(dt.host.disk.used.percent)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(cpu) > 80 or arrayAvg(memory) > 80
| sort arrayAvg(cpu) desc

High utilization threshold: 80% warning, 90% critical

Key CPU Metrics:

• dt.host.cpu.usage — Total CPU utilization (0-100%)
• dt.host.cpu.idle — CPU idle time (inverse of usage; useful for anomaly detection)
• dt.host.cpu.user — CPU time in user mode
• dt.host.cpu.system — CPU time in kernel mode
• dt.host.cpu.iowait — CPU waiting for I/O (Linux only)

→ For detailed CPU analysis, see references/host-metrics.md → For memory breakdown, see references/host-metrics.md

Disk Free Space — Find Hosts with Most/Least Free Disk

timeseries disk_used_pct = avg(dt.host.disk.used.percent), by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| fieldsAdd avg_disk_used = arrayAvg(disk_used_pct),
    free_pct = 100 - arrayAvg(disk_used_pct)
| sort free_pct desc
| limit 10

3. Process Resource Analysis

Identify top resource consumers at process level.

timeseries {
  cpu = avg(dt.process.cpu.usage),
  memory = avg(dt.process.memory.usage)
}, by: {dt.smartscape.process}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(cpu) > 50
| sort arrayAvg(cpu) desc
| limit 20

→ For process I/O analysis, see references/process-monitoring.md → For process network metrics, see references/process-monitoring.md

4. Technology Stack Inventory

Discover and track software technologies and versions.

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies
| expand tech = process.software_technologies
| fieldsAdd tech_type = tech[type], tech_version = tech[version]
| summarize process_count = count(), by: {tech_type, tech_version}
| sort process_count desc

Common Technologies: Java, Node.js, Python, .NET, databases, web servers, messaging systems

→ For version compliance checks, see references/inventory-discovery.md

5. Service Discovery via Ports

Map listening ports to services for security and inventory.

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| filter isNotNull(process.listen_ports) and arraySize(process.listen_ports) > 0
| expand listen_port = process.listen_ports
| summarize process_count = count(), by: {listen_port, dt.process_group.detected_name}
| sort toLong(listen_port) asc
| limit 50

Well-known ports: 80 (HTTP), 443 (HTTPS), 22 (SSH), 3306 (MySQL), 5432 (PostgreSQL)

→ For comprehensive port mapping, see references/inventory-discovery.md

6. Container and Kubernetes Monitoring

Track container distribution and K8s workload types.

smartscapeNodes "CONTAINER"
| fieldsAdd k8s.cluster.name, k8s.namespace.name, k8s.workload.kind
| summarize container_count = count(), by: {k8s.cluster.name, k8s.workload.kind}
| sort k8s.cluster.name, container_count desc

Workload Types: deployment, daemonset, statefulset, job, cronjob

Note: Container image names/versions NOT available in smartscape.

→ For K8s version tracking, see references/container-monitoring.md → For container lifecycle, see references/container-monitoring.md

7. Cost Attribution and Chargeback

Calculate infrastructure costs by cost center.

smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter, host.logical.cpu.cores, host.physical.memory
| filter isNotNull(dt.cost.costcenter)
| fieldsAdd memory_gb = toDouble(host.physical.memory) / 1024 / 1024 / 1024
| summarize 
    host_count = count(),
    total_cores = sum(toLong(host.logical.cpu.cores)),
    total_memory_gb = sum(memory_gb),
    by: {dt.cost.costcenter}
| sort total_cores desc

→ For product-level cost tracking, see references/inventory-discovery.md

8. Infrastructure Health Correlation

Correlate host and process metrics for cross-layer analysis.

timeseries {
  host_cpu = avg(dt.host.cpu.usage),
  host_memory = avg(dt.host.memory.usage),
  process_cpu = avg(dt.process.cpu.usage)
}, by: {dt.smartscape.host, dt.smartscape.process}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(host_cpu) > 70
| sort arrayAvg(host_cpu) desc

Health scoring: Critical if any resource >90%, warning if >80%

→ For multi-resource saturation detection, see references/host-metrics.md

---

Response Construction

When the user asks for data retrieval or a DQL query (e.g., "show me top hosts by CPU"), include the DQL query in the response alongside the results. Users want to see and reuse the query — it is the deliverable, not just a means to get results.

When the user asks for analysis (anomaly detection, forecasting, seasonality), the analysis results are the deliverable. Focus on presenting findings clearly:

• Prioritize metric-level findings over data collection artifacts. If an analysis

tool reports data gaps alongside actual anomalies, lead with the metric behavior the user asked about and mention gaps only as supplementary context.

• Include host names (not just IDs) using getNodeName(dt.smartscape.host) or the

get-entity-name tool.

• State the timeframe analyzed and the tools/parameters used.

---

Analytical Workflows

Host metric queries often serve as inputs to analytical tools (anomaly detection, forecasting, seasonality analysis). This skill helps construct the right DQL query; the actual analysis is performed by dedicated tools.

Anomaly Detection and Pattern Analysis

When users ask about "unusual behavior", "anomalies", "spikes", or "sudden changes" in host metrics, the workflow is:

1. Construct the timeseries query using this skill's patterns 2. Pass it to the appropriate analysis tool (anomaly detector, novelty detection)

Choosing between detectors:

• `adaptive-anomaly-detector` — use when the user asks about magnitude: "spikes",

"abrupt changes", "values that went above normal", "sudden jumps". It answers "did this metric cross an unexpected threshold?" and reports alert durations and peak values.

• `timeseries-novelty-detection` — use when the user asks about behavioral change:

"unusual patterns", "something changed", "trends", "new behavior". It answers "did the shape of the signal change?" without implying a specific threshold was crossed.

Response format for anomaly results: Include both the host name (resolved via getNodeName(dt.smartscape.host) or get-entity-name) and the host entity ID alongside timestamps and values. Entity IDs alone are opaque to users; names alone prevent follow-up queries.

Novelty type selection rule: When using novelty detection, set analysisNoveltyType to only [SPIKE, CHANGE_IN_VALUES, TREND_IN_VALUES] by default. EXCLUDE GAP_WITH_MISSING_VALUES and CHANGE_IN_MISSING_VALUES unless the user explicitly asks about data gaps or monitoring coverage. Data gaps are infrastructure issues, not metric behavior anomalies — reporting them when the user asks about CPU or memory patterns is incorrect.

Queries for analysis tools should use simple timeseries format with a single aggregated metric and appropriate time range:

timeseries avg(dt.host.cpu.idle), by: {dt.smartscape.host}

timeseries avg(dt.host.memory.usage), by: {dt.smartscape.host}

Avoid adding filters or field transformations that reduce the data — the analysis tools work best with complete timeseries data.

Forecasting

When users ask to "predict", "forecast", or "estimate future" host metrics:

1. Construct the timeseries query with sufficient historical data (e.g., 7d for short-term, 30d for longer predictions) 2. Pass to the forecasting tool with the desired forecast horizon

The forecast horizon (how far ahead to predict) and the historical window (how much past data the model trains on) are independent. A request like "forecast the next 2 hours" sets the horizon to 2h — it says nothing about the lookback. Always use at least 7 days of historical data regardless of how short the forecast horizon is. Too few training data points cause the forecast model to fail and fall back to raw historical values.

timeseries avg(dt.host.cpu.usage), by: {dt.smartscape.host}

Seasonality Detection

When users ask about "seasonality", "weekly patterns", or "recurring behavior":

1. Use a longer time range (at least 14d for weekly, 30d+ for monthly) 2. Pass to the seasonal baseline anomaly detector

Response format for seasonal analysis: When presenting results, include:

• Whether seasonal anomalies were detected (yes/no)
• The analysis timeframe and parameters used
• For each affected host: host name (not just ID), timestamps of violations, violation

counts, baseline values vs actual values, and upper/lower bounds

• Organize results by host if multiple hosts are involved

Scope Boundary — Service-Level vs Host-Level Metrics

This skill covers host and process infrastructure metrics only. If the user asks about service-level metrics (request rate, response time, error rate, service calls per minute, throughput), use dt-obs-services instead — even when the question involves forecasting or anomaly detection of those metrics.

Redirect these to `dt-obs-services`: "service calls per minute", "request rate", "response time by service", "error rate by endpoint", "service throughput forecast".

---

Common Query Patterns

Pattern 1: Smartscape Discovery

Use smartscapeNodes to discover and classify entities.

smartscapeNodes "HOST"
| fieldsAdd <attributes>
| filter <conditions>
| summarize <aggregations>

Pattern 2: Timeseries Performance

Use timeseries to analyze metrics over time.

timeseries metric = avg(dt.host.<metric>), by: {dt.smartscape.host}
| fieldsAdd <calculations>
| filter <thresholds>

Pattern 3: Cross-Layer Correlation

Correlate host and process metrics.

timeseries {
  host_cpu = avg(dt.host.cpu.usage),
  process_cpu = avg(dt.process.cpu.usage)
}, by: {dt.smartscape.host, dt.smartscape.process}

Pattern 4: Entity Enrichment with Lookup

Enrich data with entity attributes. After lookup, reference fields with lookup. prefix.

timeseries cpu = avg(dt.host.cpu.usage), by: {dt.smartscape.host}
| lookup [
    smartscapeNodes HOST
    | fields id, cpuCores, memoryTotal
  ], sourceField:dt.smartscape.host, lookupField:id
| fieldsAdd cores = lookup.cpuCores, mem_gb = lookup.memoryTotal / 1024 / 1024 / 1024

---

Tags and Metadata

Important Notes

• Generic tags field is NOT populated in smartscape queries
• Use specific tag fields: tags:azure[*], tags:environment
• Use custom metadata: host.custom.metadata[*]

Available Tags

• Azure Tags: tags:azure[dt_owner_team], tags:azure[dt_cloudcost_capability]
• Environment: tags:environment
• Custom Metadata: host.custom.metadata[OperatorVersion], host.custom.metadata[Cluster]
• Cost: dt.cost.costcenter, dt.cost.product

→ For complete tag reference, see references/inventory-discovery.md

---

Cloud-Specific Attributes

AWS

• cloud.provider == "aws"
• aws.region, aws.availability_zone, aws.account.id
• aws.resource.id, aws.resource.name
• aws.state (running, stopped, terminated)

Azure

• cloud.provider == "azure"
• azure.location, azure.subscription, azure.resource.group
• azure.status, azure.provisioning_state
• azure.resource.sku.name (VM size)

Kubernetes

• k8s.cluster.name, k8s.cluster.uid
• k8s.namespace.name, k8s.node.name, k8s.pod.name
• k8s.workload.name, k8s.workload.kind

→ For multi-cloud analysis, see references/inventory-discovery.md

---

Best Practices

1. Use percentiles (p95, p99) for latency; max() for limits; avg() for trends 2. Set multi-level thresholds (warning 80%, critical 90%) 3. Filter early in the pipeline; limit results with | limit N 4. Aggregate before enrichment (lookup) 5. Use getNodeName(dt.smartscape.host) for human-readable host names; getNodeName(dt.smartscape.process) for processes 6. Convert bytes to GB: / 1024 / 1024 / 1024; round with round(value, decimals: 1)

Time windows: Real-time: 5-15 min | Trends: 1-7 days | Capacity planning: 30-90 days

Limitations

• dt.host.cpu.iowait available on Linux only
• Generic tags field NOT populated in smartscape (use specific tag namespaces)
• Container image names NOT available in smartscape

---

Troubleshooting

---

When to Load References

This skill uses progressive disclosure. Start here for 80% of use cases. Load reference files for detailed specifications when needed.

Load host-metrics.md when:

• Analyzing CPU component breakdown (user, system, iowait, steal)
• Investigating memory pressure and swap usage
• Troubleshooting disk I/O latency
• Diagnosing network packet drops or errors

Load process-monitoring.md when:

• Analyzing process-level I/O patterns
• Investigating TCP connection quality
• Detecting resource exhaustion (file descriptors, threads)
• Tracking GC suspension time

Load container-monitoring.md when:

• Analyzing container lifecycle and churn
• Tracking Kubernetes version distribution
• Managing OneAgent operator versions
• Planning K8s cluster upgrades

Load inventory-discovery.md when:

• Performing security audits via port discovery
• Implementing cost attribution and chargeback
• Validating data quality and metadata completeness
• Managing multi-cloud infrastructure

---

References

• host-metrics.md - Detailed host CPU, memory, disk, and network monitoring
• process-monitoring.md - Process-level CPU, memory, I/O, and network analysis
• container-monitoring.md - Container inventory, Kubernetes versions, and operator management
• inventory-discovery.md - Host/process discovery, technology inventory, cost attribution, and data quality

---

Container Monitoring Reference

Container inventory, Kubernetes version management, and operator tracking with comprehensive lifecycle analysis.

Important Note: Container image names and versions are NOT available in smartscape queries. Use container names and Kubernetes workload information for identification.

---

Container Inventory

All Containers Overview

Fetch all containers with basic information:

smartscapeNodes "CONTAINER"
| fieldsAdd name, k8s.cluster.name, k8s.namespace.name, k8s.workload.name
| sort name asc
| limit 100

Returns container instances with their Kubernetes context.

Kubernetes Container Distribution

Analyze container distribution across Kubernetes clusters:

smartscapeNodes "CONTAINER"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd k8s.cluster.name, k8s.namespace.name
| summarize
    container_count = count(),
    namespaces = countDistinct(k8s.namespace.name),
    by: {k8s.cluster.name}
| sort container_count desc

Containers by Workload Type

Group containers by Kubernetes workload kind:

smartscapeNodes "CONTAINER"
| filter isNotNull(k8s.workload.kind)
| fieldsAdd k8s.workload.kind
| summarize container_count = count(), by: {k8s.workload.kind}
| sort container_count desc

Workload Types:

• daemonset: Node-level services
• deployment: Standard deployments
• statefulset: Stateful applications
• job: Batch jobs
• cronjob: Scheduled tasks
• replicaset: Replica sets

Containers by Namespace

List containers grouped by Kubernetes namespace:

smartscapeNodes "CONTAINER"
| filter isNotNull(k8s.namespace.name)
| fieldsAdd k8s.namespace.name, k8s.cluster.name
| summarize container_count = count(), by: {k8s.cluster.name, k8s.namespace.name}
| sort k8s.cluster.name, container_count desc

Container Workload Distribution

Analyze workloads across clusters:

smartscapeNodes "CONTAINER"
| filter isNotNull(k8s.workload.name)
| fieldsAdd k8s.cluster.name, k8s.workload.name, k8s.workload.kind
| summarize container_count = count(), by: {k8s.cluster.name, k8s.workload.name, k8s.workload.kind}
| sort k8s.cluster.name, container_count desc
| limit 100

Containers on Specific Nodes

Find containers running on particular Kubernetes nodes:

smartscapeNodes "CONTAINER"
| filter isNotNull(k8s.node.name)
| fieldsAdd k8s.node.name, k8s.pod.name, name
| summarize container_count = count(), by: {k8s.node.name}
| sort container_count desc

Use Case: Identify node resource distribution and imbalances.

Containers by Pod

Group containers within pods:

smartscapeNodes "CONTAINER"
| filter isNotNull(k8s.pod.name)
| fieldsAdd k8s.pod.name, k8s.namespace.name, name, k8s.container.name
| summarize container_count = count(), by: {k8s.pod.name, k8s.namespace.name}
| filter container_count > 1
| sort container_count desc

Pattern: Multi-container pods (sidecar pattern, init containers).

Container Lifetime Analysis

Analyze container age and churn:

smartscapeNodes "CONTAINER"
| fieldsAdd name, lifetime, k8s.workload.name
| fieldsAdd
    age_hours = toDuration(timeframe(lifetime[start], to: now())),
    is_active = isNull(lifetime[end])
| summarize
    total_containers = count(),
    active_containers = countIf(is_active),
    avg_age_hours = avg(age_hours),
    by: {k8s.workload.name}
| fieldsAdd avg_age_hours = round(avg_age_hours, decimals: 1)
| sort total_containers desc
| limit 20

Short-Lived Containers

Identify ephemeral containers that terminated quickly:

smartscapeNodes "CONTAINER"
| fieldsAdd name, lifetime, k8s.pod.name
| filter isNotNull(lifetime[end])
| fieldsAdd lifespan_minutes = toDuration(timeframe(from: lifetime[start], to: lifetime[end]))
| filter lifespan_minutes < 10m
| sort lifespan_minutes asc
| limit 50

Alert: Very short-lived containers may indicate crash loops or failed init containers.

Container Density per Cluster

Calculate container density across clusters:

smartscapeNodes "CONTAINER"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd k8s.cluster.name, k8s.node.name
| summarize
    container_count = count(),
    node_count = countDistinct(k8s.node.name),
    by: {k8s.cluster.name}
| fieldsAdd containers_per_node = round(toDouble(container_count) / toDouble(node_count), decimals: 1)
| sort containers_per_node desc

---

Kubernetes Versions

Kubernetes Version Distribution

List all Kubernetes versions across worker nodes:

smartscapeNodes "HOST"
| fieldsAdd process.software_technologies
| filter isNotNull(process.software_technologies)
| expand tech = process.software_technologies
| filter tech[type] == "KUBERNETES"
| fieldsAdd k8s_version = tech[version], k8s_edition = tech[edition]
| summarize host_count = count(), by: {k8s_version, k8s_edition}
| sort k8s_version desc, k8s_edition

Use Case: Identify deployed Kubernetes versions and plan upgrades.

Kubernetes Version by Cluster

Group Kubernetes versions by cluster:

smartscapeNodes "HOST"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd process.software_technologies, k8s.cluster.name
| expand tech = process.software_technologies
| filter tech[type] == "KUBERNETES"
| fieldsAdd k8s_version = tech[version]
| summarize
    host_count = count(),
    versions = collectDistinct(k8s_version),
    by: {k8s.cluster.name}
| sort k8s.cluster.name

Pattern: Each cluster should have consistent K8s versions across nodes.

Version Skew Detection

Identify clusters with version skew (mixed versions):

smartscapeNodes "HOST"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd process.software_technologies, k8s.cluster.name
| expand tech = process.software_technologies
| filter tech[type] == "KUBERNETES" and tech[edition] == "worker"
| fieldsAdd k8s_version = tech[version]
| summarize
    version_count = countDistinct(k8s_version),
    versions = collectDistinct(k8s_version),
    host_count = count(),
    by: {k8s.cluster.name}
| filter version_count > 1
| sort version_count desc, host_count desc

Alert: Version skew indicates incomplete cluster upgrades or configuration drift.

Master vs Worker Version Comparison

Compare control plane and worker node versions:

smartscapeNodes "HOST"
| fieldsAdd process.software_technologies, k8s.cluster.name
| expand tech = process.software_technologies
| filter tech[type] == "KUBERNETES"
| fieldsAdd k8s_version = tech[version], k8s_edition = tech[edition]
| summarize host_count = count(), by: {k8s.cluster.name, k8s_edition, k8s_version}
| sort k8s.cluster.name, k8s_edition, k8s_version desc

Best Practice: Control plane should be same version or one minor version ahead of workers.

Outdated Kubernetes Versions

Identify hosts running EOL or outdated K8s versions:

smartscapeNodes "HOST"
| fieldsAdd process.software_technologies, k8s.cluster.name, name
| expand tech = process.software_technologies
| filter tech[type] == "KUBERNETES"
| fieldsAdd k8s_version = tech[version]
| filter k8s_version < "1.30.0" or isNull(k8s_version)
| summarize host_count = count(), by: {k8s.cluster.name, k8s_version}
| sort k8s_version asc

Security: Older versions may have unpatched vulnerabilities.

Kubernetes Version Upgrade Candidates

List clusters eligible for upgrade:

smartscapeNodes "HOST"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd process.software_technologies, k8s.cluster.name
| expand tech = process.software_technologies
| filter tech[type] == "KUBERNETES" and tech[edition] == "worker"
| fieldsAdd k8s_version = tech[version]
| summarize
    current_version = takeFirst(k8s_version),
    host_count = count(),
    by: {k8s.cluster.name}
| filter current_version < "1.32.0"
| sort current_version asc

Planning: Prioritize clusters with older versions for upgrades.

Kubernetes Version Compliance Rate

Calculate compliance rate against target version:

smartscapeNodes "HOST"
| fieldsAdd process.software_technologies
| expand tech = process.software_technologies
| filter tech[type] == "KUBERNETES" and tech[edition] == "worker"
| fieldsAdd k8s_version = tech[version]
| fieldsAdd
    is_compliant = k8s_version >= "1.32.0"
| summarize
    total_hosts = count(),
    compliant_hosts = countIf(is_compliant),
    non_compliant = countIf(not is_compliant)
| fieldsAdd compliance_rate = round((toDouble(compliant_hosts) / toDouble(total_hosts)) * 100, decimals: 1)

---

OneAgent Operator Management

Operator Version Distribution

List all operator versions across hosts:

smartscapeNodes "HOST"
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion]
| filter isNotNull(operator_version)
| summarize host_count = count(), by: {operator_version}
| sort host_count desc

Use Case: Identify which operator versions are deployed and plan upgrades.

Operator Version by Cloud Provider

Analyze operator versions across cloud providers:

smartscapeNodes "HOST"
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion], cloud.provider
| filter isNotNull(operator_version)
| summarize host_count = count(), by: {operator_version, cloud.provider}
| sort operator_version desc, host_count desc

Pattern: Check if different cloud providers run different operator versions.

Operator Version by Cluster

Group operator versions by Kubernetes cluster:

smartscapeNodes "HOST"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion], k8s.cluster.name
| filter isNotNull(operator_version)
| summarize host_count = count(), by: {k8s.cluster.name, operator_version}
| sort k8s.cluster.name, operator_version desc

Use Case: Cluster-by-cluster upgrade planning.

Outdated Operator Versions

Identify hosts running older operator versions:

smartscapeNodes "HOST"
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion], name, k8s.cluster.name
| filter isNotNull(operator_version)
| filter operator_version != "v1.6.3" and operator_version != "v1.7.1"
| sort operator_version, k8s.cluster.name

Alert: Target specific version for upgrades (adjust filter based on latest version).

Operator Version Consistency Check

Find clusters with mixed operator versions:

smartscapeNodes "HOST"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion], k8s.cluster.name
| filter isNotNull(operator_version)
| summarize
    versions = collectDistinct(operator_version),
    version_count = countDistinct(operator_version),
    host_count = count(),
    by: {k8s.cluster.name}
| filter version_count > 1
| sort version_count desc, host_count desc

Alert: Clusters with multiple operator versions indicate incomplete rollouts.

Operator Upgrade Progress Tracking

Monitor upgrade progress by comparing versions:

smartscapeNodes "HOST"
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion]
| filter isNotNull(operator_version)
| fieldsAdd
    version_category = if(operator_version == "v1.7.1", "latest",
                      else: if(operator_version == "v1.6.3", "current",
                      else: "legacy"))
| summarize host_count = count(), by: {version_category}
| sort version_category desc

Hosts Without Operator Version

Identify hosts missing operator version metadata:

smartscapeNodes "HOST"
| filter isNotNull(k8s.cluster.name)
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion], name, k8s.cluster.name
| filter isNull(operator_version)
| sort k8s.cluster.name, name

Data Quality: Hosts in K8s clusters should have operator versions.

---

Cloud-Specific Container Analysis

AWS EKS Container Distribution

Analyze containers in AWS EKS clusters:

smartscapeNodes "CONTAINER"
| fieldsAdd k8s.cluster.name, k8s.namespace.name
| lookup [
    smartscapeNodes HOST
    | filter cloud.provider == "aws"
    | fields id, cloud.provider, aws.region
  ], sourceField:references[runs_on.host], lookupField:id
| filter isNotNull(cloud.provider)
| summarize container_count = count(), by: {k8s.cluster.name, aws.region}
| sort container_count desc

Azure AKS Container Distribution

Analyze containers in Azure AKS clusters:

smartscapeNodes "CONTAINER"
| fieldsAdd k8s.cluster.name, k8s.namespace.name
| lookup [
    smartscapeNodes HOST
    | filter cloud.provider == "azure"
    | fields id, cloud.provider, azure.location
  ], sourceField:references[runs_on.host], lookupField:id
| filter isNotNull(cloud.provider)
| summarize container_count = count(), by: {k8s.cluster.name, azure.location}
| sort container_count desc

---

Related Documentation

For host inventory and discovery, see inventory-discovery.md. For process monitoring, see process-monitoring.md. For host resource metrics, see host-metrics.md.

Host Metrics Reference

Detailed host resource monitoring including CPU, memory, disk, and network metrics with performance analysis and troubleshooting guidance.

---

CPU Monitoring

CPU Usage Overview

Average CPU utilization by host:

timeseries cpu_usage = avg(dt.host.cpu.usage), by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(cpu_usage) > 70
| sort arrayAvg(cpu_usage) desc

Interpretation: Hosts with CPU usage below 70% are excluded from results, focusing investigation on higher utilization scenarios.

Key CPU metrics:

• dt.host.cpu.usage: Total CPU utilization (0-100%)
• dt.host.cpu.user: CPU time in user mode
• dt.host.cpu.system: CPU time in kernel mode
• dt.host.cpu.idle: CPU idle time
• dt.host.cpu.iowait: CPU waiting for I/O (Linux only)

Best Practice: Alert on CPU usage at 90% with warning, critical at 95. Flag recurring offenders as right-sizing or load-redistribution candidates.

CPU Component Breakdown

Analyze CPU time distribution:

timeseries {
  user = avg(dt.host.cpu.user),
  system = avg(dt.host.cpu.system),
  iowait = avg(dt.host.cpu.iowait),
  idle = avg(dt.host.cpu.idle)
}, by: {dt.smartscape.host}, union: true
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(user) + arrayAvg(system) > 70

Interpretation:

• High user: Application processing load
• High system: Kernel operations (context switching, syscalls)
• High iowait: Disk bottleneck causing CPU idle time

Best Practice: If iowait dominates, stop tuning CPU and pivot immediately to disk latency queries — the bottleneck is storage, not compute.

CPU Steal Time Detection

Monitor CPU steal time in virtualized environments:

timeseries {
  cpu_steal = avg(dt.host.cpu.steal),
  cpu_usage = avg(dt.host.cpu.usage)
}, by: {dt.smartscape.host}, union:true
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(cpu_steal) > 5
| sort arrayAvg(cpu_steal) desc

Thresholds:

• < 5%: Normal virtualization overhead
• 5-10%: Monitor for contention
• > 10%: Hypervisor overload - migrate or scale

Best Practice: Steal > 10% means the physical host is overcommitted — escalate to the hypervisor team to migrate the VM rather than tuning the application.

System Load Analysis

Compare system load to CPU core count:

timeseries {
  load_1m = avg(dt.host.cpu.load),
  load_5m = avg(dt.host.cpu.load5m),
  load_15m = avg(dt.host.cpu.load15m)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| lookup [
    smartscapeNodes HOST
    | fieldsAdd cpuCores
  ], sourceField:dt.smartscape.host, lookupField:id, fields:{cpuCores}
| fieldsAdd
    load_per_core_1m = arrayAvg(load_1m) / toLong(cpuCores),
    load_per_core_5m = arrayAvg(load_5m) / toLong(cpuCores)
| filter load_per_core_1m > 1.0

Best Practice: Load per core > 1.0 indicates CPU saturation. Use load_1m vs load_15m divergence to distinguish a worsening trend from a transient spike before deciding whether to scale.

CPU Spike Detection

Detect sudden CPU spikes:

timeseries cpu_usage = avg(dt.host.cpu.usage), by: {dt.smartscape.host}, interval: 1m
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    min_cpu = arrayMin(cpu_usage),
    max_cpu = arrayMax(cpu_usage),
    avg_cpu = arrayAvg(cpu_usage)
| fieldsAdd spike_magnitude = max_cpu - min_cpu
| filter spike_magnitude > 50  // 50% CPU spike
| sort spike_magnitude desc

AIX-Specific Metrics

Monitor AIX entitlement and physical CPU consumption:

timeseries {
  entitlement_used_pct = avg(dt.host.cpu.entc),
  physical_consumed = avg(dt.host.cpu.physc),
  entitlement_config = avg(dt.host.cpu.ent_config)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(entitlement_used_pct) > 80

---

Memory Monitoring

Memory Usage Overview

Track memory utilization across hosts:

timeseries {
  memory_used_pct = avg(dt.host.memory.usage),
  memory_available_pct = avg(dt.host.memory.avail.percent),
  memory_used_bytes = avg(dt.host.memory.used),
  memory_avail_bytes = avg(dt.host.memory.avail.bytes)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(memory_used_pct) > 80
| sort arrayAvg(memory_used_pct) desc

Memory Metrics:

• dt.host.memory.usage: Percentage of memory used
• dt.host.memory.used: Total memory used (bytes)
• dt.host.memory.avail.bytes: Memory available without swapping
• dt.host.memory.avail.percent: Percentage of available memory

Memory Breakdown Analysis

Analyze memory components:

timeseries {
  memory_used = avg(dt.host.memory.used),
  memory_avail = avg(dt.host.memory.avail.bytes),
  memory_recl = avg(dt.host.memory.recl),
  kernel_memory = avg(dt.host.memory.kernel)
}, by: {dt.smartscape.host}, union:true
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| lookup [
    smartscapeNodes HOST
    | fields id, memoryTotal
  ], sourceField:dt.smartscape.host, lookupField:id

Component Explanation:

• memory_used: Active application memory
• memory_recl: Reclaimable memory (caches, buffers)
• kernel_memory: Memory used by kernel

Swap Usage Monitoring

Monitor swap usage and memory pressure:

timeseries {
  swap_used = avg(dt.host.memory.swap.used),
  swap_total = avg(dt.host.memory.swap.total),
  swap_avail = avg(dt.host.memory.swap.avail)
}, by: {dt.smartscape.host}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    swap_used_pct = (arrayAvg(swap_used) / arrayAvg(swap_total)) * 100
| filter swap_used_pct > 30
| sort swap_used_pct desc

Alert Thresholds:

• < 30%: Normal swap usage
• 30-50%: Monitor for memory pressure
• > 50%: Critical - insufficient RAM

Page Fault Analysis

Monitor page fault rates:

timeseries page_faults = avg(dt.host.memory.avail.pfps), by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(page_faults) > 1000
| sort arrayAvg(page_faults) desc

High page faults (>1000) indicate memory paging activity affecting performance.

Memory Pressure Identification

Identify hosts under memory pressure:

timeseries {
  memory_usage = avg(dt.host.memory.usage),
  swap_used = avg(dt.host.memory.swap.used),
  swap_total = avg(dt.host.memory.swap.total),
  page_faults = avg(dt.host.memory.avail.pfps)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| fieldsAdd memory_usage_avg = arrayAvg(memory_usage)
| fieldsAdd swap_used_pct = (arrayAvg(swap_used) / arrayAvg(swap_total)) * 100
// Heuristic score for ranking memory pressure severity; not an industry-standard metric.
| fieldsAdd memory_pressure_score = if(memory_usage_avg > 90, 3,
        else: if(memory_usage_avg > 80, 2, else: 1)) +
        if(swap_used_pct > 50, 3,
        else: if(swap_used_pct > 30, 2, else: 0))
| filter memory_pressure_score >= 3
| sort memory_pressure_score desc

Memory Leak Detection

Detect continuously increasing memory usage:

timeseries memory_used = avg(dt.host.memory.used), by: {dt.smartscape.host}, interval: 15m
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    first_value = arrayFirst(memory_used),
    last_value = arrayLast(memory_used)
| fieldsAdd
    memory_increase = last_value - first_value,
    increase_pct = toDouble(last_value - first_value) * 100 / first_value
| filter increase_pct > 20  // 20% increase over time window
| sort increase_pct desc

---

Disk Monitoring

Disk Space Usage

Track disk space utilization:

timeseries {
  disk_used_pct = avg(dt.host.disk.used.percent),
  disk_used_bytes = avg(dt.host.disk.used),
  disk_avail_bytes = avg(dt.host.disk.avail)
}, by: {dt.smartscape.host, dt.smartscape.disk}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(disk_used_pct) > 80
| sort arrayAvg(disk_used_pct) desc

Alert Thresholds:

• < 80%: Normal usage
• 80-90%: Warning - plan cleanup
• > 90%: Critical - immediate action required
• Very large or auto-scaling cloud disks can keep percentages low despite high absolute usage; pair this with disk_avail_bytes for capacity decisions.

Disk I/O Performance

Monitor disk read/write performance:

timeseries {
  read_bytes_per_sec = avg(dt.host.disk.bytes_read),
  write_bytes_per_sec = avg(dt.host.disk.bytes_written),
  read_ops_per_sec = avg(dt.host.disk.read_ops),
  write_ops_per_sec = avg(dt.host.disk.write_ops)
}, by: {dt.smartscape.host, dt.smartscape.disk}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    total_throughput = arrayAvg(read_bytes_per_sec) + arrayAvg(write_bytes_per_sec)
| filter total_throughput > 50000000  // > 50 MB/s
| sort total_throughput desc

Disk Latency Analysis

Identify disk latency issues:

timeseries {
  read_latency = avg(dt.host.disk.read_time),
  write_latency = avg(dt.host.disk.write_time),
  util_pct = avg(dt.host.disk.util_time),
  queue_length = avg(dt.host.disk.queue_length)
}, by: {dt.smartscape.host, dt.smartscape.disk}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(read_latency) > 20 or arrayAvg(write_latency) > 20
| sort arrayAvg(read_latency) desc

Performance Indicators:

• < 10ms: Good (SSD/NVMe)
• 10-20ms: Acceptable (SSD under load)
• > 20ms: Bottleneck - investigate
• Queue length > 2: I/O saturation

Inode Exhaustion Detection

Monitor inode usage (Linux):

timeseries {
  inodes_avail_pct = avg(dt.host.disk.inodes_avail),
  inodes_total = avg(dt.host.disk.inodes_total)
}, by: {dt.smartscape.host, dt.smartscape.disk}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    inodes_used_pct = 100 - arrayAvg(inodes_avail_pct)
| filter inodes_used_pct > 80
| sort inodes_used_pct desc

Inode exhaustion can prevent file creation even with available space.

Disk Full Prediction

Predict when disks will fill:

timeseries {
  disk_used = avg(dt.host.disk.used),
  disk_avail = avg(dt.host.disk.avail)
}, by: {dt.smartscape.host, dt.smartscape.disk}, interval: 1h
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    current_used = arrayLast(disk_used),
    current_avail = arrayLast(disk_avail),
    first_used = arrayFirst(disk_used)
| fieldsAdd growth_rate_per_hour = (current_used - first_used) / 24  // Assuming 24h window
// Negative growth indicates cleanup or transient data effects and is excluded from "time to full" prediction.
| fieldsAdd hours_until_full = if(growth_rate_per_hour > 0, current_avail / growth_rate_per_hour, else: -1)
| filter hours_until_full > 0 and hours_until_full < 48
| sort hours_until_full asc

---

Network Monitoring

Network Interface Utilization

Monitor network interface usage:

timeseries {
  bytes_received = avg(dt.host.net.nic.bytes_rx),
  bytes_sent = avg(dt.host.net.nic.bytes_tx),
  link_util_rx_pct = avg(dt.host.net.nic.link_util_rx),
  link_util_tx_pct = avg(dt.host.net.nic.link_util_tx)
}, by: {dt.smartscape.host, dt.smartscape.network_interface}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(link_util_rx_pct) > 70 or arrayAvg(link_util_tx_pct) > 70
| sort arrayAvg(link_util_rx_pct) desc

Alert Thresholds:

• < 70%: Normal usage
• 70-85%: High utilization - monitor
• > 85%: Network saturation

Packet Drop Detection

Identify packet drops:

timeseries {
  packets_dropped_rx = avg(dt.host.net.nic.packets.dropped_rx),
  packets_dropped_tx = avg(dt.host.net.nic.packets.dropped_tx),
  total_packets_rx = avg(dt.host.net.nic.packets.rx),
  total_packets_tx = avg(dt.host.net.nic.packets.tx)
}, by: {dt.smartscape.host, dt.smartscape.network_interface}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    drop_rate_rx_pct = (arrayAvg(packets_dropped_rx) / arrayAvg(total_packets_rx)) * 100,
    drop_rate_tx_pct = (arrayAvg(packets_dropped_tx) / arrayAvg(total_packets_tx)) * 100
| filter drop_rate_rx_pct > 1 or drop_rate_tx_pct > 1
| sort drop_rate_rx_pct desc

Warning: Drop rate > 1% indicates network congestion or buffer overflow.

Network Error Analysis

Monitor packet errors:

timeseries {
  packet_errors_rx = avg(dt.host.net.nic.packets.errors_rx),
  packet_errors_tx = avg(dt.host.net.nic.packets.errors_tx),
  total_packets_rx = avg(dt.host.net.nic.packets.rx),
  total_packets_tx = avg(dt.host.net.nic.packets.tx)
}, by: {dt.smartscape.host, dt.smartscape.network_interface}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    error_rate_rx_pct = (arrayAvg(packet_errors_rx) / arrayAvg(total_packets_rx)) * 100,
    error_rate_tx_pct = (arrayAvg(packet_errors_tx) / arrayAvg(total_packets_tx)) * 100
| filter error_rate_rx_pct > 0.1 or error_rate_tx_pct > 0.1
| sort error_rate_rx_pct desc

Critical: Error rate > 0.1% suggests physical layer issues (cables, NICs).

Network Quality Score

Calculate network quality score:

timeseries {
  packets_rx = avg(dt.host.net.nic.packets.rx),
  packets_tx = avg(dt.host.net.nic.packets.tx),
  dropped_rx = avg(dt.host.net.nic.packets.dropped_rx),
  dropped_tx = avg(dt.host.net.nic.packets.dropped_tx),
  errors_rx = avg(dt.host.net.nic.packets.errors_rx),
  errors_tx = avg(dt.host.net.nic.packets.errors_tx)
}, by: {dt.smartscape.host, dt.smartscape.network_interface}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| fieldsAdd drop_rate = ((arrayAvg(dropped_rx) + arrayAvg(dropped_tx)) / (arrayAvg(packets_rx) + arrayAvg(packets_tx))) * 100
| fieldsAdd error_rate = ((arrayAvg(errors_rx) + arrayAvg(errors_tx)) / (arrayAvg(packets_rx) + arrayAvg(packets_tx))) * 100
| fieldsAdd quality_score = 100 - (drop_rate * 10) - (error_rate * 20)
| filter quality_score < 90
| sort quality_score asc

Network Spike Detection

Detect sudden network traffic spikes:

timeseries bytes_rx = avg(dt.host.net.nic.bytes_rx),
          bytes_tx = avg(dt.host.net.nic.bytes_tx),
    by: {dt.smartscape.host}, interval: 1m
| fieldsAdd total_throughput = bytes_rx[] + bytes_tx[]
| fieldsAdd
    avg_throughput = arrayAvg(total_throughput),
    max_throughput = arrayMax(total_throughput),
    host_name = getNodeName(dt.smartscape.host)
| fieldsAdd
    spike_ratio = max_throughput / avg_throughput
| filter spike_ratio > 5
| sort spike_ratio desc

---

Resource Saturation

Multi-Resource Saturation Detection

Detect resource saturation across multiple metrics:

timeseries {
  cpu = avg(dt.host.cpu.usage),
  memory = avg(dt.host.memory.usage),
  disk_util = avg(dt.host.disk.util_time),
  network_rx = avg(dt.host.net.nic.link_util_rx),
  network_tx = avg(dt.host.net.nic.link_util_tx)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| fieldsAdd
    cpu_avg = arrayAvg(cpu),
    memory_avg = arrayAvg(memory),
    disk_util_avg = arrayAvg(disk_util),
    network_util_avg = (arrayAvg(network_rx) + arrayAvg(network_tx)) / 2
| fieldsAdd saturated_resources =
        (if(cpu_avg > 85, 1, else: 0)) +
        (if(memory_avg > 85, 1, else: 0)) +
        (if(disk_util_avg > 85, 1, else: 0)) +
        (if(network_util_avg > 85, 1, else: 0))
| filter saturated_resources >= 2
| sort saturated_resources desc

Infrastructure Capacity Planning

Analyze resource trends for capacity planning:

timeseries {
  cpu = avg(dt.host.cpu.usage),
  memory = avg(dt.host.memory.usage),
  disk = avg(dt.host.disk.used.percent)
}, by: {dt.smartscape.host}, interval: 1h
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    cpu_avg = arrayAvg(cpu),
    cpu_p95 = arrayPercentile(cpu, 95),
    memory_avg = arrayAvg(memory),
    memory_p95 = arrayPercentile(memory, 95),
    disk_avg = arrayAvg(disk)
| fieldsAdd
    cpu_capacity_remaining = 100 - cpu_p95,
    memory_capacity_remaining = 100 - memory_p95
| filter cpu_capacity_remaining < 20 or memory_capacity_remaining < 20
| sort cpu_capacity_remaining asc

---

Related Documentation

For process-level resource monitoring, see process-monitoring.md. For infrastructure discovery and inventory, see inventory-discovery.md. For container metrics, see container-monitoring.md.

Inventory and Discovery Reference

Comprehensive reference for host/process discovery, technology inventory, port mapping, cost attribution, data quality, and multi-cloud management.

---

Host Inventory

All Hosts Overview

Fetch all hosts with basic information:

smartscapeNodes "HOST"
| fieldsAdd name, os.type, os.version, host.logical.cpu.cores, host.physical.memory
| sort name asc

Host by Operating System

Group and count hosts by OS type:

smartscapeNodes "HOST"
| fieldsAdd os.type, os.version
| summarize host_count = count(), by: {os.type}
| sort host_count desc

OS Types: LINUX, WINDOWS, AIX, SOLARIS, ZOS

Virtualization Analysis

Identify physical vs virtual hosts:

smartscapeNodes "HOST"
| fieldsAdd name, hypervisor.type, host.logical.cpu.cores
| summarize
    total_hosts = count(),
    virtual_hosts = countIf(isNotNull(hypervisor.type)),
    physical_hosts = countIf(isNull(hypervisor.type)),
    by: {hypervisor.type}

Hypervisor Types: VMWARE, KVM, HYPERV, XEN

Cloud vs On-Premise Classification

Classify hosts by deployment type:

smartscapeNodes "HOST"
| fieldsAdd
    name,
    cloud.provider,
    aws.region,
    azure.location,
    hypervisor.type
| fieldsAdd deployment_type = if(
    isNotNull(cloud.provider), cloud.provider,
    else: if(isNotNull(hypervisor.type), "On-Premise Virtual", else: "On-Premise Physical")
  )
| summarize host_count = count(), by: {deployment_type}
| sort host_count desc

---

Technology Inventory

Technology Stack Overview

List all detected technologies:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies
| filter isNotNull(process.software_technologies) and arraySize(process.software_technologies) > 0
| expand tech = process.software_technologies
| fieldsAdd tech_type = tech[type]
| summarize process_count = count(), by: {tech_type}
| sort process_count desc

Use Case: Comprehensive technology stack visibility.

Technology Versions

Track technology versions for each type:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies
| filter isNotNull(process.software_technologies)
| expand tech = process.software_technologies
| fieldsAdd tech_type = tech[type], tech_version = tech[version], tech_edition = tech[edition]
| filter isNotNull(tech_version) and tech_version != ""
| summarize process_count = count(), by: {tech_type, tech_version}
| sort tech_type, process_count desc

Java Processes Distribution

Analyze Java deployment landscape:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies, dt.process_group.detected_name
| expand tech = process.software_technologies
| filter tech[type] == "JAVA"
| fieldsAdd java_version = tech[version]
| summarize process_count = count(), by: {java_version}
| sort process_count desc

Database Technologies

Identify all database processes:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies, dt.process_group.detected_name
| expand tech = process.software_technologies
| filter in(tech[type], {"APACHE_CASSANDRA", "ELASTIC_SEARCH", "MONGO_DB", "MYSQL", "POSTGRESQL", "ORACLE_DB", "MSSQL", "REDIS", "COUCHDB"})
| fieldsAdd db_type = tech[type], db_version = tech[version]
| summarize
    process_count = count(),
    process_groups = countDistinct(dt.process_group.detected_name),
    by: {db_type, db_version}
| sort process_count desc

Pattern: Database deployment across infrastructure.

Messaging and Streaming Technologies

List message queues and streaming platforms:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies, dt.process_group.detected_name
| expand tech = process.software_technologies
| filter in(tech[type], {"APACHE_KAFKA", "RABBIT_MQ", "AMQP", "ACTIVE_MQ"})
| fieldsAdd tech_type = tech[type], tech_version = tech[version]
| summarize process_count = count(), by: {tech_type, tech_version}
| sort process_count desc

Web Servers and Proxies

Track web server and proxy technologies:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies, dt.process_group.detected_name
| expand tech = process.software_technologies
| filter in(tech[type], {"NGINX", "ENVOY", "APACHE_HTTP_SERVER", "IIS"})
| fieldsAdd tech_type = tech[type]
| summarize
    process_count = count(),
    process_groups = collectDistinct(dt.process_group.detected_name),
    by: {tech_type}
| sort process_count desc

Application Runtime Technologies

Analyze runtime environments:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies
| expand tech = process.software_technologies
| filter in(tech[type], {"NODE_JS", "PYTHON", "DOTNET", "CLR", "GO", "RUBY"})
| fieldsAdd tech_type = tech[type], tech_version = tech[version]
| summarize process_count = count(), by: {tech_type}
| sort process_count desc

---

Port Discovery

Port Usage Overview

List all listening ports and process counts:

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| filter isNotNull(process.listen_ports) and arraySize(process.listen_ports) > 0
| expand listen_port = process.listen_ports
| summarize process_count = countDistinct(id), by: {listen_port}
| sort toLong(listen_port) asc

Use Case: Network security auditing and port inventory.

Well-Known Port Analysis

Identify services on standard ports:

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| filter isNotNull(process.listen_ports)
| expand listen_port = process.listen_ports
| filter toLong(listen_port) <= 1024
| summarize
    process_count = countDistinct(id),
    services = collectDistinct(dt.process_group.detected_name),
    by: {listen_port}
| sort toLong(listen_port) asc

Standard Ports:

• 22: SSH
• 25: SMTP
• 53: DNS
• 80: HTTP
• 443: HTTPS
• 111: RPC
• 3306: MySQL
• 5432: PostgreSQL
• 1521: Oracle
• 1433: MS SQL Server
• 9092: Kafka
• 5672: RabbitMQ

Processes by Port

Find all processes listening on a specific port:

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name, name
| expand listen_port = process.listen_ports
| filter listen_port == "443"
| limit 50

Example: Replace "443" with your target port.

Port Conflict Detection

Identify ports with multiple different process types:

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| expand listen_port = process.listen_ports
| summarize
    process_count = countDistinct(id),
    process_types = countDistinct(dt.process_group.detected_name),
    services = collectDistinct(dt.process_group.detected_name),
    by: {listen_port}
| filter process_types > 1
| sort process_types desc, process_count desc

Alert: Same port used by different process types may indicate misconfiguration.

Web Services Port Mapping

Identify HTTP/HTTPS services:

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| expand listen_port = process.listen_ports
| filter in(listen_port, {80, 443, 8080, 8443, 3000, 4000, 5000, 9090})
| summarize
    process_count = countDistinct(id),
    by: {listen_port, dt.process_group.detected_name}
| sort toLong(listen_port) asc

Database Ports

Find all database services:

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| expand listen_port = process.listen_ports
| filter in(listen_port, {3306, 5432, 1521, 1433, 27017, 6379, 9042, 7000})
| summarize
    process_count = countDistinct(id),
    by: {listen_port, dt.process_group.detected_name}
| sort toLong(listen_port) asc

Port Mapping:

• 3306: MySQL
• 5432: PostgreSQL
• 1521: Oracle
• 1433: MS SQL Server
• 27017: MongoDB
• 6379: Redis
• 9042: Cassandra

---

Multi-Cloud Hosts

Cloud Provider Distribution

Categorize hosts by cloud provider:

smartscapeNodes "HOST"
| fieldsAdd name, cloud.provider, aws.region, azure.location
| fieldsAdd provider = if(
    isNotNull(cloud.provider), cloud.provider,
    else: "on-premise"
  )
| summarize host_count = count(), by: {provider}
| sort host_count desc

Cloud Providers: aws, azure, gcp, alibaba_cloud

AWS Hosts by Region

Group AWS EC2 instances by region:

smartscapeNodes "HOST"
| filter isNotNull(aws.region)
| fieldsAdd name, aws.region, aws.availability_zone, aws.state
| summarize host_count = count(), by: {aws.region}
| sort host_count desc

Common Regions: us-east-1, us-west-2, eu-west-1, ap-southeast-1

AWS Account Inventory

List hosts grouped by AWS account:

smartscapeNodes "HOST"
| filter isNotNull(aws.account.id)
| fieldsAdd name, aws.account.id, aws.region, aws.resource.type
| summarize
    host_count = count(),
    regions = collectDistinct(aws.region),
    by: {aws.account.id}
| sort host_count desc

Use Case: Multi-account AWS organization management.

AWS Instance States

Monitor EC2 instance states:

smartscapeNodes "HOST"
| filter isNotNull(aws.state)
| fieldsAdd name, aws.state, aws.region, aws.resource.id
| summarize host_count = count(), by: {aws.state}
| sort host_count desc

AWS States: running, stopped, stopping, terminated, pending

Azure Hosts by Location

Group Azure VMs by location:

smartscapeNodes "HOST"
| filter isNotNull(azure.location)
| fieldsAdd name, azure.location, azure.subscription, azure.status
| summarize host_count = count(), by: {azure.location}
| sort host_count desc

Common Locations: eastus, westeurope, southeastasia, northeurope

Azure Resource Groups

Organize Azure VMs by resource group:

smartscapeNodes "HOST"
| filter isNotNull(azure.resource.group)
| fieldsAdd name, azure.resource.group, azure.location, azure.subscription
| summarize host_count = count(), by: {azure.resource.group}
| sort host_count desc

Azure Subscriptions

Track Azure VMs by subscription:

smartscapeNodes "HOST"
| filter isNotNull(azure.subscription)
| fieldsAdd azure.subscription, azure.location
| summarize
    host_count = count(),
    locations = collectDistinct(azure.location),
    by: {azure.subscription}
| sort host_count desc

---

Cost Attribution

Cost Center Distribution

List all cost centers and their infrastructure footprint:

smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter, dt.cost.product
| filter isNotNull(dt.cost.costcenter) or isNotNull(dt.cost.product)
| summarize host_count = count(), by: {dt.cost.costcenter, dt.cost.product}
| sort host_count desc

Use Case: Chargeback and cost allocation reporting.

Cost Center by Cloud Provider

Analyze cost centers across cloud providers:

smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter, cloud.provider
| filter isNotNull(dt.cost.costcenter)
| summarize host_count = count(), by: {dt.cost.costcenter, cloud.provider}
| sort dt.cost.costcenter, host_count desc

Product Cost Breakdown

Group infrastructure costs by product:

smartscapeNodes "HOST"
| fieldsAdd dt.cost.product, dt.cost.costcenter
| filter isNotNull(dt.cost.product)
| summarize host_count = count(), by: {dt.cost.product, dt.cost.costcenter}
| sort dt.cost.product, host_count desc

Resource Costs by Cost Center

Calculate resource consumption per cost center:

smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter, host.logical.cpu.cores, host.physical.memory
| filter isNotNull(dt.cost.costcenter)
| fieldsAdd memory_gb = toDouble(host.physical.memory) / 1024 / 1024 / 1024
| summarize
    host_count = count(),
    total_cores = sum(toLong(host.logical.cpu.cores)),
    total_memory_gb = sum(memory_gb),
    by: {dt.cost.costcenter}
| fieldsAdd total_memory_gb = round(total_memory_gb, decimals: 0)
| sort total_cores desc

Metric: Physical resource allocation per cost center.

Hosts Without Cost Attribution

Identify hosts missing cost metadata:

smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter, dt.cost.product, name, cloud.provider
| filter isNull(dt.cost.costcenter) and isNull(dt.cost.product)
| summarize host_count = count(), by: {cloud.provider}
| sort host_count desc

Action: Tag unattributed infrastructure for cost tracking.

Cost Attribution Coverage Rate

Calculate percentage of hosts with cost tags:

smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter
| summarize
    total_hosts = count(),
    attributed_hosts = countIf(isNotNull(dt.cost.costcenter)),
    unattributed_hosts = countIf(isNull(dt.cost.costcenter))
| fieldsAdd coverage_rate = round((toDouble(attributed_hosts) / toDouble(total_hosts)) * 100, decimals: 1)

Target: >90% cost attribution coverage

---

Tags and Metadata

Important Notes

• Generic tags field is NOT populated in smartscape queries
• Use specific tag fields: tags:azure[*], tags:environment
• Use custom metadata: host.custom.metadata[*]

Azure Resource Tags

List all Azure-specific tags:

smartscapeNodes "HOST"
| filter isNotNull(azure.location)
| fieldsAdd `tags:azure`[dt_owner_team], `tags:azure`[dt_owner_capability]
| filter isNotNull(`tags:azure`[dt_owner_team])
| summarize host_count = count(), by: {`tags:azure`[dt_owner_team], `tags:azure`[dt_owner_capability]}
| sort host_count desc

Azure Pattern: Tags prefixed with tags:azure for resource organization.

Azure Cost Tags

Analyze Azure cost allocation tags:

smartscapeNodes "HOST"
| filter isNotNull(azure.location)
| fieldsAdd
    capability = `tags:azure`[dt_cloudcost_capability],
    service = `tags:azure`[dt_cloudcost_service],
    cluster = `tags:azure`[dt_cloudcost_clustername]
| filter isNotNull(capability)
| summarize host_count = count(), by: {capability, service}
| sort host_count desc

Use Case: Cost allocation and chargeback reporting.

Azure Owner Tags

Track resource ownership via Azure tags:

smartscapeNodes "HOST"
| filter isNotNull(azure.location)
| fieldsAdd
    owner_team = `tags:azure`[dt_owner_team],
    owner_email = `tags:azure`[dt_owner_email],
    owner_capability = `tags:azure`[dt_owner_capability]
| filter isNotNull(owner_team)
| summarize host_count = count(), by: {owner_team, owner_capability}
| sort host_count desc

Custom Metadata - Operator Version

Track OneAgent operator versions:

smartscapeNodes "HOST"
| fieldsAdd operator_version = host.custom.metadata[OperatorVersion]
| filter isNotNull(operator_version)
| summarize host_count = count(), by: {operator_version}
| sort host_count desc

Custom Metadata - Cluster Information

Query custom cluster metadata:

smartscapeNodes "HOST"
| fieldsAdd cluster = host.custom.metadata[Cluster]
| filter isNotNull(cluster)
| summarize host_count = count(), by: {cluster}
| sort host_count desc

---

Data Quality

AWS Hosts Missing Account IDs

Identify AWS hosts without account attribution:

smartscapeNodes "HOST"
| filter cloud.provider == "aws"
| fieldsAdd aws.account.id, aws.region, name
| summarize
    total_hosts = count(),
    missing_account = countIf(isNull(aws.account.id)),
    has_account = countIf(isNotNull(aws.account.id)),
    by: {aws.region}
| fieldsAdd missing_pct = round((toDouble(missing_account) / toDouble(total_hosts)) * 100, decimals: 1)
| sort missing_pct desc

Data Quality Issue: AWS hosts should always have account IDs.

Hosts Without OS Information

Find hosts missing operating system details:

smartscapeNodes "HOST"
| fieldsAdd os.type, os.version, name
| filter isNull(os.type) or isNull(os.version)
| summarize host_count = count(), by: {os.type}

Expected: All hosts should report OS type and version.

Kubernetes Nodes Without Cluster Info

Identify K8s nodes missing cluster metadata:

smartscapeNodes "HOST"
| fieldsAdd k8s.node.name, k8s.cluster.name, k8s.cluster.uid, name
| filter isNotNull(k8s.node.name)
| filter isNull(k8s.cluster.name) or isNull(k8s.cluster.uid)
| sort name

Alert: Kubernetes nodes must have cluster information.

Processes Without Technology Detection

Find processes missing software technology metadata:

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies, dt.process_group.detected_name
| filter isNull(process.software_technologies) or arraySize(process.software_technologies) == 0
| filter dt.process_group.detected_name != "Short-lived processes"
| summarize process_count = count(), by: {dt.process_group.detected_name}
| sort process_count desc
| limit 50

Expected: Most processes should have detected technologies.

Hosts Missing Memory Information

Identify hosts without memory data:

smartscapeNodes "HOST"
| fieldsAdd host.physical.memory, name, cloud.provider
| filter isNull(host.physical.memory) or toLong(host.physical.memory) == 0
| summarize host_count = count(), by: {cloud.provider}

Metadata Completeness Score

Calculate overall metadata completeness:

smartscapeNodes "HOST"
| fieldsAdd os.type, cloud.provider, host.logical.cpu.cores, host.physical.memory
| summarize
    total_hosts = count(),
    has_os = countIf(isNotNull(os.type)),
    has_cpu = countIf(isNotNull(host.logical.cpu.cores)),
    has_memory = countIf(isNotNull(host.physical.memory))
| fieldsAdd
    os_completeness = round((toDouble(has_os) / toDouble(total_hosts)) * 100, decimals: 1),
    cpu_completeness = round((toDouble(has_cpu) / toDouble(total_hosts)) * 100, decimals: 1),
    memory_completeness = round((toDouble(has_memory) / toDouble(total_hosts)) * 100, decimals: 1)

Target: >90% completeness for all fields

---

Related Documentation

For host resource metrics, see host-metrics.md. For process monitoring, see process-monitoring.md. For container and Kubernetes monitoring, see container-monitoring.md.

Process Monitoring Reference

Detailed process-level monitoring including CPU, memory, I/O, and network metrics with troubleshooting guidance.

---

Process CPU and Memory

Top CPU-Consuming Processes

Identify resource-intensive processes:

timeseries cpu_usage = avg(dt.process.cpu.usage),
    by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(cpu_usage) > 50
| sort arrayAvg(cpu_usage) desc
| limit 20

Metric: dt.process.cpu.usage - Process CPU percentage (100% = 1 full core).

Process Memory Usage

Monitor process memory consumption:

timeseries {
  memory_bytes = avg(dt.process.memory.working_set_size),
  memory_pct = avg(dt.process.memory.usage)
}, by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    memory_gb = arrayAvg(memory_bytes) / 1024 / 1024 / 1024
| filter arrayAvg(memory_pct) > 20 or memory_gb > 4
| sort arrayAvg(memory_pct) desc
| limit 20

Memory Leak Detection

Detect processes with continuously growing memory:

timeseries memory_bytes = avg(dt.process.memory.working_set_size),
    by: {dt.smartscape.process}, interval: 15m
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    first_value = arrayFirst(memory_bytes),
    last_value = arrayLast(memory_bytes)
| fieldsAdd
    growth_bytes = last_value - first_value,
    growth_pct = toDouble(last_value - first_value) * 100 / first_value
| filter growth_pct > 30
| sort growth_pct desc

Page Fault Analysis

Monitor process page faults:

timeseries page_faults = avg(dt.process.memory.page_faults),
    by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(page_faults) > 1000
| sort arrayAvg(page_faults) desc

High page faults indicate insufficient memory or swapping.

GC Suspension Time

Monitor garbage collection impact (Java, .NET):

timeseries gc_suspension = avg(dt.process.cpu.group_suspension_time),
    by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(gc_suspension) > 100000  // > 100ms
| sort arrayAvg(gc_suspension) desc

Resource Exhaustion Events

Detect memory and thread exhaustion:

timeseries {
  memory_exhausted = sum(dt.process.mem.exhausted_mem),
  threads_exhausted = sum(dt.process.threads_exhausted)
}, by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arraySum(memory_exhausted) > 0 or arraySum(threads_exhausted) > 0
| sort arraySum(memory_exhausted) desc

Critical Alert: Any exhaustion event indicates serious issues (OOM, thread pool exhaustion).

File Descriptor Usage

Monitor file descriptor limits (Linux):

timeseries {
  fd_used = avg(dt.process.handles.file_descriptors_used),
  fd_max = avg(dt.process.handles.file_descriptors_max),
  fd_pct_used = avg(dt.process.handles.file_descriptors_percent_used)
}, by: {dt.smartscape.process}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(fd_pct_used) > 80
| sort arrayAvg(fd_pct_used) desc

Process Health Score

Calculate process health based on resource usage:

timeseries {
  cpu = avg(dt.process.cpu.usage),
  memory = avg(dt.process.memory.usage),
  page_faults = avg(dt.process.memory.page_faults)
}, by: {dt.smartscape.process}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| fieldsAdd cpu_avg = arrayAvg(cpu)
| fieldsAdd memory_avg = arrayAvg(memory)
| fieldsAdd page_faults_avg = arrayAvg(page_faults)
| fieldsAdd health_score = 100 -
        if(cpu_avg > 80, 30, else: if(cpu_avg > 60, 15, else: 0)) -
        if(memory_avg > 80, 30, else: if(memory_avg > 60, 15, else: 0)) -
        if(page_faults_avg > 5000, 20, else: if(page_faults_avg > 1000, 10, else: 0))
| filter health_score < 70
| sort health_score asc

---

Process I/O

Top I/O Consuming Processes

Identify processes with highest I/O:

timeseries {
  bytes_read = avg(dt.process.io.bytes_read),
  bytes_written = avg(dt.process.io.bytes_written),
  bytes_total = avg(dt.process.io.bytes_total)
}, by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    io_mb_per_sec = arrayAvg(bytes_total) / 1024 / 1024
| filter io_mb_per_sec > 10
| sort arrayAvg(bytes_total) desc
| limit 20

Read-Heavy Processes

Identify read-intensive processes:

timeseries {
  bytes_read = avg(dt.process.io.bytes_read),
  bytes_written = avg(dt.process.io.bytes_written)
}, by: {dt.smartscape.process}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    read_write_ratio = arrayAvg(bytes_read) / arrayAvg(bytes_written)
| filter read_write_ratio > 5 and arrayAvg(bytes_read) > 10000000
| sort arrayAvg(bytes_read) desc

Write-Heavy Processes

Identify write-intensive processes:

timeseries {
  bytes_read = avg(dt.process.io.bytes_read),
  bytes_written = avg(dt.process.io.bytes_written)
}, by: {dt.smartscape.process}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    write_read_ratio = arrayAvg(bytes_written) / arrayAvg(bytes_read)
| filter write_read_ratio > 5 and arrayAvg(bytes_written) > 10000000
| sort arrayAvg(bytes_written) desc

Requested vs Actual I/O

Compare requested I/O to actual I/O (Linux/AIX):

timeseries {
  req_bytes_read = avg(dt.process.io.req_bytes_read),
  req_bytes_write = avg(dt.process.io.req_bytes_write),
  actual_bytes_read = avg(dt.process.io.bytes_read),
  actual_bytes_written = avg(dt.process.io.bytes_written)
}, by: {dt.smartscape.process}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    read_cache_hit_pct = ((arrayAvg(req_bytes_read) - arrayAvg(actual_bytes_read)) / arrayAvg(req_bytes_read)) * 100,
    write_cache_hit_pct = ((arrayAvg(req_bytes_write) - arrayAvg(actual_bytes_written)) / arrayAvg(req_bytes_write)) * 100
| filter arrayAvg(req_bytes_read) > 0 and arrayAvg(req_bytes_write) > 0
| sort read_cache_hit_pct asc

Metric Explanation:

• req_bytes_read/write: Requested I/O (includes cache)
• bytes_read/written: Actual disk I/O (storage layer)
• Low cache hit rate indicates poor caching effectiveness

I/O Spike Detection

Detect sudden I/O spikes:

timeseries io_total = avg(dt.process.io.bytes_total),
    by: {dt.smartscape.process}, interval: 1m
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    avg_io = arrayAvg(io_total),
    max_io = arrayMax(io_total)
| fieldsAdd spike_ratio = max_io / avg_io
| filter spike_ratio > 10
| sort spike_ratio desc

---

Process Network

Network Traffic by Process

Identify network-intensive processes:

timeseries {
  bytes_sent = avg(dt.process.network.bytes_tx),
  bytes_received = avg(dt.process.network.bytes_rx),
  throughput = avg(dt.process.network.throughput)
}, by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    total_traffic = arrayAvg(bytes_sent) + arrayAvg(bytes_received)
| filter total_traffic > 1000000  // > 1 MB/s
| sort total_traffic desc
| limit 20

Request Rate Analysis

Monitor request rates per process:

timeseries requests_per_sec = avg(dt.process.network.load),
    by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(requests_per_sec) > 100
| sort arrayAvg(requests_per_sec) desc

TCP Connection Quality

Monitor TCP session health:

timeseries {
  new_sessions = avg(dt.process.network.sessions.new_aggr),
  session_timeouts = avg(dt.process.network.sessions.timeout_aggr),
  session_resets = avg(dt.process.network.sessions.reset_aggr)
}, by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    timeout_rate_pct = (arrayAvg(session_timeouts) / arrayAvg(new_sessions)) * 100,
    reset_rate_pct = (arrayAvg(session_resets) / arrayAvg(new_sessions)) * 100
| filter timeout_rate_pct > 5 or reset_rate_pct > 5
| sort timeout_rate_pct desc

Alert Thresholds:

• Timeout rate > 5%: Connection establishment issues
• Reset rate > 5%: Unexpected connection terminations

Network Latency Monitoring

Monitor round-trip time and latency:

timeseries {
  rtt_ms = avg(dt.process.network.round_trip),
  latency_ms = avg(dt.process.network.latency)
}, by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(rtt_ms) > 100 or arrayAvg(latency_ms) > 100
| sort arrayAvg(rtt_ms) desc

Metrics:

• round_trip: TCP handshake RTT
• latency: Time between data send and ACK

Packet Retransmission Analysis

Monitor packet retransmissions:

timeseries {
  retransmit_packets = avg(dt.process.network.packets.re_tx_aggr),
  retransmit_base = avg(dt.process.network.packets.base_re_tx_aggr),
  packets_received = avg(dt.process.network.packets.rx)
}, by: {dt.smartscape.process, dt.process_group.id}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    retransmit_pct = (arrayAvg(retransmit_packets) / arrayAvg(retransmit_base)) * 100
| filter retransmit_pct > 1
| sort retransmit_pct desc

Warning: Retransmit rate > 1% indicates network quality degradation.

Process Network Health Score

Calculate network health score:

timeseries {
  new_sessions = avg(dt.process.network.sessions.new_aggr),
  timeouts = avg(dt.process.network.sessions.timeout_aggr),
  resets = avg(dt.process.network.sessions.reset_aggr),
  retransmits = avg(dt.process.network.packets.re_tx_aggr),
  base_retransmits = avg(dt.process.network.packets.base_re_tx_aggr)
}, by: {dt.smartscape.process}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    timeout_rate = (arrayAvg(timeouts) / arrayAvg(new_sessions)) * 100,
    reset_rate = (arrayAvg(resets) / arrayAvg(new_sessions)) * 100,
    retransmit_rate = (arrayAvg(retransmits) / arrayAvg(base_retransmits)) * 100
| fieldsAdd
    health_score = 100 -
        if(timeout_rate > 10, 40, else: if(timeout_rate > 5, 20, else: 0)) -
        if(reset_rate > 10, 40, else: if(reset_rate > 5, 20, else: 0)) -
        if(retransmit_rate > 5, 20, else: if(retransmit_rate > 1, 10, else: 0))
| filter health_score < 80
| sort health_score asc

Bandwidth Consumption Ranking

Rank processes by bandwidth consumption:

timeseries throughput = avg(dt.process.network.throughput),
    by: {dt.smartscape.process}
| fieldsAdd
    process_name = getNodeName(dt.smartscape.process),
    avg_throughput = arrayAvg(throughput),
    max_throughput = arrayMax(throughput),
    total_data = arraySum(throughput)
| fieldsAdd
    avg_mbps = avg_throughput / 125000,  // Convert to Mbps
    total_gb = total_data / 1024 / 1024 / 1024
| sort total_gb desc
| limit 20

---

Process Inventory and Lifecycle

All Processes Overview

Fetch all process instances:

smartscapeNodes "PROCESS"
| fieldsAdd name, dt.process_group.detected_name, process.containerized
| sort name asc
| limit 100

Process Groups Summary

Group processes by process group:

smartscapeNodes "PROCESS"
| fieldsAdd dt.process_group.id, dt.process_group.detected_name
| summarize instance_count = count(), by: {dt.process_group.id, dt.process_group.detected_name}
| sort instance_count desc

Key Concept: Process groups aggregate similar processes (e.g., multiple Apache workers).

Containerized vs Native Processes

Compare containerized and native processes:

smartscapeNodes "PROCESS"
| fieldsAdd process.containerized, dt.process_group.detected_name
| summarize
    total_processes = count(),
    containerized = countIf(process.containerized == true),
    native = countIf(process.containerized == false or isNull(process.containerized)),
    by: {process.containerized}

Process Technology Detection

Identify processes by detected technology:

smartscapeNodes "PROCESS"
| fieldsAdd name, dt.process_group.detected_name, process.bitness
| filter contains(dt.process_group.detected_name, "Java")
    or contains(dt.process_group.detected_name, "Node")
    or contains(dt.process_group.detected_name, "Python")
    or contains(dt.process_group.detected_name, "Apache")
    or contains(dt.process_group.detected_name, "nginx")
| sort dt.process_group.detected_name
| limit 50

Common Technologies: Java, Node.js, Python, Apache, nginx, IIS, MySQL, PostgreSQL, Oracle

Process Listen Ports

Identify processes by listening ports:

smartscapeNodes "PROCESS"
| fieldsAdd name, process.listen_ports, dt.process_group.detected_name
| filter isNotNull(process.listen_ports) and arraySize(process.listen_ports) > 0
| expand listen_port = process.listen_ports
| summarize
    process_count = countDistinct(id),
    by: {listen_port, dt.process_group.detected_name}
| sort toLong(listen_port) asc

Use Case: Discover port conflicts or identify services by standard ports (80, 443, 3306, etc.).

---

Related Documentation

For host-level resource monitoring, see host-metrics.md. For container and Kubernetes monitoring, see container-monitoring.md. For technology inventory and discovery, see inventory-discovery.md.