Huawei Cloud Ces Ecs Monitoring

Huawei Cloud ECS Monitoring Skill

You are a professional Huawei Cloud monitoring assistant responsible for querying and analyzing ECS instance metrics using Cloud Eye Service (CES). Follow the structured workflow to provide comprehensive monitoring insights.

1. Overview

Functional Overview

Huawei Cloud ECS monitoring skill uses Cloud Eye Service (CES) to provide comprehensive monitoring and metric query capabilities for Elastic Cloud Server instances. Supports real-time monitoring of CPU, memory, disk, network, and system metrics, historical data query, and common metric analysis.

Architecture Diagram

User Request → Huawei Cloud CLI (hcloud) → Cloud Eye Service (CES) → ECS Instance
                    ↓
                IAM Permission Verification
                    ↓
                Monitoring Data Return

Application Scenarios

• Monitor ECS instance CPU, memory, disk, and network utilization
• Query historical metrics for performance analysis
• Set up basic monitoring dashboards
• Troubleshoot performance bottlenecks
• Analyze resource usage trends
• Check system and custom metrics

User Scenario Examples

1. Basic monitoring request: "Check my ECS instance performance" 2. Specific metric query: "Show CPU and memory usage for instance ecs-server-01" 3. Historical data analysis: "Show disk IO trends for the last 7 days" 4. Troubleshooting: "My ECS instance is slow, check all metrics"

2. Prerequisites

CLI Installation and Verification

Before starting any operations, you must install and verify Huawei Cloud CLI (hcloud):

Verify Installation:

hcloud --version

If not installed, follow the detailed installation guide: See references/cli-installation-guide.md for complete installation instructions for:

• macOS
• Linux
• Windows

Configuration Method

Configure Huawei Cloud credentials:

hcloud configure init

Follow the interactive prompts to set:

• Access Key ID
• Secret Access Key
• Region
• Project ID (optional)

Security Rules

[MUST] At the start of the Core Workflow (before any CLI invocation):

hcloud configure list

Security Rules:

• NEVER read, echo, or print AK/SK values (e.g., echo $HUAWEICLOUD_ACCESS_KEY is FORBIDDEN)
• NEVER ask the user to input AK/SK directly in the conversation or command line
• ONLY use hcloud configure list to check credential status

If no valid configuration exists, STOP here:

1. Obtain credentials from Huawei Cloud Console 2. Configure credentials outside of this session (via hcloud configure init in terminal) 3. Return and re-run after hcloud configure list shows valid configuration

IAM Permission Requirements

This skill requires the following minimum IAM permissions:

• ecs:cloudServers:list - List ECS instances
• ecs:cloudServers:get - Get ECS instance details
• ces:metrics:list - List available metrics
• ces:metricData:get - Get metric data

Additional optional permissions (e.g., ces:alarms:list, ces:alarmTemplates:list) and detailed policy configuration: references/iam-policies.md

Permission Failure Handling

When any operation encounters a permission error (e.g., "Access denied", "Insufficient permissions"), refer to references/iam-policies.md for the complete handling process, including required permission list, JSON policy templates, and IAM console configuration steps.

3. KooCLI Command Format Standards

[MUST] Before executing any CLI command, read references/related-commands.md for command format standards.

Key Rules:

• Use proper command structure: hcloud <service> <command> <parameters>
• Always specify region: --cli-region=<region-id>
• For ECS commands: use ecs service
• For CES commands: use ces service
• Use proper JSON formatting for complex parameters

[MUST] Command Format - Every hcloud CLI command should follow Huawei Cloud CLI standards.

4. Core Workflow/Process

Step 1: List Available ECS Instances

First, list all ECS instances in the current region to help users identify the target instance.

hcloud ECS NovaListServers --cli-region=<region-id> --limit=50

Step 2: Query Common Monitoring Metrics

Based on user requirements, query relevant monitoring metrics. If no specific metrics are requested, show common metrics:

Common ECS Metrics (Default Display) - SYS.ECS Namespace:

1. CPU Utilization (cpu_util) 2. Memory Utilization (mem_util) 3. Disk Read/Write Rate (disk_read_bytes_rate, disk_write_bytes_rate) 4. Network In/Out Rate (network_incoming_bytes_rate_inband, network_outgoing_bytes_rate_inband) 5. Disk Utilization (disk_util_inband)

Note: The above are SYS.ECS (base monitoring) metrics available without an agent.

For OS-level monitoring (AGT.ECS namespace), which requires the Telescope agent,

see references/ces-metrics-reference.md for the complete metric list including

cpu_usage, mem_usedPercent, disk_usedPercent, load_average1, etc.

Other related metrics can be found in references/ces-metrics-reference.md.

Namespace Selection and Fallback Strategy:

When querying ECS metrics, follow this namespace selection logic:

1. Default: Query SYS.ECS metrics first (no agent required, available for all instances) 2. Fallback to AGT.ECS: If any individual SYS.ECS metric query returns no data, attempt to retrieve the corresponding metric from the AGT.ECS namespace (e.g., cpu_util → cpu_usage). See references/ces-metrics-reference.md for the complete fallback mapping table. 3. AGT.ECS only metrics: Some metrics only exist in AGT.ECS namespace (e.g., load_average1, net_tcp_total, disk_readTime, disk_inodesUsedPercent). Query these directly with --metrics.N.namespace="AGT.ECS" and --period=60.

Common reasons for SYS.ECS metrics returning no data:

• The instance image does not have UVP VMTools installed (affects mem_util, disk_util_inband, network_*_inband)
• The instance was recently created and metrics have not yet been generated (wait 5-10 minutes)
• The instance is not in ACTIVE state

Command example - SYS.ECS query:

hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-id>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=<region-id>

Command example - AGT.ECS query (when SYS.ECS has no data, or for AGT.ECS-only metrics):

hcloud CES BatchListMetricData \
  --metrics.1.namespace="AGT.ECS" \
  --metrics.1.metric_name="cpu_usage" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-id>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=60 \
  --filter="average" \
  --cli-region=<region-id>

Other relevant commands are documented in references/related-commands.md.

Step 3: Analyze and Compare Metrics

Based on the monitoring data returned:

• Compare current values against recommended thresholds
• Identify metrics approaching or exceeding thresholds
• For AGT.ECS metrics, verify the monitoring agent is installed
• Cross-reference related metrics (e.g., CPU usage with load average)
• [MUST] Handle empty data: If SYS.ECS metric query returns no data points:

1. Check if the instance is in ACTIVE state 2. Try the corresponding AGT.ECS metric (see fallback mapping in Step 2) 3. If AGT.ECS also returns no data, check if the Telescope agent is installed 4. Inform the user about possible reasons (UVP VMTools missing, instance too new, agent not installed)

Step 4: Format and Present Results

Present monitoring data in a clear, actionable format:

• Show metric values with timestamps
• Identify trends and anomalies
• Provide recommendations if thresholds are exceeded
• Suggest next steps for optimization

Optional Path: Alarm Management

If users need to view or manage alarms:

# List alarms
hcloud CES ListAlarms --cli-region=<region-id>

# List alarm templates
hcloud CES ListAlarmTemplates --cli-region=<region-id>

5. Core Commands

refer to '../references/related-commands.md'

6. Parameter Description

Required Parameters

Optional Parameters

Time Range Options

• Last 1 hour (default)
• Last 6 hours
• Last 24 hours
• Last 7 days
• Custom range (user specified)

Note: period=60 (1-minute granularity) is only available for AGT.ECS metrics. SYS.ECS metrics have a minimum period of 300 (5 minutes).

Namespace

• SYS.ECS - Basic monitoring (no agent required, minimum granularity: 5 minutes / period=300)
• AGT.ECS - OS monitoring (Telescope Agent required, minimum granularity: 1 minute / period=60)

For detailed namespace descriptions and metric availability, see references/ces-metrics-reference.md.

filter

Value Range: Supports average, variance, min, max, sum

• average: Average value
• variance: Variance
• min: Minimum value
• max: Maximum value
• sum: Sum value

7. Output Format

Monitoring Report Format

## ECS Monitoring Report
**Instance**: <instance-name> (<instance-id>)
**Region**: <region>
**Time Range**: <start-time> to <end-time>

### Key Metrics Summary
- CPU Utilization: XX.XX% (avg), XX.XX% (max), XX.XX% (min)
- Memory Utilization: XX.XX% (avg), XX.XX% (max), XX.XX% (min)
- Disk Read Rate: XX.XX MB/s (avg)
- Disk Write Rate: XX.XX MB/s (avg)
- Network Inbound: XX.XX Mbps (avg)
- Network Outbound: XX.XX Mbps (avg)

### Detailed Metrics
| Time | CPU Usage | Memory Usage | Disk Read | Disk Write | Network In | Network Out |
|------|-----------|--------------|-----------|------------|------------|-------------|
| ...  | ...       | ...          | ...       | ...        | ...        | ...         |

### Recommendations
1. [If CPU > 80%] Consider scaling up instance type or optimizing application
2. [If Memory > 85%] Consider adding memory or optimizing memory usage
3. [If Disk > 90%] Consider expanding disk or cleaning up files
4. [Network bottlenecks] Consider optimizing network configuration

8. Verification Method

Skill verification and testing methods: references/verification-method.md

Basic Verification Steps

1. Environment verification: Ensure Huawei Cloud CLI is installed and configured 2. Permission verification: Verify IAM permissions are sufficient 3. Function verification: Test core monitoring functionality 4. Error handling verification: Test handling of various error scenarios

Test Cases

• Normal scenario: Successfully query monitoring data
• Insufficient permissions scenario: Handle permission errors
• Instance not found scenario: Handle instance lookup failures
• Network error scenario: Handle connection issues

9. Best Practices

Please refer to references/best-practices.md for detailed best practices, including metric selection guidelines, alerting strategy, monitoring frequency recommendations, performance optimization, and cost optimization.

Core Principles

1. Default to common metrics - When user doesn't specify, default to showing common metrics 2. Provide actionable insights - Not just raw data, provide analysis and recommendations 3. Batch query related metrics - Reduce API call frequency by querying multiple metrics in a single request

10. Reference Documents

Refer to documents in the references/ directory for more information:

• cli-installation-guide.md: Huawei Cloud CLI installation and configuration guide
• ces-metrics-reference.md: Complete list of CES metrics for ECS
• iam-policies.md: Required IAM permissions and policies
• best-practices.md: Monitoring best practices and optimization tips
• troubleshooting-guide.md: Common issues and solutions
• verification-method.md: Skill verification and testing methods
• acceptance-criteria.md: Quality standards and acceptance criteria
• related-commands.md: Related command reference

11. Notes

Security Tips

• Credential security: Never expose AK/SK in code, logs, or conversations
• Principle of least privilege: Grant only necessary IAM permissions

For more security best practices (key rotation, IAM conditions, account separation), see references/iam-policies.md.

Limitations

• API limits: Be aware of Huawei Cloud API rate limits
• Data retention: Monitoring data has limited retention time

For specific limits (max data points, query frequency, batch limits), see references/ces-metrics-reference.md.

Known Issues

1. Data latency: Monitoring data may have 1-2 minutes delay 2. Metric availability: Newly created instances may take several minutes to start reporting metrics

Troubleshooting

For common errors and detailed solutions, see references/troubleshooting-guide.md.

Support and Feedback

• Huawei Cloud official documentation: https://support.huaweicloud.com/usermanual-ecs/ecs_03_1001.html
• CLI reference documentation: https://support.huaweicloud.com/function-hcli/index.html
• Issue feedback: Through Huawei Cloud ticket system or technical support channels

Huawei Cloud ECS Monitoring Skill Acceptance Criteria

Overview

This document defines the acceptance criteria for the Huawei Cloud ECS Monitoring skill. These criteria ensure the skill meets quality standards, functional requirements, and user expectations before being considered complete and ready for use.

Functional Acceptance Criteria

1. Prerequisites Verification

AC-001: The skill must verify Huawei Cloud CLI installation before execution.

• Criteria: Skill checks hcloud --version and provides installation instructions if CLI is not found
• Validation: Run skill without Huawei Cloud CLI installed, verify helpful error message
• Success Metric: 100% of users receive clear installation guidance

AC-002: The skill must validate Huawei Cloud credentials configuration.

• Criteria: Skill checks hcloud configure list and validates credentials
• Validation: Run with invalid/missing credentials, verify appropriate error handling
• Success Metric: Proper authentication error messages for all credential issues

2. ECS Instance Management

AC-003: The skill must list all ECS instances in the specified region.

• Criteria: hcloud ECS NovaListServers --cli-region=<region-id> executes successfully
• Validation: Verify instance listing includes name, ID, status, and creation time
• Success Metric: 100% accurate instance listing with no missing data

AC-004: The skill must retrieve detailed ECS instance information.

• Criteria: hcloud ECS NovaShowServer --server_id=<instance-uuid> --cli-region=<region-id> returns complete instance details
• Validation: Verify all instance attributes are accessible (flavor, image, addresses, etc.)
• Success Metric: Complete instance metadata retrieval

3. Metric Query Capabilities

AC-005: The skill must list available CES metrics for ECS instances.

• Criteria: hcloud CES ListMetrics --namespace="SYS.ECS" returns all available metrics
• Validation: Verify CPU, memory, disk, and network metrics are listed
• Success Metric: All standard ECS metrics accessible

AC-006: The skill must query metric data with flexible time ranges.

• Criteria: Support for last 1 hour, 6 hours, 24 hours, 7 days, and custom ranges
• Validation: Test each time range option with valid metric data returned
• Success Metric: 100% time range flexibility

AC-007: The skill must support multiple aggregation methods.

• Criteria: Support for average, max, min, sum, and sampleCount statistics
• Validation: Verify each aggregation method produces correct results
• Success Metric: All aggregation methods functional

4. Core Monitoring Metrics

AC-008: The skill must monitor CPU utilization.

• Criteria: Query cpu_util (SYS.ECS) or cpu_usage (AGT.ECS) metric with proper dimensions and formatting
• Validation: Verify CPU percentage values are within 0-100% range
• Success Metric: Accurate CPU monitoring with trend analysis

AC-009: The skill must monitor memory utilization.

• Criteria: Query mem_util (SYS.ECS) or mem_usedPercent (AGT.ECS) metric with proper dimensions
• Validation: Verify memory percentage values are within 0-100% range
• Success Metric: Accurate memory monitoring

AC-010: The skill must monitor disk metrics.

• Criteria: Query disk usage, read/write rates with proper mount point dimensions
• Validation: Verify disk metrics include mount point specification
• Success Metric: Complete disk monitoring across all mounted filesystems

AC-011: The skill must monitor network metrics.

• Criteria: Query network inbound/outbound rates with proper NIC dimensions
• Validation: Verify network metrics include NIC specification
• Success Metric: Complete network interface monitoring

5. Data Presentation

AC-012: The skill must present data in clear, actionable format.

• Criteria: Output includes timestamps, metric values, and human-readable units
• Validation: Verify output is understandable without additional processing
• Success Metric: 100% user comprehension of presented data

AC-013: The skill must identify performance trends and anomalies.

• Criteria: Analysis of metric trends over time with threshold comparisons
• Validation: Verify trend detection for increasing/decreasing patterns
• Success Metric: Accurate trend identification

AC-014: The skill must provide optimization recommendations.

• Criteria: Suggestions based on metric thresholds (e.g., CPU > 80%, Memory > 85%)
• Validation: Verify recommendations are appropriate for detected issues
• Success Metric: Actionable recommendations for all common issues

Technical Acceptance Criteria

1. Performance Requirements

AC-101: Response time for single metric query must be under 5 seconds.

• Criteria: hcloud CES BatchListMetricData completes within 5 seconds for single metric
• Validation: Measure response time across 100 queries, calculate 95th percentile
• Success Metric: 95% of queries complete within 5 seconds

AC-102: Response time for multiple metrics query must be under 10 seconds.

• Criteria: Query with 6+ metrics completes within 10 seconds
• Validation: Test with CPU, memory, disk, and network metrics combined
• Success Metric: 95% of multi-metric queries complete within 10 seconds

AC-103: Instance listing must complete within 3 seconds.

• Criteria: hcloud ECS NovaListServers --cli-region=<region-id> returns results within 3 seconds
• Validation: Test with varying numbers of instances (1-100)
• Success Metric: 99% of instance listings complete within 3 seconds

2. Reliability Requirements

AC-104: The skill must handle API rate limits gracefully.

• Criteria: Implement retry logic with exponential backoff for rate limit errors
• Validation: Simulate rate limiting, verify retry behavior
• Success Metric: Automatic recovery from rate limiting

AC-105: The skill must handle network timeouts and failures.

• Criteria: Implement timeout handling and connection failure recovery
• Validation: Simulate network failures, verify graceful degradation
• Success Metric: Clear error messages for network issues

AC-106: The skill must validate all inputs before API calls.

• Criteria: Validate instance IDs, metric names, time ranges, and regions
• Validation: Test with invalid inputs, verify validation errors
• Success Metric: 100% input validation coverage

3. Security Requirements

AC-107: The skill must not expose credentials in logs or output.

• Criteria: No Access Key ID or Secret Access Key in any output
• Validation: Search all output for credential patterns
• Success Metric: Zero credential exposure incidents

AC-108: The skill must enforce least privilege IAM permissions.

• Criteria: Document and verify minimum required permissions
• Validation: Test with minimal IAM policy, verify all functions work
• Success Metric: Functionality with documented minimum permissions

AC-109: The skill must validate IAM permissions before execution.

• Criteria: Check required permissions and provide clear error if missing
• Validation: Test with insufficient permissions, verify helpful error messages
• Success Metric: Clear permission guidance for all permission errors

User Experience Acceptance Criteria

1. Documentation Quality

AC-201: All commands must have clear examples in documentation.

• Criteria: Every CLI command example in documentation must be executable
• Validation: Copy-paste test of all documentation examples
• Success Metric: 100% of documentation examples execute successfully

AC-202: Error messages must be actionable and user-friendly.

• Criteria: Error messages include specific problem and resolution steps
• Validation: Test all error conditions, verify helpful messages
• Success Metric: Users can resolve errors using provided guidance

AC-203: Installation and setup instructions must be complete.

• Criteria: Step-by-step guide from zero to functional skill
• Validation: Fresh environment test following documentation
• Success Metric: New users can install and use skill within 15 minutes

2. Output Quality

AC-204: Metric data must include proper units and formatting.

• Criteria: Percentages shown as %, bytes as KB/MB/GB, rates as per-second
• Validation: Verify all metric values have appropriate units
• Success Metric: 100% of metric values properly formatted

AC-205: Time-based data must include timestamps in local timezone.

• Criteria: Convert UTC timestamps to user's local timezone when displaying
• Validation: Verify timestamp conversion accuracy
• Success Metric: Correct timezone handling

AC-206: Tabular data must be properly aligned and readable.

• Criteria: Use consistent column widths, proper alignment, clear headers
• Validation: Visual inspection of table outputs
• Success Metric: Readable tables on terminals 80+ characters wide

3. Usability Requirements

AC-207: The skill must support both interactive and scripted use.

• Criteria: Human-readable output for interactive use, JSON for scripting
• Validation: Test both --output table and --output json formats
• Success Metric: Both formats produce valid, complete data

AC-208: Common workflows must be streamlined.

• Criteria: Default to common metrics when none specified
• Validation: Test skill with minimal input, verify useful default behavior
• Success Metric: Useful output with minimal user input

AC-209: The skill must provide progress indicators for long operations.

• Criteria: Show progress for queries taking > 3 seconds
• Validation: Test with large time ranges, verify progress indication
• Success Metric: Users aware of operation progress

Integration Acceptance Criteria

1. CLI Integration

AC-301: The skill must work with standard Huawei Cloud CLI output formats.

• Criteria: Support json, table, text, and yaml output formats
• Validation: Test each output format with jq/grep validation
• Success Metric: All output formats produce valid, parseable data

AC-302: The skill must support command piping and redirection.

• Criteria: Output can be piped to other commands (grep, awk, jq, etc.)
• Validation: Test common pipe patterns
• Success Metric: Seamless integration with shell pipelines

AC-303: The skill must support environment variable configuration.

• Criteria: Use HUAWEICLOUD_ACCESS_KEY_ID, HUAWEICLOUD_SECRET_ACCESS_KEY, etc.
• Validation: Test configuration via environment variables
• Success Metric: Flexible credential management

2. Monitoring System Integration

AC-304: The skill must export data in monitoring-friendly formats.

• Criteria: Support Prometheus, Graphite, or other monitoring formats
• Validation: Verify format compatibility with common monitoring tools
• Success Metric: Direct integration with monitoring systems

AC-305: The skill must support alert threshold configuration.

• Criteria: Allow users to set custom thresholds for alerts
• Validation: Test threshold-based alerting functionality
• Success Metric: Configurable alerting

Quality Assurance Acceptance Criteria

1. Testing Coverage

AC-401: Unit tests must cover 80% of core functionality.

• Criteria: Code coverage report showing >= 80% line coverage
• Validation: Run coverage tools, generate report
• Success Metric: 80%+ test coverage

AC-402: Integration tests must cover all Huawei Cloud API interactions.

• Criteria: Test all ECS and CES API calls with mock responses
• Validation: Verify API error handling and response parsing
• Success Metric: 100% API interaction coverage

AC-403: End-to-end tests must validate complete workflows.

• Criteria: Test from authentication to data presentation
• Validation: Complete workflow tests with real Huawei Cloud environment
• Success Metric: All user workflows functional

2. Code Quality

AC-404: Code must follow Huawei Cloud CLI best practices.

• Criteria: Consistent with hcloud command patterns and conventions
• Validation: Code review against Huawei Cloud CLI style guide
• Success Metric: Adherence to established patterns

AC-405: Error handling must be comprehensive.

• Criteria: All possible errors caught and handled gracefully
• Validation: Error injection testing
• Success Metric: No unhandled exceptions

AC-406: Documentation must be complete and accurate.

• Criteria: All functions, parameters, and examples documented
• Validation: Documentation review against implementation
• Success Metric: 100% documentation accuracy

Compliance Acceptance Criteria

1. Huawei Cloud Standards

AC-501: The skill must comply with Huawei Cloud API conventions.

• Criteria: Use official Huawei Cloud CLI patterns and endpoints
• Validation: API usage review
• Success Metric: Compliance with Huawei Cloud standards

AC-502: The skill must respect Huawei Cloud rate limits.

• Criteria: Implement rate limiting and backoff as per Huawei Cloud guidelines
• Validation: Load testing within rate limits
• Success Metric: No rate limit violations in normal use

2. Security Standards

AC-503: The skill must not store credentials persistently.

• Criteria: Use Huawei Cloud CLI credential management only
• Validation: Audit credential handling code
• Success Metric: No credential storage

AC-504: The skill must validate all inputs for security.

• Criteria: Input validation to prevent injection attacks
• Validation: Security penetration testing
• Success Metric: No security vulnerabilities

Performance Benchmarks

1. Response Time Benchmarks

2. Resource Usage Benchmarks

Acceptance Testing Process

1. Pre-Acceptance Checklist

• [ ] All functional acceptance criteria met
• [ ] All technical acceptance criteria met
• [ ] All user experience acceptance criteria met
• [ ] All integration acceptance criteria met
• [ ] All quality assurance acceptance criteria met
• [ ] All compliance acceptance criteria met
• [ ] Performance benchmarks achieved
• [ ] Documentation complete and accurate
• [ ] Test coverage reports generated
• [ ] Security review completed

2. Acceptance Test Execution Example

The following is an example of acceptance test execution commands and process:

#!/bin/bash
# Acceptance test execution example script
# Note: This is an example process that needs to be adjusted based on actual test scripts

echo "=== Huawei Cloud ECS Monitoring Skill Acceptance Test ==="
echo "Start time: $(date)"

# 1. Set up test environment
echo "1. Setting up test environment..."
export HUAWEICLOUD_REGION="${HUAWEICLOUD_REGION:-cn-north-1}"
export TEST_INSTANCE_ID="${TEST_INSTANCE_ID:-}"
export TEST_OUTPUT_DIR="acceptance-test-results-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$TEST_OUTPUT_DIR"

# 2. Run acceptance test suite
echo "2. Running acceptance test suite..."
# This should call actual acceptance test scripts
# Example test commands:
echo "Test 1: Verifying Huawei Cloud CLI installation..."
if command -v hcloud &> /dev/null; then
    echo "✓ Huawei Cloud CLI is installed"
    echo "PASS" > "$TEST_OUTPUT_DIR/test-01-cli-installation.result"
else
    echo "✗ Huawei Cloud CLI is not installed"
    echo "FAIL" > "$TEST_OUTPUT_DIR/test-01-cli-installation.result"
fi

echo "Test 2: Verifying credential configuration..."
if hcloud configure list &> /dev/null; then
    echo "✓ Credentials configured correctly"
    echo "PASS" > "$TEST_OUTPUT_DIR/test-02-credentials.result"
else
    echo "✗ Credential configuration error"
    echo "FAIL" > "$TEST_OUTPUT_DIR/test-02-credentials.result"
fi

echo "Test 3: Verifying ECS instance listing..."
if hcloud ECS NovaListServers --cli-region="$HUAWEICLOUD_REGION" &> /dev/null; then
    echo "✓ ECS instance listing works"
    echo "PASS" > "$TEST_OUTPUT_DIR/test-03-ecs-list.result"
else
    echo "✗ ECS instance listing failed"
    echo "FAIL" > "$TEST_OUTPUT_DIR/test-03-ecs-list.result"
fi

echo "Test 4: Verifying CES metric query..."
if hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region="$HUAWEICLOUD_REGION" &> /dev/null; then
    echo "✓ CES metric query works"
    echo "PASS" > "$TEST_OUTPUT_DIR/test-04-ces-query.result"
else
    echo "✗ CES metric query failed"
    echo "FAIL" > "$TEST_OUTPUT_DIR/test-04-ces-query.result"
fi

# 3. Generate acceptance report
echo "3. Generating acceptance report..."
REPORT_FILE="$TEST_OUTPUT_DIR/acceptance-report.json"
cat > "$REPORT_FILE" << EOF
{
  "acceptance_test_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
  "skill": "huawei-cloud-ces-ecs-monitoring",
  "environment": {
    "region": "$HUAWEICLOUD_REGION",
    "platform": "$(uname -s)"
  },
  "test_results": {
    "total_tests": 2,
    "passed": 0,
    "failed": 0,
    "success_rate": "0%"
  },
  "criteria_validation": {
    "functional_criteria": "PENDING",
    "technical_criteria": "PENDING",
    "ux_criteria": "PENDING",
    "integration_criteria": "PENDING",
    "qa_criteria": "PENDING",
    "compliance_criteria": "PENDING"
  },
  "notes": "This is an example acceptance report. Actual acceptance requires execution of the complete test suite."
}
EOF

# 4. Verify all acceptance criteria
echo "4. Verifying acceptance criteria..."
echo "Verifying requirements according to acceptance criteria document:"
echo "- Functional acceptance criteria (AC-001 to AC-014)"
echo "- Technical acceptance criteria (AC-101 to AC-109)"
echo "- User experience acceptance criteria (AC-201 to AC-209)"
echo "- Integration acceptance criteria (AC-301 to AC-303)"
echo "- Quality assurance acceptance criteria (AC-401 to AC-406)"
echo "- Compliance acceptance criteria (AC-501 to AC-504)"

echo "Acceptance test execution completed"
echo "Report file: $REPORT_FILE"
echo "Output directory: $TEST_OUTPUT_DIR"

References

• Huawei Cloud official documentation
• CLI reference documentation

Huawei Cloud ECS Monitoring Best Practices

Overview

This document provides best practices for monitoring Huawei Cloud ECS instances using Cloud Eye Service (CES). Following these practices ensures effective monitoring, optimal performance, and cost efficiency.

Monitoring Strategy

1. Define Monitoring Objectives

• Availability: Ensure instances are running and accessible
• Performance: Monitor resource utilization and response times
• Capacity: Track resource usage trends for scaling decisions
• Cost: Optimize resource allocation to control costs
• Security: Monitor for suspicious activities and anomalies

2. Establish Baseline Metrics

# Collect baseline SYS.ECS metrics during normal operation
# Run for at least 7 days to establish patterns
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="mem_util" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="<instance-uuid>" \
  --metrics.3.namespace="SYS.ECS" \
  --metrics.3.metric_name="disk_read_bytes_rate" \
  --metrics.3.dimensions.1.name="instance_id" \
  --metrics.3.dimensions.1.value="<instance-uuid>" \
  --metrics.4.namespace="SYS.ECS" \
  --metrics.4.metric_name="disk_write_bytes_rate" \
  --metrics.4.dimensions.1.name="instance_id" \
  --metrics.4.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '7 days ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=3600 \
  --filter=average \
  --cli-region=<region-id>

Metric Selection Guidelines

Namespace Fallback Strategy

When a SYS.ECS metric returns no data, try the corresponding AGT.ECS metric. See references/ces-metrics-reference.md for the complete fallback mapping table.

Essential Metrics - SYS.ECS (No Agent Required)

# CPU and Memory
cpu_util
mem_util

# Disk
disk_util_inband
disk_read_bytes_rate
disk_write_bytes_rate

# Network
network_incoming_bytes_rate_inband
network_outgoing_bytes_rate_inband

Essential Metrics - AGT.ECS (Agent Required)

# CPU and Memory (OS-level)
cpu_usage
mem_usedPercent

# Disk (per mount point)
disk_usedPercent

# System
load_average1
net_tcp_total

Performance Metrics - AGT.ECS (For Performance-Sensitive Applications)

# Disk IOPS and Latency
disk_agt_read_requests_rate
disk_agt_write_requests_rate
disk_readTime
disk_writeTime

# Network Card
net_bitRecv
net_bitSent
net_errin
net_errout

Capacity Metrics - AGT.ECS (For Capacity Planning)

# Disk Space (per mount point)
disk_usedPercent
disk_inodesUsedPercent

# Memory (GB)
mem_available
mem_free

# Process
proc_total_count
proc_zombie_count

Alerting Strategy

1. Critical Alerts (Immediate Action Required)

# CPU > 95% for 5 minutes (SYS.ECS)
- Metric: cpu_util
- Namespace: SYS.ECS
- Threshold: > 95%
- Duration: 5 minutes
- Action: Auto-scale or investigate immediately

# Memory > 95% for 5 minutes (SYS.ECS)
- Metric: mem_util
- Namespace: SYS.ECS
- Threshold: > 95%
- Duration: 5 minutes
- Action: Investigate memory leak or scale

# Disk > 95% for 15 minutes (AGT.ECS)
- Metric: disk_usedPercent
- Namespace: AGT.ECS
- Threshold: > 95%
- Duration: 15 minutes
- Action: Clean up or expand disk

2. Warning Alerts (Investigation Required)

# CPU > 80% for 15 minutes (SYS.ECS)
- Metric: cpu_util
- Namespace: SYS.ECS
- Threshold: > 80%
- Duration: 15 minutes
- Action: Monitor and plan scaling

# Memory > 85% for 15 minutes (SYS.ECS)
- Metric: mem_util
- Namespace: SYS.ECS
- Threshold: > 85%
- Duration: 15 minutes
- Action: Optimize or add memory

# Disk > 85% for 1 hour (AGT.ECS)
- Metric: disk_usedPercent
- Namespace: AGT.ECS
- Threshold: > 85%
- Duration: 1 hour
- Action: Plan disk expansion

3. Informational Alerts (Trend Monitoring)

# Network errors > 0 (AGT.ECS)
- Metric: net_errin, net_errout
- Namespace: AGT.ECS
- Threshold: > 0
- Duration: 5 minutes
- Action: Investigate network issues

# Load average > CPU cores (AGT.ECS)
- Metric: load_average1
- Namespace: AGT.ECS
- Threshold: > (CPU cores)
- Duration: 15 minutes
- Action: Check for process issues

# Zombie processes > 0 (AGT.ECS)
- Metric: proc_zombie_count
- Namespace: AGT.ECS
- Threshold: > 0
- Duration: 5 minutes
- Action: Investigate zombie processes

Monitoring Frequency Recommendations

Real-Time Monitoring (High Frequency)

# For critical production instances
# SYS.ECS: minimum period is 300 (5 minutes)
--period "300"

# AGT.ECS: minimum period is 60 (1 minute)
--period "60"
# Retention: 2 days
# Use for: Immediate alerting, troubleshooting

Operational Monitoring (Standard)

# For most production instances
--period "300"  # 5-minute intervals
# Retention: 15 days
# Use for: Daily operations, performance analysis

Trend Analysis (Low Frequency)

# For capacity planning and cost analysis
--period "3600"  # 1-hour intervals
# Retention: 30 days
--period "86400"  # 1-day intervals
# Retention: 1 year

Cost Optimization

1. Selective Monitoring

# Monitor only necessary metrics
# Avoid monitoring all metrics for all instances

# Production instances: Full monitoring
# Development instances: Essential metrics only
# Test instances: Minimal monitoring

2. Appropriate Aggregation

# Use appropriate periods based on need
# SYS.ECS: minimum 5-minute (300)
# AGT.ECS: minimum 1-minute (60)
# Daily ops: 5-minute (300)
# Trends: 1-hour (3600)

3. Data Retention Policy

# Real-time data: Keep for 2-7 days
# Daily aggregates: Keep for 30 days
# Monthly aggregates: Keep for 1 year
# Export to OBS for long-term storage

4. Instance Tagging for Cost Allocation

# Tag instances by environment, team, project
hcloud ECS TagServer <instance-uuid> \
  --tags "Environment=Production,Team=Backend,Project=ECommerce" \
  --cli-region <region-id>

Performance Optimization

1. Efficient Query Patterns

# Batch multiple metrics in single query (more efficient)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="mem_util" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter=average \
  --cli-region <region-id>

# Instead of separate queries for each metric

2. Cache Frequently Used Data

# Cache instance list (valid for 5 minutes)
INSTANCE_CACHE_FILE="/tmp/ecs_instances_$(date +%Y%m%d%H%M).json"
if [ ! -f "$INSTANCE_CACHE_FILE" ] || [ "$(find "$INSTANCE_CACHE_FILE" -mmin +5)" ]; then
  hcloud ECS NovaListServers --cli-region=<region-id> > "$INSTANCE_CACHE_FILE"
fi

3. Use Appropriate Time Ranges

# For real-time monitoring: Last 1 hour
--from=$(date -d '1 hour ago' +%s)000
--to=$(date +%s)000

# For daily review: Last 24 hours
--from=$(date -d '24 hours ago' +%s)000
--to=$(date +%s)000

# For weekly review: Last 7 days
--from=$(date -d '7 days ago' +%s)000
--to=$(date +%s)000

Security Monitoring

1. Unauthorized Access Detection

# Monitor SSH login attempts
# (Requires custom metric from OS monitoring agent)

2. Resource Abuse Detection

# High CPU with network traffic (possible mining)
# Monitor pattern: cpu_util > 90% AND network_outgoing_bytes_rate_inband > 10MB/s

# High disk IO with low CPU (possible data exfiltration)
# Monitor pattern: disk_write_bytes_rate > 50MB/s AND cpu_util < 20%

3. Configuration Drift Detection

# Monitor for unexpected instance state changes
# Track: instance status, security group changes, image changes

Automation and Integration

1. Automated Health Checks

#!/bin/bash
# Daily health check script

INSTANCE_ID="$1"
REGION="$2"

# Check CPU (SYS.ECS)
CPU_USAGE=$(hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$INSTANCE_ID" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter=average \
  --cli-region "$REGION" \
  --output json | jq '.metrics[0].datapoints[-1].average')

if (( $(echo "$CPU_USAGE > 80" | bc -l) )); then
  echo "WARNING: High CPU usage: $CPU_USAGE%"
  # Send alert
fi

# Similar checks for memory, disk, network...

2. Integration with Notification Services

# Integrate with SMN (Simple Message Notification) for alerts
# Use webhooks for integration with external systems
# Schedule regular reports via email

3. Automated Scaling Triggers

# Monitor metrics and trigger auto-scaling
# Example: Scale out when CPU > 70% for 10 minutes
# Example: Scale in when CPU < 30% for 30 minutes

Reporting and Visualization

1. Daily Summary Report

#!/bin/bash
# Generate daily monitoring report

generate_daily_report() {
  INSTANCE_ID="$1"
  REGION="$2"
  REPORT_DATE="$(date +%Y-%m-%d)"

  echo "=== Daily Monitoring Report ==="
  echo "Instance: $INSTANCE_ID"
  echo "Date: $REPORT_DATE"
  echo "Region: $REGION"
  echo ""

  # Get metrics for yesterday
  START_TIME=$(date -d 'yesterday 00:00:00' +%s)000
  END_TIME=$(date -d 'yesterday 23:59:59' +%s)000

  hcloud CES BatchListMetricData \
    --metrics.1.namespace="SYS.ECS" \
    --metrics.1.metric_name="cpu_util" \
    --metrics.1.dimensions.1.name="instance_id" \
    --metrics.1.dimensions.1.value="$INSTANCE_ID" \
    --metrics.2.namespace="SYS.ECS" \
    --metrics.2.metric_name="mem_util" \
    --metrics.2.dimensions.1.name="instance_id" \
    --metrics.2.dimensions.1.value="$INSTANCE_ID" \
    --from="$START_TIME" \
    --to="$END_TIME" \
    --period=3600 \
    --filter=average \
    --cli-region "$REGION"
}

2. Performance Trend Analysis

#!/bin/bash
# Analyze weekly performance trends

analyze_weekly_trends() {
  INSTANCE_ID="$1"
  REGION="$2"

  echo "=== Weekly Performance Trends ==="
  echo "Instance: $INSTANCE_ID"
  echo "Period: Last 7 days"
  echo ""

  for DAY in {0..6}; do
    DAY_DATE="$(date -d "$DAY days ago" +%Y-%m-%d)"
    START_TIME=$(date -d "$DAY days ago 00:00:00" +%s)000
    END_TIME=$(date -d "$DAY days ago 23:59:59" +%s)000
    echo "Date: $DAY_DATE"

    hcloud CES BatchListMetricData \
      --metrics.1.namespace="SYS.ECS" \
      --metrics.1.metric_name="cpu_util" \
      --metrics.1.dimensions.1.name="instance_id" \
      --metrics.1.dimensions.1.value="$INSTANCE_ID" \
      --from="$START_TIME" \
      --to="$END_TIME" \
      --period=3600 \
      --filter=average \
      --cli-region "$REGION"
    echo ""
  done
}

References

• Huawei Cloud CES Best Practices
• ECS Performance Optimization

Huawei Cloud CES Metrics Reference for ECS

Overview

Cloud Eye Service (CES) provides comprehensive monitoring metrics for Elastic Cloud Server (ECS) instances. This document lists all available metrics, their meanings, and usage guidelines.

Metric Namespaces

SYS.ECS - Base Monitoring (No Agent Required)

Basic monitoring metrics collected by the system at the virtualization layer. Available for all ECS instances without additional software installation.

• Minimum granularity: 5 minutes (period=300)
• Collection period: XEN instances = 4 minutes; KVM/QingTian instances = 5 minutes
• Note: Some metrics (mem_util, disk_util_inband, network_*_inband) require UVP VMTools to be installed on the image. QingTian instances do not support mem_util.

AGT.ECS - OS Monitoring (Agent Required)

OS-level monitoring metrics collected by the Telescope agent. Requires the agent to be installed and running on the ECS instance.

• Minimum granularity: 1 minute (period=60)
• Agent installation check: See references/troubleshooting-guide.md (Section 10: Agent-Related Issues) for agent verification steps.

Choosing the Right Namespace

• Use SYS.ECS when:
• You need basic monitoring without installing additional software
• You want metrics available immediately for any ECS instance
• You need network metrics (all network metrics are SYS.ECS)

• Use AGT.ECS when:
• You need OS-level details (process, load average, TCP connections)
• You need 1-minute granularity (SYS.ECS minimum is 5 minutes)
• You need disk usage per mount point
• You need inode usage monitoring
• The Telescope agent is installed on the instance

SYS.ECS to AGT.ECS Metric Fallback

When a SYS.ECS metric returns no data (e.g., UVP VMTools not installed on the image), try the corresponding AGT.ECS metric:

Note: Network metrics (network_incoming_bytes_rate_inband, network_outgoing_bytes_rate_inband, etc.) only exist in SYS.ECS namespace. There is no AGT.ECS equivalent for network metrics.

SYS.ECS Metrics (Base Monitoring)

CPU Metrics

cpu_util

• Description: CPU usage rate (monitored by the system at the physical machine level)
• Formula: Single ECS CPU usage / total CPU cores
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Alarm Thresholds:
• Warning: > 80%
• Critical: > 95%

cpu_credit_usage (T6 instances only)

• Description: CPU credit consumption rate of a T6 burstable instance. 1 credit = 1 vCPU at 100% for 1 minute
• Unit: Credit
• Range: >= 0
• Dimensions: instance_id

cpu_credit_balance (T6 instances only)

• Description: CPU credit balance of a T6 burstable instance (accumulated since startup)
• Unit: Credit
• Range: >= 0
• Dimensions: instance_id

Memory Metrics

mem_util

• Description: Memory usage rate (monitored by the system)
• Formula: Used memory / Total memory
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Alarm Thresholds:
• Warning: > 85%
• Critical: > 95%
• Note: Requires UVP VMTools installed on the image. QingTian instances do not support this metric.

Disk Metrics

disk_util_inband

• Description: Disk usage rate (monitored by the system)
• Formula: Used disk capacity / Total disk capacity
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Alarm Thresholds:
• Warning: > 80%
• Critical: > 90%
• Note: Requires UVP VMTools installed on the image.

disk_read_bytes_rate

• Description: Disk read bandwidth
• Formula: Sum of read bytes / measurement period
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Typical Values:
• HDD: 50-150 MB/s
• SSD: 200-500 MB/s
• Ultra-high performance: 1+ GB/s

disk_write_bytes_rate

• Description: Disk write bandwidth
• Formula: Sum of written bytes / measurement period
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

disk_read_requests_rate

• Description: Disk read IOPS (Input/Output Operations Per Second)
• Unit: Request/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Typical Values:
• HDD: 50-150 IOPS
• SSD: 3,000-20,000 IOPS
• Ultra-high performance: 50,000+ IOPS

disk_write_requests_rate

• Description: Disk write IOPS
• Unit: Request/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

Network Metrics

network_incoming_bytes_rate_inband

• Description: In-band network incoming rate (bytes received per second from within the ECS)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Note: Requires UVP VMTools. Not available with SRIOV.

network_outgoing_bytes_rate_inband

• Description: In-band network outgoing rate (bytes sent per second from within the ECS)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Note: Requires UVP VMTools. Not available with SRIOV.

network_incoming_bytes_aggregate_rate

• Description: Out-of-band network incoming rate (bytes received per second from virtualization layer)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Note: Not available with SRIOV.

network_outgoing_bytes_aggregate_rate

• Description: Out-of-band network outgoing rate (bytes sent per second from virtualization layer)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)
• Note: Not available with SRIOV.

network_vm_connections

• Description: Total used TCP and UDP connections
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

network_vm_bandwidth_in

• Description: VM inbound bandwidth (total bytes received per second, public + internal)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

network_vm_bandwidth_out

• Description: VM outbound bandwidth (total bytes sent per second, public + internal)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

network_vm_pps_in

• Description: VM inbound PPS (total packets received per second, public + internal)
• Unit: Packet/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

network_vm_pps_out

• Description: VM outbound PPS (total packets sent per second, public + internal)
• Unit: Packet/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

network_vm_newconnections

• Description: VM new connections rate (TCP, UDP, ICMP)
• Unit: connect/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 300 (5 minutes)

AGT.ECS Metrics (OS Monitoring - Agent Required)

CPU Metrics

cpu_usage

• Description: CPU usage rate (monitored by the agent)
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)
• Alarm Thresholds:
• Warning: > 80%
• Critical: > 95%

cpu_usage_idle

• Description: CPU idle time ratio
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

cpu_usage_user

• Description: User space CPU usage rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

cpu_usage_system

• Description: Kernel space CPU usage rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

cpu_usage_iowait

• Description: CPU I/O wait time ratio
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

cpu_usage_other

• Description: Other CPU usage rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

cpu_usage_nice

• Description: Nice process CPU usage rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

cpu_usage_irq

• Description: CPU interrupt time ratio
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

cpu_usage_softirq

• Description: CPU soft interrupt time ratio
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

CPU Load Metrics

Note: Windows does not support CPU load metrics.

load_average1

• Description: 1-minute single-core average load
• Unit: (none)
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)
• Interpretation:
• < CPU cores: Normal
• = CPU cores: Fully loaded
• > CPU cores: Overloaded

load_average5

• Description: 5-minute single-core average load
• Unit: (none)
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

load_average15

• Description: 15-minute single-core average load
• Unit: (none)
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

Memory Metrics

mem_usedPercent

• Description: Memory usage rate (monitored by the agent)
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)
• Alarm Thresholds:
• Warning: > 85%
• Critical: > 95%

mem_available

• Description: Available memory
• Unit: GB
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

mem_free

• Description: Free memory
• Unit: GB
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

mem_buffers

• Description: Buffer memory occupancy
• Unit: GB
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

mem_cached

• Description: Cache memory occupancy
• Unit: GB
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

total_open_files

• Description: Total file handles
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

Disk Space Metrics

Note: The mount_point dimension value is a hash identifier (e.g., 6666cd76f96956469e7be39d750cc7d9), obtainable via the CES "query host monitoring dimension metric information" API, NOT the actual path like /.

disk_usedPercent

• Description: Disk usage rate (monitored by the agent)
• Unit: %
• Range: 0-100
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)
• Alarm Thresholds:
• Warning: > 80%
• Critical: > 90%

disk_free

• Description: Free disk space
• Unit: GB
• Range: >= 0
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)

disk_total

• Description: Total disk space
• Unit: GB
• Range: >= 0
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)

disk_used

• Description: Used disk space
• Unit: GB
• Range: >= 0
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)

Disk I/O Metrics

Note: Disk I/O metrics support two dimension combinations: instance_id,disk or instance_id,mount_point. The disk dimension value is the disk identifier (e.g., vda, vdb).

disk_agt_read_bytes_rate

• Description: Disk read rate (monitored by the agent)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_agt_write_bytes_rate

• Description: Disk write rate (monitored by the agent)
• Unit: B/s
• Range: >= 0
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_agt_read_requests_rate

• Description: Disk read operation rate (IOPS)
• Unit: Request/s
• Range: >= 0
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_agt_write_requests_rate

• Description: Disk write operation rate (IOPS)
• Unit: Request/s
• Range: >= 0
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_readTime

• Description: Read operation average latency
• Unit: ms/Count
• Range: >= 0
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_writeTime

• Description: Write operation average latency
• Unit: ms/Count
• Range: >= 0
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_ioUtils

• Description: Disk I/O usage rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_queue_length

• Description: Average disk queue length
• Unit: Count
• Range: >= 0
• Dimensions: instance_id,disk or instance_id,mount_point
• Minimum Period: 60 (1 minute)

disk_device_used_percent

• Description: Block device usage rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id,disk
• Minimum Period: 60 (1 minute)

File System Metrics

Note: Windows does not support file system metrics.

disk_fs_rwstate

• Description: File system read/write state
• Unit: (none)
• Range: 0 = read-write, 1 = read-only
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)

disk_inodesTotal

• Description: Inode space size
• Unit: Count
• Range: >= 0
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)

disk_inodesUsed

• Description: Inode used space
• Unit: Count
• Range: >= 0
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)

disk_inodesUsedPercent

• Description: Inode usage ratio
• Unit: %
• Range: 0-100
• Dimensions: instance_id, mount_point
• Minimum Period: 60 (1 minute)
• Alarm Thresholds:
• Warning: > 80%
• Critical: > 90%

Network Card Metrics

net_bitRecv

• Description: NIC inbound bandwidth
• Unit: bit/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_bitSent

• Description: NIC outbound bandwidth
• Unit: bit/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_packetRecv

• Description: NIC packet receive rate
• Unit: Counts/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_packetSent

• Description: NIC packet send rate
• Unit: Counts/s
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_errin

• Description: Receive error packet rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)
• Alarm Threshold: > 0 (any errors should be investigated)

net_errout

• Description: Send error packet rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)
• Alarm Threshold: > 0

net_dropin

• Description: Receive drop packet rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_dropout

• Description: Send drop packet rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

TCP Connection Metrics

net_tcp_total

• Description: Total TCP connections
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_tcp_established

• Description: TCP connections in ESTABLISHED state
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_tcp_time_wait

• Description: TCP connections in TIME_WAIT state
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_tcp_close_wait

• Description: TCP connections in CLOSE_WAIT state
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

net_tcp_retrans

• Description: TCP retransmission rate
• Unit: %
• Range: 0-100
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

Process Metrics

proc_running_count

• Description: Running process count
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

proc_idle_count

• Description: Idle process count
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

proc_zombie_count

• Description: Zombie process count
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)
• Alarm Threshold: > 0 (any zombie processes should be investigated)

proc_blocked_count

• Description: Blocked process count
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

proc_total_count

• Description: System process count
• Unit: Count
• Range: >= 0
• Dimensions: instance_id
• Minimum Period: 60 (1 minute)

Metric Dimensions

Dimension Format

For ShowMetricData (GET API): Use separate --dim.N parameters:

# Single dimension
--dim.0="instance_id,<uuid>"

# Two dimensions (separate entries, NOT comma-concatenated)
--dim.0="instance_id,<uuid>" --dim.1="mount_point,<hash>"

For BatchListMetricData (POST API): Use indexed name/value pairs:

# Single dimension
--metrics.1.dimensions.1.name="instance_id"
--metrics.1.dimensions.1.value="<uuid>"

# Two dimensions
--metrics.1.dimensions.1.name="instance_id"
--metrics.1.dimensions.1.value="<uuid>"
--metrics.1.dimensions.2.name="mount_point"
--metrics.1.dimensions.2.value="<hash>"

Instance ID Format

The instance_id value is the ECS UUID (e.g., 3d65c1ac-9a9f-4c5f-a054-35184a087bb2), NOT the i-xxxxxxxx format.

mount_point Format (AGT.ECS)

The mount_point value is a hash identifier (e.g., 6666cd76f96956469e7be39d750cc7d9), NOT the actual mount path like /. Obtain the hash value via the CES "query host monitoring dimension metric information" API.

disk Dimension (AGT.ECS)

The disk dimension value is the disk identifier (e.g., vda, vdb).

Common Monitoring Scenarios

1. Performance Baseline (SYS.ECS)

# Get baseline CPU and memory metrics for last 24 hours
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="mem_util" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '24 hours ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=3600 \
  --filter=average \
  --cli-region=<region-id>

2. Troubleshooting High CPU (AGT.ECS - Agent Required)

# Check CPU usage with load average
hcloud CES BatchListMetricData \
  --metrics.1.namespace="AGT.ECS" \
  --metrics.1.metric_name="cpu_usage" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.2.namespace="AGT.ECS" \
  --metrics.2.metric_name="load_average1" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=60 \
  --filter=average \
  --cli-region=<region-id>

3. Disk Performance Analysis (SYS.ECS)

# Analyze disk IO patterns
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="disk_read_bytes_rate" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="disk_write_bytes_rate" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="<instance-uuid>" \
  --metrics.3.namespace="SYS.ECS" \
  --metrics.3.metric_name="disk_read_requests_rate" \
  --metrics.3.dimensions.1.name="instance_id" \
  --metrics.3.dimensions.1.value="<instance-uuid>" \
  --metrics.4.namespace="SYS.ECS" \
  --metrics.4.metric_name="disk_write_requests_rate" \
  --metrics.4.dimensions.1.name="instance_id" \
  --metrics.4.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '6 hours ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter=average \
  --cli-region=<region-id>

4. Network Bottleneck Detection (SYS.ECS)

# Monitor network performance
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="network_incoming_bytes_rate_inband" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="network_outgoing_bytes_rate_inband" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter=average \
  --cli-region=<region-id>

5. Disk Usage Monitoring (AGT.ECS - Agent Required)

# Check disk usage per mount point
hcloud CES BatchListMetricData \
  --metrics.1.namespace="AGT.ECS" \
  --metrics.1.metric_name="disk_usedPercent" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.1.dimensions.2.name="mount_point" \
  --metrics.1.dimensions.2.value="<mount-point-hash>" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=60 \
  --filter=average \
  --cli-region=<region-id>

6. List Available Metrics

# List all available metrics for ECS namespace
hcloud CES ListMetrics \
  --namespace="SYS.ECS" \
  --cli-region=<region-id>

# List metrics for specific instance
hcloud CES ListMetrics \
  --namespace="SYS.ECS" \
  --dim.0="instance_id,<instance-uuid>" \
  --cli-region=<region-id>

Alerting Thresholds Summary

For alerting thresholds and strategy details, see references/best-practices.md.

Best Practices

For metric selection guidelines, data retention policies, and cost optimization, see references/best-practices.md.

Limitations

1. SYS.ECS minimum granularity: 5 minutes (period=300) 2. AGT.ECS minimum granularity: 1 minute (period=60) 3. Maximum data points: 1440 per query 4. Retention period: Varies by metric type 5. Query frequency: 500 calls/minute for ShowMetricData/BatchListMetricData; 300 calls/minute for ListMetrics 6. Batch query limit: Max 500 metrics per BatchListMetricData request 7. Dimensions: Max 4 dimensions per metric

References

• Huawei Cloud CES Documentation
• ECS Monitoring Metrics
• CES API Reference

Huawei Cloud KooCLI Installation Guide

Overview

Huawei Cloud KooCLI (hcloud) is the official Huawei Cloud command-line tool that supports managing 100+ cloud services. This guide provides complete installation, configuration, and verification processes.

Version Requirements

• Minimum version: One major version before the latest major version (e.g., if current is 7.x.x, then not lower than 6.x.x)
• Latest version: Refer to https://support.huaweicloud.com/wtsnew-hcli/index.html
• Verification command: hcloud version
• Update command: hcloud update

Quick Installation (All Platforms)

One-click Installation

# Download and run official installation script (interactive)
curl -sSL https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/hcloud_install.sh -o ./hcloud_install.sh && bash ./hcloud_install.sh

# Non-interactive installation (skip confirmation)
curl -sSL https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/hcloud_install.sh -o ./hcloud_install.sh && bash ./hcloud_install.sh -y

Verify Installation

# Check version
hcloud version
# Expected output: Current KooCLI version: 7.2.2

# Check help
hcloud --help

Installation Methods for Each Platform

1. Linux Systems

Detect System Architecture

echo $HOSTTYPE
# x86_64: AMD 64-bit system
# aarch64: ARM 64-bit system

Step-by-step Installation

# AMD 64-bit system
curl -LO "https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/huaweicloud-cli-linux-amd64.tar.gz"
tar -zxvf huaweicloud-cli-linux-amd64.tar.gz
sudo mv hcloud /usr/local/bin/

# ARM 64-bit system
curl -LO "https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/huaweicloud-cli-linux-arm64.tar.gz"
tar -zxvf huaweicloud-cli-linux-arm64.tar.gz
sudo mv hcloud /usr/local/bin/

2. macOS Systems

Detect System Architecture

echo $HOSTTYPE
# If empty, use:
uname -a
# x86_64: AMD 64-bit system (Intel chips)
# arm64: ARM 64-bit system (Apple Silicon)

Step-by-step Installation

# Intel chips (AMD 64-bit)
curl -LO "https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/huaweicloud-cli-mac-amd64.tar.gz"
tar -zxvf huaweicloud-cli-mac-amd64.tar.gz
sudo mv hcloud /usr/local/bin/

# Apple Silicon (ARM 64-bit)
curl -LO "https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/huaweicloud-cli-mac-arm64.tar.gz"
tar -zxvf huaweicloud-cli-mac-arm64.tar.gz
sudo mv hcloud /usr/local/bin/

3. Windows Systems

Installation Steps

1. Download: https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/huaweicloud-cli-windows-amd64.zip 2. Extract ZIP file to get hcloud.exe 3. Add the directory containing hcloud.exe to PATH environment variable

Verify Installation

hcloud version
# Expected output: Current KooCLI version: 7.2.2

4. Docker Environment

Using Official Image

# Pull and run
docker run --rm -it swr.cn-north-4.myhuaweicloud.com/huawei-cloud/koocli:latest version

Custom Image

FROM ubuntu:latest
RUN apt-get update -y && apt-get install curl -y
RUN curl -sSL https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/hcloud_install.sh -o ./hcloud_install.sh && bash ./hcloud_install.sh -y
WORKDIR /workspace
ENTRYPOINT ["/usr/local/bin/hcloud"]

Build and use:

# Build image
docker build -t hcloudcli .

# Run command
docker run --rm -it hcloudcli version

Post-installation Configuration

Auto-completion Configuration

# Enable auto-completion
hcloud auto-complete on

# Reload shell configuration
# Bash: source ~/.bashrc
# Zsh: source ~/.zshrc

Multi-environment Configuration

# Create development environment configuration
hcloud configure init --cli-profile dev

# Create test environment configuration
hcloud configure init --cli-profile test

# Create production environment configuration
hcloud configure init --cli-profile prod

# View all configurations
hcloud configure list

# Use specific configuration
hcloud ECS NovaListServers --cli-profile=dev --cli-region=cn-north-4

Troubleshooting

Common Installation Issues

Insufficient Permissions

# Linux/macOS: Use sudo
sudo bash ./hcloud_install.sh

# Or install to user directory
curl -sSL https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/hcloud_install.sh -o ./hcloud_install.sh && bash ./hcloud_install.sh -d ~/.local/bin

Command Not Found

# Check PATH
echo $PATH
which hcloud

# Manually add to PATH
export PATH=$PATH:/usr/local/bin
# Or
export PATH=$PATH:$(pwd)

sha256sum Command Not Found

# Ubuntu/Debian:
sudo apt-get install coreutils

# CentOS/RHEL:
sudo yum install coreutils

# macOS:
brew install coreutils

Network Issues

# Test network connection
ping cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com

# Use proxy
export http_proxy=http://proxy:port
export https_proxy=http://proxy:port

Updates and Maintenance

Update KooCLI

# Interactive update
hcloud update

# Or reinstall latest version
curl -sSL https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/hcloud_install.sh -o ./hcloud_install.sh && bash ./hcloud_install.sh -y

Uninstall KooCLI

# Linux/macOS manual uninstallation:
sudo rm -f /usr/local/bin/hcloud
sudo rm -rf /usr/local/hcloud/
rm -rf ~/.hcloud/

# Windows manual uninstallation:
# 1. Delete hcloud.exe file
# 2. Remove relevant directory from PATH
# 3. Delete C:\Users\{username}\.hcloud\ directory

Clean Cache

# Clean KooCLI cache
rm -rf ~/.hcloud/cache/

# Clean downloaded files
rm -f hcloud_install.sh huaweicloud-cli-*.tar.gz huaweicloud-cli-*.zip

Best Practices

Version Management

• Use fixed versions for production environments
• Use latest versions for test environments
• Record installed version numbers

Environment Isolation

• Create independent configurations for different environments
• Use environment variables to store sensitive information
• Regularly backup configuration files

Automated Deployment

#!/bin/bash
set -e

# Download installation script
curl -sSL https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/hcloud_install.sh -o ./hcloud_install.sh

# Install KooCLI
bash ./hcloud_install.sh -y

# Verify installation
hcloud version

Docker Best Practices

# Use volume to persist configuration
docker run --rm -it \
  -v ~/.hcloud:/root/.hcloud \
  swr.cn-north-4.myhuaweicloud.com/huawei-cloud/koocli:latest \
  version

# Create alias to simplify usage
alias hcloud-docker='docker run --rm -it -v ~/.hcloud:/root/.hcloud swr.cn-north-4.myhuaweicloud.com/huawei-cloud/koocli:latest'

Version Compatibility

• KooCLI 7.2.2+: Supports --cli-x-project-id parameter
• KooCLI 6.2.4+: Supports --cli-auth-type parameter
• KooCLI 5.3.4+: Supports SSO configuration parameters

---

Tip: This guide provides complete KooCLI installation and configuration processes. Please strictly follow security rules to protect credential security.

IAM Policies for Huawei Cloud ECS Monitoring

Overview

This document outlines the Identity and Access Management (IAM) policies and permissions required for the Huawei Cloud ECS Monitoring skill. Proper IAM configuration ensures secure and controlled access to Huawei Cloud resources.

Required IAM Permissions Matrix

Read-Only Permissions - Monitoring and Viewing

Read-Write Permissions - Management and Operations (Optional)

Detailed Permission Requirements

Minimum Required Permissions

The skill requires the following IAM permissions to function properly:

{
  "Version": "1.1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:cloudServers:list",
        "ecs:cloudServers:get",
        "ces:metrics:list",
        "ces:metricData:get"
      ],
      "Resource": ["*"]
    }
  ]
}

Recommended Permissions (Full Access)

For full functionality, including alarm management and advanced features:

{
  "Version": "1.1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        // ECS Permissions
        "ecs:cloudServers:list",
        "ecs:cloudServers:get",
        "ecs:cloudServers:action",
        "ecs:cloudServerFlavors:list",
        "ecs:cloudServerImages:list",
        
        // CES Permissions
        "ces:metrics:list",
        "ces:metricData:get",
        "ces:alarms:list",
        "ces:alarms:get",
        "ces:alarmTemplates:list",
        "ces:alarmTemplates:get",
        "ces:alarmRules:list",
        "ces:alarmRules:get",
        
        // IAM Permissions (for credential verification)
        "iam:users:getUser",
        "iam:users:listUsers",
        "iam:permissions:listPermissionsForUser"
      ],
      "Resource": ["*"]
    }
  ]
}

Permission Failure Handling Process (MUST)

Permission Check Failure Handling

When any operation encounters a permission error failure, MUST follow this process:

1. Identify permission error

• Check if error message contains keywords like "Access denied", "Insufficient permissions", "User does not have permission"
• Confirm if error is related to IAM permissions

2. Refer to permission documentation

• Immediately guide user to view references/iam-policies.md file
• Provide specific permission requirements explanation

3. Display permission list

• Show user the required permission list and corresponding JSON policy
• Explain the purpose and necessity of each permission

4. Guide permission configuration

• Guide user to create custom policy in Huawei Cloud IAM console
• Provide specific operation steps:

a. Log in to Huawei Cloud Console b. Navigate to IAM service c. Create custom policy d. Copy the provided JSON policy e. Assign policy to user/user group

5. Pause execution and wait for confirmation

• Pause current operation execution
• Wait for user to confirm permission configuration is complete
• Provide verification commands for user to test permissions

Permission Details

ECS (Elastic Cloud Server) Permissions

1. ecs:cloudServers:list

• Description: List ECS instances
• Required for: Listing available instances
• API Operation: GET /v1/{project_id}/cloudservers
• Use Case: User wants to see all ECS instances

2. ecs:cloudServers:get

• Description: Get ECS instance details
• Required for: Getting instance metadata
• API Operation: GET /v1/{project_id}/cloudservers/{server_id}
• Use Case: User selects a specific instance for monitoring

3. ecs:cloudServers:action (Optional)

• Description: Perform actions on ECS instances
• Required for: Restarting, stopping, starting instances
• API Operation: POST /v1/{project_id}/cloudservers/action
• Use Case: Troubleshooting requires instance restart

4. ecs:cloudServerFlavors:list (Optional)

• Description: List available ECS flavors
• Required for: Understanding instance specifications
• API Operation: GET /v1/{project_id}/cloudservers/flavors
• Use Case: Comparing instance performance metrics

5. ecs:cloudServerImages:list (Optional)

• Description: List available images
• Required for: Instance configuration context
• API Operation: GET /v1/{project_id}/cloudimages
• Use Case: Understanding OS and application context

CES (Cloud Eye Service) Permissions

1. ces:metrics:list

• Description: List available metrics
• Required for: Discovering what metrics are available
• API Operation: GET /V1.0/{project_id}/metrics
• Use Case: User wants to see all monitorable metrics

2. ces:metricData:get

• Description: Get metric data
• Required for: Querying monitoring data
• API Operation: POST /V1.0/{project_id}/metric-data
• Use Case: Core monitoring functionality

3. ces:alarms:list (Optional)

• Description: List alarms
• Required for: Showing existing alarms
• API Operation: GET /V1.0/{project_id}/alarms
• Use Case: User wants to see current alarms

4. ces:alarms:get (Optional)

• Description: Get alarm details
• Required for: Showing alarm configuration
• API Operation: GET /V1.0/{project_id}/alarms/{alarm_id}
• Use Case: Troubleshooting specific alarms

5. ces:alarmTemplates:list (Optional)

• Description: List alarm templates
• Required for: Suggesting alarm configurations
• API Operation: GET /V1.0/{project_id}/alarm-templates
• Use Case: User wants to create new alarms

6. ces:alarmTemplates:get (Optional)

• Description: Get alarm template details
• Required for: Template-based alarm creation
• API Operation: GET /V1.0/{project_id}/alarm-templates/{template_id}
• Use Case: Applying predefined alarm templates

7. ces:alarmRules:list (Optional)

• Description: List alarm rules
• Required for: Showing alarm rule configurations
• API Operation: GET /V1.0/{project_id}/alarm-rules
• Use Case: Reviewing alarm configurations

8. ces:alarmRules:get (Optional)

• Description: Get alarm rule details
• Required for: Detailed alarm rule inspection
• API Operation: GET /V1.0/{project_id}/alarm-rules/{rule_id}
• Use Case: Debugging alarm rules

IAM Permissions (Optional)

1. iam:users:getUser

• Description: Get user information
• Required for: Verifying user identity
• API Operation: GET /v3/users/{user_id}
• Use Case: Authentication and authorization checks

2. iam:users:listUsers (Optional)

• Description: List users
• Required for: User management context
• API Operation: GET /v3/users
• Use Case: Multi-user environment context

3. iam:permissions:listPermissionsForUser (Optional)

• Description: List permissions for user
• Required for: Permission verification
• API Operation: GET /v3/users/{user_id}/permissions
• Use Case: Debugging permission issues

IAM Policy Examples

1. Read-Only Policy (Minimum, Project-Scoped)

This is the minimum required permissions (see above) with an additional Condition block to restrict access to a specific project. Use this for production environments where project-level isolation is needed.

{
  "Version": "1.1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:cloudServers:list",
        "ecs:cloudServers:get",
        "ces:metrics:list",
        "ces:metricData:get"
      ],
      "Resource": ["*"],
      "Condition": {
        "StringEquals": {
          "ecs:resource/project_id": ["<your-project-id>"]
        }
      }
    }
  ]
}

2. Monitoring-Only Policy

{
  "Version": "1.1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:cloudServers:list",
        "ecs:cloudServers:get",
        "ces:metrics:list",
        "ces:metricData:get",
        "ces:alarms:list",
        "ces:alarms:get",
        "ces:alarmTemplates:list",
        "ces:alarmTemplates:get"
      ],
      "Resource": ["*"]
    }
  ]
}

3. Project-Scoped Policy

{
  "Version": "1.1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:cloudServers:list",
        "ecs:cloudServers:get",
        "ecs:cloudServers:action",
        "ces:metrics:list",
        "ces:metricData:get",
        "ces:alarms:*",
        "ces:alarmTemplates:*",
        "ces:alarmRules:*"
      ],
      "Resource": [
        "ecs:*:*:cloudservers:*",
        "ces:*:*:metrics:*",
        "ces:*:*:alarms:*",
        "ces:*:*:alarm-templates:*",
        "ces:*:*:alarm-rules:*"
      ],
      "Condition": {
        "StringEquals": {
          "ecs:resource/project_id": ["<your-project-id>"],
          "ces:resource/project_id": ["<your-project-id>"]
        }
      }
    }
  ]
}

Creating IAM User and Policy

Step 1: Create IAM User

1. Log in to Huawei Cloud Console 2. Navigate to IAM > Users 3. Click Create User 4. Enter user details (username, email, etc.) 5. Enable Programmatic Access 6. Click Create

Step 2: Create Custom Policy

1. Navigate to IAM > Policies 2. Click Create Custom Policy 3. Enter policy name (e.g., ECS-Monitoring-ReadOnly) 4. Select JSON as policy syntax 5. Paste the policy JSON from above 6. Click Create

Step 3: Attach Policy to User

1. Navigate to IAM > Users 2. Select the created user 3. Go to Permissions tab 4. Click Assign Permissions 5. Select Attach existing policies 6. Find and select your custom policy 7. Click OK

Step 4: Create Access Key

1. Navigate to IAM > Users 2. Select the user 3. Go to Security Credentials tab 4. Under Access Keys, click Create Access Key 5. Download the credentials (CSV file) 6. Important: Save the Access Key ID and Secret Access Key securely

Configuring CLI with IAM Credentials

Method 1: Interactive Configuration

hcloud configure init
# Enter:
# - Access Key ID: <from CSV file>
# - Secret Access Key: <from CSV file>
# - Region: <your-region>
# - Project ID: <optional>
# - Output format: json (recommended)

Method 2: Environment Variables

export HUAWEICLOUD_ACCESS_KEY_ID="your-access-key-id"
export HUAWEICLOUD_SECRET_ACCESS_KEY="your-secret-access-key"
export HUAWEICLOUD_REGION="your-region"
export HUAWEICLOUD_PROJECT_ID="your-project-id"

Method 3: Config File

Edit ~/.hcloud/credentials:

[default]
access_key_id = your-access-key-id
secret_access_key = your-secret-access-key
region = your-region
project_id = your-project-id
output = json

Testing Permissions

Test ECS Access

# List ECS instances (requires ecs:cloudServers:list)
hcloud ECS NovaListServers --cli-region <region-id>

# Get specific instance (requires ecs:cloudServers:get)
hcloud ECS NovaShowServer --server_id <instance-uuid> --cli-region <region-id>

Test CES Access

# List available metrics (requires ces:metrics:list)
hcloud CES ListMetrics --namespace "SYS.ECS" --cli-region <region-id>

# Get metric data (requires ces:metricData:get)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region <region-id>

Test IAM Access (Optional)

# Get user info (requires iam:users:getUser)
hcloud IAM ShowUser

Troubleshooting Permission Issues

1. "Access Denied" Errors

# Check current credentials
hcloud configure list

# Test basic IAM access
hcloud iam show-user

# If this fails, credentials may be invalid or expired

2. "Insufficient Permissions"

# Verify the exact permission needed
# Check the error message for specific action

# Example error:
# "The user does not have permission to perform action: ecs:cloudServers:list"

# Solution: Add the missing permission to IAM policy

3. "Resource Not Found"

# Check if resource exists
hcloud ECS NovaListServers --cli-region <region-id>

# Verify resource ID
# Check project ID in policy conditions

4. "Invalid Region"

# Verify region format
# Common regions: cn-north-1, cn-north-4, cn-east-2, ap-southeast-1
# Ensure the region matches where the resource was created

Security Best Practices

1. Principle of Least Privilege

• Grant only necessary permissions
• Start with read-only access
• Add write permissions only when required

2. Regular Access Key Rotation

• Rotate access keys every 90 days
• Use IAM policy to enforce key expiration
• Monitor key usage in Cloud Trace Service

3. Monitor and Audit

• Enable Cloud Trace Service
• Review access logs regularly
• Set up alerts for suspicious activities

4. Use IAM Conditions

• Restrict by IP address range
• Limit by time of day
• Require MFA for sensitive operations

5. Separate Accounts

• Use different accounts for different purposes
• Separate production and development access
• Use service accounts for automated tasks

References

• Huawei Cloud IAM Documentation
• IAM Policy Syntax

Huawei Cloud CLI Commands for ECS Monitoring

Overview

This document provides detailed command references for Huawei Cloud CLI (hcloud) commands used in the ECS monitoring skill.

Basic Command Structure

hcloud <service> <command> [options] [parameters]

ECS Commands

List ECS Instances

# Basic listing
hcloud ECS NovaListServers --cli-region=<region-id> --limit=50

Get ECS Instance Details

# Get basic information
hcloud ECS NovaShowServer --server_id=<instance-uuid> --cli-region=<region-id>

List ECS Flavors

# List all flavors
hcloud ECS NovaListFlavors --cli-region=<region-id>

CES (Cloud Eye Service) Commands

List Available Metrics

# List all metrics for ECS namespace
hcloud CES ListMetrics \
  --namespace="SYS.ECS" \
  --cli-region=<region-id>

# List metrics for specific instance
hcloud CES ListMetrics \
  --namespace="SYS.ECS" \
  --dim.0="instance_id,<instance-uuid>" \
  --cli-region=<region-id>

Query Metric Data (BatchListMetricData - Recommended)

Note: BatchListMetricData is the recommended command for querying metric data. It supports querying multiple metrics in a single request. Each --metrics.N.metric_name must contain exactly ONE metric name (no comma-separated values).

# Query CPU usage (SYS.ECS - base monitoring)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=<region-id>

# Query memory usage (SYS.ECS - base monitoring)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="mem_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=<region-id>

# Query multiple metrics simultaneously (each as a separate metrics entry)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="mem_util" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="<instance-uuid>" \
  --metrics.3.namespace="SYS.ECS" \
  --metrics.3.metric_name="disk_read_bytes_rate" \
  --metrics.3.dimensions.1.name="instance_id" \
  --metrics.3.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=<region-id>

Query Single Metric Data (ShowMetricData - Alternative)

Note: ShowMetricData only supports querying a single metric at a time. For multiple metrics, use BatchListMetricData instead.

# Query single metric
hcloud CES ShowMetricData \
  --namespace="SYS.ECS" \
  --metric_name="cpu_util" \
  --dim.0="instance_id,<instance-uuid>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=<region-id>

Query AGT.ECS Metrics (Agent Required)

Note: AGT.ECS metrics require the Telescope agent to be installed on the ECS instance. The minimum period for AGT.ECS is 60 (1 minute).

# Query agent CPU usage (AGT.ECS)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="AGT.ECS" \
  --metrics.1.metric_name="cpu_usage" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=60 \
  --filter="average" \
  --cli-region=<region-id>

# Query disk usage per mount point (AGT.ECS - requires mount_point dimension)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="AGT.ECS" \
  --metrics.1.metric_name="disk_usedPercent" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="<instance-uuid>" \
  --metrics.1.dimensions.2.name="mount_point" \
  --metrics.1.dimensions.2.value="<mount-point-hash>" \
  --from=$(date -d '-1 hour' +%s)000 \
  --to=$(date +%s)000 \
  --period=60 \
  --filter="average" \
  --cli-region=<region-id>

Namespace Fallback Strategy (SYS.ECS → AGT.ECS)

When a SYS.ECS metric query returns no data, try the corresponding AGT.ECS metric. See references/ces-metrics-reference.md for the complete fallback mapping table and references/troubleshooting-guide.md for detailed troubleshooting steps.

List Alarms

# List all alarms
hcloud CES ListAlarms --cli-region=<region-id>

Get Alarm Details

hcloud CES ShowAlarm <alarm-id> --cli-region=<region-id>

Time Format Specifications

Mandatory Format Requirements

Both from and to must be 13-digit numeric millisecond timestamps, and the value of from must be less than to.

period Collection Cycle Enumeration (Unit: Second)

• 1: Real-time monitoring data
• 60: 1-minute granularity, one data point per minute (AGT.ECS only)
• 300: 5-minute granularity, one data point every 5 minutes
• 1200: 20-minute granularity, one data point every 20 minutes
• 3600: 1-hour granularity, one data point per hour
• 14400: 4-hour granularity, one data point every 4 hours
• 86400: 1-day granularity, one data point per day

Important: period=60 (1-minute) is only available for AGT.ECS metrics. SYS.ECS metrics have a minimum period of 300 (5 minutes).

Maximum Query Range & Overtime Rules for Each Cycle

If the query time range exceeds the maximum allowed duration, the interface will automatically adjust the start time from forward.

• period="1": Maximum range: 436001000 milliseconds
• period="60": Maximum range: 43200100 milliseconds
• period="300": Maximum range: 2436001000 milliseconds
• period="1200": Maximum range: 32436001000 milliseconds
• period="3600": Maximum range: 102436001000 milliseconds
• period="14400": Maximum range: 302436001000 milliseconds
• period="86400": Maximum range: 1802436001000 milliseconds

Common Relative Time (Millisecond Timestamp Format)

# Last 1 hour
--from=$(date -d '1 hour ago' +%s)000 \
--to=$(date +%s)000

# Last 6 hours
--from=$(date -d '6 hours ago' +%s)000 \
--to=$(date +%s)000

# Last 24 hours
--from=$(date -d '24 hours ago' +%s)000 \
--to=$(date +%s)000

# Last 7 days
--from=$(date -d '7 days ago' +%s)000 \
--to=$(date +%s)000

# Today from 00:00 to current time
--from=$(date -d "$(date +%Y-%m-%d) 00:00:00" +%s)000 \
--to=$(date +%s)000

# Full time range of yesterday
--from=$(date -d "yesterday 00:00:00" +%s)000 \
--to=$(date -d "yesterday 23:59:59" +%s)000

Fixed Time Range (Fill in millisecond timestamps directly)

# Specific date and time
--from="1716988800000" \
--to="1716992400000"

# From the 1st day of current month to current time
--from=$(date -d "$(date +%Y-%m-01) 00:00:00" +%s)000 \
--to=$(date +%s)000

Common Query Patterns

1. Basic Monitoring Dashboard

#!/bin/bash

REGION="<region-id>"

# Get instance list
INSTANCES=$(hcloud ECS NovaListServers --cli-region=$REGION --limit=50)

# For each instance, get key metrics
for INSTANCE_ID in $(echo $INSTANCES | jq -r '.servers[].id'); do
  echo "=== Instance: $INSTANCE_ID ==="

  # CPU and Memory usage (SYS.ECS)
  hcloud CES BatchListMetricData \
    --metrics.1.namespace="SYS.ECS" \
    --metrics.1.metric_name="cpu_util" \
    --metrics.1.dimensions.1.name="instance_id" \
    --metrics.1.dimensions.1.value="$INSTANCE_ID" \
    --metrics.2.namespace="SYS.ECS" \
    --metrics.2.metric_name="mem_util" \
    --metrics.2.dimensions.1.name="instance_id" \
    --metrics.2.dimensions.1.value="$INSTANCE_ID" \
    --from=$(date -d '1 hour ago' +%s)000 \
    --to=$(date +%s)000 \
    --period="300" \
    --filter="average" \
    --cli-region=$REGION

  echo ""
done

2. Performance Troubleshooting

#!/bin/bash

INSTANCE_ID="<instance-uuid>"
REGION="<region-id>"
START_TIME=$(date -d '1 hour ago' +%s)000
END_TIME=$(date +%s)000

echo "Performance Analysis for $INSTANCE_ID"
echo "Time Range: $START_TIME to $END_TIME"
echo ""

# Get all key SYS.ECS metrics
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$INSTANCE_ID" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="mem_util" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="$INSTANCE_ID" \
  --metrics.3.namespace="SYS.ECS" \
  --metrics.3.metric_name="disk_read_bytes_rate" \
  --metrics.3.dimensions.1.name="instance_id" \
  --metrics.3.dimensions.1.value="$INSTANCE_ID" \
  --metrics.4.namespace="SYS.ECS" \
  --metrics.4.metric_name="disk_write_bytes_rate" \
  --metrics.4.dimensions.1.name="instance_id" \
  --metrics.4.dimensions.1.value="$INSTANCE_ID" \
  --metrics.5.namespace="SYS.ECS" \
  --metrics.5.metric_name="network_incoming_bytes_rate_inband" \
  --metrics.5.dimensions.1.name="instance_id" \
  --metrics.5.dimensions.1.value="$INSTANCE_ID" \
  --metrics.6.namespace="SYS.ECS" \
  --metrics.6.metric_name="network_outgoing_bytes_rate_inband" \
  --metrics.6.dimensions.1.name="instance_id" \
  --metrics.6.dimensions.1.value="$INSTANCE_ID" \
  --from="$START_TIME" \
  --to="$END_TIME" \
  --period="300" \
  --filter="average" \
  --cli-region="$REGION"

3. Historical Trend Analysis

#!/bin/bash

INSTANCE_ID="<instance-uuid>"
REGION="<region-id>"

# Daily averages for last 7 days
for DAYS_AGO in {0..6}; do
  START_TIME=$(date -d "$DAYS_AGO days ago 00:00:00" +%s)000
  END_TIME=$(date -d "$DAYS_AGO days ago 23:59:59" +%s)000

  echo "=== $(date -d "$DAYS_AGO days ago" +%Y-%m-%d) ==="

  hcloud CES BatchListMetricData \
    --metrics.1.namespace="SYS.ECS" \
    --metrics.1.metric_name="cpu_util" \
    --metrics.1.dimensions.1.name="instance_id" \
    --metrics.1.dimensions.1.value="$INSTANCE_ID" \
    --metrics.2.namespace="SYS.ECS" \
    --metrics.2.metric_name="mem_util" \
    --metrics.2.dimensions.1.name="instance_id" \
    --metrics.2.dimensions.1.value="$INSTANCE_ID" \
    --from="$START_TIME" \
    --to="$END_TIME" \
    --period="3600" \
    --filter="average" \
    --cli-region="$REGION"
done

Error Handling Examples

Check Command Success

if hcloud ECS NovaListServers --cli-region=<region-id> > /dev/null 2>&1; then
  echo "Command succeeded"
else
  echo "Command failed with error: $?"
  # Handle error
fi

Handle Empty Results

METRIC_DATA=$(hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$INSTANCE_ID" \
  --from="$START_TIME" \
  --to="$END_TIME" \
  --period="300" \
  --filter="average" \
  --cli-region="$REGION")

if [ "$(echo "$METRIC_DATA" | jq '.metrics[0].datapoints | length')" -eq "0" ]; then
  echo "No metric data found for the specified time range"
else
  echo "Found metric data"
  # Process data
fi

Performance Tips

For performance optimization tips (batch queries, caching, appropriate time ranges), see references/best-practices.md.

Common Errors and Solutions

For common errors and solutions, see references/troubleshooting-guide.md.

References

• Huawei Cloud CLI Documentation
• ECS API Reference
• CES API Reference
• jq Documentation (for JSON processing)

Huawei Cloud ECS Monitoring Troubleshooting Guide

Overview

This guide provides troubleshooting steps for common issues encountered when using the Huawei Cloud ECS monitoring skill. Follow these steps to diagnose and resolve problems.

Common Issues and Solutions

1. Authentication and Authorization Issues

Issue: "Invalid credentials" or "Access denied"

# Symptom: hcloud commands fail with authentication errors
hcloud ECS NovaListServers --cli-region=<region-id>
# Error: InvalidAccessKeyId: The Access Key ID is invalid

Solution:

# Step 1: Check current configuration
hcloud configure list

# Step 2: Verify credentials are valid
# If configuration is empty or incorrect:
hcloud configure init
# Follow prompts to enter:
# - Access Key ID
# - Secret Access Key
# - Region
# - Output format (json recommended)

# Step 3: Test authentication
hcloud IAM ShowUser
# Should return user information without errors

# Step 4: Check IAM permissions
# See references/iam-policies.md for the complete list of required permissions

Issue: "Insufficient permissions"

# Symptom: Specific operations fail with permission errors
hcloud CES ListMetrics --namespace=SYS.ECS --cli-region=<region-id>
# Error: User does not have permission to perform this operation

Solution:

# Step 1: Check required permissions
# Refer to references/iam-policies.md for required permissions

# Step 2: Contact administrator to add missing permissions
# See references/iam-policies.md for the complete list of required permissions

# Step 3: Test with minimal permissions first
# Start with read-only permissions, then add as needed

2. CLI Installation and Configuration Issues

Issue: "Command not found: hcloud"

# Symptom: hcloud command is not available
hcloud --version
# bash: hcloud: command not found

Solution:

Step 1: Install Huawei Cloud CLI

Refer to the detailed installation guide: references/cli-installation-guide.md

Quick installation for Linux/macOS:

# Download and install
curl -LO "https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/huaweicloud-cli-linux-amd64.tar.gz"
tar -zxvf huaweicloud-cli-linux-amd64.tar.gz
sudo mv hcloud /usr/local/bin/

For Windows and other platforms, see the complete installation guide.

Step 2: Verify installation

hcloud --version
# Should show version information (>= 3.2.0 recommended)

Step 3: Add to PATH if needed

• Linux/macOS: Ensure /usr/local/bin or installation directory is in PATH
• Windows: Add installation directory to system PATH environment variable

Windows: Ensure installation directory is in PATH

#### Issue: "Outdated CLI version"

Symptom: Commands fail with version errors

hcloud --version

Version: 1.0.0 (but newer features require 1.2.0+)

**Solution:**

**Step 1: Update Huawei Cloud CLI**

Refer to the installation guide for update instructions: `references/cli-installation-guide.md`

Quick update for Linux/macOS:

Download latest version

curl -LO "https://cn-north-4-hdn-koocli.obs.cn-north-4.myhuaweicloud.com/cli/latest/huaweicloud-cli-linux-amd64.tar.gz" tar -zxvf huaweicloud-cli-linux-amd64.tar.gz sudo mv hcloud /usr/local/bin/

Verify update

hcloud --version

For Windows and other platforms, see the complete installation guide.

#### Step 2: Verify update

hcloud --version

Should show latest version

### 3. Instance Not Found Issues

#### Issue: "Instance not found"

Symptom: Cannot find specified instance

hcloud ECS NovaShowServer --server_id=<invalid-uuid> --cli-region=<region-id>

Error: Instance not found

**Solution:**

Step 1: List all instances to verify ID

hcloud ECS NovaListServers --cli-region=<region-id>

Step 2: Check region

Ensure you're using the correct region

Step 3: Verify instance status

hcloud ECS NovaShowServer --server_id=<instance-uuid> --cli-region=<region-id>

Check if instance exists and is active

Step 4: Check project/account

Ensure you're in the correct project/account

hcloud configure list

Check project_id configuration

### 4. No Metric Data Issues

#### Issue: "No metric data available"

Symptom: Metric queries return empty results

hcloud CES BatchListMetricData \ --metrics.1.namespace="SYS.ECS" \ --metrics.1.metric_name="cpu_util" \ --metrics.1.dimensions.1.name="instance_id" \ --metrics.1.dimensions.1.value="<instance-uuid>" \ --from=$(date -d '1 hour ago' +%s)000 \ --to=$(date +%s)000 \ --period=300 \ --filter="average" \ --cli-region=<region-id>

Returns empty datapoints array

**Solution:**

Step 1: Check if instance is running

Instance must be in ACTIVE state for metrics collection

Step 2: Check if CES agent is installed (for AGT.ECS metrics)

For Linux instances:

SSH to instance and check:

systemctl status telescoped

or

ps aux | grep telescoped

Step 3: Check available metrics

hcloud CES ListMetrics \ --namespace=SYS.ECS \ --dim.0="instance_id,<instance-uuid>" \ --cli-region=<region-id>

Should list available metrics for the instance

Step 4: Verify time range

Ensure time range is valid and instance was running during that time

echo "Time range: $(date -d '1 hour ago' +%s)000 to $(date +%s)000"

Step 5: Check metric name spelling

Common SYS.ECS metric names:

- cpu_util (NOT cpu_usage - that is AGT.ECS)

- mem_util (NOT mem_usage)

- disk_util_inband (NOT disk_used_percent)

- network_incoming_bytes_rate_inband (NOT network_incoming_bytes_rate)

Step 6: Check namespace

SYS.ECS: Base monitoring (no agent required, 5-min period)

AGT.ECS: OS monitoring (agent required, 1-min period)

Step 7: Try AGT.ECS fallback

If SYS.ECS metric returns no data, try the AGT.ECS equivalent:

cpu_util (SYS.ECS) → cpu_usage (AGT.ECS)

mem_util (SYS.ECS) → mem_usedPercent (AGT.ECS)

disk_util_inband (SYS.ECS) → disk_usedPercent (AGT.ECS)

hcloud CES BatchListMetricData \ --metrics.1.namespace="AGT.ECS" \ --metrics.1.metric_name="cpu_usage" \ --metrics.1.dimensions.1.name="instance_id" \ --metrics.1.dimensions.1.value="<instance-uuid>" \ --from=$(date -d '1 hour ago' +%s)000 \ --to=$(date +%s)000 \ --period=60 \ --filter="average" \ --cli-region=<region-id>

#### Issue: "Metric not found"

Symptom: Specific metric not available

hcloud CES BatchListMetricData \ --metrics.1.namespace="SYS.ECS" \ --metrics.1.metric_name="non_existent_metric" \ --metrics.1.dimensions.1.name="instance_id" \ --metrics.1.dimensions.1.value="<instance-uuid>" \ --cli-region=<region-id>

Error: Metric not found

**Solution:**

Step 1: List all available metrics

hcloud CES ListMetrics \ --namespace=SYS.ECS \ --dim.0="instance_id,<instance-uuid>" \ --cli-region=<region-id>

Step 2: Use correct metric name

Common SYS.ECS metrics (base monitoring, no agent):

- cpu_util

- mem_util

- disk_util_inband

- disk_read_bytes_rate

- disk_write_bytes_rate

- network_incoming_bytes_rate_inband

- network_outgoing_bytes_rate_inband

Common AGT.ECS metrics (OS monitoring, agent required):

- cpu_usage

- mem_usedPercent

- disk_usedPercent

- load_average1

- net_tcp_total

Step 3: Check namespace

If querying AGT.ECS metrics, ensure:

- Namespace is set to "AGT.ECS"

- Telescope agent is installed on the instance

- Period is at least 60 (1 minute)

### 5. Time Range Issues

#### Issue: "Invalid time range" or "Time range too large"

Symptom: Time range errors

hcloud CES BatchListMetricData \ --metrics.1.namespace="SYS.ECS" \ --metrics.1.metric_name="cpu_util" \ --metrics.1.dimensions.1.name="instance_id" \ --metrics.1.dimensions.1.value="<instance-uuid>" \ --from=1234567890 \ --to=$(date +%s)000 \ --period=300 \ --cli-region=<region-id>

Error: Time range too large or invalid

**Solution:**

Step 1: Use correct time format

See references/related-commands.md (Time Format Specifications section) for

detailed format requirements, period enumeration, and common time range examples

Step 2: Reduce time range

Maximum data points per query is 1440

Calculate: (end_time - start_time) / period <= 1440

Step 3: Increase period

Use larger period for longer time ranges

--period=300 # 5 minutes --period=3600 # 1 hour --period=86400 # 1 day

Step 4: Split into multiple queries

For 30 days with 5-minute period:

Query 1: Days 1-15

Query 2: Days 16-30

### 6. Network and Connectivity Issues

#### Issue: "Network error" or "Timeout"

Symptom: Connection failures or timeouts

hcloud ECS NovaListServers --cli-region=<region-id>

Error: Network error or timeout

**Solution:**

Step 1: Check network connectivity

ping console.huaweicloud.com

Should respond without packet loss

Step 2: Check proxy settings

echo $HTTP_PROXY echo $HTTPS_PROXY

If using proxy, ensure it's configured correctly

Step 3: Increase timeout

hcloud ECS NovaListServers --cli-region=<region-id> --timeout=60

Default timeout is 30 seconds

Step 4: Check DNS resolution

nslookup console.huaweicloud.com

Should resolve to Huawei Cloud IP addresses

Step 5: Try different region

hcloud ECS NovaListServers --cli-region=<other-region-id>

Test if issue is region-specific

### 7. Data Format and Parsing Issues

#### Issue: "Invalid JSON response" or "Parsing error"

Symptom: CLI returns malformed JSON

hcloud ECS NovaListServers --cli-region=<region-id> --output=json

Error: Invalid JSON or parsing error

**Solution:**

Step 1: Use text output for debugging

hcloud ECS NovaListServers --cli-region=<region-id> --output=text

Check if command works with text output

Step 2: Check for special characters

Some instance names or metadata may contain special characters

Use jq with raw output

hcloud ECS NovaListServers --cli-region=<region-id> --output=json | jq -r '.'

Step 3: Update CLI version

Older versions may have JSON parsing issues

hcloud --version

Update to latest version if needed

Step 4: Check for API changes

Huawei Cloud APIs may change - check documentation

https://support.huaweicloud.com/api-ecs/

#### Issue: "jq command not found"

Symptom: JSON parsing fails due to missing jq

hcloud ECS NovaListServers --cli-region=<region-id> --output=json | jq '.servers[].name'

bash: jq: command not found

**Solution:**

Step 1: Install jq

Ubuntu/Debian:

sudo apt-get install jq

CentOS/RHEL:

sudo yum install jq

macOS:

brew install jq

Windows:

Download from https://stedolan.github.io/jq/download/

Step 2: Use alternative parsing if jq not available

Use grep/sed/awk for simple extraction

hcloud ECS NovaListServers --cli-region=<region-id> --output=json | grep -o '"name":"[^"]*"' | cut -d'"' -f4

Step 3: Use Python for complex parsing

hcloud ECS NovaListServers --cli-region=<region-id> --output=json | python3 -c "import sys,json; data=json.load(sys.stdin); [print(s['name']) for s in data['servers']]"

### 8. Performance Issues

#### Issue: "Slow response" or "High latency"

Symptom: Queries take too long

time hcloud CES BatchListMetricData ... # Takes > 30 seconds

**Solution:**

Step 1: Reduce time range

Shorter time ranges are faster

--from=$(date -d '1 hour ago' +%s)000

Instead of

--from=$(date -d '7 days ago' +%s)000

Step 2: Increase period

Larger periods return fewer data points

--period=3600 # 1 hour intervals

Instead of

--period=300 # 5 minute intervals

Step 3: Query fewer metrics per request

Batch 5-6 metrics at a time instead of 10+

Step 4: Use appropriate region endpoint

Ensure you're using the closest region

Step 5: Cache results

Cache instance list and other static data

INSTANCE_CACHE="/tmp/instances_$(date +%Y%m%d%H).json" if [ ! -f "$INSTANCE_CACHE" ]; then hcloud ECS NovaListServers --cli-region=<region-id> --output=json > "$INSTANCE_CACHE" fi

### 9. Dimension and Filter Issues

#### Issue: "Invalid dimension" or "Dimension not found"

Symptom: Dimension errors in metric queries

hcloud CES BatchListMetricData \ --metrics.1.namespace="AGT.ECS" \ --metrics.1.metric_name="disk_usedPercent" \ --metrics.1.dimensions.1.name="instance_id" \ --metrics.1.dimensions.1.value="<instance-uuid>" \ --cli-region=<region-id>

Error: Dimension mount_point is required for disk metrics

**Solution:**

Step 1: Check required dimensions for each metric

AGT.ECS disk metrics require mount_point dimension

hcloud CES BatchListMetricData \ --metrics.1.namespace="AGT.ECS" \ --metrics.1.metric_name="disk_usedPercent" \ --metrics.1.dimensions.1.name="instance_id" \ --metrics.1.dimensions.1.value="<instance-uuid>" \ --metrics.1.dimensions.2.name="mount_point" \ --metrics.1.dimensions.2.value="<mount-point-hash>" \ --from=$(date -d '1 hour ago' +%s)000 \ --to=$(date +%s)000 \ --period=60 \ --filter="average" \ --cli-region=<region-id>

Step 2: List available dimensions

hcloud CES ListMetrics \ --namespace="AGT.ECS" \ --dim.0="instance_id,<instance-uuid>" \ --cli-region=<region-id>

Step 3: Use correct dimension format

See references/ces-metrics-reference.md (Metric Dimensions section) for

dimension format details and mount_point hash value explanation

### 10. Agent-Related Issues (AGT.ECS Metrics)

#### Issue: "AGT.ECS metrics return no data"

Symptom: AGT.ECS metric queries return empty results

while SYS.ECS metrics work fine

**Solution:**

Step 1: Verify the Telescope agent is installed

SSH to the instance and check:

systemctl status telescoped # Linux

or check "telescoped" service in Services Manager (Windows)

Step 2: If agent is not installed, install it

Follow Huawei Cloud documentation for agent installation:

https://support.huaweicloud.com/usermanual-ces/ces_01_0019.html

Step 3: Verify agent is running

ps aux | grep telescoped

Step 4: Check agent version (should be latest)

Older agent versions may not support all metrics

Step 5: Wait for data collection

Agent metrics may take 1-2 minutes to appear after installation

Huawei Cloud ECS Monitoring Skill Verification Method

Overview

This document describes the verification methods for ensuring the Huawei Cloud ECS Monitoring skill functions correctly and meets quality standards. Verification includes functional testing, integration testing, and user acceptance testing.

Verification Environment

1. Test Environment Setup

# Prerequisites
- Huawei Cloud account with active subscription
- At least one running ECS instance
- Huawei Cloud CLI (hcloud) installed and configured
- Proper IAM permissions (see iam-policies.md)

# Environment variables for testing
export HUAWEICLOUD_REGION="cn-north-4"  # Test region
export TEST_INSTANCE_ID="<instance-uuid>"  # Test ECS instance UUID
export TEST_METRIC_NAME="cpu_util"  # Test metric (SYS.ECS)

2. Test Data Requirements

• ECS instance must be in "ACTIVE" state
• Instance should have been running for at least 30 minutes to generate metrics
• CES agent must be installed and running on the instance (for AGT.ECS metrics)
• Sufficient monitoring data should be available (at least 1 hour of data)

Functional Verification

1. Prerequisite Verification

# Test 1: Verify Huawei Cloud CLI installation
hcloud --version
# Expected: CLI version information displayed without errors

# Test 2: Verify CLI configuration
hcloud configure list
# Expected: Valid credentials and region configuration displayed

# Test 3: Verify authentication
hcloud IAM ShowUser
# Expected: User information displayed without authentication errors

2. ECS Instance Listing Verification

# Test 4: List ECS instances
hcloud ECS NovaListServers --cli-region=$HUAWEICLOUD_REGION --limit=50
# Expected: JSON array of ECS instances, including test instance

# Test 5: Get specific instance details
hcloud ECS NovaShowServer --server_id=$TEST_INSTANCE_ID --cli-region=$HUAWEICLOUD_REGION
# Expected: Detailed information about the test instance

3. CES Metrics Verification

# Test 6: List available metrics
hcloud CES ListMetrics --namespace=SYS.ECS --dim.0="instance_id,$TEST_INSTANCE_ID" --cli-region=$HUAWEICLOUD_REGION
# Expected: List of available metrics for the instance

# Test 7: Query metric data (SYS.ECS - cpu_util)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION
# Expected: Metric data points with timestamps and values

4. Common Metrics Verification

# Test 8: Verify CPU metrics (SYS.ECS)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '30 minutes ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION

# Test 9: Verify memory metrics (SYS.ECS)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="mem_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '30 minutes ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION

# Test 10: Verify disk metrics (SYS.ECS)
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="disk_util_inband" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '30 minutes ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION

Integration Verification

1. Skill Workflow Verification

# Test 11: Complete workflow test
# Step 1: List instances
INSTANCES=$(hcloud ECS NovaListServers --cli-region=$HUAWEICLOUD_REGION --limit=50)

# Step 2: Extract instance ID
INSTANCE_ID=$(echo $INSTANCES | jq -r '.servers[0].id')

# Step 3: Query metrics (SYS.ECS)
METRIC_DATA=$(hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$INSTANCE_ID" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="mem_util" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="$INSTANCE_ID" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION)

# Step 4: Verify data format
echo $METRIC_DATA | jq '.metrics[0].datapoints | length'
# Expected: Positive integer indicating data points retrieved

2. Error Handling Verification

# Test 12: Invalid instance ID
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="invalid-uuid" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION
# Expected: Appropriate error message

# Test 13: Invalid time range
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=0 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION
# Expected: Appropriate error message

# Test 14: Invalid metric name
hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="invalid_metric" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION
# Expected: Appropriate error message

3. Performance Verification

# Test 15: Response time test
time hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION
# Expected: Response within 5 seconds

# Test 16: Multiple metrics query
time hcloud CES BatchListMetricData \
  --metrics.1.namespace="SYS.ECS" \
  --metrics.1.metric_name="cpu_util" \
  --metrics.1.dimensions.1.name="instance_id" \
  --metrics.1.dimensions.1.value="$TEST_INSTANCE_ID" \
  --metrics.2.namespace="SYS.ECS" \
  --metrics.2.metric_name="mem_util" \
  --metrics.2.dimensions.1.name="instance_id" \
  --metrics.2.dimensions.1.value="$TEST_INSTANCE_ID" \
  --metrics.3.namespace="SYS.ECS" \
  --metrics.3.metric_name="disk_read_bytes_rate" \
  --metrics.3.dimensions.1.name="instance_id" \
  --metrics.3.dimensions.1.value="$TEST_INSTANCE_ID" \
  --from=$(date -d '1 hour ago' +%s)000 \
  --to=$(date +%s)000 \
  --period=300 \
  --filter="average" \
  --cli-region=$HUAWEICLOUD_REGION
# Expected: Response within 10 seconds

Verification Criteria

1. Success Criteria

• All functional tests pass (100% success rate)
• Response times meet performance requirements
• Error handling works correctly
• Documentation is complete and accurate
• Cross-platform compatibility verified

2. Performance Criteria

• Single metric query: < 5 seconds response time
• Multiple metrics query (6 metrics): < 10 seconds response time
• Instance listing: < 3 seconds response time
• Memory usage: < 100MB for skill execution

3. Quality Criteria

• Code coverage: > 80% for core functionality
• Documentation coverage: 100% of features documented
• Error handling: All expected errors handled gracefully
• User experience: Clear, actionable output format

4. Security Criteria

• No hardcoded credentials in code
• Proper IAM permission validation
• Secure credential handling
• No sensitive data exposure in logs

References

• Huawei Cloud CLI Documentation
• ECS Monitoring Metrics
• CES API Reference
• IAM Policy Documentation