Huawei Cloud Msot Msopprof Operator Profiler

Huawei Cloud msOT msopprof Operator Profiler

Overview

This skill provides operator-level performance profiling capabilities for Ascend NPU.

Architecture: Profiling Configuration → Data Collection → Report Generation → Analysis

Related Skills:

• huawei-cloud-ascend-profiler-db-explorer - Profiling database analysis and

query

• huawei-cloud-ascend-small-model-migrate - Migration workflow that uses

performance data

Architecture Components

This skill involves the following cloud services and components:

• msopprof: Huawei Cloud operator profiling tool for Ascend NPU
• CANN: AI Computing Platform for NPU runtime support
• Ascend NPU: Target hardware for performance profiling
• Profiling Database: Storage for collected performance data

Architecture Diagram:

┌─────────────────────────────────────────────────────────────┐
│           msOT msopprof Operator Profiler Skill            │
├─────────────────────────────────────────────────────────────┤
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐ │
│  │  Profiling   │───▶│  Data        │───▶│  Report      │ │
│  │  Config      │    │  Collection  │    │  Generation  │ │
│  └──────────────┘    └──────────────┘    └──────────────┘ │
│         │                   │                   │          │
│         ▼                   ▼                   ▼          │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐ │
│  │  Mode        │    │  Operator    │    │  Data        │ │
│  │  Selection   │    │  Execution   │    │  Export      │ │
│  │  (Device/Sim)│    │  Monitoring  │    │              │ │
│  └──────────────┘    └──────────────┘    └──────────────┘ │
└─────────────────────────────────────────────────────────────┘

Use Cases

Typical Problem Scenarios:

• Collecting operator-level performance data on Ascend NPU
• Profiling model inference performance
• Identifying performance bottlenecks through operator execution
• Comparing operator performance between device and simulator modes
• Generating performance analysis reports for model optimization

Typical User Phrases:

• "Profile operator performance on Ascend NPU"
• "Collect performance data using msopprof"
• "Generate performance analysis report"
• "Compare device vs simulator profiling"
• "Identify performance bottlenecks"
• "AscendCollectionOperatorPerformanceData"
• "msopprofPerformance"
• "PerformanceAnalysisReport"

Scope

Supported:

• Operator performance data collection
• Device mode profiling
• Simulator mode profiling
• Report generation

Not supported:

• System-level profiling
• Non-Ascend platforms

Core Workflow

1. Profiling Configuration

• Set up profiling parameters
• Configure collection mode (device/simulator)

2. Data Collection

• Execute profiling on target model
• Collect operator performance data

3. Report Generation

• Generate profiling reports
• Output performance metrics

4. Data Export

• Export data to profiling database
• Prepare for further analysis

Enhanced Features

Best Practices Knowledge Base

This skill integrates a searchable knowledge base containing migration success stories and optimization patterns:

Features:

• Case Study Repository: Searchable database of migration success stories

for common model architectures

• Operator Optimization Recipes: Proven optimization patterns with code

snippets for common operators

• Similarity Matching: Recommends proven solutions based on model

architecture similarity

• Performance Patterns: Collection of known performance patterns and

anti-patterns

• Expert Recommendations: Curated tips from Ascend optimization experts

Knowledge Base Structure:

• Case Studies: YOLO, ResNet, UNet migration success stories
• Operator Recipes: Conv2D, MatMul, Attention optimization patterns
• Performance Patterns: Common bottleneck patterns and solutions
• Expert Tips: Optimization best practices from field experts

YOLOv8 Migration Success Story

Model Information

• Model: YOLOv8s
• Input: 640x640
• Target: Ascend 910B

Challenges Encountered

1. Issue: NMS operator fallback to CPU

• Solution: Implemented AscendC NMS operator
• Gain: 30% latency reduction

2. Issue: Memory bandwidth bottleneck

• Solution: Optimized data layout and batch processing
• Gain: 15% throughput improvement

3. Issue: Custom activation function

• Solution: Replaced with supported operators
• Gain: Stable NPU execution

Final Results

• Latency: 8.2 ms → 5.1 ms (-37.8%)
• Throughput: 122 FPS → 196 FPS (+60.7%)
• Accuracy: 60.2% → 60.4% (+0.2%)

Key Takeaways

• Always check operator coverage before migration
• Implement custom operators for critical path
• Optimize memory access patterns

Operator Optimization Recipes

• Conv2D: Padding overhead → Use native NPU padding (5-10% gain)
• MatMul: Memory layout → Optimal tiling config (10-15% gain)
• Attention: FlashAttention → Enable NPU FlashAttention (20-30% gain)
• NMS: CPU fallback → AscendC implementation (25-35% gain)

Reference Documents

• Acceptance Criteria - Functional

acceptance criteria

• Verification Method - Verification approach
• Troubleshooting - Common issues and solutions

Prerequisites

• msopprof >= 1.0.0 installed
• CANN >= 7.0.0 installed
• Ascend NPU driver installed
• Operator code to be analyzed

Core Commands

# Collect operator performance data
msopprof --output=/path/to/output \
  --mode=device \
  ./my_operator

# Analyze performance report
python3 scripts/analyze_profile.py --data /path/to/output

Parameter Confirmation

• output: Performance data output path (Required)
• mode: Collection mode (device/simulator) (Optional)
• operator: Operator executable file path (Required)

Experience: Some Projects Require Simulator-Compatible Build

This is an experience case, not a prerequisite for using `msprof op

simulator`. The conclusion should be stated as: **Some projects / template

libraries / build links require separate "simulator-compatible" executables

for simulation scenarios**, not simply "all simulations need sim build".

Symptoms

When using an existing executable to launch simulation, you may see errors like:

terminate called after throwing an instance of 'std::__ios_failure'
  what():  basic_filebuf::xsgetn error reading the file: Bad address
[WARN]  Child process killed by signal 6

Why This Is Not an Absolute Rule

Official documentation provides two types of information:

1. Some projects explicitly require simulator build

• E.g., catkins/template library scenarios require --simulator in build

2. Some official examples state the same executable runs on both device and simulator

This means "whether simulator build is required" depends on the project's build configuration, not a rigid rule of msOpProf.

Therefore, when encountering this error, the correct root cause statement is:

The executable being launched may not be a simulator-compatible build, or

is missing simulator-specific build options or dependencies.

Common Root Causes

Typical scenarios:

• Project separates device/simulator build outputs
• Template library or scripts require explicit --simulator
• Executable defaults to device runtime only
• Simulator requires specific libraries, architecture params, or linking that

differ from device build

Recommended Troubleshooting Sequence

1. Check Project Documentation or Build Scripts

Look for indicators like:

• --simulator
• sim
• *_sim
• Separate simulator target/profile
• Architecture or run mode parameters

2. Check Simulator Environment

Verify:

export LD_LIBRARY_PATH=${INSTALL_DIR}/tools/simulator/Ascendxxxyy/lib:$LD_LIBRARY_PATH

Where Ascendxxxyy matches your actual simulator type.

3. If Build Supports It, Generate Simulator-Compatible Build

Example A: Template Library / catkins Style

bash scripts/build.sh --simulator 00_basic_matmul

Example B: Project-Level CMake Option (Example Only)

cmake -DCMAKE_ASC_RUN_MODE=sim -DCMAKE_ASC_ARCHITECTURES=dav-2201 ..
make -j

Note: These CMake variables are project-specific examples, not `msprof op

simulator` parameters. Support depends on your project.

Verification Criteria

When the issue is resolved, you should see:

• No more signal 6 / Bad address
• Simulation execution phase proceeds normally
• Output directory contains dump/ and simulator/
• simulator/ contains per-core subdirectories, trace.json,

visualize_data.bin, etc.

Scenarios Where This Does Not Apply

Do not conclude "requires sim build" when:

1. Using --config or --export mode (no app launch path involved) 2. Official examples already state the same executable works on both device and simulator 3. Root cause is actually:

• Wrong LD_LIBRARY_PATH
• Missing simulator installation
• Architecture mismatch

Acceptance Criteria

Functional Acceptance Criteria

1. Mode Selection

• AC-1.1: Should correctly identify device vs simulator mode
• Verification: Check mode determination logic
• AC-1.2: Should recommend appropriate mode based on requirements
• Verification: Verify recommendation reasoning
• AC-1.3: Should explain mode differences clearly
• Verification: Check output explanation

2. Command Generation

• AC-2.1: Should generate correct msprof op command
• Verification: Compare with reference commands
• AC-2.2: Should include required parameters
• Verification: Check parameter completeness
• AC-2.3: Should set correct output directory
• Verification: Verify path in command

3. Result Interpretation

• AC-3.1: Should interpret CSV files correctly
• Verification: Compare with actual data
• AC-3.2: Should explain visualize_data.bin usage
• Verification: Verify visualization guidance
• AC-3.3: Should identify bottleneck operators
• Verification: Check TOP analysis

4. Report Generation

• AC-4.1: Should follow fixed 4-section report template
• Verification: Verify report structure
• AC-4.2: Should include TOP5 items
• Verification: Check report completeness
• AC-4.3: Should provide actionable suggestions
• Verification: Verify recommendation quality

Correct/Error Pattern Comparison

Mode Selection

Correct: Choose device when real NPU available

# User has real NPU and wants real hardware bottleneck analysis
# -> Recommend device mode
msprof op --output=./output ./execute_op

Error: Use simulator when device mode needed

# User asks for real hardware analysis but uses simulator
msprof op simulator ...  # Wrong mode

Parameter Usage

Correct: Use --kernel-name only for application mode

msprof op --kernel-name="Add|Sub" --output=./output ./execute_op
# Correct for application

Error: Apply device parameters to simulator

msprof op simulator --kernel-name="Add" ...
# Wrong: --kernel-name not valid for simulator

Report Format

Correct: Follow fixed template

## 1. Operator Basic Information

| Field | Content |
|-------|---------|
| Mode | device |

## 2. Key Data TOP5

| Name | Metric | Value |

## 3. Core Bottleneck TOP5

| Name | Conclusion | Evidence |

## 4. Optimization Suggestions TOP5

| Name | Suggestion | Related Bottleneck |

Error: Use custom report format

# Profiling Results

Top operators:
1. Add
2. Mul
...

Non-Functional Acceptance Criteria

• NAC-1.1: Command generation time < 5 seconds
• NAC-1.2: Report generation time < 10 seconds
• NAC-1.3: Mode selection accuracy > 95%

Test Cases Summary

Positive Test Cases

1. TC-001: Device mode with application input 2. TC-002: Device mode with config input 3. TC-003: Simulator mode with application input 4. TC-004: Simulator mode with export input 5. TC-005: Fixed report generation 6. TC-006: visualize_data.bin interpretation

Negative Test Cases

1. TC-N01: Use device parameters for simulator 2. TC-N02: Use simulator parameters for device 3. TC-N03: Missing required parameters 4. TC-N04: Report missing required sections 5. TC-N05: Invalid mode selection for input type

Device Tuning Guide

This document focuses on `msprof op` Device Mode Profiling. For command-level pipeline, per-core breakdown, or complex dump analysis, see simulator-tuning-guide.md.

1. Scope and Prerequisites

Scope

• Have real Ascend device available
• Need real hardware metrics: time, bandwidth, cache, roofline, core load,

compute pipeline, etc.

• Input modes supported:
• application: executable file
• config: JSON + .o

Prerequisites Checklist

1. Application runs correctly before profiling 2. Output path and config path contain no symlinks; parent directory permissions match tool requirements 3. If source-level profiling is needed, operator compiled with -g 4. If using range replay, confirm user code has mstx markers

2. Compilation Preparation

If you need source-level profiling, cache breakdown, or source mapping, add debug info during operator compilation:

# Example: add -g to operator compile configuration
add_ops_compile_options(ALL OPTIONS -g)

Then recompile and redeploy the operator package.

Note: -g embeds debug info, which may impact performance. Control access

permissions accordingly. Official documentation does not support -O0.

3. Collection Mode Selection

3.1 Application Mode

# Single operator default collection
msprof op --output=./output ./execute_add_op

# Full metrics + Roofline
msprof op --aic-metrics=Roofline,Default --output=./output ./execute_add_op

3.2 Multi-Operator Application

# Collect first 10 matching Add/Sub operators
msprof op --launch-count=10 --kernel-name="Add|Sub" --output=./output ./test

# Skip first 3 operators, collect 5 starting from 4th
msprof op --launch-skip-before-match=3 --launch-count=5 --output=./output ./test

Notes:

• --kernel-name only valid in application mode
• --launch-skip-before-match count does not require kernel-name first

3.3 Config Mode

msprof op --config=./add_test.json --aic-metrics=Default --output=./output

This mode is useful when you cannot directly launch the app but have JSON + .o configuration.

4. Replay Mode Selection

• kernel (--replay-mode=kernel): Default; focus on single operator kernel.

Most stable.

• application (--replay-mode=application): Preserve app-level context / L2

state. May have partial data in visualize_data.bin.

• range (--replay-mode=range --mstx=on): Profile specific operator range.

Most limitations; check compatibility first.

Range Mode Limitations

• Must use with --mstx=on
• Only supports specific core types (see A2/A3 docs)
• Cannot use with MemoryDetail, TimelineDetail, Source simultaneously
• Not recommended with --kill=on
• Check version-specific help for merged operator limitations

5. Metric Selection Guide

• Confirm time anomaly: Default → OpBasicInfo.csv, PipeUtilization.csv
• Compute vs memory bound: Roofline,Default → Roofline in

visualize_data.bin

• Core load imbalance: Occupancy,Default → Per-core time/IO/cache rate
• Source-level profiling: Source,Default → visualize_data.bin

(needs -g)

• L2 / Memory detail: MemoryDetail → L2 hit rate, GM copy, MTE bandwidth
• Timeline detail: TimelineDetail,Default → Limited to A2/A3 scenarios
• Pipe pipeline: PipeTimeline → Atlas 350 only
• Lightweight basic info: BasicInfo → Only OpBasicInfo.csv

6. Result Analysis

6.1 CSV Quick Analysis

• OpBasicInfo.csv: Total time anomaly → Check operator name, block dim,

total time

• PipeUtilization.csv: Compute/copy imbalance → Check per-pipe time ratio
• ArithmeticUtilization.csv: Low compute unit utilization → Check

Cube/Vector time and ratio

• Memory.csv: Main path bandwidth issue → Check UB/L1/L2/GM read/write

bandwidth

• MemoryUB.csv: Large block variance → Check core imbalance
• L2Cache.csv: Low hit rate → Check L2 Hit/Miss
• ResourceConflictRatio.csv: High resource conflict → Check bank conflict

ratio

6.2 visualize_data.bin

Import into MindStudio Insight to view:

• Compute/memory heatmap
• Roofline bottleneck analysis
• Cache heatmap
• Source-level profiling
• Compute pipeline

6.3 trace.json

In device mode, trace.json is mainly for compute/communication pipeline. Its meaning differs from simulator mode trace.json.

7. Key Visualization Interpretation

7.1 Compute/Memory Heatmap

Focus on three aspects:

1. Core Load Analysis (Occupancy): If max/min differ significantly (>10% threshold), load is imbalanced. 2. Compute Load Analysis: Check if Cube/Vector utilization is low. 3. Memory Load Analysis: Check if MTE path bandwidth is a bottleneck.

7.2 Roofline

Use as "bottleneck quick check":

• Point near compute roofline: Compute Bound
• Point near bandwidth line: Memory Bound
• Neither: May be Latency Bound; drill into pipeline/memory/compute

7.3 Cache Heatmap

Answers:

• Which source code regions have L2 Hit/Miss anomalies?
• Is low hit rate concentrated in specific regions?

Requires:

• Source enabled
• Operator compiled with -g
• Core type / operator type supports this view

7.4 Source-Level Profiling

Left side shows source code level, right side shows instruction level. Useful for correlating "high-time code path" with "high-time instruction".

7.5 Compute Pipeline

This is often a source of confusion:

• User guide describes supported scope for merged operators
• Pipeline view shows MC2/LCCL/ASC details

Guidance:

• For general questions, say "supports merged operator scenarios"
• For specific operator support, check version help + version docs

8. Common Errors

1. Enabling all metrics at once

• Correct: Run Default first, then add Roofline/Source/MemoryDetail

2. Treating `TimelineDetail` as default for all profiling

• It has many limitations; Source is more generally applicable

3. Permission issues

• Tool may fail silently if output/config directory permissions are wrong

4. Confusing device `trace.json` with simulator pipeline view

• Same filename, different meaning

9. Recommended Tuning Sequence

1. Default: Confirm genuine performance issue exists 2. Roofline: Determine high-level direction (compute/memory/latency) 3. MemoryDetail or Source: Drill down based on direction 4. Occupancy: Check core imbalance 5. For compute or core-specific: Check trace.json/PipeTimeline/PcSampling

Simulator Tuning Guide

This document focuses on `msprof op simulator`. It is suitable for scenarios without real hardware, or when instruction-level analysis is needed.

1. Scope and Mode Boundaries

Questions Suitable for Simulator

• Is there a bubble in the instruction pipeline?
• Are MTE and VECTOR/CUBE properly parallel?
• Are SET_FLAG/WAIT_FLAG causing waits?
• Which code paths and instructions have highest time?
• Do you need per-core breakdown, per-core trace.json, IO rate waveform?

Common Confusions to Avoid

• Simulator --aic-metrics only supports:
• PipeUtilization
• ResourceConflictRatio
• PMSampling
• TimelineDetail is a device mode feature, not a simulator parameter
• Simulator trace.json is instruction pipeline view; device trace.json

is compute/communication pipeline

2. Pre-Start Checks

2.1 How to Specify Simulator Type

Two methods:

1. --soc-version=Ascendxxxyy 2. LD_LIBRARY_PATH=${INSTALL_DIR}/tools/simulator/Ascendxxxyy/lib:$LD_LIBRARY_PATH

Notes:

• application/export: Both methods work
• config: Use LD_LIBRARY_PATH; --soc-version is not valid here

2.2 Is -g Needed?

If you need code path mapping, call stack, or source-level profiling, compile with -g.

2.3 Is Build Simulator-Compatible?

Not all projects can directly run "device executable" in simulation:

• Some official examples/projects can run on both device and simulator
• Some template libraries or build rules require explicit simulator build

If you encounter signal 6, Bad address, std::__ios_failure, see experiences/simulator-needs-sim-build.md.

3. Data Collection Modes

3.1 Application Mode

# Method 1: Explicit simulator type
msprof op simulator --soc-version=Ascend910B4 \
  --output=./output_sim ./execute_add_op

# Method 2: Via environment variable
export LD_LIBRARY_PATH=${INSTALL_DIR}/tools/simulator/Ascend910B4/lib:$LD_LIBRARY_PATH
msprof op simulator --output=./output_sim ./execute_add_op

3.2 Config Mode

export LD_LIBRARY_PATH=${INSTALL_DIR}/tools/simulator/Ascend910B4/lib:$LD_LIBRARY_PATH
msprof op simulator --config=./add_test.json --output=./output_sim

Notes:

• Under --config, no need for --soc-version
• --kernel-name is not valid for --config

3.3 Export Mode (Parse Existing dump)

msprof op simulator --soc-version=Ascend910B4 \
  --export=./dump_dir --output=./output_sim

--export directory requirements:

• Should contain dump data and related kernel files
• For code path mapping, should include aicore_binary.o
• Pure dump without aicore_binary.o still allows pipeline parsing,

but no code mapping

4. Simulator-Specific Parameters

--soc-version

Check ${INSTALL_DIR}/tools/simulator/ directory names, e.g.:

• Ascend910B4
• Ascend910_9391
• Ascend310B4
• Ascend950

--timeout

For "large operator, long simulation" scenarios:

msprof op simulator --soc-version=Ascend910B4 \
  --timeout=5 --output=./output_sim ./app

Notes:

• Unit is minutes, range [1, 2880]
• On timeout, tool kills simulation process and proceeds to parsing

--core-id

Parse specific cores only, useful for evenly distributed operators:

msprof op simulator --soc-version=Ascend910B4 \
  --core-id="0|31" --output=./output_sim ./app

Notes:

• Range [0, 49]
• Only affects parsing, not simulation
• Not valid for `PMSampling`

--dump

msprof op simulator --soc-version=Ascend910B4 \
  --dump=on --output=./output_sim ./app

Notes:

• Default off
• For A2/A3, controls whether to keep dump
• For some Atlas products, parameter may not apply; dump follows normal process
• Only for single-process scenarios

5. Metrics and Views

Default Metrics

Simulator default provides:

• PipeUtilization
• ResourceConflictRatio

Meaning: Even without --aic-metrics, you get basic pipeline view and sync details.

PipeUtilization

• Shows instruction pipeline only
• Good for first look at overall execution order

ResourceConflictRatio

• Beyond pipeline, provides SET/WAIT FLAG sync details
• Good for analyzing sync waits or conflicts

PMSampling

msprof op simulator --soc-version=Ascend910B4 \
  --aic-metrics=PMSampling --output=./output_sim ./app

Purpose:

• Shows memory path IO rate waveform
• Key paths:
• GM <-> L1
• GM <-> UB
• GM <-> other

Notes:

• Not enabled by default
• Parses all cores; --core-id has no effect

6. Output Structure

Single Operator Common Structure

OPPROF_{timestamp}_XXX/
├── dump/
└── simulator/
    ├── core0.veccore0/
    │   ├── core0.veccore0_code_exe.csv
    │   ├── core0.veccore0_instr_exe.csv
    │   └── trace.json
    ├── core0.veccore1/
    │   ├── core0.veccore1_code_exe.csv
    │   ├── core0.veccore1_instr_exe.csv
    │   └── trace.json
    ├── ...
    ├── visualize_data.bin
    └── trace.json

Troubleshooting

1. Command Execution Issues

Issue: msprof command not found

Symptom: msprof: command not found

Solution:

# Check installation
which msprof

# Source environment
source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh

# Or add to PATH
export PATH=$PATH:/usr/local/Ascend/ascend-toolkit/latest/compiler/bin

Issue: msprof op permission denied

Symptom: Permission denied: ./execute_op

Solution:

# Fix permissions
chmod 755 ./execute_op

# Check SELinux (if applicable)
getenforce
setenforce 0  # Temporarily disable if needed

2. Device Mode Issues

Issue: --kernel-name not effective

Root Cause: --kernel-name only works with application mode

Solution:

# Verify you're using application mode
msprof op --kernel-name="Add" --output=./output ./execute_op  # Correct

# For config mode, use json matching instead
msprof op --config=./test.json --output=./output

Issue: signal 6 error

Symptom: Program terminated with signal 6

Root Cause: Usually application crash, not msprof issue

Solution:

# Run application directly first
./execute_op

# Check application logs
# Fix application bugs before profiling

Issue: TimelineDetail not available

Root Cause: TimelineDetail only for specific chips/modes

Solution:

# Check supported capabilities
msprof op --help | grep -i timeline

# Use Default instead
msprof op --aic-metrics=Default --output=./output ./execute_op

3. Simulator Mode Issues

Issue: signal 6 / Bad address

Symptom: signal 6 or Bad address during simulation

Root Cause: Simulator may need sim-compatible build

Solution:

# Check simulator build requirements
# Some operators need special compilation for simulation

# Try with --dump disabled
msprof op simulator --soc-version=Ascend910B4 \
  --output=./output --dump=off ./execute_op

Issue: --soc-version not effective

Root Cause: --soc-version only works with application/export, not config

Solution:

# For config mode, use LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${INSTALL_DIR}/tools/simulator/Ascend910B4/lib:$LD_LIBRARY_PATH
msprof op simulator --config=./test.json --output=./output

Issue: --core-id not working for PMSampling

Root Cause: --core-id does not filter PMSampling output

Solution:

# Run without --core-id for full PMSampling
msprof op simulator --aic-metrics=PMSampling \
  --output=./output ./execute_op

# Parse results to filter specific core
grep "core0" output/simulator/core0.*/instr_exe.csv

4. Output Issues

Issue: OPPROF directory not created

Root Cause: Permission or path issue

Solution:

# Use absolute path
msprof op --output=/tmp/profiling ./execute_op

# Check directory permissions
mkdir -p /tmp/profiling
chmod 777 /tmp/profiling

Issue: CSV files missing

Root Cause: Collection too short or metrics not enabled

Solution:

# Use Default metrics explicitly
msprof op --aic-metrics=Default --output=./output ./execute_op

# Increase iteration count
msprof op --launch-count=100 --output=./output ./execute_op

Issue: visualize_data.bin empty

Root Cause: Visualization metrics not collected

Solution:

# Collect with Roofline or Occupancy
msprof op --aic-metrics=Roofline --output=./output ./execute_op

# Check file size
ls -la output_npu/visualize_data.bin

5. Performance Issues

Issue: Profiling too slow

Solution:

# Reduce launch count
msprof op --launch-count=10 --output=./output ./execute_op

# Use simulator for quick iteration
msprof op simulator --output=./output ./execute_op

# Skip unnecessary metrics
msprof op --aic-metrics=BasicInfo --output=./output ./execute_op

Issue: Output too large

Solution:

# Limit core analysis
msprof op --core-id=0 --output=./output ./execute_op

# Use timeout
msprof op --timeout=60 --output=./output ./execute_op

Quick Diagnostic Commands

# Check msprof version
msprof --version

# Check help
msprof op --help

# List supported metrics
msprof op --help | grep -A50 "aic-metrics"

# Test basic profiling
msprof op --output=/tmp/test ./execute_op

# Check output
ls -la /tmp/test/

Verification Methods

Prerequisite Verification

1. Verify msprof Installation

# Check msprof version
msprof --version

# Check msprof op subcommand
msprof op --help

# Verify msprof op simulator
msprof op simulator --help

2. Verify Environment

# For device mode: Check NPU availability
npu-smi info

# For simulator mode: Check simulator installation
ls -la ${ASCEND_TOOLKIT_PATH}/tools/simulator/

# Check CANN version
cat /usr/local/Ascend/ascend-toolkit/latest/version.ini

3. Verify Target Application

# Check operator executable exists
ls -la ./execute_op

# Test executable runs
./execute_op

# Verify compilation with debug info (for source analysis)
file ./execute_op

Functional Verification

1. Mode Selection Verification

# Test mode determination logic
# User query: "profile operator on real NPU"
# Expected: device mode

# User query: "simulate without hardware"
# Expected: simulator mode

2. Command Generation Verification

# Generate command
msprof op --output=./output_npu ./execute_op

# Verify output directory created
ls -la output_npu/

# Check OpBasicInfo.csv generated
cat output_npu/OpBasicInfo.csv

3. Result Interpretation Verification

# Read CSV results
head -20 output_npu/OpBasicInfo.csv

# Check key metrics
# - Task Duration (us)
# - aiv_vec_ratio
# - aiv_scalar_ratio

# Verify visualize_data.bin exists
ls -la output_npu/visualize_data.bin

4. Report Verification

# Generate report following template
# Should contain:
# 1. Operator Basic Information
# 2. Key Data TOP5
# 3. Core Bottleneck TOP5
# 4. Optimization Suggestions TOP5

# Verify TOP5 items present
grep -c "##" report.md  # Should be 4 sections

End-to-End Verification Script

#!/bin/bash
set -e

echo "=== 1. Verify msprof Installation ==="
msprof --version
msprof op --help | head -5

echo "=== 2. Verify NPU (for device mode) ==="
npu-smi info | head -10

echo "=== 3. Verify Target Application ==="
ls -la ./execute_op
file ./execute_op

echo "=== 4. Run Profiling ==="
rm -rf output_npu
msprof op --output=./output_npu ./execute_op

echo "=== 5. Verify Output ==="
ls -la output_npu/
test -f output_npu/OpBasicInfo.csv && echo "OpBasicInfo.csv: OK"
test -f output_npu/PipeUtilization.csv && echo "PipeUtilization.csv: OK"

echo "=== All verifications passed ==="

Verification Checklist

• msprof version: >= 7.0
• msprof op available: Command recognized
• NPU available: npu-smi returns info
• Target executable: Exists and runs
• Output directory: OPPROF_* created
• OpBasicInfo.csv: File exists and readable
• PipeUtilization.csv: File exists
• visualize_data.bin: File exists
• Report sections: 4 required sections present