Huawei Cloud Ascend Op Mfu Calculator

Huawei Cloud Ascend Operator MFU Calculator

Overview

This skill calculates MFU (Machine FLOP Utilization) for operators like matmul/GEMM/FlashAttention on Ascend NPU, providing clear formulas and derivation process.

Architecture: Input Validation → FLOPs Calculation → Achieved TFLOPs/s → MFU Calculation → Result Analysis

Related Skills:

• huawei-cloud-ascend-profiler-db-explorer - Profiling data analysis for operator performance data

Prerequisites

1. Python 3.8+ installed 2. Basic understanding of FLOPs calculation concepts

Usage Scenarios

Typical Problem Scenarios:

• Evaluating how well an operator utilizes Ascend NPU compute power
• Comparing performance of different operator implementations
• Identifying optimization opportunities for matrix operations

Typical User Utterances:

• "Calculate MFU for my GEMM operator"
• "What's the machine FLOP utilization for FlashAttention?"
• "Analyze my matmul operator performance efficiency"

Workflow

1. Input Collection: Gather operator parameters (matrix dimensions, data types, execution time) 2. FLOPs Calculation: Compute theoretical FLOPs for the operation 3. Achieved Performance: Calculate achieved TFLOPs/s from execution time 4. MFU Calculation: Apply formula MFU = Achieved FLOPs / Peak FLOPs 5. Result Analysis: Provide interpretation and optimization suggestions

MFU Calculation Formula

MFU = (Achieved FLOPs / Peak FLOPs) × 100%

Where:

• Achieved FLOPs = Operation FLOPs / Execution Time
• Peak FLOPs = Hardware-specific peak performance (e.g., Ascend 910B: 256 TFLOPs for FP16)

Reference Documents

Enhanced Features

Intelligent Bottleneck Diagnoser

• AI-powered bottleneck diagnosis that analyzes profiling data to identify root causes automatically
• Classifies bottlenecks into categories: memory-bound, compute-bound, communication-bound, or operator-fallback
• Provides actionable optimization recommendations with priority ranking
• Includes pattern matching for known performance anti-patterns

Parameter Confirmation

MFU Calculation Methodology

Overview

This document provides detailed methodology for calculating Machine FLOP Utilization (MFU) for operators on Ascend NPU. MFU is a key metric to evaluate how effectively an operator utilizes the hardware computing power.

1. MFU Definition

Machine FLOP Utilization (MFU) is defined as:

$$ \text{MFU} = \frac{\text{Achieved FLOPs}}{\text{Peak FLOPs}} = \frac{\text{Actual FLOPs produced in computation}}{\text{Theoretical FLOPs executable by hardware in the same time}} $$

Unit Conventions

• FLOPs: Number of floating-point operations
• TFLOPs/s: Trillions of floating-point operations per second
• Execution Time: Typically in milliseconds (ms)

Conversion Factors

Achieved FLOPs/s = FLOPs / Execution Time (seconds)
Achieved TFLOPs/s = Achieved FLOPs/s ÷ 1e12
MFU = Achieved TFLOPs/s ÷ Peak TFLOPs/s

2. Peak FLOPs Reference

Ascend 910B Series

3. FLOPs Calculation Formulas

3.1 Matmul / GEMM

Standard Matrix Multiplication (M, K) × (K, N):

$$ \text{FLOPs} \approx 2 \times M \times N \times K $$

Batched Matmul (B, M, K) × (B, K, N):

$$ \text{FLOPs} \approx 2 \times B \times M \times N \times K $$

3.2 Linear Layer

Input: $(B, L, D_\text{in})$, Weight: $(D_\text{in}, D_\text{out})$

$$ \text{FLOPs} \approx 2 \times B \times L \times D_\text{in} \times D_\text{out} $$

3.3 Attention QK^T

$Q=(B, H, L_q, D_h),\ K=(B, H, L_k, D_h)$

$$ \text{FLOPs} \approx 2 \times B \times H \times L_q \times L_k \times D_h $$

3.4 FlashAttention

Common Layout (BNSD/BSND/BSH/SBH)

$$ \text{full_attention} = 2 \times q_b \times q_n \times q_s \times k_s \times (q_d + k_d) $$

Adjustment based on sparse_mode:

• sparse_mode == 0: $\text{FLOPs} = \text{full_attention}$
• sparse_mode == 2 or 3 with $q_s == k_s$: $\text{FLOPs} = \text{full_attention} \times 0.5$

TND Layout

$$ \text{FLOPs} = 2 \times N \times (D_q + D_k) \times \sum_{i} \text{q_lens}[i] \times \text{kv_lens}[i] $$

4. Step-by-Step Calculation Guide

Step 1: Gather Input Information

Collect the following from user or profiler:

• Operator type (matmul, GEMM, FlashAttention, etc.)
• Tensor dimensions (batch, sequence length, hidden dimension, etc.)
• Execution time (in milliseconds)
• Hardware peak FLOPs (from table above)

Step 2: Calculate Operator FLOPs

Use the appropriate formula based on operator type:

# Example: Matmul FLOPs calculation
def calculate_matmul_flops(M, N, K, batch=1):
    return 2 * batch * M * N * K

Step 3: Compute Achieved FLOPs/s

def calculate_achieved_tflops(flops, time_ms):
    time_s = time_ms / 1000
    flops_per_sec = flops / time_s
    tflops_per_sec = flops_per_sec / 1e12
    return tflops_per_sec

Step 4: Calculate MFU

def calculate_mfu(achieved_tflops, peak_tflops):
    return achieved_tflops / peak_tflops

5. Worked Examples

Example 1: Simple Matmul

Input:

• Matrix A: (1024, 512)
• Matrix B: (512, 1024)
• Execution time: 0.1 ms
• Hardware: Ascend 910B1 (378.88 TFLOPs/s)

Calculation: 1. FLOPs = 2 × 1024 × 1024 × 512 = 1,073,741,824 2. Achieved TFLOPs/s = 1,073,741,824 / 0.0001 / 1e12 = 10.74 TFLOPs/s 3. MFU = 10.74 / 378.88 ≈ 0.0283 → 2.83%

Example 2: Batched Matmul

Input:

• Batch size: 32
• Input shape: (32, 64, 128)
• Weight shape: (32, 128, 256)
• Execution time: 0.5 ms
• Hardware: Ascend 910B1 (378.88 TFLOPs/s)

Calculation: 1. FLOPs = 2 × 32 × 64 × 256 × 128 = 134,217,728 × 32 = 4,294,967,296 2. Achieved TFLOPs/s = 4,294,967,296 / 0.0005 / 1e12 = 8.59 TFLOPs/s 3. MFU = 8.59 / 378.88 ≈ 0.0227 → 2.27%

Example 3: Attention QK^T

Input:

• Batch: 4
• Heads: 12
• Query length: 1024
• Key length: 1024
• Head dimension: 64
• Execution time: 0.2 ms
• Hardware: Ascend 910B1 (378.88 TFLOPs/s)

Calculation: 1. FLOPs = 2 × 4 × 12 × 1024 × 1024 × 64 = 6,442,450,944 2. Achieved TFLOPs/s = 6,442,450,944 / 0.0002 / 1e12 = 32.21 TFLOPs/s 3. MFU = 32.21 / 378.88 ≈ 0.0850 → 8.50%

6. Interpretation Guidelines

7. Common Pitfalls

7.1 Unit Mismatch

Problem: Mixing milliseconds and seconds in calculations.

Solution: Always convert time to seconds before calculating FLOPs/s.

# Correct
time_s = time_ms / 1000

# Incorrect
# flops_per_sec = flops / time_ms  # Wrong units!

7.2 Missing Batch Dimension

Problem: Forgetting to multiply by batch size.

Solution: Always include batch dimension in FLOPs calculation.

7.3 Incorrect Peak FLOPs

Problem: Using wrong peak FLOPs for the hardware.

Solution: Verify the correct model and precision mode with the user.

7.4 Ignoring Sparse Mode

Problem: Not adjusting for causal mask or sparse attention.

Solution: Apply sparse_mode adjustment factors for FlashAttention.

8. Troubleshooting Low MFU

Checklist for Investigation

1. Check operator dimensions - Are they optimal for the hardware? 2. Verify execution time - Is it accurate from profiler? 3. Check memory bandwidth - Is the operator memory-bound? 4. Review operator configuration - Are tiling parameters optimal? 5. Consider fusion - Can multiple operators be fused? 6. Check parallelism - Is the computation properly parallelized?

Optimization Recommendations

• Increase batch size if memory allows
• Optimize tensor layouts for better memory access
• Use hardware-specific optimizations (e.g., Tensor Core)
• Consider operator fusion to reduce overhead
• Adjust tiling parameters for better cache utilization

9. Best Practices

1. Always verify peak FLOPs - Confirm with official documentation 2. Use profiler data - Get accurate execution times from profiling 3. Document assumptions - Note when using approximate values 4. Include uncertainty - Mention if results are estimates 5. Compare with baseline - Track improvements over time 6. Consider end-to-end - MFU is one metric; consider overall performance

10. References

• Ascend 910B Series Technical Specifications
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• GEMM Performance Optimization Guide

Verification Method

Verify MFU (Machine FLOP Utilization) calculation.

MFU Definition

$$ MFU = \frac{Achieved\ FLOPs/s}{Peak\ FLOPs/s} = \frac{Actual\ FLOPs}{Peak\ FLOPs \times Time} $$

Peak FLOPS Reference

FLOPs Calculation

Matmul/GEMM: (M, K) x (K, N) -> (M, N)

$$ FLOPs = 2 \times M \times K \times N $$

Example Calculation

For matmul (1024, 1024) x (1024, 1024):

• FLOPs = 2 x 1024 x 1024 x 1024 = 2,147,483,648
• If execution time = 1ms
• Achieved = 2.15 TFLOPS/s
• MFU on 910B3 = 2.15 / 294.91 = 0.73% (per chip)

Verification Steps

1. Get Operator Dimensions

# From profiling data or model definition
M, K, N = 1024, 1024, 1024

2. Calculate FLOPs

flops = 2 * M * K * N
print(f"FLOPs: {flops:,}")

3. Get Execution Time

# From profiling data
import pandas as pd
df = pd.read_csv("OpBasicInfo.csv")
exec_time_us = df[df["Op Name"] == "matmul"]["Task Duration(us)"].values[0]
exec_time_s = exec_time_us * 1e-6

4. Calculate MFU

achieved_tflops = flops / exec_time_s / 1e12
peak_tflops = 294.91  # 910B3
mfu = achieved_tflops / peak_tflops
print(f"MFU: {mfu:.2%}")

Acceptance Criteria

Common Issues

1. MFU > 100%

Cause: Wrong peak FLOPS or dimension error

Solution: Verify chip model and dimensions

2. Very Low MFU

Possible Causes:

• Memory-bound operation
• Small matrix size
• Suboptimal kernel

3. Negative MFU

Cause: Calculation error

Solution: Check all values are positive