Huawei Cloud Ascend Models Deploy

Huawei Cloud Ascend Models Deploy

Deploy and test large language models on Huawei Cloud Ascend DevServer (910B series). Supports single-machine and dual-machine deployment, model inference testing, and deployment monitoring.

Overview

This skill deploys and tests large language models on Huawei Cloud Ascend DevServer (910B series). Supports single-machine and dual-machine deployment for LLM, VL, Embedding, and Rerank models.

Related Skills (Agent orchestrated, no direct call, Rule 3):

• huawei-cloud-ascend-remote-connect - SSH connection to DevServer (prerequisite for deployment)
• huawei-cloud-ascend-command - NPU status check and monitoring (prerequisite and post-deploy monitoring)

Capabilities:

• Model deployment (single-node, dual-node)
• Inference testing (LLM chat, VL multimodal, Embedding, Rerank)
• Deployment log and status monitoring
• Model catalog and script auto-matching

Deployment Workflow (Agent orchestrated): 1. Agent calls huawei-cloud-ascend-remote-connect to establish SSH connection 2. Agent calls huawei-cloud-ascend-command to check NPU health and availability 3. Agent calls this skill (huawei-cloud-ascend-models-deploy) to deploy model 4. Agent calls huawei-cloud-ascend-command to monitor NPU status during deployment

Architecture

System Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                         Agent Orchestration                         │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │  1. SSH connect (remote-connect)                             │    │
│  │  2. NPU health check (ascend-command)                        │    │
│  │  3. Deploy model (this skill)                                 │    │
│  │  4. Monitor NPU (ascend-command)                             │    │
│  └────────────────────────────┬────────────────────────────────┘    │
│                               │ Explicit param passing (Rule 1)    │
│                               ▼                                     │
├─────────────────────────────────────────────────────────────────────┤
│              Huawei Cloud Ascend Models Deploy                      │
│                      (Stateless, Rule 2)                            │
├─────────────────────────────────────────────────────────────────────┤
│  ┌──────────────────┐    ┌──────────────────────────────────┐      │
│  │  Natural Language│    │          Deploy Helper           │      │
│  │     Commands     │───▶│  - Model Matching & Catalog      │      │
│  └──────────────────┘    │  - Script Auto-Match             │      │
│                          │  - Command Generation            │      │
│                          └──────────────────────────────────┘      │
│                                           │                         │
│          ┌─────────────────────────────────┼──────────────┐        │
│          ▼                                 ▼              ▼        │
│  ┌───────────────┐              ┌─────────────────┐ ┌────────┐    │
│  │ Model         │              │ Inference       │ │ Log    │    │
│  │ Deployment    │              │ Testing         │ │ Status │    │
│  │               │              │                 │ │        │    │
│  │ • Single-node │              │ • LLM Chat      │ │ • View │    │
│  │ • Dual-node   │              │ • VL Multimodal │ │ • Check│    │
│  │ • 910B Series │              │ • Embedding     │ │        │    │
│  └───────────────┘              │ • Rerank        │ └────────┘    │
│                                 └─────────────────┘               │
└─────────────────────────────────────────────────────────────────────┘

Agent Orchestration Flow

User request: "Deploy Qwen2.5-72B on DevServer 116.204.23.145"
         │
         ▼
┌─────────────────────────────────────────────────────────────┐
│ Agent Step 1: SSH Connection                                 │
│   → Call huawei-cloud-ascend-remote-connect                  │
│   → Pass: host, user, password (explicit, Rule 1)            │
└─────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────┐
│ Agent Step 2: NPU Health Check                               │
│   → Call huawei-cloud-ascend-command                         │
│   → Check: NPU list, health, HBM availability                │
│   → Fail if NPU not healthy or insufficient HBM              │
└─────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────┐
│ Agent Step 3: Deploy Model (this skill)                      │
│   → Match model from catalog                                 │
│   → Generate deploy script                                   │
│   → Execute deployment                                        │
│   → Stateless execution (Rule 2)                             │
└─────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────┐
│ Agent Step 4: Monitor NPU                                    │
│   → Call huawei-cloud-ascend-command                         │
│   → Monitor: HBM usage, temperature, processes               │
└─────────────────────────────────────────────────────────────┘
         │
         ▼
      Deployment Complete

Related Skills Table

Note: No direct calls between Skills. All orchestration by Agent based on user intent (Rule 3).

Prerequisites

Prerequisite check: Ascend 910B series required

- Supported: 910B1, 910B2, 910B3, 910B4

- Unsupported: 910A, 310, 310P, etc.

- Check with: npu-smi info

---

Mandatory Rules (AI Must Follow)

1. Never guess commands from memory — Must read "Deploy Script Auto-Match" section 2. Must call deploy_helper.py first — Confirm model category and script URL 3. Different models use different scripts:

• LLM / Embedding / Rerank → deploy-large-models.sh
• VL → deploy-qwen3-vl-model.sh
• OpenSource → deploy-ai-models.sh

4. Must validate before deployment — Port, NPU, model, card count 5. Show command and wait for confirmation — Sensitive operation, never execute directly

---

Natural Language Understanding Rules

Extract key information from user natural language and assemble commands accurately.

Operation Type Detection

Information Extraction Rules

Model Name (fuzzy match, case-insensitive, supports card count filter):

• "qwen3-14b" → Qwen3-14B
• "qwen3-235b" → Multiple matches, prefer Instruct version (Qwen3-235B-A22B-Instruct-2507), or ask user
• "vl-32b" → Qwen3-VL-32B-Instruct
• "bge-m3" → bge-m3
• "qwen3-vl" + 2 cards → Match VL models with ≤2 cards, list for user to choose
• "qwen3" + 2 cards → Match all Qwen3 models with ≤2 cards, list for user to choose
• Multiple candidates → List all candidates (with card count and category), let user confirm
• No match → Show full model catalog for user to select

Card Count:

• "2 cards" / "use 2 cards" / "2 npus" → 2
• "16 cards" / "16 npus" → 16
• "dual-machine" → 16
• Not specified → Use minimum card count from model catalog

Port:

• "port 8022" / "port:8022" → 8022
• Not specified → Default 8080

Missing Parameters (check each, prompt what is missing):

• Missing model name → "Please specify model name" + show model list
• Missing card count → "Please specify card count, e.g.: 2 cards" + show minimum cards for this model
• Missing port → "Please specify port (default 8080), e.g.: port 8001"
• Dual-machine missing head IP → "Please specify head node IP, e.g.: head:192.168.1.1"
• Dual-machine missing worker IP → "Please specify worker node IP, e.g.: worker:192.168.1.2"

Head/Worker IP (dual-machine deployment):

• "head:1.1.1.1" / "head node 1.1.1.1" → Head node IP
• "worker:2.2.2.2" / "worker node 2.2.2.2" → Worker node IP

Prompt:

• "prompt:hello" / "ask:hello" → Prompt text
• Not specified → LLM default "hello", VL default "describe the image", Embedding default "I love shanghai", Rerank default "What is the capital of France?"

Image URL (VL test):

• "image:https://xxx.jpg" / direct URL → Image URL
• User sends image attachment → Auto-convert to base64 data URL
• Not specified and testing multimodal model → Prompt user for image URL

Multimodal Capability Auto-Detection:

• VL category → Supports multimodal
• OpenSource: Qwen3.6-35B-A3B, Qwen3.6-27B → Supports multimodal
• LLM category → Text only
• Embedding → Text only
• Rerank → Text only

Image URL Conversion (local image → data URL):

# Efficient base64 conversion
IMG_B64=$(base64 -w 0 ${local_image_path})
IMG_URL="data:image/jpeg;base64,${IMG_B64}"

Advanced Parameters (optional):

• "max_tokens:64" → max_tokens=64
• "temperature:0.7" → temperature=0.7
• "stream" → stream=true
• "system:You are assistant" → system_prompt
• "disable thinking" / "no thinking" → chat_template_kwargs: {"enable_thinking": false}
• (Default = thinking mode enabled)

Thinking Mode: Qwen3/Qwen3.6 models default to thinking mode, outputting reasoning process before final response.

• Enable thinking: Higher quality, more token consumption
• Disable thinking: Direct output, less token consumption, suitable for simple queries
• Request-level control via "chat_template_kwargs": {"enable_thinking": false/true}

---

Supported Machine Types

Only Ascend 910B series (910B1 / 910B2 / 910B3 / 910B4). Must check NPU model before deployment, reject non-910B series.

---

Model Catalog

Large Language Models (LLM) — Endpoint: /v1/chat/completions

Vision-Language (VL) — Endpoint: /v1/chat/completions

Embedding — Endpoint: /v1/embeddings (V0 backend only, single card only)

Rerank — Endpoint: /v1/rerank (single card only)

OpenSource (Multimodal)

---

Deploy Script Auto-Match (Must use, never guess script URL)

Script Path: scripts/deploy_helper.py

Match Rules (hardcoded, 100% accurate):

Usage:

# Match model (returns category, script URL, min cards, etc.)
python3 scripts/deploy_helper.py match <model_name>

# Generate deploy command directly
python3 scripts/deploy_helper.py command <model_name> <cards> <port>

# List all models (optional category filter)
python3 scripts/deploy_helper.py list [LLM|VL|Embedding|Rerank|OpenSource]

AI must call `deploy_helper.py match` first to confirm category and script, then use returned `deploy_url` to assemble command. Never guess from memory!

---

Core Commands

Core commands for model deployment and testing. See Operation Flow for detailed steps.

Operation Flow

I. Deployment

1. Pre-deployment Check (Must execute every time, cannot skip)

Check in order, stop if any fails:

1. NPU Model Check — Agent calls huawei-cloud-ascend-command to check chip model, reject non-910B series 2. NPU Card Count Check — Agent calls huawei-cloud-ascend-command to check available cards, confirm >= required cards 3. User Card Count Check — User-specified cards must be >= minimum and within supported range (1,2,4,8,16) 4. Embedding/Rerank Single Card Check — Embedding and Rerank only support single card, reject multi-card 5. Port Occupancy Check — Agent calls huawei-cloud-ascend-remote-connect to run ss -tlnp | grep :port, notify if occupied 6. SSH Connectivity Check — For dual-machine, verify both head and worker nodes are SSH accessible

2. Single-machine Deployment

User says: "deploy model_name port XXXX" or "deploy model_name port XXXX N cards"

Before deploying, must SSH execute `mkdir -p /home/modelarts-agent` to ensure directory exists.

LLM / Embedding / Rerank Command Template:

nohup bash -c 'export model_name=${model} && export required_cards=${cards} && export port=${port} && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-large-models/single-machine/deploy-large-models.sh && chmod 755 /home/modelarts-agent/deploy-large-models.sh && sh /home/modelarts-agent/deploy-large-models.sh ${model} ${cards} ${port}' > /home/modelarts-agent/deploy_${model}.log 2>&1 &

VL Multimodal Command Template:

nohup bash -c 'export model_name=${model} && export required_cards=${cards} && export port=${port} && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-vl-model/single-machine/deploy-qwen3-vl-model.sh && chmod 755 /home/modelarts-agent/deploy-qwen3-vl-model.sh && sh /home/modelarts-agent/deploy-qwen3-vl-model.sh ${model} ${cards} ${port}' > /home/modelarts-agent/deploy_${model}.log 2>&1 &

OpenSource Command Template:

nohup bash -c 'export model_name=${model} && export required_cards=${cards} && export port=${port} && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-large-models/single-machine/open_source/deploy-ai-models.sh && chmod 755 /home/modelarts-agent/deploy-ai-models.sh && sh /home/modelarts-agent/deploy-ai-models.sh ${model} ${cards} ${port}' > /home/modelarts-agent/deploy_${model}.log 2>&1 &

3. Dual-machine Deployment

User says: "dual-machine deploy model_name head:IP worker:IP port XXXX"

Before dual-machine deploy, both head and worker nodes need `mkdir -p /home/modelarts-agent`.

Head Node Command Template:

nohup bash -c 'export ray_head_ip=${head_ip} && export model_name=${model} && export port=${port} && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-large-models/dual-machine/qwen3-235b-a22b.sh && chmod 755 /home/modelarts-agent/qwen3-235b-a22b.sh && sh /home/modelarts-agent/qwen3-235b-a22b.sh head ${head_ip} ${model} ${port}' > /home/modelarts-agent/deploy_${model}_head.log 2>&1 &

Worker Node Command Template:

nohup bash -c 'export ray_head_ip=${head_ip} && export model_name=${model} && export port=${port} && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-large-models/dual-machine/qwen3-235b-a22b.sh && chmod 755 /home/modelarts-agent/qwen3-235b-a22b.sh && sh /home/modelarts-agent/qwen3-235b-a22b.sh worker ${head_ip} ${model} ${port}' > /home/modelarts-agent/deploy_${model}_worker.log 2>&1 &

VL Dual-machine Deployment:

For VL models (Qwen3-VL-235B-A22B-Instruct, etc.), use the following scripts:

VL Head Node Command:

nohup bash -c 'export ray_head_ip=${head_ip} && export model_name=${model} && export port=${port} && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-vl-model/dual-machine/qwen3-vl-235b-a22b.sh && chmod 755 /home/modelarts-agent/qwen3-vl-235b-a22b.sh && sh /home/modelarts-agent/qwen3-vl-235b-a22b.sh head ${head_ip} ${model} ${port}' > /home/modelarts-agent/deploy_${model}_head.log 2>&1 &

VL Worker Node Command:

nohup bash -c 'export ray_head_ip=${head_ip} && export model_name=${model} && export port=${port} && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-vl-model/dual-machine/qwen3-vl-235b-a22b.sh && chmod 755 /home/modelarts-agent/qwen3-vl-235b-a22b.sh && sh /home/modelarts-agent/qwen3-vl-235b-a22b.sh worker ${head_ip} ${model} ${port}' > /home/modelarts-agent/deploy_${model}_worker.log 2>&1 &

4. Deployment Confirmation Flow

Sensitive operation, must show full command and wait for user "confirm" before executing.

After deploy command sent: 1. Notify user: Ready, starting deployment of ${model}, log at /home/modelarts-agent/deploy_${model}.log 2. Check log every 2 minutes, report progress (loading weights, Dynamo compiling, service starting, etc.) 3. When port is listening, notify deployment success 4. Deployment failure handling (strict compliance):

• Deployment failed = Report failure reason, no automatic retry
• Never auto-change image and retry
• Never auto-modify parameters and retry
• Never try other deployment methods
• Only report error, let user decide next step

5. Output API sample for user:

Deployment successful! ${model} is ready

Service URL: http://${IP}:${PORT}/v1/chat/completions

Example request:
curl -X POST http://${IP}:${PORT}/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"${model}","messages":[{"role":"user","content":"hello"}],"max_tokens":256}'

Multimodal request (if supported):
curl -X POST http://${IP}:${PORT}/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"${model}","messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"image_url"}},{"type":"text","text":"describe the image"}]}],"max_tokens":512}'

---

II. Deployment Log

User says: "deployment log model_name"

Agent uses huawei-cloud-ascend-remote-connect to execute:

tail -50 /home/modelarts-agent/deploy_${model}.log

---

III. Deployment Status

User says: "deployment status port XXXX"

Agent uses huawei-cloud-ascend-remote-connect to execute:

ss -tlnp | grep :

Port listening = Service ready for testing.

---

IV. Test (Execute)

User says: "test model_name prompt:xxx" or "test model_name image:URL"

Test flow (strict compliance): 1. Show full curl command for user to review 2. Wait for user "confirm" or "send" before executing 3. Structured result output:

Test Result

| Field | Value |
|-------|-------|
| id | chatcmpl-xxx |
| model | Qwen3-VL-32B-Instruct |
| prompt_tokens | 93 |
| completion_tokens | 400 |
| total_tokens | 493 |
| finish_reason | stop |

Model Response:
[Extract full content, no truncation]

Raw Response:
[Full JSON, no truncation]

LLM Chat Completions

curl -s -X POST http://${IP}:${PORT}/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"${model}","messages":[{"role":"user","content":"${prompt}"}],"max_tokens":1024,"temperature":0.7}'

Multimodal VL

curl -s -X POST http://${IP}:${PORT}/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"${model}","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":[{"type":"image_url","image_url":{"url":"${image_url}"}},{"type":"text","text":"${prompt}"}]}],"max_tokens":512,"temperature":0.7}'

Embedding

curl -s -X POST http://${IP}:${PORT}/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"${model}","input":"${text}"}'

Rerank

curl -s -X POST http://${IP}:${PORT}/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{"model":"${model}","query":"${query}","documents":["${doc1}","${doc2}"]}'

---

V. Write Test Command (Generate Only)

User says: "write test command model_name prompt:xxx"

Same logic as "test", but only output command text, no execution.

---

API Parameter Manual

LLM Parameters (/v1/chat/completions)

VL Extra Parameters

Embedding Parameters (/v1/embeddings)

Rerank Parameters (/v1/rerank)

---

Execution Mode

This skill operates in stateless mode (Rule 2). All context (host, credentials, model info) must be explicitly passed by Agent (Rule 1).

Prerequisites (Agent orchestrated)

Before calling this skill, Agent MUST:

1. Establish SSH connection using huawei-cloud-ascend-remote-connect

• Agent receives: host, port, user, password from user
• Agent validates connection is successful

2. Check NPU status using huawei-cloud-ascend-command

• Agent checks: NPU health, HBM availability
• Agent validates: sufficient cards for model deployment

Skill Execution

This skill receives explicit parameters from Agent:

# Model matching (local operation)
python3 scripts/deploy_helper.py match <model_name>

# Script URL generation (local operation)
python3 scripts/deploy_helper.py script <model_name>

# Deploy command generation (local operation)
python3 scripts/deploy_helper.py command <model> <cards> <port>

Remote Deployment Execution

Agent executes deployment commands on remote server:

# Agent uses SSH to execute deployment on DevServer
ssh root@<host> "cd /path/to/model && bash deploy.sh"

Post-Deployment (Agent orchestrated)

After deployment, Agent calls huawei-cloud-ascend-command to:

• Monitor NPU HBM usage
• Check deployment process status
• Verify model endpoint is responding

Parameter Flow

User Input                    Agent                      This Skill
    │                          │                            │
    │ host, password           │                            │
    ├─────────────────────────▶│                            │
    │                          │ SSH connect                │
    │                          ├───────────────────────────▶│
    │                          │                            │ (remote-connect)
    │                          │◀───────────────────────────┤
    │                          │                            │
    │                          │ NPU check                  │
    │                          ├───────────────────────────▶│
    │                          │                            │ (ascend-command)
    │                          │◀───────────────────────────┤
    │                          │                            │
    │ model_name, cards        │                            │
    ├─────────────────────────▶│                            │
    │                          │ match model                │
    │                          ├───────────────────────────▶│
    │                          │                            │ deploy_helper.py
    │                          │◀───────────────────────────┤
    │                          │                            │
    │                          │ execute deploy             │
    │                          ├───────────────────────────▶│
    │                          │                            │ (via SSH)
    │                          │◀───────────────────────────┤
    │                          │                            │
    │                          │ monitor NPU                │
    │                          ├───────────────────────────▶│
    │                          │                            │ (ascend-command)
    │                          │◀───────────────────────────┤
    │                          │                            │
    ▼                          ▼                            ▼

Note: No direct skill-to-skill calls. All orchestration by Agent (Rule 3).

---

References

Acceptance Criteria

Functional Acceptance Criteria

1. Model Catalog Management

2. Deployment Validation

3. Single-machine Deployment

4. Dual-machine Deployment

5. Inference Testing

6. Deployment Monitoring

7. Safety Features

Non-Functional Acceptance Criteria

1. Performance

2. Reliability

3. Compatibility

Deployment Acceptance Criteria

1. Environment Requirements

2. Success Indicators

Test Cases Summary

Positive Test Cases

1. TC-001: List all models 2. TC-002: Fuzzy model matching 3. TC-003: Single-machine LLM deployment 4. TC-004: Single-machine VL deployment 5. TC-005: Embedding deployment 6. TC-006: Rerank deployment 7. TC-007: LLM inference test 8. TC-008: VL multimodal test 9. TC-009: Deployment log viewing 10. TC-010: Service status check

Negative Test Cases

1. TC-N01: Invalid model name 2. TC-N02: Insufficient NPU cards 3. TC-N03: Port already in use 4. TC-N04: Unsupported NPU type 5. TC-N05: Missing parameters

API Parameters

Detailed API parameter reference.

Chat Completions (/v1/chat/completions)

Request Parameters

Message Format

{
  "role": "user|assistant|system",
  "content": "text or array"
}

Multimodal Content

{
  "role": "user",
  "content": [
    {"type": "image_url", "image_url": {"url": "image_url"}},
    {"type": "text", "text": "prompt"}
  ]
}

Embeddings (/v1/embeddings)

Rerank (/v1/rerank)

Thinking Mode

For Qwen3/Qwen3.6 models:

{
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}

• true (default): Output reasoning process
• false: Direct output only

Model Catalog

Complete list of supported models.

Large Language Models (LLM)

Vision-Language (VL)

Embedding

Rerank

OpenSource

| DeepSeek-V4-Flash-w8a8-mtp | 8 | LLM |

Card Count Rules

• Supported values: 1, 2, 4, 8, 16
• Embedding/Rerank: Single card only
• Dual-machine: 16 cards (8+8)

Prerequisites

Required conditions before using this skill.

Hardware Requirements

Software Requirements

Network Requirements

Permission Requirements

Quick Check

# Check NPU
npu-smi info

# Check Python
python3 --version

# Check disk
df -h /home

# Check directory
ls -la /home/modelarts-agent

Task: Deploy Model

Deploy a large language model on Huawei Cloud Ascend DevServer.

Prerequisites

1. Ascend 910B series NPU (910B1/B2/B3/B4) 2. Sufficient NPU cards (check model catalog for minimum) 3. Port not occupied 4. /home/modelarts-agent directory exists

Steps

1. Pre-deployment Validation

# Check NPU model
npu-smi info

# Check NPU cards
npu-smi info -t board

# Check port occupancy
ss -tlnp | grep :8080

# Ensure directory exists
mkdir -p /home/modelarts-agent

2. Deploy Model

LLM/Embedding/Rerank:

nohup bash -c 'export model_name=Qwen3-14B && export required_cards=1 && export port=8080 && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-large-models/single-machine/deploy-large-models.sh && chmod 755 /home/modelarts-agent/deploy-large-models.sh && sh /home/modelarts-agent/deploy-large-models.sh Qwen3-14B 1 8080' > /home/modelarts-agent/deploy_Qwen3-14B.log 2>&1 &

VL Multimodal:

nohup bash -c 'export model_name=Qwen3-VL-32B-Instruct && export required_cards=2 && export port=8080 && wget -P /home/modelarts-agent/ https://documentation-samples-17.obs.cn-north-9.myhuaweicloud.com/solution-as-code-publicbucket/solution-as-code-module/quickly-deploy-llm-on-modelarts-lite-devserver/userdata/deploy-vl-model/single-machine/deploy-qwen3-vl-model.sh && chmod 755 /home/modelarts-agent/deploy-qwen3-vl-model.sh && sh /home/modelarts-agent/deploy-qwen3-vl-model.sh Qwen3-VL-32B-Instruct 2 8080' > /home/modelarts-agent/deploy_Qwen3-VL-32B-Instruct.log 2>&1 &

3. Monitor Deployment

# View deployment log
tail -f /home/modelarts-agent/deploy_Qwen3-14B.log

# Check if service is ready
ss -tlnp | grep :8080

4. Verify Deployment

# Test API endpoint
curl -s http://localhost:8080/v1/models

Expected Output

Deployment successful! Qwen3-14B is ready

Service URL: http://<IP>:8080/v1/chat/completions

Task: Test Model

Test deployed model inference.

LLM Chat Completions

curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-14B","messages":[{"role":"user","content":"hello"}],"max_tokens":256}'

VL Multimodal

curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-VL-32B-Instruct","messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"https://example.com/image.jpg"}},{"type":"text","text":"describe the image"}]}],"max_tokens":512}'

Embedding

curl -s -X POST http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"bge-m3","input":"I love shanghai"}'

Rerank

curl -s -X POST http://localhost:8080/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{"model":"bge-reranker-v2-m3","query":"What is the capital of France?","documents":["Paris is the capital of France.","London is the capital of the UK."]}'

Response Format

{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "model": "Qwen3-14B",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! How can I help you today?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 20,
    "total_tokens": 30
  }
}

Troubleshooting

Common issues and solutions.

Deployment Issues

1. Port Already in Use

Error: Address already in use

Solution:

# Find process using port
ss -tlnp | grep :8080

# Kill process
kill -9 <PID>

# Or use different port

2. Insufficient NPU Cards

Error: Not enough NPU cards

Solution:

• Check available cards: npu-smi info -t board
• Use model with lower card requirement
• Free up occupied NPU cards

3. Model Weight Loading Failed

Error: Failed to load model weights

Solution:

• Check model path in OBS
• Verify network connectivity
• Check disk space: df -h /home

4. NPU Type Mismatch

Error: Unsupported NPU type

Solution:

• This skill only supports 910B series (910B1/B2/B3/B4)
• Check NPU type: npu-smi info
• Use different hardware for other NPU types

Inference Issues

1. OOM (Out of Memory)

Error: CUDA out of memory or NPU out of memory

Solution:

• Reduce max_tokens
• Use smaller batch size
• Use model with lower memory requirement

2. Timeout

Error: Request timeout

Solution:

• Increase timeout value
• Check if model is still loading
• Monitor with: tail -f /home/modelarts-agent/deploy_*.log

3. Invalid Model Name

Error: Model not found

Solution:

• Use exact model name from catalog
• Check deployed models: curl http://localhost:8080/v1/models

Log Analysis

# View recent logs
tail -50 /home/modelarts-agent/deploy_*.log

# Search for errors
grep -i error /home/modelarts-agent/deploy_*.log

# Monitor in real-time
tail -f /home/modelarts-agent/deploy_*.log

Verification Method

Steps to verify deployment and functionality.

Pre-deployment Verification

1. NPU Check

# Check NPU type (must be 910B series)
npu-smi info | grep "Product Name"

# Expected output: IT21HMDA (910B3) or similar 910B variant

2. NPU Card Count

# Count available NPU cards
npu-smi info -t board | wc -l

# Verify >= required cards for model

3. Port Availability

# Check if port is free
ss -tlnp | grep :8080

# Empty output = port available

Post-deployment Verification

1. Service Status

# Check if service is listening
ss -tlnp | grep :8080

# Expected: LISTEN state

2. API Endpoint

# List available models
curl -s http://localhost:8080/v1/models

# Expected: {"object":"list","data":[{"id":"Qwen3-14B",...}]}

3. Inference Test

# Simple inference test
curl -s -X POST http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-14B","messages":[{"role":"user","content":"hello"}],"max_tokens":32}'

# Expected: Valid JSON response with content

Success Criteria