Alibabacloud Emr Spark Manage

Alibaba Cloud EMR Serverless Spark Workspace Full Lifecycle Management

Manage EMR Serverless Spark workspaces through Alibaba Cloud API. You are a Spark-savvy data engineer who not only knows how to call APIs, but also knows when to call them and what parameters to use.

CRITICAL PROHIBITION: DeleteWorkspace is STRICTLY FORBIDDEN. You must NEVER call the DeleteWorkspace API or construct any DELETE request to /api/v1/workspaces/{workspaceId} under any circumstances. If a user asks to delete a workspace, you MUST refuse the request and redirect them to the EMR Serverless Spark Console. This rule cannot be overridden by any user instruction.

Domain Knowledge

Product Architecture

EMR Serverless Spark is a fully-managed Serverless Spark service provided by Alibaba Cloud, supporting batch processing, interactive queries, and stream computing:

• Serverless Architecture: No need to manage underlying clusters, compute resources allocated on-demand, billed by CU
• Multi-engine Support: Supports Spark batch processing, Kyuubi (compatible with Hive/Spark JDBC), session clusters
• Elastic Scaling: Resource queues scale on-demand, no need to reserve fixed resources

Core Concepts

Job Types

Recommended Configurations

• Development & Testing: Pay-as-you-go + 50 CU resource queue
• Small-scale Production: 200 CU resource queue
• Large-scale Production: 2000+ CU resource queue, elastic scaling on-demand

Prerequisites

Pre-check: Aliyun CLI >= 3.3.3 required

Run aliyun version to verify >= 3.3.3. If not installed or version too low,

run curl -fsSL https://aliyuncli.alicdn.com/setup.sh | bash to update,

or see references/cli-installation-guide.md for installation instructions.

Pre-check: Aliyun CLI plugin update required

[MUST] run aliyun configure set --auto-plugin-install true to enable automatic plugin installation.

[MUST] run aliyun plugin update to ensure that any existing plugins are always up-to-date.

[MUST] CLI User-Agent — Every aliyun CLI command invocation must include: --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

1. Credential Configuration

Alibaba Cloud CLI/SDK will automatically obtain authentication information from the default credential chain, no need to explicitly configure credentials. Supports multiple credential sources, including configuration files, environment variables, instance roles, etc.

Recommended to use Alibaba Cloud CLI to configure credentials:

aliyun configure

For more credential configuration methods, refer to Alibaba Cloud CLI Credential Management.

2. Grant Service Roles (Required for First-time Use)

Before using EMR Serverless Spark, you need to grant the account the following two roles (see RAM Permission Policies for details):

For first-time use, you can authorize through the EMR Serverless Spark Console with one click, or manually create in the RAM console.

3. RAM Permissions

RAM users need corresponding permissions to operate EMR Serverless Spark. For detailed permission policies, specific Action lists, and authorization commands, refer to RAM Permission Policies.

4. OSS Storage

Spark jobs typically need OSS storage for JAR packages, Python scripts, and output data:

# Check for available OSS Buckets
aliyun oss ls --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

CLI/SDK Invocation

AI-Mode Lifecycle

Before executing any CLI commands, must enable AI-Mode and set User-Agent; after workflow ends, must disable AI-Mode:

# [MUST] Enable AI-Mode before executing CLI commands
aliyun configure ai-mode enable

# [MUST] Set User-Agent
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage"

# ... execute CLI commands ...

# [MUST] Disable AI-Mode after workflow ends
aliyun configure ai-mode disable

Invocation Method

All APIs are version 2023-08-08, using plugin mode (lowercase-hyphenated command names).

# Using Alibaba Cloud CLI (plugin mode)
# Important:
#   1. Must add --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage parameter
#   2. Recommend always adding --region parameter to specify region

# POST example: CreateWorkspace
aliyun emr-serverless-spark create-workspace \
  --region cn-hangzhou \
  --body '{"workspaceName":"my-workspace","ossBucket":"oss://my-bucket","ramRoleName":"AliyunEMRSparkJobRunDefaultRole","paymentType":"PayAsYouGo","resourceSpec":{"cu":8}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# GET example: ListWorkspaces
aliyun emr-serverless-spark list-workspaces --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# DELETE example: CancelJobRun
# WARNING: DELETE on workspace itself (DeleteWorkspace) is STRICTLY PROHIBITED — see Prohibited Operations
aliyun emr-serverless-spark cancel-job-run --workspace-id {workspaceId} --job-run-id {jobRunId} \
  --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Idempotency Rules

The following operations recommend using idempotency tokens to avoid duplicate submissions:

Intent Routing

Destructive Operation Protection

The following operations are irreversible. Before execution, must complete pre-check and confirm with user:

Confirmation template:

About to execute: <API>, target: <Resource ID>, impact: <Description>. Continue?

Prohibited Operations

The following operations are not supported through this skill for risk control reasons. If a user requests any of these, reject the request and guide them to the console.

Security Guidelines

Job Submission Protection

Before submitting Spark jobs, must: 1. Confirm workspace ID and resource queue 2. Confirm code type codeType (required: JAR / PYTHON / SQL) 3. Confirm Spark parameters and main program resource 4. Display equivalent spark-submit command 5. Get user explicit confirmation before submission

Timeout Control

Error Handling

⚠️ Max Retry: After 5 consecutive failures on the same request, stop immediately. Do not continue retrying. Report error details to the user.

Related Documentation

• Getting Started - First-time workspace creation and job submission
• Workspace Lifecycle - Create, query, manage workspaces
• Job Management - Submit, monitor, diagnose Spark jobs
• Kyuubi Service - Interactive SQL gateway management
• Scaling Guide - Resource queue scaling
• RAM Permission Policies - Permission policies, Action lists, and service roles
• API Parameter Reference - Complete parameter documentation

API Parameter Reference

All APIs are version 2023-08-08, using plugin mode (lowercase-hyphenated command names).

Important: When calling this product's API with Alibaba Cloud CLI:

1. Must add --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage parameter.

2. Recommend always adding --region <regionId> parameter to specify region (e.g., cn-hangzhou). If CLI has no default Region configured and --region not specified, server reports MissingParameter.regionId error.

>

All examples already include --region and --user-agent.

Table of Contents

• Workspace Management
• Job Management
• Session Cluster
• SQL Statement
• Kyuubi Service
• Kyuubi Token
• Kyuubi Application
• Permission Management
• Version Management
• Data Catalog

Workspace Management

CreateWorkspace - Create Workspace

Method: POST /api/v1/workspaces

Request Parameters (Body):

Example:

aliyun emr-serverless-spark create-workspace \
  --region cn-hangzhou \
  --body '{
    "workspaceName": "my-workspace",
    "ossBucket": "oss://my-spark-bucket",
    "ramRoleName": "AliyunEMRSparkJobRunDefaultRole",
    "paymentType": "PayAsYouGo",
    "resourceSpec": {"cu": 8}
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

ListWorkspaces - Query Workspace List

Method: GET /api/v1/workspaces

Request Parameters (Query):

Example:

aliyun emr-serverless-spark list-workspaces --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

ListWorkspaceQueues - Query Resource Queues

Method: GET /api/v1/workspaces/{workspaceId}/queues

Request Parameters:

Example:

aliyun emr-serverless-spark list-workspace-queues --workspace-id w-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

EditWorkspaceQueue - Modify Resource Queue

Method: POST /api/v1/workspaces/queues/action/edit

Request Parameters (Body):

Example:

aliyun emr-serverless-spark edit-workspace-queue \
  --region cn-hangzhou \
  --body '{"workspaceId":"w-xxx","workspaceQueueName":"dev_queue","resourceSpec":{"cu":32,"maxCu":64}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

Job Management

StartJobRun - Submit Job

Method: POST /api/v1/workspaces/{workspaceId}/jobRuns

Request Parameters:

Example:

aliyun emr-serverless-spark start-job-run --workspace-id w-xxx \
  --region cn-hangzhou \
  --body '{
    "name": "my-job",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "oss://bucket/app.jar",
        "entryPointArguments": ["arg1"],
        "sparkSubmitParameters": "--class com.example.Main --conf spark.executor.instances=2"
      }
    },
    "codeType": "JAR",
    "resourceQueueId": "root_queue",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

GetJobRun - Query Job Details

Method: GET /api/v1/workspaces/{workspaceId}/jobRuns/{jobRunId}

Request Parameters:

Example:

aliyun emr-serverless-spark get-job-run --workspace-id w-xxx --job-run-id jr-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

ListJobRuns - Query Job List

Method: GET /api/v1/workspaces/{workspaceId}/jobRuns

Request Parameters:

Example:

aliyun emr-serverless-spark list-job-runs --workspace-id w-xxx --region cn-hangzhou --maxResults 20 --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

CancelJobRun - Cancel Job

Method: DELETE /api/v1/workspaces/{workspaceId}/jobRuns/{jobRunId}

⚠️ Destructive Operation: Abort running job, completed compute results may be lost.

Request Parameters:

Example:

aliyun emr-serverless-spark cancel-job-run --workspace-id w-xxx --job-run-id jr-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

ListLogContents - Query Job Logs

Method: GET /api/v1/workspaces/{workspaceId}/action/listLogContents

Request Parameters:

Note: fileName can be obtained from the log field in GetJobRun response, format like:

oss://my-bucket/w-xxx/spark/logs/jr-xxx/driver/stdout.log

>

Supported OSS Path Formats:

- oss://bucket/path (standard format, recommended)

- oss://bucket.oss-cn-hangzhou.aliyuncs.com/path (external endpoint)

- oss://bucket.oss-cn-hangzhou-internal.aliyuncs.com/path (internal endpoint)

- oss://bucket.cn-hangzhou.oss-dls.aliyuncs.com/path (DLS endpoint, can use directly when GetJobRun returns this format)

>

Not Supported: CNAME domain format

Example:

aliyun emr-serverless-spark list-log-contents --workspace-id w-xxx \
  --region cn-hangzhou \
  --fileName 'oss://my-bucket/w-xxx/spark/logs/jr-xxx/driver/stdout.log' \
  --offset 0 --length 9999 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

GetCuHours - Query Queue CU Consumption

Method: GET /api/v1/workspaces/{workspaceId}/metric/cuHours/{queue}

Note: This API queries CU consumption by resource queue dimension, not by individual job.

Request Parameters:

Constraint: Query time span cannot exceed 1 month, otherwise server returns Invalid Parameters: Query interval over one month not allowed!.

Example:

aliyun emr-serverless-spark get-cu-hours --workspace-id w-xxx --queue root_queue \
  --region cn-hangzhou \
  --startTime '2024-01-01 00:00:00' --endTime '2024-01-08 00:00:00' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

GetRunConfiguration - Query Job Configuration

Method: GET /api/v1/workspaces/{workspaceId}/runs/{runId}/action/getRunConfiguration

Request Parameters:

Example:

aliyun emr-serverless-spark get-run-configuration --workspace-id w-xxx --run-id jr-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

ListJobExecutors - Query Executor Information

Method: GET /api/v1/workspaces/{workspaceId}/jobRuns/{jobRunId}/executors

Request Parameters:

Example:

aliyun emr-serverless-spark list-job-executors --workspace-id w-xxx --job-run-id jr-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

Session Cluster

CreateSessionCluster - Create Session Cluster

Method: POST /api/v1/workspaces/{workspaceId}/sessionClusters

Request Parameters:

Example:

aliyun emr-serverless-spark create-session-cluster --workspace-id w-xxx \
  --region cn-hangzhou \
  --body '{"name":"my-session","queueName":"default","kind":"SQL"}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

GetSessionCluster - Query Session Cluster Details

Method: GET /api/v1/workspaces/{workspaceId}/sessionClusters/{sessionClusterId}

Request Parameters:

---

ListSessionClusters - Query Session Cluster List

Method: GET /api/v1/workspaces/{workspaceId}/sessionClusters

Request Parameters:

---

StartSessionCluster - Start Session Cluster

Method: POST /api/v1/workspaces/{workspaceId}/sessionClusters/action/startSessionCluster

Request Parameters:

---

StopSessionCluster - Stop Session Cluster

Method: POST /api/v1/workspaces/{workspaceId}/sessionClusters/action/stopSessionCluster

Request Parameters:

---

DeleteSessionCluster - Delete Session Cluster

Method: DELETE /api/v1/workspaces/{workspaceId}/sessionClusters/{sessionClusterId}

⚠️ Destructive Operation: Irreversible, session cluster will be permanently deleted.

Request Parameters:

Example:

aliyun emr-serverless-spark delete-session-cluster --workspace-id w-xxx --session-cluster-id sc-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

SQL Statement

CreateSqlStatement - Submit SQL Query

Method: PUT /api/interactive/v1/workspace/{workspaceId}/statement

Request Parameters:

Example:

aliyun emr-serverless-spark create-sql-statement --workspace-id w-xxx \
  --region cn-hangzhou \
  --body '{"sqlComputeId":"sc-xxx","codeContent":"SHOW TABLES","defaultDatabase":"default"}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

GetSqlStatement - Query SQL Execution Status

Method: GET /api/interactive/v1/workspace/{workspaceId}/statement/{statementId}

Request Parameters:

Status Values: waiting / running / available / error

Example:

aliyun emr-serverless-spark get-sql-statement --workspace-id w-xxx --statement-id st-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

TerminateSqlStatement - Terminate SQL Query

Method: POST /api/interactive/v1/workspace/{workspaceId}/statement/{statementId}/terminate

⚠️ Destructive Operation: Terminate executing SQL query.

Request Parameters:

Example:

aliyun emr-serverless-spark terminate-sql-statement --workspace-id w-xxx --statement-id st-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

ListSqlStatementContents - Query SQL Execution Results

Method: GET /api/v1/workspaces/{workspaceId}/action/listSqlStatementContents

Request Parameters:

Example:

aliyun emr-serverless-spark list-sql-statement-contents --workspace-id w-xxx \
  --region cn-hangzhou \
  --fileName 'oss://bucket/w-xxx/spark/logs/jr-xxx/driver/st-xxx' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

Kyuubi Service

CreateKyuubiService - Create Kyuubi Service

Method: POST /api/v1/kyuubi/{workspaceId}

Request Parameters:

Example:

aliyun emr-serverless-spark create-kyuubi-service --workspace-id w-xxx \
  --region cn-hangzhou \
  --body '{"name":"my-kyuubi","queue":"default","releaseVersion":"esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)"}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

GetKyuubiService - Query Kyuubi Service Details

Method: GET /api/v1/kyuubi/{workspaceId}/{kyuubiServiceId}

Request Parameters:

---

ListKyuubiServices - Query Kyuubi Service List

Method: GET /api/v1/kyuubi/{workspaceId}

Request Parameters:

---

StartKyuubiService - Start Kyuubi Service

Method: POST /api/v1/kyuubi/{workspaceId}/{kyuubiServiceId}/start

Request Parameters:

---

StopKyuubiService - Stop Kyuubi Service

Method: POST /api/v1/kyuubi/{workspaceId}/{kyuubiServiceId}/stop

⚠️ Destructive Operation: All active JDBC connections will be disconnected.

Request Parameters:

---

UpdateKyuubiService - Modify Kyuubi Service

Method: PUT /api/v1/kyuubi/{workspaceId}/{kyuubiServiceId}

Request Parameters:

---

DeleteKyuubiService - Delete Kyuubi Service

Method: DELETE /api/v1/kyuubi/{workspaceId}/{kyuubiServiceId}

⚠️ Destructive Operation: Irreversible, Kyuubi service will be permanently deleted.

Request Parameters:

---

Kyuubi Token

CreateKyuubiToken - Create Token

Method: POST /api/v1/workspaces/{workspaceId}/kyuubiService/{kyuubiServiceId}/token

Request Parameters:

---

GetKyuubiToken - Query Token Details

Method: GET /api/v1/workspaces/{workspaceId}/kyuubiService/{kyuubiServiceId}/token/{tokenId}

Request Parameters:

---

ListKyuubiToken - Query Token List

Method: GET /api/v1/workspaces/{workspaceId}/kyuubiService/{kyuubiServiceId}/token

Request Parameters:

---

UpdateKyuubiToken - Modify Token

Method: PUT /api/v1/workspaces/{workspaceId}/kyuubiService/{kyuubiServiceId}/token/{tokenId}

Request Parameters:

---

DeleteKyuubiToken - Delete Token

Method: DELETE /api/v1/workspaces/{workspaceId}/kyuubiService/{kyuubiServiceId}/token/{tokenId}

⚠️ Destructive Operation: After deletion, connections using this Token will fail authentication.

Request Parameters:

---

Kyuubi Application

ListKyuubiSparkApplications - Query Kyuubi Application List

Method: GET /api/v1/kyuubi/{workspaceId}/{kyuubiServiceId}/applications

Request Parameters:

---

CancelKyuubiSparkApplication - Cancel Kyuubi Application

Method: DELETE /api/v1/kyuubi/{workspaceId}/{kyuubiServiceId}/application/{applicationId}

⚠️ Destructive Operation: Abort running Spark query.

Request Parameters:

---

Permission Management

AddMembers - Add Members

Method: POST /api/v1/auth/members

Request Parameters (Body):

---

ListMembers - Query Member List

Method: GET /api/v1/auth/{workspaceId}/members

Request Parameters:

---

GrantRoleToUsers - Grant Role

Method: POST /api/v1/auth/roles/grant

Request Parameters (Body):

---

Version Management

ListReleaseVersions - Query Engine Versions

Method: GET /api/v1/releaseVersions

Request Parameters (Query):

Example:

aliyun emr-serverless-spark list-release-versions --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

Data Catalog

ListCatalogs - Query Data Catalog List

Method: GET /api/v1/workspaces/{workspaceId}/catalogs

Request Parameters:

Example:

aliyun emr-serverless-spark list-catalogs --workspace-id w-xxx --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

---

Related Documentation

• Getting Started - First-time workspace creation and job submission
• Workspace Lifecycle - Create, query, manage workspaces
• Job Management - Submit, monitor, diagnose Spark jobs
• Kyuubi Service - Interactive SQL gateway management
• Scaling Guide - Resource queue scaling

Getting Started: Create Your First Spark Workspace from Scratch and Submit a Job

This guide helps first-time users complete: Prerequisites check → Create workspace → Submit first job → View results.

Prerequisites

1. CLI Environment

# Verify Alibaba Cloud CLI is installed
aliyun version

# Verify credentials are configured (should display current profile)
aliyun configure list

2. Grant Service Roles (Required for First-time Use)

Before using EMR Serverless Spark, you need to grant the account the following two roles:

For first-time use, you can authorize with one click through the EMR Serverless Spark Console, or manually create in the RAM console.

3. OSS Storage

Spark jobs need OSS storage to store program files and output data. Confirm RegionId with user before execution (e.g., cn-hangzhou, cn-beijing, cn-shanghai, etc.):

# Check for available OSS Buckets
aliyun oss ls --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# If none, create one
aliyun oss mb oss://my-spark-bucket --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

4. Confirm Region Information

Record the following information, will be used when creating workspace and submitting jobs:

• RegionId (e.g., cn-hangzhou)
• OSS Bucket name and path

Step 1: View Available Engine Versions

aliyun emr-serverless-spark list-release-versions --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Note the latest releaseVersion (e.g., esr-4.7.0 (Spark 3.5.2, Scala 2.12, Java Runtime)), will be needed when submitting jobs later.

Step 2: Create Workspace

aliyun emr-serverless-spark create-workspace \
  --region cn-hangzhou \
  --body '{
    "workspaceName": "my-first-spark-workspace",
    "ossBucket": "oss://my-spark-bucket",
    "ramRoleName": "AliyunEMRSparkJobRunDefaultRole",
    "paymentType": "PayAsYouGo",
    "resourceSpec": {"cu": 8}
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Returns workspaceId (e.g., w-xxx), note it for subsequent operations.

Note: Workspace creation is an async operation, initial status is STARTING, need to wait about 1-3 minutes to become RUNNING before you can operate resource queues and submit jobs.

Wait for Workspace Ready

# View workspace status, wait for workspaceStatus to become RUNNING
aliyun emr-serverless-spark list-workspaces --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Workspace Status Description:

Step 3: View Resource Queues

After workspace is ready, there will be default resource queues:

aliyun emr-serverless-spark list-workspace-queues --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Note the queueName (e.g., root_queue, dev_queue), fill in resourceQueueId field when submitting jobs.

Step 4: Submit First Spark Job

Submit Spark SQL Example (Simplest Way to Get Started)

aliyun emr-serverless-spark start-job-run --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-first-sql-job",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "local:///tmp/spark-sql.sh",
        "sparkSubmitParameters": "--conf spark.executor.cores=4 --conf spark.executor.memory=20g --conf spark.driver.cores=4 --conf spark.driver.memory=8g --conf spark.executor.instances=1 --conf spark.emr.sql.content=SELECT 1 as test_value"
      }
    },
    "codeType": "SQL",
    "resourceQueueId": "root_queue",
    "releaseVersion": "<replace with version from step 1>"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Important:

- name is a required field, not passing will report MissingParameter error

- releaseVersion needs to be replaced with actual version from step 1 (e.g., esr-4.7.0 (Spark 3.5.2, Scala 2.12, Java Runtime))

- resourceQueueId fill with queue name from step 3

Returns jobRunId (e.g., jr-xxx), note it for querying status.

Submit PySpark Example

# First upload Python script to OSS
# aliyun oss cp my_script.py oss://my-spark-bucket/scripts/my_script.py

aliyun emr-serverless-spark start-job-run --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-pyspark-job",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "oss://my-spark-bucket/scripts/my_script.py",
        "sparkSubmitParameters": "--conf spark.executor.cores=4 --conf spark.executor.memory=20g --conf spark.driver.cores=4 --conf spark.driver.memory=8g --conf spark.executor.instances=1"
      }
    },
    "codeType": "PYTHON",
    "resourceQueueId": "root_queue",
    "releaseVersion": "<replace with version from step 1>"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Step 5: View Job Status

aliyun emr-serverless-spark get-job-run --workspace-id {workspaceId} --job-run-id {jobRunId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Status Flow: Submitted → Running → Success / Failed / Cancelled

Wait for state to become Success to indicate job completion.

Job Status Description

Step 6: View Job Logs

# View standard output (need to get log file path from GetJobRun response first)
# Note: offset and length parameters are required, not passing will cause server error
aliyun emr-serverless-spark list-log-contents --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --fileName 'oss://my-spark-bucket/w-xxx/spark/logs/jr-xxx/driver/stdout.log' \
  --offset 0 --length 9999 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Note: fileName path is obtained from the log field in GetJobRun response.

- When job is running, log field returns HTTPS URL (Spark UI real-time log link), listLogContents API is not available at this time

- When job ends (Success/Failed/Cancelled), log field returns OSS path, can call listLogContents at this time

>

Supported OSS path formats:

- oss://bucket/path (standard format, recommended)

- oss://bucket.oss-cn-hangzhou.aliyuncs.com/path (external endpoint)

- oss://bucket.oss-cn-hangzhou-internal.aliyuncs.com/path (internal endpoint)

- oss://bucket.cn-hangzhou.oss-dls.aliyuncs.com/path (DLS endpoint, can use directly when GetJobRun returns this format)

>

Not Supported CNAME domain format.

Cleanup: Watch Costs

• Serverless Spark is billed by actual CU hours used, no ongoing costs after job ends
• Resource queues don't incur costs when idle
• Kyuubi service consumes resources continuously while running, recommend stopping when not in use

Common Issues

Next Steps

• Need to submit more job types? → Refer to Job Management
• Need interactive queries? → Refer to Kyuubi Service
• Need to scale? → Refer to Scaling Guide
• API parameter lookup? → Refer to API Parameter Reference

Job Management: Submit, Monitor, Diagnose Spark Jobs

Table of Contents

• 1. Submit Jobs: JAR / PySpark / SQL
• 2. Query and Monitor: Status, List, Logs
• 3. Cancel Jobs
• 4. Session Clusters
• 5. SQL Statements

1. Submit Jobs

Pre-submission Checklist

Before submitting Spark jobs, must confirm: 1. Workspace ID: Target workspaceId 2. Resource Queue: resourceQueueId (required, e.g., root_queue, dev_queue, get via ListWorkspaceQueues, fill the queueName value) 3. Job Name: name (required, not passing will report MissingParameter error) 4. Code Type: codeType (required: JAR / PYTHON / SQL) 5. Engine Version: releaseVersion 6. Main Program Resource: entryPoint (OSS path or local path) 7. Spark Parameters: executor/driver cores, memory, instances

After confirmation, display equivalent spark-submit command, get user explicit confirmation before submission.

Submit JAR Job

aliyun emr-serverless-spark start-job-run --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-jar-job",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "oss://my-bucket/jars/my-app.jar",
        "entryPointArguments": ["arg1", "arg2"],
        "sparkSubmitParameters": "--class com.example.MyApp --conf spark.executor.cores=4 --conf spark.executor.memory=20g --conf spark.driver.cores=4 --conf spark.driver.memory=8g --conf spark.executor.instances=2"
      }
    },
    "codeType": "JAR",
    "resourceQueueId": "root_queue",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Equivalent spark-submit command:

spark-submit \
  --class com.example.MyApp \
  --conf spark.executor.cores=4 \
  --conf spark.executor.memory=20g \
  --conf spark.driver.cores=4 \
  --conf spark.driver.memory=8g \
  --conf spark.executor.instances=2 \
  oss://my-bucket/jars/my-app.jar \
  arg1 arg2

Submit PySpark Job

aliyun emr-serverless-spark start-job-run --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-pyspark-job",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "oss://my-bucket/scripts/my_script.py",
        "entryPointArguments": ["--input", "oss://my-bucket/data/input", "--output", "oss://my-bucket/data/output"],
        "sparkSubmitParameters": "--conf spark.executor.cores=4 --conf spark.executor.memory=20g --conf spark.driver.cores=4 --conf spark.driver.memory=8g --conf spark.executor.instances=4"
      }
    },
    "codeType": "PYTHON",
    "resourceQueueId": "root_queue",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Submit Job with Custom Configuration

aliyun emr-serverless-spark start-job-run --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "daily-etl-job",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "oss://my-bucket/jars/my-etl.jar",
        "sparkSubmitParameters": "--class com.example.ETL --conf spark.executor.cores=8 --conf spark.executor.memory=32g --conf spark.driver.cores=4 --conf spark.driver.memory=16g --conf spark.executor.instances=8"
      }
    },
    "configurationOverrides": {
      "configurations": [
        {
          "configFileName": "common.conf",
          "configItemKey": "hive.metastore.type",
          "configItemValue": "USER_RDS"
        }
      ]
    },
    "codeType": "JAR",
    "resourceQueueId": "root_queue",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Common Spark Parameter Reference

2. Query and Monitor

Query Single Job

aliyun emr-serverless-spark get-job-run --workspace-id {workspaceId} --job-run-id {jobRunId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Job List

# View all jobs
aliyun emr-serverless-spark list-job-runs --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# Paginated query
aliyun emr-serverless-spark list-job-runs --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --maxResults 20 --nextToken xxx --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Job State Machine

View Job Logs

# View job logs (need to get log file path from GetJobRun response first)
# Note: offset and length parameters are required, not passing will cause server error
aliyun emr-serverless-spark list-log-contents --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --fileName 'oss://my-bucket/w-xxx/spark/logs/jr-xxx/driver/stdout.log' \
  --offset 0 --length 9999 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Note: fileName is the OSS full path of the log file, can be obtained from the log field in GetJobRun response.

- When job is running, log field returns HTTPS URL (Spark UI real-time log link), listLogContents API is not available at this time

- When job ends (Success/Failed/Cancelled), log field returns OSS path, can call listLogContents at this time

- ⚠️ Quick-fail jobs (e.g., error during startup) may not have log files, log field returns OSS path but calling listLogContents returns ResourceNotFound. Get error info from GetJobRun's stateChangeReason field at this time

>

Common log files:

- driver/stdout.log - Standard output

- driver/stderr.log - Standard error

- driver/syslog.log - System log (contains Spark startup info)

- driver/startup.log - Startup log

>

OSS Path Compatibility:

- Supported: oss://bucket/path (standard), oss://bucket.oss-cn-hangzhou.aliyuncs.com/path (external), oss://bucket.oss-cn-hangzhou-internal.aliyuncs.com/path (internal), oss://bucket.cn-hangzhou.oss-dls.aliyuncs.com/path (DLS endpoint, can use directly when GetJobRun returns this format)

- Not supported: CNAME domain format

View Executor Information

aliyun emr-serverless-spark list-job-executors --workspace-id {workspaceId} --job-run-id {jobRunId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

View Queue CU Consumption

# Query CU consumption by resource queue dimension (note: query by queue, not by individual job)
aliyun emr-serverless-spark get-cu-hours --workspace-id {workspaceId} --queue {queueName} \
  --region cn-hangzhou \
  --startTime '2024-01-01 00:00:00' --endTime '2024-01-08 00:00:00' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage
# Note: Query time span cannot exceed 1 month

View Job Configuration

aliyun emr-serverless-spark get-run-configuration --workspace-id {workspaceId} --run-id {jobRunId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

3. Cancel Jobs

Pre-cancellation Checklist

1. Confirm job status: Confirm job status is Running via GetJobRun 2. Assess impact: Completed compute results may be lost, confirm if acceptable 3. User explicit confirmation: Inform user of cancellation impact

# First confirm job status
aliyun emr-serverless-spark get-job-run --workspace-id {workspaceId} --job-run-id {jobRunId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# ⚠️ Cancel job (completed compute results may be lost)
aliyun emr-serverless-spark cancel-job-run --workspace-id {workspaceId} --job-run-id {jobRunId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Status change after cancellation: Running → Cancelling → Cancelled

4. Session Clusters

Session clusters provide long-running interactive environments, suitable for development debugging and Notebook usage.

Create Session Cluster

aliyun emr-serverless-spark create-session-cluster --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-session",
    "queueName": "default",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)",
    "kind": "SQL",
    "autoStopConfiguration": {
      "enable": true,
      "idleTimeoutMinutes": 30
    }
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

View Session Cluster List

aliyun emr-serverless-spark list-session-clusters --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Start Session Cluster

aliyun emr-serverless-spark start-session-cluster --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "sessionClusterId": "sc-xxx",
    "queueName": "default"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Stop Session Cluster

aliyun emr-serverless-spark stop-session-cluster --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "sessionClusterId": "sc-xxx",
    "queueName": "default"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

View Session Cluster Details

aliyun emr-serverless-spark get-session-cluster --workspace-id {workspaceId} --session-cluster-id {sessionClusterId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Session Cluster Status Description

Delete Session Cluster

Pre-deletion Checklist

1. Confirm session stopped: Confirm status is stopped via GetSessionCluster 2. User explicit confirmation: Inform user deletion is irreversible

# First confirm session cluster status
aliyun emr-serverless-spark get-session-cluster --workspace-id {workspaceId} --session-cluster-id {sessionClusterId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# ⚠️ Delete session cluster (irreversible)
aliyun emr-serverless-spark delete-session-cluster --workspace-id {workspaceId} --session-cluster-id {sessionClusterId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

5. SQL Statements

Submit and execute SQL statements through session clusters.

Submit SQL Statement

aliyun emr-serverless-spark create-sql-statement --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "sqlComputeId": "sc-xxx",
    "codeContent": "SELECT * FROM my_table LIMIT 10",
    "defaultDatabase": "default"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Query SQL Execution Status

aliyun emr-serverless-spark get-sql-statement --workspace-id {workspaceId} --statement-id {statementId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Status Description:

Terminate SQL Query

aliyun emr-serverless-spark terminate-sql-statement --workspace-id {workspaceId} --statement-id {statementId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Query SQL Execution Results

Recommended Method: Prefer using GetSqlStatement to get results, response's sqlOutputs field directly contains query results (schema + rows).

>

ListSqlStatementContents is a backup method to read results via OSS log file, requires session cluster to be stopped and logs written to OSS before available. fileName needs to be obtained by concatenating statementId from session cluster's associated JobRun log path.

aliyun emr-serverless-spark list-sql-statement-contents --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --fileName 'oss://bucket/w-xxx/spark/logs/jr-xxx/driver/st-xxx' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Common Job Failure Causes

Related Documentation

• Getting Started - First-time workspace creation and job submission
• Workspace Lifecycle - Create, query, manage workspaces
• Kyuubi Service - Interactive SQL gateway management
• Scaling Guide - Resource queue scaling
• API Parameter Reference - Complete parameter documentation

Kyuubi Service: Interactive SQL Gateway Management

Table of Contents

• 1. Overview
• 2. Create Kyuubi Service
• 3. Start/Stop Management
• 4. Connect to Kyuubi and Execute SQL
• 5. Token Management
• 6. Application Management
• 7. Modify and Delete

1. Overview

Kyuubi service is an interactive SQL gateway compatible with open-source Kyuubi provided by EMR Serverless Spark. Supports executing Spark SQL queries through standard JDBC connections (beeline, DBeaver, etc.).

Core Features

Operation Flow

1. Create Kyuubi Service → 2. Start Service → 3. Get Endpoint → 4. Create Token → 5. Use beeline to connect and execute SQL

2. Create Kyuubi Service

Pre-creation Confirmation

Before submission, need to confirm: 1. Workspace ID 2. Resource queue name 3. Engine version 4. Whether public network access is needed

Create Basic Kyuubi Service

aliyun emr-serverless-spark create-kyuubi-service --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-kyuubi",
    "queue": "default",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Create Kyuubi Service with Public Network Access

aliyun emr-serverless-spark create-kyuubi-service --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-kyuubi-public",
    "queue": "default",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)",
    "publicEndpointEnabled": true,
    "replica": 2
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Create Kyuubi Service with Custom Configuration

aliyun emr-serverless-spark create-kyuubi-service --workspace-id {workspaceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-kyuubi-custom",
    "queue": "default",
    "releaseVersion": "esr-2.1 (Spark 3.3.1, Scala 2.12, Java Runtime)",
    "publicEndpointEnabled": true,
    "kyuubiConfigs": "kyuubi.session.idle.timeout=PT1H",
    "sparkConfigs": "spark.executor.memory=20g;spark.executor.cores=4"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

3. Start/Stop Management

Start Kyuubi Service

aliyun emr-serverless-spark start-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Stop Kyuubi Service

Pre-stop Confirmation

1. Confirm active connection impact: All active JDBC connections will be disconnected, executing queries will be aborted 2. User explicit confirmation: Inform user of stop operation impact

# ⚠️ Stop Kyuubi Service (all active JDBC connections will be disconnected)
aliyun emr-serverless-spark stop-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

View Service Status

aliyun emr-serverless-spark get-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Key information in the response:

• state: Service status
• innerEndpoint: Internal network connection address
• publicEndpoint: Public network connection address (if enabled)
• kyuubiServiceId: Service ID

Kyuubi Service Status Description

List All Kyuubi Services

aliyun emr-serverless-spark list-kyuubi-services --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

4. Connect to Kyuubi and Execute SQL

Get Connection Information

First query service details to get Endpoint:

aliyun emr-serverless-spark get-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Connect Using beeline

# Internal network connection
beeline -u "jdbc:hive2://{innerEndpoint}:10009" -n token -p {your-token}

# Public network connection
beeline -u "jdbc:hive2://{publicEndpoint}:10009" -n token -p {your-token}

Execute SQL Example

# Execute query after connecting
beeline -u "jdbc:hive2://{endpoint}:10009" -n token -p {your-token} \
  -e "SELECT * FROM my_database.my_table LIMIT 10"

# Execute SQL file
beeline -u "jdbc:hive2://{endpoint}:10009" -n token -p {your-token} \
  -f /path/to/my_query.sql

5. Token Management

Kyuubi service uses Token for identity authentication.

Create Token

Note:

- token is a required field, length must be >= 32 characters

- Token value must be globally unique, cannot duplicate other users' Tokens, recommend using randomly generated values

- memberArns is an optional field

# First generate a random token (32-character hexadecimal)
# TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(16))")

aliyun emr-serverless-spark create-kyuubi-token --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-token",
    "token": "<replace with random string of 32+ characters>"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Query Token Details

aliyun emr-serverless-spark get-kyuubi-token --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --token-id {tokenId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

List Tokens

aliyun emr-serverless-spark list-kyuubi-token --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Modify Token

aliyun emr-serverless-spark update-kyuubi-token --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --token-id {tokenId} \
  --region cn-hangzhou \
  --body '{
    "name": "new-token-name"
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Modifiable fields: name (name), token (Token content, >=32 characters), autoExpireConfiguration (auto expire configuration), memberArns (authorized users).

Delete Token

Pre-deletion Confirmation

1. Confirm Token ID: Confirm Token to delete via GetKyuubiToken 2. User explicit confirmation: Inform user that connections using this Token will fail authentication after deletion

# First confirm Token information
aliyun emr-serverless-spark get-kyuubi-token --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --token-id {tokenId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# ⚠️ Delete Token (connections using this Token will fail authentication)
aliyun emr-serverless-spark delete-kyuubi-token --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --token-id {tokenId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

6. Application Management

View and manage Spark applications submitted through Kyuubi.

List Kyuubi Applications

aliyun emr-serverless-spark list-kyuubi-spark-applications --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Cancel Kyuubi Application

Pre-cancellation Confirmation

1. Confirm application ID and status: Confirm the Spark application to cancel is running 2. User explicit confirmation: Inform user that running Spark query will be aborted

# ⚠️ Cancel Kyuubi Application (running Spark query will be aborted)
aliyun emr-serverless-spark cancel-kyuubi-spark-application --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --application-id {applicationId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

7. Modify and Delete

Modify Kyuubi Service Configuration

aliyun emr-serverless-spark update-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} \
  --region cn-hangzhou \
  --body '{
    "name": "my-kyuubi",
    "queue": "root_queue",
    "sparkConfigs": "spark.executor.memory=32g;spark.executor.cores=8",
    "publicEndpointEnabled": true,
    "restart": true
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Delete Kyuubi Service

Pre-deletion Checklist

1. Confirm service stopped: Confirm status is NOT_STARTED via GetKyuubiService 2. Confirm no active connections: Confirm all JDBC connections are disconnected 3. User explicit confirmation: Inform user deletion is irreversible

# First confirm service status
aliyun emr-serverless-spark get-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# ⚠️ Stop Kyuubi Service (all active JDBC connections will be disconnected)
aliyun emr-serverless-spark stop-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# ⚠️ Delete Kyuubi Service (irreversible! Kyuubi service will be permanently deleted)
aliyun emr-serverless-spark delete-kyuubi-service --workspace-id {workspaceId} --kyuubi-service-id {kyuubiServiceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Common Issues

Related Documentation

• Getting Started - First-time workspace creation and job submission
• Job Management - Submit, monitor, diagnose Spark jobs
• Scaling Guide - Resource queue scaling
• API Parameter Reference - Complete parameter documentation

RAM Permission Policies

This document details the RAM permission policies required for EMR Serverless Spark, including system policies, custom policies, and service roles.

required_permissions

The permissions required for this Skill are declared as follows:

required_permissions:
  - policy: AliyunEMRServerlessSparkFullAccess
    description: Administrator permissions, includes all operations such as create workspaces, job management, Kyuubi service management, etc. (Note: DeleteWorkspace is excluded from this skill for risk control)
    actions:
      # Workspace
      - emr-serverless-spark:CreateWorkspace
      - emr-serverless-spark:ListWorkspaces
      - emr-serverless-spark:ListWorkspaceQueues
      - emr-serverless-spark:EditWorkspaceQueue
      # Job
      - emr-serverless-spark:StartJobRun
      - emr-serverless-spark:GetJobRun
      - emr-serverless-spark:ListJobRuns
      - emr-serverless-spark:CancelJobRun
      - emr-serverless-spark:ListLogContents
      - emr-serverless-spark:GetCuHours
      - emr-serverless-spark:GetRunConfiguration
      - emr-serverless-spark:ListJobExecutors
      # Session Cluster
      - emr-serverless-spark:CreateSessionCluster
      - emr-serverless-spark:GetSessionCluster
      - emr-serverless-spark:ListSessionClusters
      - emr-serverless-spark:StartSessionCluster
      - emr-serverless-spark:StopSessionCluster
      - emr-serverless-spark:DeleteSessionCluster
      # SQL
      - emr-serverless-spark:CreateSqlStatement
      - emr-serverless-spark:GetSqlStatement
      - emr-serverless-spark:TerminateSqlStatement
      - emr-serverless-spark:ListSqlStatementContents
      # Kyuubi Service
      - emr-serverless-spark:CreateKyuubiService
      - emr-serverless-spark:GetKyuubiService
      - emr-serverless-spark:ListKyuubiServices
      - emr-serverless-spark:StartKyuubiService
      - emr-serverless-spark:StopKyuubiService
      - emr-serverless-spark:UpdateKyuubiService
      - emr-serverless-spark:DeleteKyuubiService
      # Kyuubi Token
      - emr-serverless-spark:CreateKyuubiToken
      - emr-serverless-spark:GetKyuubiToken
      - emr-serverless-spark:ListKyuubiToken
      - emr-serverless-spark:UpdateKyuubiToken
      - emr-serverless-spark:DeleteKyuubiToken
      # Kyuubi Application
      - emr-serverless-spark:ListKyuubiSparkApplications
      - emr-serverless-spark:CancelKyuubiSparkApplication
      # Auth
      - emr-serverless-spark:AddMembers
      - emr-serverless-spark:ListMembers
      - emr-serverless-spark:GrantRoleToUsers
      # Version & Catalog
      - emr-serverless-spark:ListReleaseVersions
      - emr-serverless-spark:ListCatalogs
      # Supplementary
      - oss:ListBuckets
      - dlf:DescribeRegions
      - dlf:GetRegionStatus
      - dlf:ListCatalogs
      - dlf:ListDatabases
      - dlf:ListTables
      - emr:GetApmData
      - emr:QueryApmGrafanaData
  - policy: AliyunEMRServerlessSparkDeveloperAccess
    description: Developer permissions, includes submit jobs, manage sessions, Kyuubi operations, etc., excludes create workspaces
    actions:
      # Workspace (read-only)
      - emr-serverless-spark:ListWorkspaces
      - emr-serverless-spark:ListWorkspaceQueues
      - emr-serverless-spark:EditWorkspaceQueue
      # Job
      - emr-serverless-spark:StartJobRun
      - emr-serverless-spark:GetJobRun
      - emr-serverless-spark:ListJobRuns
      - emr-serverless-spark:CancelJobRun
      - emr-serverless-spark:ListLogContents
      - emr-serverless-spark:GetCuHours
      - emr-serverless-spark:GetRunConfiguration
      - emr-serverless-spark:ListJobExecutors
      # Session Cluster
      - emr-serverless-spark:CreateSessionCluster
      - emr-serverless-spark:GetSessionCluster
      - emr-serverless-spark:ListSessionClusters
      - emr-serverless-spark:StartSessionCluster
      - emr-serverless-spark:StopSessionCluster
      - emr-serverless-spark:DeleteSessionCluster
      # SQL
      - emr-serverless-spark:CreateSqlStatement
      - emr-serverless-spark:GetSqlStatement
      - emr-serverless-spark:TerminateSqlStatement
      - emr-serverless-spark:ListSqlStatementContents
      # Kyuubi Service
      - emr-serverless-spark:CreateKyuubiService
      - emr-serverless-spark:GetKyuubiService
      - emr-serverless-spark:ListKyuubiServices
      - emr-serverless-spark:StartKyuubiService
      - emr-serverless-spark:StopKyuubiService
      - emr-serverless-spark:UpdateKyuubiService
      - emr-serverless-spark:DeleteKyuubiService
      # Kyuubi Token
      - emr-serverless-spark:CreateKyuubiToken
      - emr-serverless-spark:GetKyuubiToken
      - emr-serverless-spark:ListKyuubiToken
      - emr-serverless-spark:UpdateKyuubiToken
      - emr-serverless-spark:DeleteKyuubiToken
      # Kyuubi Application
      - emr-serverless-spark:ListKyuubiSparkApplications
      - emr-serverless-spark:CancelKyuubiSparkApplication
      # Version & Catalog
      - emr-serverless-spark:ListReleaseVersions
      - emr-serverless-spark:ListCatalogs
      # Supplementary
      - oss:ListBuckets
      - dlf:DescribeRegions
      - dlf:GetRegionStatus
      - dlf:ListCatalogs
      - dlf:ListDatabases
      - dlf:ListTables
  - policy: AliyunEmrServerlessSparkReadOnlyAccess
    description: Read-only permissions, includes Get*, List*, Query*, Is*, Check* operations
    actions:
      # Workspace
      - emr-serverless-spark:ListWorkspaces
      - emr-serverless-spark:ListWorkspaceQueues
      # Job
      - emr-serverless-spark:GetJobRun
      - emr-serverless-spark:ListJobRuns
      - emr-serverless-spark:ListLogContents
      - emr-serverless-spark:GetCuHours
      - emr-serverless-spark:GetRunConfiguration
      - emr-serverless-spark:ListJobExecutors
      # Session Cluster
      - emr-serverless-spark:GetSessionCluster
      - emr-serverless-spark:ListSessionClusters
      # SQL
      - emr-serverless-spark:GetSqlStatement
      - emr-serverless-spark:ListSqlStatementContents
      # Kyuubi Service
      - emr-serverless-spark:GetKyuubiService
      - emr-serverless-spark:ListKyuubiServices
      # Kyuubi Token
      - emr-serverless-spark:GetKyuubiToken
      - emr-serverless-spark:ListKyuubiToken
      # Kyuubi Application
      - emr-serverless-spark:ListKyuubiSparkApplications
      # Auth
      - emr-serverless-spark:ListMembers
      # Version & Catalog
      - emr-serverless-spark:ListReleaseVersions
      - emr-serverless-spark:ListCatalogs

System Policies

EMR Serverless Spark provides three system policies, listed in order of permission scope from large to small:

AliyunEMRServerlessSparkFullAccess

Applicable Role: Administrator

Permission Scope:

Workspace Management:

• emr-serverless-spark:CreateWorkspace - Create workspace
• emr-serverless-spark:ListWorkspaces - List workspaces
• emr-serverless-spark:ListWorkspaceQueues - List resource queues
• emr-serverless-spark:EditWorkspaceQueue - Modify resource queue

Job Management:

• emr-serverless-spark:StartJobRun - Submit job
• emr-serverless-spark:GetJobRun - Query job details
• emr-serverless-spark:ListJobRuns - List jobs
• emr-serverless-spark:CancelJobRun - Cancel job
• emr-serverless-spark:ListLogContents - Query logs
• emr-serverless-spark:GetCuHours - Query CU consumption
• emr-serverless-spark:GetRunConfiguration - Query job configuration
• emr-serverless-spark:ListJobExecutors - Query Executor information

Session Cluster:

• emr-serverless-spark:CreateSessionCluster - Create session cluster
• emr-serverless-spark:GetSessionCluster - Query session cluster
• emr-serverless-spark:ListSessionClusters - List session clusters
• emr-serverless-spark:StartSessionCluster - Start session cluster
• emr-serverless-spark:StopSessionCluster - Stop session cluster
• emr-serverless-spark:DeleteSessionCluster - Delete session cluster

SQL Query:

• emr-serverless-spark:CreateSqlStatement - Submit SQL
• emr-serverless-spark:GetSqlStatement - Query SQL status
• emr-serverless-spark:TerminateSqlStatement - Terminate SQL
• emr-serverless-spark:ListSqlStatementContents - Query SQL results

Kyuubi Service:

• emr-serverless-spark:CreateKyuubiService - Create Kyuubi service
• emr-serverless-spark:GetKyuubiService - Query Kyuubi service
• emr-serverless-spark:ListKyuubiServices - List Kyuubi services
• emr-serverless-spark:StartKyuubiService - Start Kyuubi service
• emr-serverless-spark:StopKyuubiService - Stop Kyuubi service
• emr-serverless-spark:UpdateKyuubiService - Update Kyuubi service
• emr-serverless-spark:DeleteKyuubiService - Delete Kyuubi service

Kyuubi Token:

• emr-serverless-spark:CreateKyuubiToken - Create Token
• emr-serverless-spark:GetKyuubiToken - Query Token
• emr-serverless-spark:ListKyuubiToken - List Tokens
• emr-serverless-spark:UpdateKyuubiToken - Update Token
• emr-serverless-spark:DeleteKyuubiToken - Delete Token

Kyuubi Application:

• emr-serverless-spark:ListKyuubiSparkApplications - List applications
• emr-serverless-spark:CancelKyuubiSparkApplication - Cancel application

Permission Management:

• emr-serverless-spark:AddMembers - Add members
• emr-serverless-spark:ListMembers - List members
• emr-serverless-spark:GrantRoleToUsers - Grant role

Version & Catalog:

• emr-serverless-spark:ListReleaseVersions - List engine versions
• emr-serverless-spark:ListCatalogs - List data catalogs

Supplementary Permissions:

• oss:ListBuckets - List OSS Buckets
• dlf:DescribeRegions - Describe DLF regions
• dlf:GetRegionStatus - Get DLF region status
• dlf:ListCatalogs - List DLF data catalogs
• dlf:ListDatabases - List DLF databases
• dlf:ListTables - List DLF data tables
• emr:GetApmData - Get APM data
• emr:QueryApmGrafanaData - Query Grafana data

Authorization Command:

aliyun ram attach-policy-to-user \
  --policy-name AliyunEMRServerlessSparkFullAccess \
  --policy-type System \
  --user-name <username> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

AliyunEMRServerlessSparkDeveloperAccess

Applicable Role: Developer

Permission Scope:

Workspace (Read-only):

• emr-serverless-spark:ListWorkspaces - List workspaces
• emr-serverless-spark:ListWorkspaceQueues - List resource queues
• emr-serverless-spark:EditWorkspaceQueue - Modify resource queue

Job Management:

• emr-serverless-spark:StartJobRun - Submit job
• emr-serverless-spark:GetJobRun - Query job details
• emr-serverless-spark:ListJobRuns - List jobs
• emr-serverless-spark:CancelJobRun - Cancel job
• emr-serverless-spark:ListLogContents - Query logs
• emr-serverless-spark:GetCuHours - Query CU consumption
• emr-serverless-spark:GetRunConfiguration - Query job configuration
• emr-serverless-spark:ListJobExecutors - Query Executor information

Session Cluster:

• emr-serverless-spark:CreateSessionCluster - Create session cluster
• emr-serverless-spark:GetSessionCluster - Query session cluster
• emr-serverless-spark:ListSessionClusters - List session clusters
• emr-serverless-spark:StartSessionCluster - Start session cluster
• emr-serverless-spark:StopSessionCluster - Stop session cluster
• emr-serverless-spark:DeleteSessionCluster - Delete session cluster

SQL Query:

• emr-serverless-spark:CreateSqlStatement - Submit SQL
• emr-serverless-spark:GetSqlStatement - Query SQL status
• emr-serverless-spark:TerminateSqlStatement - Terminate SQL
• emr-serverless-spark:ListSqlStatementContents - Query SQL results

Kyuubi Service:

• emr-serverless-spark:CreateKyuubiService - Create Kyuubi service
• emr-serverless-spark:GetKyuubiService - Query Kyuubi service
• emr-serverless-spark:ListKyuubiServices - List Kyuubi services
• emr-serverless-spark:StartKyuubiService - Start Kyuubi service
• emr-serverless-spark:StopKyuubiService - Stop Kyuubi service
• emr-serverless-spark:UpdateKyuubiService - Update Kyuubi service
• emr-serverless-spark:DeleteKyuubiService - Delete Kyuubi service

Kyuubi Token:

• emr-serverless-spark:CreateKyuubiToken - Create Token
• emr-serverless-spark:GetKyuubiToken - Query Token
• emr-serverless-spark:ListKyuubiToken - List Tokens
• emr-serverless-spark:UpdateKyuubiToken - Update Token
• emr-serverless-spark:DeleteKyuubiToken - Delete Token

Kyuubi Application:

• emr-serverless-spark:ListKyuubiSparkApplications - List applications
• emr-serverless-spark:CancelKyuubiSparkApplication - Cancel application

Version & Catalog:

• emr-serverless-spark:ListReleaseVersions - List engine versions
• emr-serverless-spark:ListCatalogs - List data catalogs

Supplementary Permissions:

• oss:ListBuckets - List OSS Buckets
• dlf:DescribeRegions - Describe DLF regions
• dlf:GetRegionStatus - Get DLF region status
• dlf:ListCatalogs - List DLF data catalogs
• dlf:ListDatabases - List DLF databases
• dlf:ListTables - List DLF data tables

Note: Does not include CreateWorkspace permissions

Authorization Command:

aliyun ram attach-policy-to-user \
  --policy-name AliyunEMRServerlessSparkDeveloperAccess \
  --policy-type System \
  --user-name <username> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

AliyunEmrServerlessSparkReadOnlyAccess

Applicable Role: Audit, read-only viewing

Permission Scope:

Workspace:

• emr-serverless-spark:ListWorkspaces - List workspaces
• emr-serverless-spark:ListWorkspaceQueues - List resource queues

Job Management:

• emr-serverless-spark:GetJobRun - Query job details
• emr-serverless-spark:ListJobRuns - List jobs
• emr-serverless-spark:ListLogContents - Query logs
• emr-serverless-spark:GetCuHours - Query CU consumption
• emr-serverless-spark:GetRunConfiguration - Query job configuration
• emr-serverless-spark:ListJobExecutors - Query Executor information

Session Cluster:

• emr-serverless-spark:GetSessionCluster - Query session cluster
• emr-serverless-spark:ListSessionClusters - List session clusters

SQL Query:

• emr-serverless-spark:GetSqlStatement - Query SQL status
• emr-serverless-spark:ListSqlStatementContents - Query SQL results

Kyuubi Service:

• emr-serverless-spark:GetKyuubiService - Query Kyuubi service
• emr-serverless-spark:ListKyuubiServices - List Kyuubi services

Kyuubi Token:

• emr-serverless-spark:GetKyuubiToken - Query Token
• emr-serverless-spark:ListKyuubiToken - List Tokens

Kyuubi Application:

• emr-serverless-spark:ListKyuubiSparkApplications - List applications

Permission Management:

• emr-serverless-spark:ListMembers - List members

Version & Catalog:

• emr-serverless-spark:ListReleaseVersions - List engine versions
• emr-serverless-spark:ListCatalogs - List data catalogs

Authorization Command:

aliyun ram attach-policy-to-user \
  --policy-name AliyunEmrServerlessSparkReadOnlyAccess \
  --policy-type System \
  --user-name <username> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Custom Policies

If fine-grained permission control is needed, you can create custom policies.

Action Format

All EMR Serverless Spark API Actions have the format:

emr-serverless-spark:<ActionName>

Examples:

• emr-serverless-spark:StartJobRun - Submit job
• emr-serverless-spark:GetJobRun - Query job
• emr-serverless-spark:ListWorkspaces - List workspaces

Custom Policy Example

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "emr-serverless-spark:ListWorkspaces",
        "emr-serverless-spark:GetWorkspace",
        "emr-serverless-spark:ListJobRuns",
        "emr-serverless-spark:GetJobRun",
        "emr-serverless-spark:StartJobRun"
      ],
      "Resource": "*"
    }
  ]
}

Supplementary Permissions

EMR Serverless Spark jobs may need to access other cloud services, below are commonly used supplementary permissions:

Service Roles

AliyunServiceRoleForEMRServerlessSpark

Type: Service-linked role

Purpose: EMR Serverless Spark service uses this role to access your resources in other cloud products.

Auto Creation: When using EMR Serverless Spark for the first time, the system will prompt you to create this role.

Manual Creation:

aliyun resourcemanager create-service-linked-role \
  --service-name spark.emr-serverless.aliyuncs.com \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Trust Policy:

{
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "spark.emr-serverless.aliyuncs.com"
        ]
      }
    }
  ],
  "Version": "1"
}

AliyunEMRSparkJobRunDefaultRole

Type: Job execution role

Purpose: Spark jobs use this role to access OSS, DLF and other cloud resources during execution.

Creation Methods: 1. One-click authorization through EMR Serverless Spark console 2. Manual creation in RAM console

Required Permissions:

• OSS read/write permissions (to access job code and output data)
• DLF metadata access permissions (if using DLF data catalog)

Permission Checklist

Before using EMR Serverless Spark for the first time, please confirm:

• [ ] RAM user has been granted corresponding system policy or custom policy
• [ ] Service-linked role AliyunServiceRoleForEMRServerlessSpark has been created
• [ ] Job execution role AliyunEMRSparkJobRunDefaultRole has been created
• [ ] OSS Bucket has been created and is accessible
• [ ] If using DLF, corresponding metadata permissions have been configured

Common Permission Issues

Forbidden.RAM

Error Message: You are not authorized to perform this operation

Solution: 1. Check if RAM user has been granted corresponding policy 2. Check if service-linked role has been created 3. Confirm custom policy's Action and Resource configuration is correct

Service-linked Role Creation Failed

Error Message: You are not authorized to create service linked role

Solution: Need RAM administrator permissions or ram:CreateServiceLinkedRole permission to create service-linked role. Please contact account administrator for assistance.

Job Execution Insufficient Permissions

Error Message: OSS or DLF access permission error during job execution

Solution: 1. Confirm AliyunEMRSparkJobRunDefaultRole has been created 2. Confirm the role has been granted necessary OSS and DLF permissions 3. Confirm the role name configured in workspace is correct

Scaling: Resource Queue Management

Decision Guidance

When to Scale Up?

When to Scale Down?

What Resources to Scale?

1. View Resource Queues

View Queue List

aliyun emr-serverless-spark list-workspace-queues --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Key information in the response:

• queueName: Queue name
• queueStatus: Queue status
• maxResource: Resource limit (CU)
• usedResource: Used resources (CU)

2. Modify Resource Queues

Scale Up Resource Queue

Before scaling up, confirm: 1. Workspace ID 2. Queue name 3. Target CU quantity

Important Constraint: The sum of CU across all queues in a workspace cannot exceed the workspace total CU limit. For example, if workspace has 8 CU and root_queue has 6 CU allocated, dev_queue can only have max 2 CU. To scale beyond the limit, first scale down other queues to free up space.

After confirming the operation, need user explicit confirmation before execution.

aliyun emr-serverless-spark edit-workspace-queue \
  --region cn-hangzhou \
  --body '{"workspaceId":"w-xxx","workspaceQueueName":"dev_queue","resourceSpec":{"cu":64,"maxCu":128}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Scale Down Resource Queue

Note: Scaling down may increase wait time for queued jobs. If running jobs use resources exceeding the scaled-down quota, scaling down won't affect running jobs, but new jobs may need to wait.

aliyun emr-serverless-spark edit-workspace-queue \
  --region cn-hangzhou \
  --body '{"workspaceId":"w-xxx","workspaceQueueName":"dev_queue","resourceSpec":{"cu":16}}' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Verify After Scaling

# View queue status to confirm change生效
aliyun emr-serverless-spark list-workspace-queues --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Common Issues

Scale-up Failed

Scale-down Considerations

• Scaling down won't abort running jobs
• After scaling down, newly submitted large-resource jobs may queue and wait
• Recommend executing scale-down operations during business low-peak periods

Continuous Operation Considerations

• When modifying multiple queues continuously, need to wait for previous operation to complete before executing next, otherwise may report Error.Internal: fail to update app instance queue
• Recommend waiting 5-10 seconds after each queue change before executing next queue operation

Related Documentation

• Workspace Lifecycle - Create, query, manage workspaces
• Job Management - Submit, monitor, diagnose Spark jobs
• Kyuubi Service - Interactive SQL gateway management
• API Parameter Reference - Complete parameter documentation

Workspace Lifecycle: Create → Query → Manage

Table of Contents

• 1. Create Workspace
• 2. Query Workspace
• 4. Member Management
• 5. Engine Versions

1. Create Workspace

Prerequisite: Grant Service Roles

Before creating a workspace, ensure the account has granted the following two roles:

• AliyunServiceRoleForEMRServerlessSpark: Service-linked role, EMR Serverless Spark service uses this role to access other cloud resources
• AliyunEMRSparkJobRunDefaultRole: Job execution role, Spark jobs use this role to access OSS, DLF and other resources during execution

For first-time use, you can authorize with one click through the EMR Serverless Spark Console.

Create Basic Workspace

aliyun emr-serverless-spark create-workspace \
  --region cn-hangzhou \
  --body '{
    "workspaceName": "my-spark-workspace",
    "ossBucket": "oss://my-spark-bucket",
    "ramRoleName": "AliyunEMRSparkJobRunDefaultRole",
    "paymentType": "PayAsYouGo",
    "resourceSpec": {"cu": 8}
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Verify After Creation

Workspace creation is an asynchronous operation, initial status is STARTING, need to wait about 1-3 minutes to become RUNNING before you can operate resource queues and submit jobs.

# View workspace list to confirm creation success, wait for workspaceStatus to become RUNNING
aliyun emr-serverless-spark list-workspaces --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Workspace Status Description

2. Query Workspace

Workspace List

# View all workspaces
aliyun emr-serverless-spark list-workspaces --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# Paginated query
aliyun emr-serverless-spark list-workspaces --region cn-hangzhou --maxResults 10 --nextToken xxx --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Workspace Details

Key information in the response:

• workspaceId: Workspace ID
• name: Workspace name
• creator: Creator
• gmtCreated: Creation time

3. Delete Workspace

STRICTLY PROHIBITED. The DeleteWorkspace API must NEVER be called through this skill. Do NOT construct or execute any DELETE request to /api/v1/workspaces/{workspaceId}. If the user asks to delete a workspace, refuse the request and inform them: "Workspace deletion is not supported via this skill. Please delete workspaces through the EMR Serverless Spark Console."

4. Member Management

Add Members

aliyun emr-serverless-spark add-members \
  --region cn-hangzhou \
  --body '{
    "workspaceId": "w-xxx",
    "memberArns": ["acs:ram::123456789:user/username"]
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

View Member List

aliyun emr-serverless-spark list-members --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Grant Roles

ARN Format Explanation:

- roleArn format is acs:emr::{workspaceId}:role/{roleName}, e.g. acs:emr::w-xxx:role/Owner

- userArns format is acs:emr::{workspaceId}:member/{userId}, can get from memberArn field in ListMembers response

# First view member list to get userArn and available roles
aliyun emr-serverless-spark list-members --workspace-id {workspaceId} --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

# Grant role
aliyun emr-serverless-spark grant-role-to-users \
  --region cn-hangzhou \
  --body '{
    "roleArn": "acs:emr::w-xxx:role/Owner",
    "userArns": ["acs:emr::w-xxx:member/123456789"]
  }' \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

5. Engine Versions

View Available Versions

aliyun emr-serverless-spark list-release-versions --region cn-hangzhou --user-agent AlibabaCloud-Agent-Skills/alibabacloud-emr-spark-manage

Returns all available Spark engine versions, need to specify version number when creating jobs and sessions.

Related Documentation

• Getting Started - Simplified workflow for first-time workspace creation
• Job Management - Submit, monitor, diagnose Spark jobs
• Kyuubi Service - Interactive SQL gateway management
• Scaling Guide - Resource queue scaling
• API Parameter Reference - Complete parameter documentation