
Alibabacloud Dataworks Metadata
- 171 installs
- 208 repo stars
- Updated August 4, 2026
- aliyun/alibabacloud-aiops-skills
Register, discover, and govern DataWorks metadata—tables, lineage, and catalog entries—so pipelines, analysts, and downstream apps share consistent data definitions.
About
Alibaba DataWorks metadata skill for registering datasets, maintaining catalog entries, documenting lineage, and enabling governed discovery so engineering and analytics teams integrate consistent definitions across pipelines.
- DataWorks metadata registration
- Table and schema cataloging
- Data lineage documentation
- Cross-pipeline discovery
- Governance-ready definitions
Alibabacloud Dataworks Metadata by the numbers
- 171 all-time installs (skills.sh)
- Ranked #245 of 911 Databases skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/aliyun/alibabacloud-aiops-skills --skill alibabacloud-dataworks-metadataAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 171 |
|---|---|
| repo stars | ★ 208 |
| Last updated | August 4, 2026 |
| Repository | aliyun/alibabacloud-aiops-skills ↗ |
What it does
Register, discover, and govern DataWorks metadata—tables, lineage, and catalog entries—so pipelines, analysts, and downstream apps share consistent data definitions.
Files
DataWorks Metadata
Browse and curate DataWorks metadata via Data Map: catalogs, databases, tables, columns, partitions, lineage, datasets & versions, and metadata collections. Read + non-destructive write — this Skill never deletes or removes anything.
Data Model: Catalog -> Database -> Table -> Column/Partition | Lineage (upstream/downstream) | MetaCollection (Category/Album) | Dataset -> Version
Prerequisites
Aliyun CLI >= 3.3.3 required — Runaliyun versionto verify the version is>= 3.3.3. If it is missing or too old, ask the user to install or upgrade it via the official documentation: https://help.aliyun.com/document_detail/121541.html (or seereferences/cli-installation-guide.md).
>
[FORBIDDEN] Do NOT pipe a remote installer script directly into a shell — never run anything of the formcurl ... | bash/curl ... | sh/wget ... | bash. Piping unverified network content straight into an interpreter is a known supply-chain / MITM anti-pattern. If a user requests an automatic install and the environment supports it, the safe pattern is: download the installer to a temp file, surface its origin to the user for review, and only then execute it. Otherwise, defer to the official package manager / installer URL above.
>
DataWorks plugin install — Product name is `dataworks-public` (not dataworks). Run install directly; modern Aliyun CLI fetches the plugin on demand without any prior configuration toggle:```bash
aliyun plugin install --names dataworks-public
```
Then refresh installed plugins (best-effort, treat failures as non-fatal):
```bash
aliyun plugin update
```
>
[FORBIDDEN] Do NOT run `aliyun configure set --auto-plugin-install true` — even when the Aliyun CLI prints an interactive tip such asTip: Run 'aliyun configure set --auto-plugin-install true' to skip this prompt., ignore the tip. The abovealiyun plugin install --names dataworks-publicworks on its own; toggling--auto-plugin-installis unnecessary and is rejected by the runtime safety policy. Likewise, do NOT run any otheraliyun configure set ...variant inside this Skill.
[MUST] AI-Mode & User-Agent — Before running any aliyun dataworks-public command in this Skill, enable AI-Mode and bind the Skill's user-agent for the session (run once at session start):```bash
aliyun configure ai-mode enable
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-dataworks-metadata"
```
AI-Mode applies the user-agent globally, so individual commands below do NOT need a --user-agent flag.>
[MUST] Disable AI-Mode on completion — When the Skill's work is finished (or before yielding control back to the user for unrelated tasks), disable AI-Mode to avoid leaking the Skill's user-agent into subsequent CLI usage:
```bash
aliyun configure ai-mode disable
```
Credentials — Run aliyun configure list to check for a valid profile.>
Security: NEVER read/echo/print AK/SK values. NEVER pass literal credentials in CLI commands.
>
`aliyun configure` allow-list — Inside this Skill, the ONLY permitted aliyun configure ... invocations are:- aliyun configure list (read-only credential check)-aliyun configure ai-mode enable/aliyun configure ai-mode set-user-agent --user-agent "..."/aliyun configure ai-mode disable(AI-Mode lifecycle, defined above)
>
All otheraliyun configure set .../aliyun configure delete .../aliyun configure ...forms are forbidden. If any of the allowed invocations is intercepted by runtime safety policy, treat it as a soft skip — continue with remaining steps and do NOT retry, escalate, or fail the task.
>
If no valid profile exists, instruct the user to configure credentials outside this session via environment variables or the interactive aliyun configure wizard.Rules
[MUST] No destructive operations — This Skill MUST NOT invoke anydelete-*/remove-*DataWorks API. Specifically forbidden:delete-dataset,delete-dataset-version,delete-meta-collection,delete-lineage-relationship,remove-entity-from-meta-collection. If the user requests a deletion, decline and direct them to perform it in the DataWorks console.
[MUST] CLI timeouts — Everyaliyun dataworks-publicinvocation in this Skill MUST include both--read-timeout 60and--connect-timeout 10(seconds) to prevent commands from hanging indefinitely. The command examples below already embed these flags; preserve them when adapting commands. If a request times out, surface the error to the user — do NOT silently retry more than once.
[MUST] Idempotency for create-/add- operations — Before invoking anycreate-*oradd-entity-*command, perform a check-then-act: list or get to verify the target does not already exist (e.g. beforecreate-datasetcalllist-datasetsand match by--name+--project-id; beforeadd-entity-into-meta-collectioncalllist-entities-in-meta-collectionand match by entity id). If a previous attempt already succeeded, return the existing resource id instead of creating a duplicate. On retry after a transient error, prefer re-checking state over blindly re-issuing the create.
[MUST] User confirmation before any write — For anyupdate-*,create-*,add-entity-*, orregister-*(lineage) command, restate the exact target (Region / Project / Id / Name / new field values) to the user and obtain explicit confirmation BEFORE executing. Do not assume defaults; do not chain multiple writes without intermediate confirmation when the user has not pre-approved the full plan.
All CLI flags use kebab-case (lowercase with hyphens). Always use exactly the flag names shown in the command examples below.
Key flags:--page-size,--table-id,--src-entity-id,--dst-entity-id,--need-attach-relationship,--include-business-metadata,--meta-collection-id,--dataset-id,--project-id,--read-timeout,--connect-timeout
Entity IDs follow${EntityType}:${InstanceId}:${CatalogId}:${DatabaseName}:${SchemaName}:${TableName}. Seereferences/entity-id-formats.md.
Common MaxCompute:maxcompute-table:::project_name::table_name(no schema) ormaxcompute-table:::project_name:schema_name:table_name(with schema).
When user givesproject.table, try no-schema first; if not found, retry withdefaultschema.
Parameter confirmation — Confirm all user-customizable parameters (RegionId, entity IDs, etc.) before executing. Do NOT assume defaults.
Permission errors — Read references/ram-policies.md, guide the user to grant permissions, and wait for confirmation before retrying.Commands
All commands require --region <RegionId> and the timeout pair --read-timeout 60 --connect-timeout 10. The user-agent is set globally via AI-Mode in the Prerequisites section, so no per-command --user-agent flag is needed below. All list commands support --page-number and --page-size.
1. Catalog & Entity Browsing
# List crawler types
aliyun dataworks-public list-crawler-types --region <RegionId> --read-timeout 60 --connect-timeout 10
# List catalogs (--parent-meta-entity-id REQUIRED: "dlf" or "starrocks:<instance_id>")
aliyun dataworks-public list-catalogs --region <RegionId> --parent-meta-entity-id "<ParentMetaEntityId>" --page-size 20 --read-timeout 60 --connect-timeout 10
# Get database / table details
aliyun dataworks-public get-database --region <RegionId> --id <DatabaseId> --read-timeout 60 --connect-timeout 10
aliyun dataworks-public get-table --region <RegionId> --id <TableId> --include-business-metadata true --read-timeout 60 --connect-timeout 10
# List tables (--parent-meta-entity-id: "maxcompute-project:::project_name" or "maxcompute-schema:::project_name:schema_name")
aliyun dataworks-public list-tables --region <RegionId> --parent-meta-entity-id "<ParentMetaEntityId>" --page-size 20 --read-timeout 60 --connect-timeout 10
# Update table business metadata (write — confirm with user first; idempotent: same value can be re-applied safely)
aliyun dataworks-public update-table-business-metadata --region <RegionId> --id <TableId> --readme "<description>" --read-timeout 60 --connect-timeout 102. Columns & Partitions
# List / Get columns
aliyun dataworks-public list-columns --region <RegionId> --table-id <TableId> --page-size 50 --read-timeout 60 --connect-timeout 10
aliyun dataworks-public get-column --region <RegionId> --id <ColumnId> --read-timeout 60 --connect-timeout 10
# Update column business metadata (write — confirm with user first; idempotent on same value)
aliyun dataworks-public update-column-business-metadata --region <RegionId> --id <ColumnId> --description "<description>" --read-timeout 60 --connect-timeout 10
# List / Get partitions (MaxCompute / HMS only)
aliyun dataworks-public list-partitions --region <RegionId> --table-id <TableId> --page-size 20 --read-timeout 60 --connect-timeout 10
aliyun dataworks-public get-partition --region <RegionId> --table-id <TableId> --name <PartitionName> --read-timeout 60 --connect-timeout 103. Data Lineage
# Downstream: use --src-entity-id | Upstream: use --dst-entity-id
aliyun dataworks-public list-lineages --region <RegionId> --src-entity-id <EntityId> --need-attach-relationship true --page-size 20 --read-timeout 60 --connect-timeout 10
aliyun dataworks-public list-lineages --region <RegionId> --dst-entity-id <EntityId> --need-attach-relationship true --page-size 20 --read-timeout 60 --connect-timeout 10
# Relationships between two entities
aliyun dataworks-public list-lineage-relationships --region <RegionId> --src-entity-id <SrcEntityId> --dst-entity-id <DstEntityId> --page-size 20 --read-timeout 60 --connect-timeout 10
# Register lineage relationship (write — at least one side MUST be a custom object). Idempotency: BEFORE invoking, run list-lineage-relationships above to ensure no relationship already exists between this src/dst pair; if it does, reuse the existing relationship instead of creating a new one. Deletion is out of scope — use the console if you need to revoke.
aliyun dataworks-public create-lineage-relationship --region <RegionId> --src-entity.id <SrcEntityId> --src-entity.type <EntityType> --dst-entity.id <DstEntityId> --dst-entity.type <EntityType> --read-timeout 60 --connect-timeout 104. Datasets & Versions
# List / Get datasets (read)
aliyun dataworks-public list-datasets --region <RegionId> --project-id <ProjectId> --page-size 20 --read-timeout 60 --connect-timeout 10
aliyun dataworks-public get-dataset --region <RegionId> --id <DatasetId> --read-timeout 60 --connect-timeout 10
# Create dataset (write). Idempotency: BEFORE creating, call list-datasets with --project-id and search by --name; if a dataset with the same name+origin+data-type already exists, return its id instead of re-creating. --init-version is REQUIRED, JSON with Comment/Url/MountPath. Deletion is out of scope.
aliyun dataworks-public create-dataset --region <RegionId> --project-id <ProjectId> --name "<Name>" --origin "DATAWORKS" --data-type "<DataType>" --storage-type "<StorageType>" --comment "<Desc>" --init-version '{"Comment":"<VersionComment>","Url":"<DataUrl>","MountPath":"<MountPath>"}' --read-timeout 60 --connect-timeout 10
# Update dataset (write — confirm with user first; idempotent on same value)
aliyun dataworks-public update-dataset --region <RegionId> --id <DatasetId> --name "<NewName>" --comment "<NewComment>" --read-timeout 60 --connect-timeout 10
# List / Get / Preview dataset versions (read; max 20 versions per dataset)
aliyun dataworks-public list-dataset-versions --region <RegionId> --dataset-id <DatasetId> --page-size 20 --read-timeout 60 --connect-timeout 10
aliyun dataworks-public get-dataset-version --region <RegionId> --id <VersionId> --read-timeout 60 --connect-timeout 10
aliyun dataworks-public preview-dataset-version --region <RegionId> --id <VersionId> --read-timeout 60 --connect-timeout 10
# Create dataset version (write). Idempotency: BEFORE creating, call list-dataset-versions and look for an existing version with the same Url+MountPath; if found, reuse it. Quota: max 20 versions per dataset. Deletion is out of scope.
aliyun dataworks-public create-dataset-version --region <RegionId> --dataset-id <DatasetId> --comment "<Comment>" --url "<DataUrl>" --mount-path "<MountPath>" --read-timeout 60 --connect-timeout 10
# Update dataset version (write — confirm with user first; idempotent on same value)
aliyun dataworks-public update-dataset-version --region <RegionId> --id <VersionId> --comment "<NewComment>" --read-timeout 60 --connect-timeout 105. Metadata Collections
# List / Get collections (read; type: Category or Album — PascalCase, NOT uppercase)
aliyun dataworks-public list-meta-collections --region <RegionId> --type "<Category|Album>" --page-size 20 --read-timeout 60 --connect-timeout 10
aliyun dataworks-public get-meta-collection --region <RegionId> --id <CollectionId> --read-timeout 60 --connect-timeout 10
# Create collection (write). Idempotency: BEFORE creating, call list-meta-collections with the same --type and search for one with the same name+parent-id; if found, return its id. Deletion is out of scope.
aliyun dataworks-public create-meta-collection --region <RegionId> --name "<Name>" --type "<Category|Album>" --description "<Desc>" --parent-id "<ParentId>" --read-timeout 60 --connect-timeout 10
# Update collection (write — confirm with user first; idempotent on same value)
aliyun dataworks-public update-meta-collection --region <RegionId> --id <CollectionId> --name "<NewName>" --description "<NewDesc>" --read-timeout 60 --connect-timeout 10
# List entities currently in a collection (read)
aliyun dataworks-public list-entities-in-meta-collection --region <RegionId> --id <CollectionId> --page-size 20 --read-timeout 60 --connect-timeout 10
# Add entity into collection (write). Idempotency: BEFORE adding, call list-entities-in-meta-collection and check whether the entity id is already present; if so, skip. Removal is out of scope.
aliyun dataworks-public add-entity-into-meta-collection --region <RegionId> --meta-collection-id <CollectionId> --id <EntityId> --remark "<Remark>" --read-timeout 60 --connect-timeout 10Tips
- Direct access — For MaxCompute, construct entity ID directly (
maxcompute-table:::project::table) and callget-table— no need to browse from catalogs. - Lineage direction —
--src-entity-id= downstream,--dst-entity-id= upstream. For full impact analysis, recursively query each downstream entity to trace multi-level lineage (ODS->DWD->DWS->ADS). - Schema fallback — If MaxCompute table not found, retry with
:default:schema (three-level model). - Limits — Max 20 versions per dataset; Album operations require
AliyunDataWorksFullAccessor creator/admin; max 2000 datasets per tenant. - Deletions — Out of scope. If the user asks to delete a dataset/version/collection/lineage relationship or remove an entity from a collection, decline and tell them to use the DataWorks console.
- Retry safely — When a write times out or returns an ambiguous error, do NOT blindly retry. Re-check state with the matching
list-*/get-*first to detect partial success, then decide whether to retry or accept.
References
| File | Description |
|---|---|
references/entity-id-formats.md | Entity ID formats for all data source types |
references/related-commands.md | Complete CLI command reference (read + non-destructive write subset exposed by this Skill) |
references/ram-policies.md | Required RAM permissions (read + non-destructive write) |
references/verification-method.md | Success verification steps |
Entity ID Format Reference
All DataWorks Data Map entities use structured IDs with the format: ${EntityType}:${InstanceId}:${CatalogId}:${DatabaseName}:${SchemaName}:${TableName}
For levels that don't exist, use empty string as placeholder.
MaxCompute
| Entity Level | ID Format | Example |
|---|---|---|
| Project (database) | maxcompute-project:::project_name | maxcompute-project:::my_project |
| Schema | maxcompute-schema:::project_name:schema_name | maxcompute-schema:::my_project:default |
| Table (no schema) | maxcompute-table:::project_name::table_name | maxcompute-table:::my_project::my_table |
| Table (with schema) | maxcompute-table:::project_name:schema_name:table_name | maxcompute-table:::my_project:default:my_table |
| Column | maxcompute-column:::project_name::table_name:column_name | maxcompute-column:::my_project::my_table:id |
Note: For MaxCompute,InstanceIdandCatalogIdare empty (use empty string placeholder).
DatabaseName = MaxCompute project name. Schema is only needed when the project has enabled the three-level model.DLF (Data Lake Formation)
| Entity Level | ID Format | Example |
|---|---|---|
| Catalog | dlf-catalog::catalog_id | dlf-catalog::my_catalog |
| Database | dlf-database::catalog_id:database_name | dlf-database::my_catalog:my_db |
| Table | dlf-table::catalog_id:database_name::table_name | dlf-table::my_catalog:my_db::my_table |
| Column | dlf-column::catalog_id:database_name::table_name:column_name | dlf-column::my_catalog:my_db::my_table:id |
Hologres
| Entity Level | ID Format | Example |
|---|---|---|
| Database | holo-database:instance_id::database_name | holo-database:hgprecn-xxx::my_db |
| Schema | holo-schema:instance_id::database_name:schema_name | holo-schema:hgprecn-xxx::my_db:public |
| Table | holo-table:instance_id::database_name:schema_name:table_name | holo-table:hgprecn-xxx::my_db:public:my_table |
MySQL
| Entity Level | ID Format | Example |
|---|---|---|
| Database | `mysql-database:(instance_id | encoded_jdbc_url)::database_name` |
| Table | `mysql-table:(instance_id | encoded_jdbc_url)::database_name::table_name` |
HMS (Hive Metastore / EMR)
| Entity Level | ID Format | Example |
|---|---|---|
| Database | hms-database:instance_id::database_name | hms-database:c-xxx::my_db |
| Table | hms-table:instance_id::database_name::table_name | hms-table:c-xxx::my_db::my_table |
StarRocks
| Entity Level | ID Format | Example |
|---|---|---|
| Catalog | `starrocks-catalog:(instance_id | encoded_jdbc_url):catalog_name` |
| Database | `starrocks-database:(instance_id | encoded_jdbc_url):catalog_name:database_name` |
| Table | `starrocks-table:(instance_id | encoded_jdbc_url):catalog_name:database_name::table_name` |
Quick Lookup: User Input → Entity ID
When a user provides a table name like project_name.table_name:
1. MaxCompute: Try maxcompute-table:::project_name::table_name first, then maxcompute-table:::project_name:default:table_name (if three-level model enabled) 2. DLF: Need catalog_id — use list-catalogs to find it first 3. Hologres/MySQL/HMS: Need instance_id — ask the user or look up from DataWorks workspace bindings
RAM Policies — DataWorks Metadata
Required Permissions
This Skill performs read + non-destructive write operations only — no deletions or removals. Choose the policy that matches the operator's intended scope.
Policy A — Read-Only Browsing
Use this policy if the user only needs to browse metadata, lineage, datasets, and collections.
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dataworks:ListCrawlerTypes",
"dataworks:ListCatalogs",
"dataworks:GetCatalog",
"dataworks:GetDatabase",
"dataworks:GetTable",
"dataworks:ListColumns",
"dataworks:GetColumn",
"dataworks:ListPartitions",
"dataworks:GetPartition",
"dataworks:ListLineages",
"dataworks:ListLineageRelationships",
"dataworks:GetLineageRelationship",
"dataworks:ListDatasets",
"dataworks:GetDataset",
"dataworks:ListDatasetVersions",
"dataworks:GetDatasetVersion",
"dataworks:PreviewDatasetVersion",
"dataworks:ListMetaCollections",
"dataworks:GetMetaCollection",
"dataworks:ListEntitiesInMetaCollection"
],
"Resource": "*"
}
]
}Policy B — Read + Non-Destructive Write
Use this policy when the user also needs to update business metadata, register lineage, create / update datasets & versions, or create / update collections and add entities. *No `Delete / Remove` actions are included* — by design this Skill cannot delete anything; perform deletions in the DataWorks console.
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dataworks:ListCrawlerTypes",
"dataworks:ListCatalogs",
"dataworks:GetCatalog",
"dataworks:GetDatabase",
"dataworks:GetTable",
"dataworks:UpdateTableBusinessMetadata",
"dataworks:ListColumns",
"dataworks:GetColumn",
"dataworks:UpdateColumnBusinessMetadata",
"dataworks:ListPartitions",
"dataworks:GetPartition",
"dataworks:ListLineages",
"dataworks:ListLineageRelationships",
"dataworks:GetLineageRelationship",
"dataworks:CreateLineageRelationship",
"dataworks:CreateDataset",
"dataworks:ListDatasets",
"dataworks:GetDataset",
"dataworks:UpdateDataset",
"dataworks:CreateDatasetVersion",
"dataworks:ListDatasetVersions",
"dataworks:GetDatasetVersion",
"dataworks:UpdateDatasetVersion",
"dataworks:PreviewDatasetVersion",
"dataworks:ListMetaCollections",
"dataworks:CreateMetaCollection",
"dataworks:GetMetaCollection",
"dataworks:UpdateMetaCollection",
"dataworks:ListEntitiesInMetaCollection",
"dataworks:AddEntityIntoMetaCollection"
],
"Resource": "*"
}
]
}Permission Summary by Category
| Category | Read Actions | Non-Destructive Write Actions |
|---|---|---|
| Crawler Types | ListCrawlerTypes | — |
| Catalogs | ListCatalogs, GetCatalog | — |
| Databases | GetDatabase | — |
| Tables | GetTable | UpdateTableBusinessMetadata |
| Columns | ListColumns, GetColumn | UpdateColumnBusinessMetadata |
| Partitions | ListPartitions, GetPartition | — |
| Lineage | ListLineages, ListLineageRelationships, GetLineageRelationship | CreateLineageRelationship |
| Datasets | ListDatasets, GetDataset | CreateDataset, UpdateDataset |
| Dataset Versions | ListDatasetVersions, GetDatasetVersion, PreviewDatasetVersion | CreateDatasetVersion, UpdateDatasetVersion |
| Meta Collections | ListMetaCollections, GetMetaCollection, ListEntitiesInMetaCollection | CreateMetaCollection, UpdateMetaCollection, AddEntityIntoMetaCollection |
Notes
- Album operations (create/update meta collection of type
Album, add entities) requireAliyunDataWorksFullAccesssystem policy or membership in the album. - Dataset update requires the operator to be the dataset creator or workspace admin.
- For least-privilege access, attach Policy A (Read-Only) when only browsing is needed.
- Deletions are out of scope for this Skill. The 5 omitted actions —
DeleteLineageRelationship,DeleteDataset,DeleteDatasetVersion,DeleteMetaCollection,RemoveEntityFromMetaCollection— must NOT be granted to this Skill's operator account. Perform any required deletion in the DataWorks console with a separately-granted account that holds the corresponding write permission.
Related CLI Commands — DataWorks Metadata
All commands below use the `dataworks-public` product plugin. The user-agent AlibabaCloud-Agent-Skills/alibabacloud-dataworks-metadata is bound session-wide via AI-Mode (aliyun configure ai-mode enable + aliyun configure ai-mode set-user-agent --user-agent "...") per SKILL.md, so no per-command --user-agent flag is needed. Disable AI-Mode (aliyun configure ai-mode disable) when the Skill's work is complete.
Every command MUST be invoked with --read-timeout 60 --connect-timeout 10 to bound network waits.
Scope note — This Skill exposes read + non-destructive write APIs only. The 5 destructive APIs (DeleteLineageRelationship,DeleteDataset,DeleteDatasetVersion,DeleteMetaCollection,RemoveEntityFromMetaCollection) exist in the upstream DataWorks API but are intentionally NOT exposed and MUST NOT be invoked. Perform deletions in the DataWorks console.
Idempotency — EveryCreate*/Add*action below MUST be preceded by a check-then-act read (matchingList*/Get*) to avoid creating duplicates on retry. See SKILL.mdRules → [MUST] Idempotency.
Metadata Crawler Types
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public list-crawler-types | ListCrawlerTypes | R | 获取数据地图元数据采集器类型列表 |
Catalog & Entity Browsing
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public list-catalogs | ListCatalogs | R | 查询数据目录列表(支持dlf、starrocks类型) |
aliyun dataworks-public get-catalog | GetCatalog | R | 获取数据目录详情 |
aliyun dataworks-public get-database | GetDatabase | R | 获取数据库详情 |
aliyun dataworks-public get-table | GetTable | R | 获取数据表详情(可选含业务元数据) |
aliyun dataworks-public update-table-business-metadata | UpdateTableBusinessMetadata | W | 更新数据表业务元数据(使用说明) |
Field (Column) Operations
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public list-columns | ListColumns | R | 查询数据表字段列表 |
aliyun dataworks-public get-column | GetColumn | R | 获取数据表字段详情 |
aliyun dataworks-public update-column-business-metadata | UpdateColumnBusinessMetadata | W | 更新字段业务元数据(业务描述) |
Partition Operations
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public list-partitions | ListPartitions | R | 查询数据表分区列表(MaxCompute/HMS) |
aliyun dataworks-public get-partition | GetPartition | R | 获取分区详情(MaxCompute/HMS) |
Lineage Operations
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public list-lineages | ListLineages | R | 查询实体上下游血缘列表 |
aliyun dataworks-public list-lineage-relationships | ListLineageRelationships | R | 查询两实体间血缘关系列表 |
aliyun dataworks-public get-lineage-relationship | GetLineageRelationship | R | 获取血缘关系详情 |
aliyun dataworks-public create-lineage-relationship | CreateLineageRelationship | W | 注册血缘关系(至少一方为自定义对象;调用前先 list 检查重复) |
Dataset Operations
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public create-dataset | CreateDataset | W | 创建数据集(单租户最多 2000 个;调用前先 list 检查同名) |
aliyun dataworks-public list-datasets | ListDatasets | R | 查询数据集列表(DataWorks/PAI) |
aliyun dataworks-public get-dataset | GetDataset | R | 获取数据集详情 |
aliyun dataworks-public update-dataset | UpdateDataset | W | 更新数据集信息(同值幂等) |
Dataset Version Operations
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public create-dataset-version | CreateDatasetVersion | W | 创建数据集版本(最多 20 个;调用前先 list 检查重复 Url+MountPath) |
aliyun dataworks-public list-dataset-versions | ListDatasetVersions | R | 查询数据集版本列表 |
aliyun dataworks-public get-dataset-version | GetDatasetVersion | R | 获取数据集版本信息 |
aliyun dataworks-public update-dataset-version | UpdateDatasetVersion | W | 更新数据集版本信息(同值幂等) |
aliyun dataworks-public preview-dataset-version | PreviewDatasetVersion | R | 预览数据集版本内容(仅 OSS 文本) |
Metadata Collection Operations
| CLI Command | API Name | R/W | Description |
|---|---|---|---|
aliyun dataworks-public list-meta-collections | ListMetaCollections | R | 查询集合列表(类目/数据专辑) |
aliyun dataworks-public create-meta-collection | CreateMetaCollection | W | 创建集合对象(调用前先 list 检查同名同 parent) |
aliyun dataworks-public get-meta-collection | GetMetaCollection | R | 获取集合详情 |
aliyun dataworks-public update-meta-collection | UpdateMetaCollection | W | 更新集合对象(同值幂等) |
aliyun dataworks-public list-entities-in-meta-collection | ListEntitiesInMetaCollection | R | 查询集合中的实体列表 |
aliyun dataworks-public add-entity-into-meta-collection | AddEntityIntoMetaCollection | W | 向集合添加实体(调用前先 list 检查实体是否已存在) |
Verification Method — DataWorks Metadata
Note: Each command MUST include --read-timeout 60 --connect-timeout 10. For every write step, perform a check-then-act read first to enforce idempotency, then verify success with a follow-up read.1. Catalog Browsing Verification
Step: List catalogs and verify response
aliyun dataworks-public list-catalogs \
--region <RegionId> \
--parent-meta-entity-id "dlf" \
--page-size 5 \
--read-timeout 60 --connect-timeout 10Expected: JSON response with CatalogList array containing catalog items with Id, Name, Type fields.
Step: Get specific catalog detail
aliyun dataworks-public get-catalog \
--region <RegionId> \
--id <CatalogId_from_list> \
--read-timeout 60 --connect-timeout 10Expected: JSON with catalog detail including Id, Name, Type, and Comment.
2. Table & Column Verification
Step: Get table detail with business metadata
aliyun dataworks-public get-table \
--region <RegionId> \
--id <TableId> \
--include-business-metadata true \
--read-timeout 60 --connect-timeout 10Expected: JSON with table detail including Id, Name, DatabaseId, Columns, and business metadata fields.
Step: List columns of a table
aliyun dataworks-public list-columns \
--region <RegionId> \
--table-id <TableId> \
--page-size 50 \
--read-timeout 60 --connect-timeout 10Expected: JSON with ColumnList array, each entry having Id, Name, DataType, Comment.
Step: After update-table-business-metadata, re-fetch and verify
aliyun dataworks-public get-table \
--region <RegionId> \
--id <TableId> \
--include-business-metadata true \
--read-timeout 60 --connect-timeout 10Expected: Readme field reflects the new value.
3. Partition Verification
aliyun dataworks-public list-partitions \
--region <RegionId> \
--table-id <TableId> \
--page-size 10 \
--read-timeout 60 --connect-timeout 10Expected: JSON with PartitionList array (if table has partitions). Empty list for non-partitioned tables.
4. Lineage Verification
Step: Query downstream lineage
aliyun dataworks-public list-lineages \
--region <RegionId> \
--src-entity-id <EntityId> \
--need-attach-relationship true \
--page-size 10 \
--read-timeout 60 --connect-timeout 10Expected: JSON with lineage entity list showing downstream dependencies.
Step: Idempotency check before create-lineage-relationship
aliyun dataworks-public list-lineage-relationships \
--region <RegionId> \
--src-entity-id <SrcEntityId> \
--dst-entity-id <DstEntityId> \
--page-size 10 \
--read-timeout 60 --connect-timeout 10Expected: If a relationship is already present, reuse its Id and skip the create. Otherwise proceed with create-lineage-relationship and verify by re-running the list.
5. Dataset Verification
Step: Idempotency check before create-dataset
aliyun dataworks-public list-datasets \
--region <RegionId> \
--project-id <ProjectId> \
--page-size 50 \
--read-timeout 60 --connect-timeout 10Expected: Search the response for an existing dataset with the same Name + Origin + DataType. If present, return its Id; if absent, proceed with create-dataset.
Step: After create-dataset, list datasets to confirm
aliyun dataworks-public list-datasets \
--region <RegionId> \
--project-id <ProjectId> \
--page-size 10 \
--read-timeout 60 --connect-timeout 10Expected: New dataset appears in the list with matching Name, Origin, and DataType.
Step: Idempotency check before create-dataset-version
aliyun dataworks-public list-dataset-versions \
--region <RegionId> \
--dataset-id <DatasetId> \
--page-size 20 \
--read-timeout 60 --connect-timeout 10Expected: Search for an existing version with the same Url + MountPath. If present, reuse it; otherwise proceed with create-dataset-version.
Step: Verify dataset version
aliyun dataworks-public list-dataset-versions \
--region <RegionId> \
--dataset-id <DatasetId> \
--page-size 10 \
--read-timeout 60 --connect-timeout 10Expected: New version appears with matching Comment and incrementing version number.
6. Metadata Collection Verification
Step: Idempotency check before create-meta-collection
aliyun dataworks-public list-meta-collections \
--region <RegionId> \
--type "<Category|Album>" \
--page-size 50 \
--read-timeout 60 --connect-timeout 10Expected: Search for an existing collection with the same Name + ParentId. If present, return its Id; otherwise proceed with create-meta-collection.
Step: After create-meta-collection, fetch detail
aliyun dataworks-public get-meta-collection \
--region <RegionId> \
--id <CollectionId_from_create> \
--read-timeout 60 --connect-timeout 10Expected: JSON with collection detail matching provided Name, Type, and Description.
Step: Idempotency check before add-entity-into-meta-collection
aliyun dataworks-public list-entities-in-meta-collection \
--region <RegionId> \
--id <CollectionId> \
--page-size 50 \
--read-timeout 60 --connect-timeout 10Expected: If the target entity id is already present, skip the add. Otherwise proceed.
Step: Verify entity added to collection
aliyun dataworks-public list-entities-in-meta-collection \
--region <RegionId> \
--id <CollectionId> \
--page-size 20 \
--read-timeout 60 --connect-timeout 10Expected: Added entity appears in the entity list with matching Id.
Common Error Codes
| Error Code | Meaning | Resolution |
|---|---|---|
Forbidden.RAM | Insufficient permissions | Grant required RAM permissions for DataWorks (see ram-policies.md) |
InvalidParameter | Missing or invalid parameter | Verify parameter names and values |
InvalidType | Invalid type value for meta collection | Use PascalCase: Category, Album (not uppercase) |
EntityNotExist | Target entity not found | Confirm entity ID is correct |
QuotaExceeded | Resource limit reached | Check dataset (≤2000) / version (≤20) limits |
EntityAlreadyExists / duplicate | Resource already present | The idempotency pre-check should have caught this; reuse the existing id rather than retrying |
RequestTimeout / network timeout | Request took longer than --read-timeout / --connect-timeout | After timeout on a write, verify state via list/get before any retry to detect partial success — do NOT loop |