Azure Ml Foundry Workspace

Azure Machine Learning Workspace / Azure AI Foundry - Complete Deep-Dive Reference

Authoritative reference for every aspect of Azure Machine Learning Workspace (Azure AI Foundry) including architecture, networking, private endpoints, compute clusters, endpoint deployment, managed identities, ACR integration, storage accounts, all CLI and PowerShell commands, log reading, debugging, and Terraform integration.

---

1. ARCHITECTURE AND CORE CONCEPTS

Workspace Resource Hierarchy

Azure Subscription
  └── Resource Group
        ├── Azure ML Workspace (Microsoft.MachineLearningServices/workspaces)
        │     ├── Dependent Resources (auto-created or BYO)
        │     │     ├── Azure Storage Account (default datastore)
        │     │     ├── Azure Key Vault (secrets, connection strings)
        │     │     ├── Azure Application Insights (telemetry)
        │     │     └── Azure Container Registry (Docker images for environments)
        │     ├── Compute Targets
        │     │     ├── Compute Instances (dev/test VMs)
        │     │     ├── Compute Clusters (AmlCompute - training)
        │     │     ├── Serverless Compute (on-demand)
        │     │     ├── Kubernetes Compute (AKS / Arc-enabled)
        │     │     └── Attached Compute (Databricks, HDInsight, VMs)
        │     ├── Data Assets (versioned references to data)
        │     ├── Datastores (connections to storage)
        │     ├── Environments (Docker + conda specs)
        │     ├── Models (registered trained models)
        │     ├── Endpoints
        │     │     ├── Managed Online Endpoints (real-time)
        │     │     ├── Kubernetes Online Endpoints (BYO infra)
        │     │     ├── Batch Endpoints (large-scale scoring)
        │     │     └── Serverless Endpoints (MaaS - pay-per-token)
        │     ├── Jobs (training runs, pipelines, sweeps)
        │     ├── Components (reusable pipeline steps)
        │     ├── Schedules (recurring job triggers)
        │     └── Registries (cross-workspace sharing)
        └── AI Foundry Hub (kind=hub) + Projects (kind=project)

AI Foundry Hub/Project vs Classic Workspace

Workspace Creation - All Methods

CLI:

# Install/upgrade ML extension
az extension add --name ml --upgrade

# Create resource group
az group create --name ml-rg --location eastus

# Create workspace with all dependencies auto-created
az ml workspace create \
  --name my-ml-workspace \
  --resource-group ml-rg \
  --location eastus

# Create workspace with explicit dependencies
az ml workspace create \
  --name my-ml-workspace \
  --resource-group ml-rg \
  --location eastus \
  --storage-account /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Storage/storageAccounts/mlstorage \
  --key-vault /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.KeyVault/vaults/mlkeyvault \
  --app-insights /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Insights/components/mlinsights \
  --container-registry /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.ContainerRegistry/registries/mlacr \
  --public-network-access Disabled \
  --managed-network AllowInternetOutbound \
  --image-build-compute cpu-build-cluster \
  --enable-data-isolation true \
  --tags Environment=Production Team=DataScience

# Create AI Foundry Hub
az ml workspace create \
  --name my-ai-hub \
  --resource-group ml-rg \
  --location eastus \
  --kind hub \
  --storage-account aihubstorage \
  --key-vault aihubkeyvault

# Create AI Foundry Project within Hub
az ml workspace create \
  --name my-ai-project \
  --resource-group ml-rg \
  --location eastus \
  --kind project \
  --hub-id /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.MachineLearningServices/workspaces/my-ai-hub

# Show workspace details
az ml workspace show \
  --name my-ml-workspace \
  --resource-group ml-rg

# List all workspaces
az ml workspace list \
  --resource-group ml-rg \
  --output table

# Update workspace
az ml workspace update \
  --name my-ml-workspace \
  --resource-group ml-rg \
  --description "Updated workspace" \
  --public-network-access Disabled

# Delete workspace
az ml workspace delete \
  --name my-ml-workspace \
  --resource-group ml-rg \
  --permanently-delete --all-resources

# Diagnose workspace configuration
az ml workspace diagnose \
  --name my-ml-workspace \
  --resource-group ml-rg

PowerShell (Az.MachineLearningServices):

# Install the module
Install-Module -Name Az.MachineLearningServices -Scope CurrentUser -Repository PSGallery -Force

# Create workspace
New-AzMLWorkspace `
  -Name "my-ml-workspace" `
  -ResourceGroupName "ml-rg" `
  -Location "eastus" `
  -StorageAccountId "/subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Storage/storageAccounts/mlstorage" `
  -KeyVaultId "/subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.KeyVault/vaults/mlkeyvault" `
  -ApplicationInsightId "/subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Insights/components/mlinsights" `
  -IdentityType "SystemAssigned" `
  -PublicNetworkAccess "Disabled"

# Get workspace
Get-AzMLWorkspace -Name "my-ml-workspace" -ResourceGroupName "ml-rg"

# List workspaces
Get-AzMLWorkspace -ResourceGroupName "ml-rg"

# Update workspace
Update-AzMLWorkspace `
  -Name "my-ml-workspace" `
  -ResourceGroupName "ml-rg" `
  -Description "Updated workspace" `
  -Tag @{Environment="Production"}

# Remove workspace
Remove-AzMLWorkspace -Name "my-ml-workspace" -ResourceGroupName "ml-rg"

# Diagnose workspace
Invoke-AzMLWorkspaceDiagnose -Name "my-ml-workspace" -ResourceGroupName "ml-rg"

---

2. NETWORKING

Azure ML supports three managed network isolation modes (Disabled, AllowInternetOutbound, AllowOnlyApprovedOutbound) with the managed VNet approach recommended for production. Private endpoints provide inbound connectivity, and outbound rules control egress from compute resources.

Key DNS Zones

Key Service Tags

For full VNet configuration, private endpoint setup, NSG rules, and outbound rule management, see [references/networking.md](references/networking.md).

---

3. COMPUTE

Azure ML offers multiple compute targets: Compute Instances for dev/test, AmlCompute Clusters for scalable training, Serverless Compute for on-demand jobs without cluster management, and Kubernetes Compute for BYO infrastructure scenarios.

GPU VM SKU Quick Reference

For the complete GPU SKU table, compute instance/cluster CLI reference, serverless compute, Kubernetes attach, and debugging commands, see [references/compute.md](references/compute.md).

---

4. ENDPOINT DEPLOYMENT

Azure ML supports four endpoint types: Managed Online Endpoints (recommended for real-time inference with blue-green deployments), Batch Endpoints (large-scale scoring on compute clusters), Kubernetes Online Endpoints (BYO AKS/Arc infrastructure), and Serverless Endpoints (pay-per-token Model-as-a-Service).

Endpoint Types Quick Reference

For full endpoint creation, deployment, traffic splitting, log retrieval, and batch invocation commands, see [references/endpoints.md](references/endpoints.md).

---

5-7. IDENTITIES, ACR, AND STORAGE

Managed identities (system-assigned or user-assigned) control access between workspace, compute, endpoints, and dependent resources. ACR stores Docker images for environments and model serving, requiring Premium SKU for private endpoints and an image-build-compute cluster when behind a VNet. Storage accounts serve as the default datastore for blobs, file shares, job outputs, and MLflow artifacts.

Identity Types

Key RBAC Roles

For full identity configuration, role assignment commands, ACR integration, private ACR setup, datastore registration, and storage account details, see [references/identities-acr-storage.md](references/identities-acr-storage.md).

---

8-9. CLI AND POWERSHELL

The az ml CLI extension provides comprehensive workspace management through 20+ command groups covering workspaces, compute, jobs, models, endpoints, environments, data, datastores, components, schedules, registries, and connections. The Az.MachineLearningServices PowerShell module offers equivalent functionality for Windows-native automation.

Key az ml Command Groups

For the complete command reference, job management deep-dive, schedule management, and full PowerShell cmdlet reference, see [references/cli-powershell.md](references/cli-powershell.md).

---

10. TERRAFORM INTEGRATION

Azure ML workspaces can be fully provisioned with Terraform using the azurerm provider. A production setup includes the workspace, VNet/subnets, NSG, storage account, key vault, ACR, Application Insights, private endpoints, DNS zones, compute clusters, and RBAC role assignments.

Key Terraform Resources

For the full production-ready Terraform configuration (providers, networking, storage, key vault, ACR, workspace, compute, role assignments, and outputs), see [references/terraform.md](references/terraform.md).

---

11. TROUBLESHOOTING AND DEBUGGING

Azure ML provides multiple debugging surfaces: real-time job log streaming, deployment container logs (inference-server and storage-initializer), compute instance SSH access for system-level diagnostics, Log Analytics queries for historical analysis, and the az ml workspace diagnose command for configuration validation.

Common Error Categories

For full error reference tables, log locations, Log Analytics queries, endpoint metrics monitoring, workspace diagnostics, and the secure workspace setup checklist, see [references/troubleshooting.md](references/troubleshooting.md).

---

Additional Resources

Detailed reference files for each topic area:

• [references/networking.md](references/networking.md) -- VNet, private endpoints, DNS zones, NSG rules, service tags
• [references/compute.md](references/compute.md) -- GPU SKUs, compute instances, clusters, serverless, Kubernetes
• [references/endpoints.md](references/endpoints.md) -- Managed online, batch, Kubernetes, and serverless endpoints
• [references/identities-acr-storage.md](references/identities-acr-storage.md) -- Managed identities, ACR integration, storage accounts
• [references/cli-powershell.md](references/cli-powershell.md) -- Complete az ml CLI and PowerShell command reference
• [references/terraform.md](references/terraform.md) -- Full production-ready Terraform configuration
• [references/troubleshooting.md](references/troubleshooting.md) -- Log reading, debugging, error tables, setup checklist

External Documentation

• Azure ML Documentation
• Azure AI Foundry Documentation
• az ml CLI Reference
• Az.MachineLearningServices PowerShell Module
• Terraform AzureRM ML Workspace
• Secure Azure ML Workspace with VNet
• Managed Network Isolation
• Troubleshoot Online Endpoints
• Azure ML RBAC Roles
• GPU VM Sizes

CLI and PowerShell Reference

Complete reference for all az ml CLI commands and Az.MachineLearningServices PowerShell commands.

---

8. COMPLETE az ml CLI COMMAND REFERENCE

All az ml Subcommands

Job Management - Deep Dive

# Create and submit a job
az ml job create --file job.yml \
  --resource-group ml-rg --workspace-name my-workspace

# Stream job logs in real-time
az ml job stream --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace

# Show job details
az ml job show --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace \
  --query "{status:status, duration:properties.duration, compute:compute}" -o json

# List jobs
az ml job list \
  --resource-group ml-rg --workspace-name my-workspace \
  --output table \
  --max-results 20

# List jobs by experiment
az ml job list \
  --resource-group ml-rg --workspace-name my-workspace \
  --query "[?experiment_name=='my-experiment']" \
  --output table

# Cancel a running job
az ml job cancel --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace

# Download job outputs
az ml job download --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace \
  --output-name model \
  --download-path ./downloaded-outputs

# Download all outputs
az ml job download --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace \
  --all

# Archive/restore jobs
az ml job archive --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace
az ml job restore --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace

# Run job locally for debugging
az ml job create --file job.yml \
  --set compute=local \
  --resource-group ml-rg --workspace-name my-workspace

# Create and test local endpoint deployment
az ml online-deployment create --local \
  --file deployment.yml \
  --resource-group ml-rg --workspace-name my-workspace

Schedule Management

# Create a schedule
az ml schedule create --file schedule.yml \
  --resource-group ml-rg --workspace-name my-workspace

# List schedules
az ml schedule list \
  --resource-group ml-rg --workspace-name my-workspace \
  --output table

# Enable/disable schedule
az ml schedule enable --name my-schedule \
  --resource-group ml-rg --workspace-name my-workspace
az ml schedule disable --name my-schedule \
  --resource-group ml-rg --workspace-name my-workspace

# Delete schedule
az ml schedule delete --name my-schedule \
  --resource-group ml-rg --workspace-name my-workspace --yes

schedule.yml:

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: daily-retrain
display_name: Daily Retraining
trigger:
  type: recurrence
  frequency: day
  interval: 1
  schedule:
    hours: [2]
    minutes: [0]
  time_zone: "Eastern Standard Time"
create_job: ./pipeline.yml

---

9. POWERSHELL Az.MachineLearningServices - COMPLETE REFERENCE

# ===================== WORKSPACE COMMANDS =====================
# Create workspace
New-AzMLWorkspace -Name "ws" -ResourceGroupName "rg" -Location "eastus" `
  -IdentityType "SystemAssigned"

# Get workspace
Get-AzMLWorkspace -Name "ws" -ResourceGroupName "rg"

# List workspaces
Get-AzMLWorkspace -ResourceGroupName "rg"

# Update workspace
Update-AzMLWorkspace -Name "ws" -ResourceGroupName "rg" `
  -Description "Updated" -Tag @{env="prod"}

# Diagnose workspace
Invoke-AzMLWorkspaceDiagnose -Name "ws" -ResourceGroupName "rg"

# Remove workspace
Remove-AzMLWorkspace -Name "ws" -ResourceGroupName "rg"

# ===================== COMPUTE COMMANDS =====================
# Create compute
New-AzMLWorkspaceCompute -Name "cluster" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -Location "eastus" -ComputeType "AmlCompute" `
  -Property @{vmSize="Standard_DS3_v2"; scaleSettings=@{minNodeCount=0;maxNodeCount=4}}

# Get compute
Get-AzMLWorkspaceCompute -Name "cluster" -ResourceGroupName "rg" -WorkspaceName "ws"

# List compute
Get-AzMLWorkspaceCompute -ResourceGroupName "rg" -WorkspaceName "ws"

# Start/stop compute
Start-AzMLWorkspaceCompute -Name "instance" -ResourceGroupName "rg" -WorkspaceName "ws"
Stop-AzMLWorkspaceCompute -Name "instance" -ResourceGroupName "rg" -WorkspaceName "ws"

# Remove compute
Remove-AzMLWorkspaceCompute -Name "cluster" -ResourceGroupName "rg" -WorkspaceName "ws"

# ===================== ONLINE ENDPOINT COMMANDS =====================
# Create online endpoint
New-AzMLWorkspaceOnlineEndpoint -Name "ep" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -Location "eastus" `
  -AuthMode "Key" -IdentityType "SystemAssigned"

# Get online endpoint
Get-AzMLWorkspaceOnlineEndpoint -Name "ep" -ResourceGroupName "rg" -WorkspaceName "ws"

# List online endpoints
Get-AzMLWorkspaceOnlineEndpoint -ResourceGroupName "rg" -WorkspaceName "ws"

# Get endpoint keys
Get-AzMLWorkspaceOnlineEndpointKey -Name "ep" -ResourceGroupName "rg" -WorkspaceName "ws"

# Regenerate keys
New-AzMLWorkspaceOnlineEndpointKey -Name "ep" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -KeyType "Primary"

# Remove endpoint
Remove-AzMLWorkspaceOnlineEndpoint -Name "ep" -ResourceGroupName "rg" -WorkspaceName "ws"

# ===================== ONLINE DEPLOYMENT COMMANDS =====================
# Create deployment
New-AzMLWorkspaceOnlineDeployment -Name "blue" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -EndpointName "ep" -Location "eastus" `
  -SkuName "Default" -SkuCapacity 1

# Get deployment
Get-AzMLWorkspaceOnlineDeployment -Name "blue" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -EndpointName "ep"

# Get deployment logs
Get-AzMLWorkspaceOnlineDeploymentLog -Name "blue" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -EndpointName "ep" -Tail 200

# Remove deployment
Remove-AzMLWorkspaceOnlineDeployment -Name "blue" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -EndpointName "ep"

# ===================== BATCH ENDPOINT COMMANDS =====================
New-AzMLWorkspaceBatchEndpoint -Name "batch" -ResourceGroupName "rg" `
  -WorkspaceName "ws" -Location "eastus"
Get-AzMLWorkspaceBatchEndpoint -Name "batch" -ResourceGroupName "rg" -WorkspaceName "ws"
Remove-AzMLWorkspaceBatchEndpoint -Name "batch" -ResourceGroupName "rg" -WorkspaceName "ws"

# ===================== JOB COMMANDS =====================
# Get job
Get-AzMLWorkspaceJob -Name "job-id" -ResourceGroupName "rg" -WorkspaceName "ws"

# List jobs
Get-AzMLWorkspaceJob -ResourceGroupName "rg" -WorkspaceName "ws"

# Cancel job
Stop-AzMLWorkspaceJob -Name "job-id" -ResourceGroupName "rg" -WorkspaceName "ws"

# ===================== DATA AND DATASTORE COMMANDS =====================
Get-AzMLWorkspaceDatastore -Name "store" -ResourceGroupName "rg" -WorkspaceName "ws"
Get-AzMLWorkspaceDatastore -ResourceGroupName "rg" -WorkspaceName "ws"

# ===================== ENVIRONMENT COMMANDS =====================
Get-AzMLWorkspaceEnvironmentContainer -Name "env" -ResourceGroupName "rg" -WorkspaceName "ws"
Get-AzMLWorkspaceEnvironmentVersion -Name "env" -ResourceGroupName "rg" -WorkspaceName "ws" -Version 1

# ===================== MODEL COMMANDS =====================
Get-AzMLWorkspaceModelContainer -Name "model" -ResourceGroupName "rg" -WorkspaceName "ws"
Get-AzMLWorkspaceModelVersion -Name "model" -ResourceGroupName "rg" -WorkspaceName "ws" -Version 1

# ===================== CONNECTION COMMANDS =====================
Get-AzMLWorkspaceConnection -Name "conn" -ResourceGroupName "rg" -WorkspaceName "ws"
New-AzMLWorkspaceConnection -Name "conn" -ResourceGroupName "rg" -WorkspaceName "ws" `
  -AuthType "ApiKey" -Category "AzureOpenAI" -Target "https://myopenai.openai.azure.com"

Compute Reference

Complete reference for Azure ML compute targets including GPU SKUs, compute instances, compute clusters, serverless compute, Kubernetes compute, and debugging.

---

GPU VM SKU Reference for Azure ML

Compute Instance - Complete CLI Reference

# Create compute instance - standard
az ml compute create \
  --name dev-instance \
  --type ComputeInstance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_DS3_v2 \
  --enable-node-public-ip false \
  --idle-time-before-shutdown-minutes 30 \
  --ssh-public-access disabled

# Create GPU compute instance
az ml compute create \
  --name gpu-dev \
  --type ComputeInstance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_NC24ads_A100_v4 \
  --idle-time-before-shutdown-minutes 60 \
  --enable-node-public-ip false

# Create compute instance with setup script
az ml compute create \
  --name dev-custom \
  --type ComputeInstance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_DS3_v2 \
  --setup-scripts-creation-script "https://raw.githubusercontent.com/myorg/scripts/setup.sh"

# Start compute instance
az ml compute start \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# Stop compute instance
az ml compute stop \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# Restart compute instance
az ml compute restart \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# Show compute details
az ml compute show \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# List all compute
az ml compute list \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --output table

# List compute by type
az ml compute list \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --type ComputeInstance \
  --output table

# Connect via SSH (managed VNet with no public IP)
az ml compute connect-ssh \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# Delete compute instance
az ml compute delete \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --yes

# Update compute instance (e.g., idle shutdown)
az ml compute update \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --idle-time-before-shutdown-minutes 120

PowerShell equivalents:

# Create compute instance
New-AzMLWorkspaceCompute `
  -Name "dev-instance" `
  -ResourceGroupName "ml-rg" `
  -WorkspaceName "my-ml-workspace" `
  -Location "eastus" `
  -ComputeType "ComputeInstance" `
  -Property @{
    vmSize = "Standard_DS3_v2"
    sshSettings = @{sshPublicAccess = "Disabled"}
    idleTimeBeforeShutdown = "PT30M"
  }

# Get compute
Get-AzMLWorkspaceCompute -Name "dev-instance" -ResourceGroupName "ml-rg" -WorkspaceName "my-ml-workspace"

# Start compute
Start-AzMLWorkspaceCompute -Name "dev-instance" -ResourceGroupName "ml-rg" -WorkspaceName "my-ml-workspace"

# Stop compute
Stop-AzMLWorkspaceCompute -Name "dev-instance" -ResourceGroupName "ml-rg" -WorkspaceName "my-ml-workspace"

# Remove compute
Remove-AzMLWorkspaceCompute -Name "dev-instance" -ResourceGroupName "ml-rg" -WorkspaceName "my-ml-workspace"

Compute Cluster - Complete CLI Reference

# Create CPU cluster
az ml compute create \
  --name cpu-cluster \
  --type AmlCompute \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_DS3_v2 \
  --min-instances 0 \
  --max-instances 10 \
  --idle-time-before-scale-down 120 \
  --tier Dedicated \
  --enable-node-public-ip false \
  --identity-type SystemAssigned

# Create GPU cluster for training
az ml compute create \
  --name gpu-cluster \
  --type AmlCompute \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_NC24ads_A100_v4 \
  --min-instances 0 \
  --max-instances 4 \
  --idle-time-before-scale-down 300 \
  --tier Dedicated \
  --enable-node-public-ip false

# Create low-priority (spot) cluster for cost savings
az ml compute create \
  --name spot-gpu-cluster \
  --type AmlCompute \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_NC24ads_A100_v4 \
  --min-instances 0 \
  --max-instances 8 \
  --tier LowPriority

# Create cluster with user-assigned identity
az ml compute create \
  --name identity-cluster \
  --type AmlCompute \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_DS3_v2 \
  --min-instances 0 \
  --max-instances 4 \
  --identity-type UserAssigned \
  --user-assigned-identities /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ml-identity

# Create cluster in specific subnet (BYO VNet)
az ml compute create \
  --name subnet-cluster \
  --type AmlCompute \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --size Standard_DS3_v2 \
  --min-instances 0 \
  --max-instances 10 \
  --vnet-name ml-vnet \
  --subnet compute-subnet \
  --vnet-resource-group network-rg

# Update cluster (scale limits)
az ml compute update \
  --name gpu-cluster \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --min-instances 1 \
  --max-instances 8 \
  --idle-time-before-scale-down 600

YAML definition (compute-cluster.yml):

$schema: https://azuremlschemas.azureedge.net/latest/amlCompute.schema.json
name: gpu-cluster
type: amlcompute
size: Standard_NC24ads_A100_v4
min_instances: 0
max_instances: 4
idle_time_before_scale_down: 300
tier: dedicated
enable_node_public_ip: false
identity:
  type: system_assigned
tags:
  purpose: training
  team: data-science

# Create from YAML
az ml compute create --file compute-cluster.yml \
  --resource-group ml-rg --workspace-name my-ml-workspace

Serverless Compute

On-demand compute without cluster management. Azure provisions and deprovisions automatically.

# job.yml with serverless compute
$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
command: python train.py --epochs 50 --lr 0.001
environment: azureml:AzureML-sklearn-1.5-ubuntu22.04-py311-cpu:1
resources:
  instance_type: Standard_NC24ads_A100_v4
  instance_count: 1
queue_settings:
  job_tier: Standard

Kubernetes Compute (AKS / Arc-enabled)

# Attach existing AKS cluster
az ml compute attach \
  --name aks-compute \
  --type Kubernetes \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --resource-id /subscriptions/<sub>/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/my-aks \
  --namespace azureml

# Install Azure ML extension on AKS
az k8s-extension create \
  --name azureml \
  --extension-type Microsoft.AzureML.Kubernetes \
  --scope cluster \
  --cluster-name my-aks \
  --resource-group aks-rg \
  --cluster-type managedClusters \
  --configuration-settings \
    enableTraining=True \
    enableInference=True \
    inferenceRouterServiceType=LoadBalancer \
    allowInsecureConnections=False \
    InferenceRouterHA=True

# Verify extension
az k8s-extension show \
  --name azureml \
  --cluster-name my-aks \
  --resource-group aks-rg \
  --cluster-type managedClusters

# Detach compute
az ml compute detach \
  --name aks-compute \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

Compute Instance Logs and Debugging

# Get compute instance details (includes state, errors)
az ml compute show \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --query "{state:properties.state, errors:properties.errors, applications:properties.applications}" \
  --output json

# Check compute instance provisioning state
az ml compute show \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --query "properties.provisioningState" -o tsv

# SSH into compute instance for direct log access
az ml compute connect-ssh \
  --name dev-instance \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# Once SSHed in, check system logs:
# journalctl -u azureml -n 100
# cat /var/log/azureml/*.log
# cat /mnt/batch/tasks/startup/stdout.txt
# cat /mnt/batch/tasks/startup/stderr.txt
# df -h   (check disk space)
# nvidia-smi  (GPU status)
# top / htop  (process status)

# Check compute cluster node status
az ml compute show \
  --name gpu-cluster \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --query "{currentNodeCount:properties.currentNodeCount, targetNodeCount:properties.targetNodeCount, allocationState:properties.allocationState, errors:properties.errors}" \
  --output json

# Use Activity Log for compute events
az monitor activity-log list \
  --resource-group ml-rg \
  --resource-type Microsoft.MachineLearningServices/workspaces/computes \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --output table

Endpoints Reference

Complete reference for Azure ML endpoint deployment including managed online endpoints, batch endpoints, Kubernetes endpoints, serverless endpoints, and deployment logs.

---

Managed Online Endpoints (Recommended for Real-Time Inference)

# Create endpoint
az ml online-endpoint create \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --auth-mode key

# Create endpoint with managed identity auth
az ml online-endpoint create \
  --name secure-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --auth-mode aml_token

# Create deployment (blue)
az ml online-deployment create \
  --name blue \
  --endpoint-name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --model azureml:my-model@latest \
  --code-configuration code=./scoring scoring_script=score.py \
  --environment azureml:my-env@latest \
  --instance-type Standard_DS3_v2 \
  --instance-count 2 \
  --all-traffic

# Create deployment (green) for blue-green
az ml online-deployment create \
  --name green \
  --endpoint-name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --model azureml:my-model@2 \
  --code-configuration code=./scoring scoring_script=score.py \
  --environment azureml:my-env@latest \
  --instance-type Standard_DS3_v2 \
  --instance-count 2

# Split traffic between deployments
az ml online-endpoint update \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --traffic "blue=90 green=10"

# Shift all traffic to green
az ml online-endpoint update \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --traffic "blue=0 green=100"

# Test endpoint
az ml online-endpoint invoke \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --request-file sample-request.json

# Test specific deployment (mirror traffic)
az ml online-endpoint invoke \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --deployment-name green \
  --request-file sample-request.json

# Get endpoint scoring URI and keys
az ml online-endpoint show \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --query "scoring_uri" -o tsv

az ml online-endpoint get-credentials \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# List endpoints
az ml online-endpoint list \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --output table

# Delete deployment then endpoint
az ml online-deployment delete \
  --name blue \
  --endpoint-name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --yes

az ml online-endpoint delete \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --yes

Endpoint Deployment Logs - All Container Types

# Get inference server logs (default)
az ml online-deployment get-logs \
  --name blue \
  --endpoint-name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --lines 500

# Get storage initializer logs (model download phase)
az ml online-deployment get-logs \
  --name blue \
  --endpoint-name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --container storage-initializer \
  --lines 200

# Get inference server container logs specifically
az ml online-deployment get-logs \
  --name blue \
  --endpoint-name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --container inference-server \
  --lines 200

# Check endpoint provisioning state
az ml online-endpoint show \
  --name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --query "{state:provisioning_state, traffic:traffic}" -o json

# Check deployment provisioning state
az ml online-deployment show \
  --name blue \
  --endpoint-name my-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --query "{state:provisioning_state, instanceType:instance_type, instanceCount:instance_count}" -o json

# Use Log Analytics for comprehensive endpoint logs
az monitor log-analytics query \
  --workspace <log-analytics-workspace-id> \
  --analytics-query "
    AmlOnlineEndpointConsoleLog
    | where TimeGenerated > ago(1h)
    | where EndpointName == 'my-endpoint'
    | where DeploymentName == 'blue'
    | project TimeGenerated, Message, ContainerName
    | order by TimeGenerated desc
    | take 100
  " \
  --output table

# Query endpoint traffic metrics
az monitor log-analytics query \
  --workspace <log-analytics-workspace-id> \
  --analytics-query "
    AmlOnlineEndpointTrafficLog
    | where TimeGenerated > ago(24h)
    | where EndpointName == 'my-endpoint'
    | summarize RequestCount=count(), AvgLatencyMs=avg(RequestDuration) by bin(TimeGenerated, 1h), DeploymentName
    | order by TimeGenerated desc
  " \
  --output table

Batch Endpoints

# Create batch endpoint
az ml batch-endpoint create \
  --name my-batch-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# Create batch deployment
az ml batch-deployment create \
  --name batch-v1 \
  --endpoint-name my-batch-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --model azureml:my-model@latest \
  --compute azureml:cpu-cluster \
  --instance-count 4 \
  --mini-batch-size 100 \
  --max-concurrency-per-instance 2 \
  --output-action append_row \
  --output-file-name predictions.csv \
  --set-default

# Invoke batch scoring
az ml batch-endpoint invoke \
  --name my-batch-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --input azureml:my-scoring-data@latest

# Invoke with inline data
az ml batch-endpoint invoke \
  --name my-batch-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --input-type uri_folder \
  --input "https://mystorage.blob.core.windows.net/data/scoring/"

# Check batch job status
az ml batch-endpoint list-jobs \
  --name my-batch-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --output table

Kubernetes Online Endpoints

# Create Kubernetes online endpoint
az ml online-endpoint create \
  --name k8s-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --auth-mode key

# Create Kubernetes deployment
az ml online-deployment create \
  --name k8s-deploy \
  --endpoint-name k8s-endpoint \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --model azureml:my-model@latest \
  --code-configuration code=./scoring scoring_script=score.py \
  --environment azureml:my-env@latest \
  --compute azureml:aks-compute \
  --instance-type "defaultInstanceType" \
  --instance-count 2 \
  --all-traffic

Serverless Endpoints (Model as a Service)

# Deploy model from catalog as serverless
az ml serverless-endpoint create \
  --name phi3-serverless \
  --model-id azureml://registries/azureml/models/Phi-3-medium-128k-instruct \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# Get credentials
az ml serverless-endpoint get-credentials \
  --name phi3-serverless \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

# List serverless endpoints
az ml serverless-endpoint list \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace \
  --output table

# Delete serverless endpoint
az ml serverless-endpoint delete \
  --name phi3-serverless \
  --resource-group ml-rg \
  --workspace-name my-ml-workspace

Identities, ACR, and Storage Reference

Complete reference for Azure ML managed identities, ACR integration, and storage account configuration.

---

5. MANAGED IDENTITIES - COMPLETE REFERENCE

Identity Types and When to Use Them

Workspace Identity Configuration

# Create workspace with system-assigned identity
az ml workspace create \
  --name my-workspace \
  --resource-group ml-rg \
  --location eastus \
  --identity-type SystemAssigned

# Create workspace with user-assigned identity
az identity create \
  --name ml-workspace-identity \
  --resource-group ml-rg

az ml workspace create \
  --name my-workspace \
  --resource-group ml-rg \
  --location eastus \
  --identity-type UserAssigned \
  --user-assigned-identities /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ml-workspace-identity \
  --primary-user-assigned-identity /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ml-workspace-identity

Required Role Assignments

Workspace system-assigned identity (auto-granted for workspaces created after Nov 2024):

WS_IDENTITY=$(az ml workspace show -n my-workspace -g ml-rg --query identity.principalId -o tsv)

# Azure AI Administrator on resource group (auto for new workspaces)
az role assignment create \
  --assignee $WS_IDENTITY \
  --role "Azure AI Administrator" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg

# If using older workspace (pre Nov 2024), Contributor was auto-assigned
# For fine-grained control, assign these individually:

# Storage Blob Data Contributor on storage account
az role assignment create \
  --assignee $WS_IDENTITY \
  --role "Storage Blob Data Contributor" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Storage/storageAccounts/mlstorage

# Storage File Data Privileged Contributor on storage account
az role assignment create \
  --assignee $WS_IDENTITY \
  --role "Storage File Data Privileged Contributor" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Storage/storageAccounts/mlstorage

# Key Vault Administrator on key vault
az role assignment create \
  --assignee $WS_IDENTITY \
  --role "Key Vault Administrator" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.KeyVault/vaults/mlkeyvault

# AcrPush on container registry (for building/pushing environment images)
az role assignment create \
  --assignee $WS_IDENTITY \
  --role "AcrPush" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.ContainerRegistry/registries/mlacr

# Contributor on Application Insights
az role assignment create \
  --assignee $WS_IDENTITY \
  --role "Contributor" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Insights/components/mlinsights

Compute cluster/instance identity for data access:

COMPUTE_IDENTITY=$(az ml compute show -n gpu-cluster -g ml-rg -w my-workspace --query identity.principalId -o tsv)

# Storage Blob Data Reader (minimum for reading training data)
az role assignment create \
  --assignee $COMPUTE_IDENTITY \
  --role "Storage Blob Data Reader" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Storage/storageAccounts/mlstorage

# Storage Blob Data Contributor (if compute needs write access)
az role assignment create \
  --assignee $COMPUTE_IDENTITY \
  --role "Storage Blob Data Contributor" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Storage/storageAccounts/mlstorage

# AcrPull (for pulling Docker images from workspace ACR)
az role assignment create \
  --assignee $COMPUTE_IDENTITY \
  --role "AcrPull" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.ContainerRegistry/registries/mlacr

Managed online endpoint identity for deployment:

ENDPOINT_IDENTITY=$(az ml online-endpoint show -n my-endpoint -g ml-rg -w my-workspace --query identity.principalId -o tsv)

# AcrPull on ACR (to pull model serving images)
az role assignment create \
  --assignee $ENDPOINT_IDENTITY \
  --role "AcrPull" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.ContainerRegistry/registries/mlacr

# Storage Blob Data Reader (to access model artifacts)
az role assignment create \
  --assignee $ENDPOINT_IDENTITY \
  --role "Storage Blob Data Reader" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.Storage/storageAccounts/mlstorage

# AzureML Workspace Connection Secrets Reader (to access workspace connections/secrets)
az role assignment create \
  --assignee $ENDPOINT_IDENTITY \
  --role "Azure Machine Learning Workspace Connection Secrets Reader" \
  --scope /subscriptions/<sub>/resourceGroups/ml-rg/providers/Microsoft.MachineLearningServices/workspaces/my-workspace

User RBAC Roles for Azure ML

---

6. ACR INTEGRATION - COMPLETE REFERENCE

How Azure ML Uses ACR

Azure ML uses ACR to store Docker images for:

• Custom environments (Docker + conda builds)
• Model serving containers
• Training job containers
• Pipeline step containers

The workspace creates or associates an ACR. Environment builds push images there.

# Check workspace ACR
az ml workspace show \
  --name my-workspace \
  --resource-group ml-rg \
  --query "container_registry" -o tsv

# Create environment (triggers image build to ACR)
az ml environment create \
  --file environment.yml \
  --resource-group ml-rg \
  --workspace-name my-workspace

# List environments
az ml environment list \
  --resource-group ml-rg \
  --workspace-name my-workspace \
  --output table

# Show environment details (includes Docker image URI in ACR)
az ml environment show \
  --name my-env \
  --version 1 \
  --resource-group ml-rg \
  --workspace-name my-workspace

Image Build Compute (Required for Private ACR)

When ACR is behind a VNet, Azure ML cannot build images directly. You must configure a compute cluster for image builds.

# Set image build compute
az ml workspace update \
  --name my-workspace \
  --resource-group ml-rg \
  --image-build-compute cpu-build-cluster

# The cpu-build-cluster must:
# 1. Be in the same workspace
# 2. Be a CPU cluster (GPU not needed for Docker builds)
# 3. Have network access to ACR
# 4. Have AcrPull + AcrPush roles on the workspace ACR

Using a Private External ACR

# Create user-assigned identity for ACR access
az identity create \
  --name acr-pull-identity \
  --resource-group ml-rg

ACR_IDENTITY_ID=$(az identity show -n acr-pull-identity -g ml-rg --query id -o tsv)
ACR_IDENTITY_PRINCIPAL=$(az identity show -n acr-pull-identity -g ml-rg --query principalId -o tsv)

# Grant AcrPull on the external private ACR
az role assignment create \
  --assignee $ACR_IDENTITY_PRINCIPAL \
  --role AcrPull \
  --scope /subscriptions/<sub>/resourceGroups/acr-rg/providers/Microsoft.ContainerRegistry/registries/externalacr

# Grant workspace identity Managed Identity Operator on the pull identity
WS_IDENTITY=$(az ml workspace show -n my-workspace -g ml-rg --query identity.principalId -o tsv)
az role assignment create \
  --assignee $WS_IDENTITY \
  --role "Managed Identity Operator" \
  --scope $ACR_IDENTITY_ID

# Use external ACR image in environment
cat <<EOF > env-external-acr.yml
\$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: external-acr-env
version: 1
image: externalacr.azurecr.io/my-base-image:latest
inference_config:
  liveness_route:
    path: /health
    port: 8080
  readiness_route:
    path: /ready
    port: 8080
  scoring_route:
    path: /score
    port: 8080
EOF

az ml environment create --file env-external-acr.yml \
  --resource-group ml-rg --workspace-name my-workspace

Troubleshooting ACR/Environment Build Issues

# Check environment build status
az ml environment show \
  --name my-env --version 1 \
  --resource-group ml-rg --workspace-name my-workspace \
  --query "build_context"

# View ACR build logs
az acr task logs \
  --registry mlacr \
  --resource-group ml-rg

# List images in workspace ACR
az acr repository list \
  --name mlacr \
  --output table

# Show image tags
az acr repository show-tags \
  --name mlacr \
  --repository azureml/azureml_<env-hash> \
  --output table

# Check ACR access from compute
# SSH into compute instance, then:
# az acr login --name mlacr
# docker pull mlacr.azurecr.io/azureml/azureml_<hash>:latest

---

7. STORAGE ACCOUNTS - COMPLETE REFERENCE

Default Workspace Storage

Azure ML uses the workspace storage account for:

• Default datastore (blob container azureml-blobstore-<guid>)
• File share for notebooks (code-<guid>)
• MLflow tracking artifacts
• Job outputs and logs
• Pipeline intermediate data
• Model artifacts before registration

# Check workspace default storage
az ml workspace show \
  --name my-workspace \
  --resource-group ml-rg \
  --query "storage_account" -o tsv

# List datastores (shows connection to storage)
az ml datastore list \
  --resource-group ml-rg \
  --workspace-name my-workspace \
  --output table

# Show default datastore
az ml datastore show \
  --name workspaceblobstore \
  --resource-group ml-rg \
  --workspace-name my-workspace

Register Additional Datastores

# Register Azure Blob datastore (identity-based access - recommended)
az ml datastore create \
  --file blob-datastore.yml \
  --resource-group ml-rg \
  --workspace-name my-workspace

blob-datastore.yml:

$schema: https://azuremlschemas.azureedge.net/latest/azureBlob.schema.json
name: training-data-store
type: azure_blob
account_name: trainingdatastorage
container_name: ml-datasets
credentials:
  # Option 1: Identity-based (recommended - no keys stored)
  # Leave credentials empty, configure RBAC instead
  # Option 2: Account key
  # account_key: "<key>"
  # Option 3: SAS token
  # sas_token: "<sas>"

# Register ADLS Gen2 datastore
az ml datastore create \
  --name adls-datastore \
  --type azure_data_lake_gen2 \
  --resource-group ml-rg \
  --workspace-name my-workspace \
  --account-name mydatalakeaccount \
  --filesystem ml-filesystem

# Register Azure File share datastore
az ml datastore create \
  --name fileshare-datastore \
  --type azure_file \
  --resource-group ml-rg \
  --workspace-name my-workspace \
  --account-name mystorage \
  --file-share-name ml-files \
  --account-key "<key>"

# Test datastore connectivity
az ml datastore show \
  --name training-data-store \
  --resource-group ml-rg \
  --workspace-name my-workspace \
  --query "credentials"

Networking Reference

Complete reference for Azure ML networking, VNet integration, private endpoints, DNS zones, NSG rules, and service tags.

---

Network Isolation Modes

Azure ML supports three managed network isolation modes:

Managed Virtual Network (Recommended)

Azure ML creates and manages a VNet for you. All computes (instances, clusters, serverless, managed endpoints) run inside this managed VNet.

# Create workspace with managed VNet - AllowInternetOutbound
az ml workspace create \
  --name secure-workspace \
  --resource-group ml-rg \
  --location eastus \
  --managed-network AllowInternetOutbound

# Create workspace with managed VNet - AllowOnlyApprovedOutbound
az ml workspace create \
  --name lockdown-workspace \
  --resource-group ml-rg \
  --location eastus \
  --managed-network AllowOnlyApprovedOutbound

# Provision managed network (creates private endpoints to dependencies)
az ml workspace provision-network \
  --name secure-workspace \
  --resource-group ml-rg \
  --include-spark

# Add outbound rule - private endpoint to external storage
az ml workspace outbound-rule set \
  --name secure-workspace \
  --resource-group ml-rg \
  --rule-name external-storage-pe \
  --type private_endpoint \
  --service-resource-id /subscriptions/<sub>/resourceGroups/data-rg/providers/Microsoft.Storage/storageAccounts/externalstorage \
  --sub-resource-target blob \
  --spark-enabled false

# Add outbound rule - FQDN (for AllowOnlyApprovedOutbound)
az ml workspace outbound-rule set \
  --name lockdown-workspace \
  --resource-group ml-rg \
  --rule-name pypi-access \
  --type fqdn \
  --destination "pypi.org"

# Add outbound rule - service tag
az ml workspace outbound-rule set \
  --name secure-workspace \
  --resource-group ml-rg \
  --rule-name azure-monitor \
  --type service_tag \
  --service-tag AzureMonitor \
  --protocol TCP \
  --port-ranges "443"

# List outbound rules
az ml workspace outbound-rule list \
  --name secure-workspace \
  --resource-group ml-rg \
  --output table

# Remove outbound rule
az ml workspace outbound-rule remove \
  --name secure-workspace \
  --resource-group ml-rg \
  --rule-name external-storage-pe

Private Endpoints for Workspace

A private endpoint on your own VNet provides inbound connectivity to the workspace.

# Create VNet and subnet for private endpoint
az network vnet create \
  --name ml-vnet \
  --resource-group ml-rg \
  --address-prefix 10.0.0.0/16 \
  --subnet-name pe-subnet \
  --subnet-prefix 10.0.1.0/24

# Disable private endpoint network policies on subnet
az network vnet subnet update \
  --name pe-subnet \
  --resource-group ml-rg \
  --vnet-name ml-vnet \
  --disable-private-endpoint-network-policies true

# Create private endpoint for workspace
az network private-endpoint create \
  --name ml-workspace-pe \
  --resource-group ml-rg \
  --vnet-name ml-vnet \
  --subnet pe-subnet \
  --private-connection-resource-id $(az ml workspace show -n my-ml-workspace -g ml-rg --query id -o tsv) \
  --group-id amlworkspace \
  --connection-name ml-workspace-connection

# Create private DNS zone
az network private-dns zone create \
  --resource-group ml-rg \
  --name privatelink.api.azureml.ms

az network private-dns zone create \
  --resource-group ml-rg \
  --name privatelink.notebooks.azure.net

# Link DNS zone to VNet
az network private-dns link vnet create \
  --resource-group ml-rg \
  --zone-name privatelink.api.azureml.ms \
  --name ml-dns-link \
  --virtual-network ml-vnet \
  --registration-enabled false

az network private-dns link vnet create \
  --resource-group ml-rg \
  --zone-name privatelink.notebooks.azure.net \
  --name ml-notebooks-dns-link \
  --virtual-network ml-vnet \
  --registration-enabled false

# Create DNS zone group for automatic DNS record management
az network private-endpoint dns-zone-group create \
  --resource-group ml-rg \
  --endpoint-name ml-workspace-pe \
  --name default \
  --private-dns-zone privatelink.api.azureml.ms \
  --zone-name api

az network private-endpoint dns-zone-group add \
  --resource-group ml-rg \
  --endpoint-name ml-workspace-pe \
  --name default \
  --private-dns-zone privatelink.notebooks.azure.net \
  --zone-name notebooks

Required Private DNS Zones for Azure ML

NSG Rules for Azure ML Compute

When using BYO VNet (not managed network), the following NSG rules are required:

# Create NSG
az network nsg create \
  --name ml-compute-nsg \
  --resource-group ml-rg

# INBOUND rules
# Allow Azure Machine Learning service (for compute management)
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowAzureMLInbound \
  --priority 100 \
  --direction Inbound \
  --source-address-prefixes AzureMachineLearning \
  --destination-port-ranges 44224 \
  --protocol TCP \
  --access Allow

# Allow Azure Batch management
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowBatchNodeManagement \
  --priority 110 \
  --direction Inbound \
  --source-address-prefixes BatchNodeManagement \
  --destination-port-ranges 29876-29877 \
  --protocol TCP \
  --access Allow

# OUTBOUND rules
# Allow Azure Storage (for data access)
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowStorageOutbound \
  --priority 100 \
  --direction Outbound \
  --destination-address-prefixes Storage \
  --destination-port-ranges 443 \
  --protocol TCP \
  --access Allow

# Allow Azure ML service
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowAzureMLOutbound \
  --priority 110 \
  --direction Outbound \
  --destination-address-prefixes AzureMachineLearning \
  --destination-port-ranges 443 \
  --protocol TCP \
  --access Allow

# Allow Azure Active Directory
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowAADOutbound \
  --priority 120 \
  --direction Outbound \
  --destination-address-prefixes AzureActiveDirectory \
  --destination-port-ranges 443 \
  --protocol TCP \
  --access Allow

# Allow Azure Resource Manager
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowARMOutbound \
  --priority 130 \
  --direction Outbound \
  --destination-address-prefixes AzureResourceManager \
  --destination-port-ranges 443 \
  --protocol TCP \
  --access Allow

# Allow Azure Container Registry
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowACROutbound \
  --priority 140 \
  --direction Outbound \
  --destination-address-prefixes AzureContainerRegistry \
  --destination-port-ranges 443 \
  --protocol TCP \
  --access Allow

# Allow Azure Key Vault
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowKeyVaultOutbound \
  --priority 150 \
  --direction Outbound \
  --destination-address-prefixes AzureKeyVault \
  --destination-port-ranges 443 \
  --protocol TCP \
  --access Allow

# Allow Azure Monitor (for logging)
az network nsg rule create \
  --nsg-name ml-compute-nsg \
  --resource-group ml-rg \
  --name AllowAzureMonitorOutbound \
  --priority 160 \
  --direction Outbound \
  --destination-address-prefixes AzureMonitor \
  --destination-port-ranges 443 \
  --protocol TCP \
  --access Allow

Service Tags Reference for Azure ML

Terraform Reference

Complete Terraform configuration for production-ready Azure ML workspace deployment.

---

Full Production-Ready Terraform Configuration

# ========================================
# providers.tf
# ========================================
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
    azapi = {
      source  = "azure/azapi"
      version = "~> 2.0"
    }
  }
}

provider "azurerm" {
  features {
    key_vault {
      purge_soft_delete_on_destroy = false
    }
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

# ========================================
# data.tf
# ========================================
data "azurerm_client_config" "current" {}

# ========================================
# variables.tf
# ========================================
variable "resource_group_name" {
  type    = string
  default = "ml-production-rg"
}

variable "location" {
  type    = string
  default = "eastus"
}

variable "workspace_name" {
  type    = string
  default = "ml-prod-workspace"
}

variable "environment" {
  type    = string
  default = "production"
}

variable "managed_network_isolation_mode" {
  type        = string
  default     = "AllowInternetOutbound"
  description = "Disabled, AllowInternetOutbound, or AllowOnlyApprovedOutbound"
}

# ========================================
# resource-group.tf
# ========================================
resource "azurerm_resource_group" "ml" {
  name     = var.resource_group_name
  location = var.location
  tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
    Purpose     = "Machine Learning"
  }
}

# ========================================
# networking.tf
# ========================================
resource "azurerm_virtual_network" "ml" {
  name                = "${var.workspace_name}-vnet"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "private_endpoints" {
  name                 = "private-endpoints"
  resource_group_name  = azurerm_resource_group.ml.name
  virtual_network_name = azurerm_virtual_network.ml.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_subnet" "compute" {
  name                 = "compute"
  resource_group_name  = azurerm_resource_group.ml.name
  virtual_network_name = azurerm_virtual_network.ml.name
  address_prefixes     = ["10.0.2.0/24"]
}

# NSG for compute subnet
resource "azurerm_network_security_group" "compute" {
  name                = "${var.workspace_name}-compute-nsg"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name

  security_rule {
    name                       = "AllowAzureMLInbound"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "44224"
    source_address_prefix      = "AzureMachineLearning"
    destination_address_prefix = "*"
  }

  security_rule {
    name                       = "AllowBatchInbound"
    priority                   = 110
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_ranges    = ["29876", "29877"]
    source_address_prefix      = "BatchNodeManagement"
    destination_address_prefix = "*"
  }

  security_rule {
    name                       = "AllowAzureServicesOutbound"
    priority                   = 100
    direction                  = "Outbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "443"
    source_address_prefix      = "*"
    destination_address_prefixes = [
      "AzureMachineLearning",
      "Storage",
      "AzureActiveDirectory",
      "AzureResourceManager",
      "AzureContainerRegistry",
      "AzureKeyVault",
      "AzureMonitor",
      "AzureFrontDoor.Frontend",
      "MicrosoftContainerRegistry",
    ]
  }
}

resource "azurerm_subnet_network_security_group_association" "compute" {
  subnet_id                 = azurerm_subnet.compute.id
  network_security_group_id = azurerm_network_security_group.compute.id
}

# Private DNS Zones
locals {
  private_dns_zones = [
    "privatelink.api.azureml.ms",
    "privatelink.notebooks.azure.net",
    "privatelink.blob.core.windows.net",
    "privatelink.file.core.windows.net",
    "privatelink.vaultcore.azure.net",
    "privatelink.azurecr.io",
    "privatelink.monitor.azure.com",
  ]
}

resource "azurerm_private_dns_zone" "zones" {
  for_each            = toset(local.private_dns_zones)
  name                = each.value
  resource_group_name = azurerm_resource_group.ml.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "links" {
  for_each              = toset(local.private_dns_zones)
  name                  = "${replace(each.value, ".", "-")}-link"
  resource_group_name   = azurerm_resource_group.ml.name
  private_dns_zone_name = azurerm_private_dns_zone.zones[each.value].name
  virtual_network_id    = azurerm_virtual_network.ml.id
  registration_enabled  = false
}

# ========================================
# storage.tf
# ========================================
resource "azurerm_storage_account" "ml" {
  name                            = "mlstorage${random_string.suffix.result}"
  location                        = azurerm_resource_group.ml.location
  resource_group_name             = azurerm_resource_group.ml.name
  account_tier                    = "Standard"
  account_replication_type        = "ZRS"
  account_kind                    = "StorageV2"
  min_tls_version                 = "TLS1_2"
  https_traffic_only_enabled      = true
  allow_nested_items_to_be_public = false
  shared_access_key_enabled       = true  # Required for Azure ML

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "random_string" "suffix" {
  length  = 8
  special = false
  upper   = false
}

# Private endpoint for blob
resource "azurerm_private_endpoint" "storage_blob" {
  name                = "${azurerm_storage_account.ml.name}-blob-pe"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "storage-blob-connection"
    private_connection_resource_id = azurerm_storage_account.ml.id
    subresource_names              = ["blob"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [azurerm_private_dns_zone.zones["privatelink.blob.core.windows.net"].id]
  }
}

# Private endpoint for file
resource "azurerm_private_endpoint" "storage_file" {
  name                = "${azurerm_storage_account.ml.name}-file-pe"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "storage-file-connection"
    private_connection_resource_id = azurerm_storage_account.ml.id
    subresource_names              = ["file"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [azurerm_private_dns_zone.zones["privatelink.file.core.windows.net"].id]
  }
}

# ========================================
# keyvault.tf
# ========================================
resource "azurerm_key_vault" "ml" {
  name                       = "mlkv${random_string.suffix.result}"
  location                   = azurerm_resource_group.ml.location
  resource_group_name        = azurerm_resource_group.ml.name
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  sku_name                   = "standard"
  purge_protection_enabled   = true
  soft_delete_retention_days = 90
  enable_rbac_authorization  = true

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
  }
}

resource "azurerm_private_endpoint" "keyvault" {
  name                = "${azurerm_key_vault.ml.name}-pe"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "keyvault-connection"
    private_connection_resource_id = azurerm_key_vault.ml.id
    subresource_names              = ["vault"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [azurerm_private_dns_zone.zones["privatelink.vaultcore.azure.net"].id]
  }
}

# ========================================
# acr.tf
# ========================================
resource "azurerm_container_registry" "ml" {
  name                = "mlacr${random_string.suffix.result}"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  sku                 = "Premium"  # Required for private endpoint
  admin_enabled       = true       # Required for Azure ML

  network_rule_set {
    default_action = "Deny"
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_private_endpoint" "acr" {
  name                = "${azurerm_container_registry.ml.name}-pe"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "acr-connection"
    private_connection_resource_id = azurerm_container_registry.ml.id
    subresource_names              = ["registry"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [azurerm_private_dns_zone.zones["privatelink.azurecr.io"].id]
  }
}

# ========================================
# application-insights.tf
# ========================================
resource "azurerm_log_analytics_workspace" "ml" {
  name                = "${var.workspace_name}-logs"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  sku                 = "PerGB2018"
  retention_in_days   = 90
}

resource "azurerm_application_insights" "ml" {
  name                = "${var.workspace_name}-appinsights"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  application_type    = "web"
  workspace_id        = azurerm_log_analytics_workspace.ml.id
}

# ========================================
# ml-workspace.tf
# ========================================
resource "azurerm_machine_learning_workspace" "ml" {
  name                          = var.workspace_name
  location                      = azurerm_resource_group.ml.location
  resource_group_name           = azurerm_resource_group.ml.name
  application_insights_id       = azurerm_application_insights.ml.id
  key_vault_id                  = azurerm_key_vault.ml.id
  storage_account_id            = azurerm_storage_account.ml.id
  container_registry_id         = azurerm_container_registry.ml.id
  public_network_access_enabled = false
  image_build_compute_name      = "cpu-build-cluster"
  high_business_impact          = true

  managed_network {
    isolation_mode = var.managed_network_isolation_mode
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
  }

  depends_on = [
    azurerm_private_endpoint.storage_blob,
    azurerm_private_endpoint.storage_file,
    azurerm_private_endpoint.keyvault,
    azurerm_private_endpoint.acr,
  ]
}

# Private endpoint for workspace
resource "azurerm_private_endpoint" "workspace" {
  name                = "${var.workspace_name}-pe"
  location            = azurerm_resource_group.ml.location
  resource_group_name = azurerm_resource_group.ml.name
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "workspace-connection"
    private_connection_resource_id = azurerm_machine_learning_workspace.ml.id
    subresource_names              = ["amlworkspace"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name = "default"
    private_dns_zone_ids = [
      azurerm_private_dns_zone.zones["privatelink.api.azureml.ms"].id,
      azurerm_private_dns_zone.zones["privatelink.notebooks.azure.net"].id,
    ]
  }
}

# ========================================
# ml-compute.tf
# ========================================
# CPU cluster for image builds and general training
resource "azurerm_machine_learning_compute_cluster" "cpu_build" {
  name                          = "cpu-build-cluster"
  location                      = azurerm_resource_group.ml.location
  machine_learning_workspace_id = azurerm_machine_learning_workspace.ml.id
  vm_priority                   = "Dedicated"
  vm_size                       = "Standard_DS3_v2"

  scale_settings {
    min_node_count                       = 0
    max_node_count                       = 4
    scale_down_nodes_after_idle_duration = "PT120S"
  }

  identity {
    type = "SystemAssigned"
  }
}

# GPU cluster for training
resource "azurerm_machine_learning_compute_cluster" "gpu_training" {
  name                          = "gpu-training-cluster"
  location                      = azurerm_resource_group.ml.location
  machine_learning_workspace_id = azurerm_machine_learning_workspace.ml.id
  vm_priority                   = "Dedicated"
  vm_size                       = "Standard_NC24ads_A100_v4"

  scale_settings {
    min_node_count                       = 0
    max_node_count                       = 4
    scale_down_nodes_after_idle_duration = "PT300S"
  }

  identity {
    type = "SystemAssigned"
  }
}

# Low-priority GPU cluster for cost savings
resource "azurerm_machine_learning_compute_cluster" "gpu_spot" {
  name                          = "gpu-spot-cluster"
  location                      = azurerm_resource_group.ml.location
  machine_learning_workspace_id = azurerm_machine_learning_workspace.ml.id
  vm_priority                   = "LowPriority"
  vm_size                       = "Standard_NC24ads_A100_v4"

  scale_settings {
    min_node_count                       = 0
    max_node_count                       = 8
    scale_down_nodes_after_idle_duration = "PT120S"
  }

  identity {
    type = "SystemAssigned"
  }
}

# Compute instance for development
resource "azurerm_machine_learning_compute_instance" "dev" {
  name                          = "dev-instance"
  machine_learning_workspace_id = azurerm_machine_learning_workspace.ml.id
  virtual_machine_size          = "Standard_DS3_v2"
  authorization_type            = "personal"
  description                   = "Development compute instance"

  assign_to_user {
    object_id = data.azurerm_client_config.current.object_id
    tenant_id = data.azurerm_client_config.current.tenant_id
  }
}

# ========================================
# role-assignments.tf
# ========================================
# Workspace identity -> Storage Blob Data Contributor
resource "azurerm_role_assignment" "ws_storage_blob" {
  scope                = azurerm_storage_account.ml.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_machine_learning_workspace.ml.identity[0].principal_id
}

# Workspace identity -> Storage File Data Privileged Contributor
resource "azurerm_role_assignment" "ws_storage_file" {
  scope                = azurerm_storage_account.ml.id
  role_definition_name = "Storage File Data Privileged Contributor"
  principal_id         = azurerm_machine_learning_workspace.ml.identity[0].principal_id
}

# Workspace identity -> Key Vault Administrator
resource "azurerm_role_assignment" "ws_keyvault" {
  scope                = azurerm_key_vault.ml.id
  role_definition_name = "Key Vault Administrator"
  principal_id         = azurerm_machine_learning_workspace.ml.identity[0].principal_id
}

# Workspace identity -> AcrPush on ACR
resource "azurerm_role_assignment" "ws_acr_push" {
  scope                = azurerm_container_registry.ml.id
  role_definition_name = "AcrPush"
  principal_id         = azurerm_machine_learning_workspace.ml.identity[0].principal_id
}

# GPU cluster identity -> AcrPull
resource "azurerm_role_assignment" "gpu_acr_pull" {
  scope                = azurerm_container_registry.ml.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_machine_learning_compute_cluster.gpu_training.identity[0].principal_id
}

# GPU cluster identity -> Storage Blob Data Reader
resource "azurerm_role_assignment" "gpu_storage_reader" {
  scope                = azurerm_storage_account.ml.id
  role_definition_name = "Storage Blob Data Reader"
  principal_id         = azurerm_machine_learning_compute_cluster.gpu_training.identity[0].principal_id
}

# ========================================
# outbound-rules.tf (for managed network)
# ========================================
resource "azurerm_machine_learning_workspace_network_outbound_rule_private_endpoint" "external_storage" {
  count                = var.managed_network_isolation_mode != "Disabled" ? 1 : 0
  name                 = "external-data-storage"
  workspace_id         = azurerm_machine_learning_workspace.ml.id
  service_resource_id  = azurerm_storage_account.ml.id  # or external storage
  sub_resource_target  = "blob"
}

# ========================================
# outputs.tf
# ========================================
output "workspace_id" {
  value = azurerm_machine_learning_workspace.ml.id
}

output "workspace_name" {
  value = azurerm_machine_learning_workspace.ml.name
}

output "workspace_discovery_url" {
  value = azurerm_machine_learning_workspace.ml.discovery_url
}

output "storage_account_name" {
  value = azurerm_storage_account.ml.name
}

output "acr_login_server" {
  value = azurerm_container_registry.ml.login_server
}

output "key_vault_uri" {
  value = azurerm_key_vault.ml.vault_uri
}

All Terraform Resources for Azure ML

Troubleshooting Reference

Complete reference for Azure ML log reading, debugging, error resolution, and workspace setup checklist.

---

10. LOG READING AND DEBUGGING - COMPLETE REFERENCE

Job Logs

# Stream logs in real-time during execution
az ml job stream --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace

# View job status and details
az ml job show --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace \
  --query "{status:status, error:error, startTime:properties.startTime, endTime:properties.endTime}" -o json

# Download job logs
az ml job download --name <job-name> \
  --resource-group ml-rg --workspace-name my-workspace \
  --output-name logs \
  --download-path ./job-logs

# Query job logs via Log Analytics
az monitor log-analytics query \
  --workspace <workspace-id> \
  --analytics-query "
    AmlComputeJobEvent
    | where TimeGenerated > ago(24h)
    | where JobName == '<job-name>'
    | project TimeGenerated, EventType, Message
    | order by TimeGenerated desc
  " --output table

Compute Instance Logs

# View compute instance state and errors
az ml compute show --name dev-instance \
  --resource-group ml-rg --workspace-name my-workspace \
  --query "{state:properties.state, errors:properties.errors, lastOperation:properties.lastOperation}" -o json

# SSH into compute instance for deep debugging
az ml compute connect-ssh --name dev-instance \
  --resource-group ml-rg --workspace-name my-workspace

# On the compute instance, check these log locations:
# /var/log/azureml/                     - Azure ML agent logs
# /var/log/azureml/azureml_setup.log    - Setup/provisioning logs
# /mnt/batch/tasks/startup/stdout.txt   - Startup script output
# /mnt/batch/tasks/startup/stderr.txt   - Startup script errors
# /var/log/syslog                       - System logs
# /var/log/kern.log                     - Kernel logs (GPU driver issues)
# ~/.azureml/logs/                      - SDK-level logs
# nvidia-smi                            - GPU utilization and errors
# dmesg | grep -i gpu                   - GPU kernel messages

# Query compute events in Log Analytics
az monitor log-analytics query \
  --workspace <workspace-id> \
  --analytics-query "
    AmlComputeClusterEvent
    | where TimeGenerated > ago(24h)
    | where ComputeName == 'dev-instance'
    | project TimeGenerated, EventType, Message, NodeId
    | order by TimeGenerated desc
  " --output table

Endpoint Deployment Logs

# Inference server logs (default - shows score.py output)
az ml online-deployment get-logs \
  --name blue --endpoint-name my-endpoint \
  --resource-group ml-rg --workspace-name my-workspace \
  --lines 500

# Storage initializer logs (model download, environment setup)
az ml online-deployment get-logs \
  --name blue --endpoint-name my-endpoint \
  --resource-group ml-rg --workspace-name my-workspace \
  --container storage-initializer --lines 200

# Query Log Analytics for comprehensive endpoint logs
az monitor log-analytics query \
  --workspace <workspace-id> \
  --analytics-query "
    AmlOnlineEndpointConsoleLog
    | where TimeGenerated > ago(1h)
    | where EndpointName == 'my-endpoint'
    | project TimeGenerated, Message, ContainerName, InstanceId
    | order by TimeGenerated desc
    | take 200
  " --output table

# Monitor endpoint request metrics
az monitor log-analytics query \
  --workspace <workspace-id> \
  --analytics-query "
    AmlOnlineEndpointTrafficLog
    | where TimeGenerated > ago(24h)
    | where EndpointName == 'my-endpoint'
    | summarize
        TotalRequests = count(),
        SuccessRate = countif(ResponseCode >= 200 and ResponseCode < 300) * 100.0 / count(),
        AvgLatencyMs = avg(RequestDuration),
        P99LatencyMs = percentile(RequestDuration, 99)
      by bin(TimeGenerated, 1h), DeploymentName
    | order by TimeGenerated desc
  " --output table

# Check endpoint health metrics
az monitor metrics list \
  --resource $(az ml online-endpoint show -n my-endpoint -g ml-rg -w my-workspace --query id -o tsv) \
  --metric "RequestsPerMinute,RequestLatency,NewConnectionsPerSecond" \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \
  --interval PT1M \
  --output table

Workspace Diagnostics

# Run workspace diagnostic check
az ml workspace diagnose \
  --name my-workspace \
  --resource-group ml-rg

# This checks:
# - Storage account connectivity
# - Key vault accessibility
# - ACR connectivity
# - Application Insights configuration
# - Network configuration
# - DNS resolution
# - NSG rules
# - Private endpoint status

# Check workspace Activity Log for errors
az monitor activity-log list \
  --resource-group ml-rg \
  --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ) \
  --query "[?contains(operationName.value, 'MachineLearningServices')].{Time:eventTimestamp, Operation:operationName.value, Status:status.value, Message:properties.statusMessage}" \
  --output table

---

12. TROUBLESHOOTING - COMPREHENSIVE ERROR REFERENCE

Compute Errors

Endpoint Deployment Errors

Networking Errors

Job Errors

---

QUICK REFERENCE: SETTING UP A COMPLETE SECURE ML WORKSPACE

Step-by-step checklist for a production-ready, network-isolated Azure ML workspace:

1. Create VNet with subnets for private endpoints and compute 2. Create NSG with required service tag rules for compute subnet 3. Create storage account with firewall (Deny default), private endpoints for blob + file 4. Create key vault with RBAC enabled, purge protection, private endpoint 5. Create ACR (Premium SKU) with firewall, private endpoint 6. Create Application Insights with Log Analytics workspace 7. Create all required private DNS zones and link to VNet 8. Create ML workspace with managed-network=AllowInternetOutbound, private endpoint 9. Run az ml workspace provision-network to provision managed network 10. Create CPU cluster for image builds (set as image-build-compute) 11. Create GPU cluster for training 12. Assign RBAC roles: workspace identity -> storage, KV, ACR; compute identity -> storage, ACR 13. Run az ml workspace diagnose to verify configuration 14. Create environment, data assets, and test a simple training job 15. Create managed online endpoint and test inference

References

• Azure ML Documentation
• Azure AI Foundry Documentation
• az ml CLI Reference
• Az.MachineLearningServices PowerShell Module
• Terraform AzureRM ML Workspace
• Terraform AzureRM ML Compute Cluster
• Secure Azure ML Workspace with VNet
• Managed Network Isolation
• Troubleshoot Online Endpoints
• Azure ML Network Isolation Planning
• Azure ML RBAC Roles
• GPU VM Sizes