Adf Master

Azure Data Factory Master Knowledge Base

Deprecated Features

Apache Airflow Workflow Orchestration Manager - DEPRECATED

Status: Deprecated since early 2025. Available only for existing customers. Retirement Date: Not yet announced, but no new deployments permitted. Impact: New customers cannot provision Apache Airflow in Azure Data Factory.

Deprecation Details:

• Apache Airflow Workflow Orchestration Manager is deprecated with no retirement date set
• Only existing deployments can continue using this feature
• No new Airflow integrations can be created in ADF

Migration Path:

• Recommended: Migrate to Fabric Data Factory with native Airflow support
• Alternative: Use standalone Apache Airflow deployments (Azure Container Instances, AKS, or VM-based)
• Alternative: Migrate orchestration logic to native ADF pipelines with control flow activities

Why Deprecated:

• Microsoft focus shifted to Fabric Data Factory as the unified data integration platform
• Fabric provides modern orchestration capabilities superseding Airflow integration
• Limited adoption and maintenance burden for standalone Airflow feature in ADF

Action Required:

• If using Airflow in ADF: Migrate to Fabric Data Factory, standalone Airflow, or native ADF patterns
• For new projects: Do NOT use Airflow in ADF
• Monitor Microsoft announcements for official retirement timeline

Reference:

• Microsoft Roadmap: https://www.directionsonmicrosoft.com/roadmaps/ref/azure-data-factory-roadmap/

Feature Updates (2025-2026)

Microsoft Fabric Integration (GA)

ADF Mounting in Fabric:

• Bring existing ADF pipelines into Fabric workspaces without rebuilding
• Generally Available since June 2025
• Seamless integration enables hybrid ADF + Fabric workflows

Cross-Workspace Pipeline Orchestration:

• New Invoke Pipeline activity supports cross-platform calls
• Invoke pipelines across Fabric, Azure Data Factory, and Synapse
• Managed VNet support for secure cross-workspace communication

Variable Libraries:

• Environment-specific variables for CI/CD automation
• Automatic value substitution during workspace promotion
• Eliminates separate parameter files per environment

Connector Enhancements:

• ServiceNow V2 (V1 End of Support)
• Enhanced PostgreSQL and Snowflake connectors
• Native OneLake connectivity for zero-copy integration

Node.js 20.x Requirement for CI/CD

CRITICAL: As of 2025, npm package @microsoft/azure-data-factory-utilities requires Node.js 20.x

Breaking Change:

• Older Node.js versions (14.x, 16.x, 18.x) may cause package incompatibility errors
• Update CI/CD pipelines to use Node.js 20.x or compatible versions

GitHub Actions:

- name: Setup Node.js
  uses: actions/setup-node@v4
  with:
    node-version: '20.x'

Azure DevOps:

- task: UseNode@1
  inputs:
    version: '20.x'

Official Documentation Sources

Primary Microsoft Learn Resources

Main Documentation Hub:

• URL: https://learn.microsoft.com/en-us/azure/data-factory/
• Last Updated: February 2025
• Coverage: Complete ADF documentation including tutorials, concepts, how-to guides, and reference materials
• Key Topics: Pipelines, datasets, triggers, linked services, data flows, integration runtimes, monitoring

Introduction to Azure Data Factory:

• URL: https://learn.microsoft.com/en-us/azure/data-factory/introduction
• Summary: Managed cloud service for complex hybrid ETL, ELT, and data integration projects
• Key Features: 90+ built-in connectors, serverless architecture, code-free UI, single-pane monitoring

Context7 Library Documentation

Library ID: /websites/learn_microsoft_en-us_azure_data-factory

• Trust Score: 7.5
• Code Snippets: 10,839
• Topics: CI/CD, ARM templates, pipeline patterns, data flows, monitoring, troubleshooting

How to Access:

Use Context7 MCP tool to fetch latest documentation:
mcp__context7__get-library-docs:
  - context7CompatibleLibraryID: /websites/learn_microsoft_en-us_azure_data-factory
  - topic: "CI/CD continuous integration deployment pipelines ARM templates"
  - tokens: 8000

CI/CD Deployment

Detailed CI/CD coverage — ARM template generation, the PrePostDeploymentScript.ps1 pattern (stop/start triggers around deploys, cleanup of removed resources), and complete GitHub Actions + Azure DevOps pipeline YAMLs — lives in references/cicd-deployment.md. Load that reference when wiring continuous deployment for an ADF instance or troubleshooting a deploy pipeline.

Troubleshooting Resources

Official Troubleshooting Guide

URL: https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide Last Updated: January 2025

Common Issues Covered: 1. Template parameter validation errors 2. Integration Runtime type cannot be changed 3. ARM template size exceeds 4MB limit 4. Git connection problems 5. Authentication failures 6. Deployment errors

Diagnostic Logs

Enable Diagnostic Settings:

Azure Portal → Data Factory → Diagnostic settings → Add diagnostic setting
Send to: Log Analytics workspace

Logs to Enable:
- PipelineRuns
- TriggerRuns
- ActivityRuns
- SandboxPipelineRuns
- SandboxActivityRuns

Kusto Queries for Troubleshooting:

// Failed pipeline runs in last 24 hours
ADFPipelineRun
| where Status == "Failed"
| where TimeGenerated > ago(24h)
| project TimeGenerated, PipelineName, RunId, Status, ErrorMessage, Parameters
| order by TimeGenerated desc

// Failed CI/CD deployments
ADFActivityRun
| where ActivityType == "ExecutePipeline"
| where Status == "Failed"
| where TimeGenerated > ago(7d)
| project TimeGenerated, PipelineName, ActivityName, ErrorCode, ErrorMessage
| order by TimeGenerated desc

// Performance analysis
ADFActivityRun
| where TimeGenerated > ago(7d)
| extend DurationMinutes = datetime_diff('minute', End, Start)
| summarize AvgDuration = avg(DurationMinutes) by ActivityType, ActivityName
| where AvgDuration > 10
| order by AvgDuration desc

Common Error Patterns

Error: "Template parameters are not valid"

• Cause: Deleted triggers still referenced in parameters
• Solution: Regenerate ARM template or use PrePostDeploymentScript cleanup

Error: "Updating property type is not supported"

• Cause: Trying to change Integration Runtime type
• Solution: Delete and recreate IR (not in-place update)

Error: "Operation timed out"

• Cause: Network connectivity, large data volume, insufficient compute
• Solution: Increase timeout, optimize query, increase DIUs

Error: "Authentication failed"

• Cause: Service principal expired, missing permissions, wrong credentials
• Solution: Verify credentials, check role assignments, renew if expired

Best Practices

Repository Structure

Recommended Folder Layout:

repository-root/
├── adf-resources/          # ADF JSON files (if using npm approach)
│   ├── dataset/
│   ├── pipeline/
│   ├── trigger/
│   ├── linkedService/
│   └── integrationRuntime/
├── .github/
│   └── workflows/          # GitHub Actions workflows
│       ├── adf-build.yml
│       └── adf-deploy.yml
├── azure-pipelines/        # Azure DevOps pipelines
│   ├── build.yml
│   └── release.yml
├── parameters/             # Environment-specific parameters
│   ├── ARMTemplateParametersForFactory.dev.json
│   ├── ARMTemplateParametersForFactory.test.json
│   └── ARMTemplateParametersForFactory.prod.json
├── package.json            # npm configuration
└── README.md

Git Configuration

Only Configure Git on Development ADF:

• Development: Git-integrated for source control
• Test: CI/CD deployment only (no Git)
• Production: CI/CD deployment only (no Git)

Rationale: Prevents accidental manual changes in higher environments

Multi-Environment Strategy

Environment Flow:
Dev (Git) → Build → Test → Approval → Production
            ↓
        ARM Templates

Parameter Management:

• Separate parameter file per environment
• Store secrets in Azure Key Vault
• Reference Key Vault in parameter files
• Never commit secrets to source control

Monitoring and Alerting

Set up alerts for:

• Build pipeline failures
• Deployment failures
• Pipeline run failures
• Performance degradation
• Cost anomalies

Recommended Tools:

• Azure Monitor (Metrics and Alerts)
• Log Analytics (Kusto queries)
• Application Insights (for custom logging)
• Azure Advisor (optimization recommendations)

Additional Resources

GitHub Repositories

Official Azure Data Factory Samples:

• URL: https://github.com/Azure/Azure-DataFactory
• Path: SamplesV2/ContinuousIntegrationAndDelivery/
• Contents: PrePostDeploymentScript.Ver2.ps1, example pipelines, documentation

Community Examples:

• Search GitHub for "azure-data-factory-cicd" for real-world examples
• Many organizations publish their CI/CD patterns as reference

Community Support

Microsoft Q&A:

• URL: https://learn.microsoft.com/en-us/answers/tags/130/azure-data-factory
• Active community, Microsoft employees respond

Stack Overflow:

• Tag: azure-data-factory
• Large knowledge base of resolved issues

Azure Status:

• URL: https://status.azure.com
• Check for service outages and incidents

When to Fetch Latest Information

Situations requiring current documentation: 1. npm package version updates 2. New ADF features or activities 3. Changes to ARM template schema 4. Updates to PrePostDeploymentScript 5. New GitHub Actions or Azure DevOps tasks 6. Breaking changes or deprecations

How to Fetch:

• Use WebFetch for Microsoft Learn articles
• Check npm for latest package version
• Use Context7 for comprehensive topic coverage
• Review Azure Data Factory GitHub for script updates

This knowledge base should be your starting point for all Azure Data Factory questions. Always verify critical information with the latest official documentation when making production decisions.

Progressive Disclosure References

For detailed JSON schemas and complete reference materials, see:

• Activity Types: references/activity-types.md - Complete JSON schemas for all activity types (Copy, ForEach, IfCondition, Switch, Until, Lookup, ExecutePipeline, WebActivity, DatabricksJob, SetVariable, AppendVariable, Wait, Fail, GetMetadata)
• Expression Functions: references/expression-functions.md - Complete reference for all ADF expression functions (string, collection, logical, conversion, math, date/time, pipeline/activity references)
• Linked Services: references/linked-services.md - Complete JSON configurations for all connector types (Blob Storage, ADLS Gen2, Azure SQL, Synapse, Fabric Lakehouse/Warehouse, Databricks, Key Vault, REST, SFTP, Snowflake, PostgreSQL)
• Triggers: references/triggers.md - Complete JSON schemas for schedule, tumbling window, and event triggers
• Datasets: references/datasets.md - Complete JSON schemas for all dataset types with parameterization patterns

For machine learning and analytics patterns, see the dedicated skill:

• ML & Analytics: adf-master:adf-ml-analytics - Azure ML pipelines, batch endpoints, Azure AI Services, Databricks ML/MLflow, SQL-to-Storage archival, feature engineering with Data Flows

ADF Activity Types - Complete JSON Reference

Copy Activity

The Copy Activity moves data between supported data stores.

Basic Structure

{
  "name": "CopyData",
  "type": "Copy",
  "dependsOn": [],
  "policy": {
    "timeout": "0.12:00:00",
    "retry": 2,
    "retryIntervalInSeconds": 30,
    "secureOutput": false,
    "secureInput": false
  },
  "typeProperties": {
    "source": { },
    "sink": { },
    "enableStaging": false,
    "parallelCopies": 4,
    "dataIntegrationUnits": 4,
    "translator": { }
  },
  "inputs": [{ "referenceName": "SourceDataset", "type": "DatasetReference" }],
  "outputs": [{ "referenceName": "SinkDataset", "type": "DatasetReference" }]
}

Source Types

AzureSqlSource:

{
  "source": {
    "type": "AzureSqlSource",
    "sqlReaderQuery": "SELECT * FROM dbo.TableName WHERE Date >= '@{pipeline().parameters.StartDate}'",
    "queryTimeout": "02:00:00",
    "partitionOption": "None",
    "isolationLevel": "ReadCommitted"
  }
}

DelimitedTextSource:

{
  "source": {
    "type": "DelimitedTextSource",
    "storeSettings": {
      "type": "AzureBlobStorageReadSettings",
      "recursive": true,
      "wildcardFolderPath": "raw/2025",
      "wildcardFileName": "*.csv",
      "enablePartitionDiscovery": false
    },
    "formatSettings": {
      "type": "DelimitedTextReadSettings",
      "skipLineCount": 0,
      "compressionProperties": null
    }
  }
}

ParquetSource:

{
  "source": {
    "type": "ParquetSource",
    "storeSettings": {
      "type": "AzureBlobStorageReadSettings",
      "recursive": true,
      "wildcardFileName": "*.parquet"
    }
  }
}

JsonSource:

{
  "source": {
    "type": "JsonSource",
    "storeSettings": {
      "type": "AzureBlobStorageReadSettings",
      "recursive": true
    },
    "formatSettings": {
      "type": "JsonReadSettings"
    }
  }
}

RestSource:

{
  "source": {
    "type": "RestSource",
    "httpRequestTimeout": "00:01:40",
    "requestInterval": "00.00:00:00.010",
    "requestMethod": "GET",
    "additionalHeaders": {
      "Authorization": "Bearer @{pipeline().parameters.Token}"
    },
    "paginationRules": {
      "AbsoluteUrl": "$.nextLink"
    }
  }
}

Sink Types

ParquetSink:

{
  "sink": {
    "type": "ParquetSink",
    "storeSettings": {
      "type": "AzureBlobStorageWriteSettings",
      "copyBehavior": "FlattenHierarchy"
    },
    "formatSettings": {
      "type": "ParquetWriteSettings",
      "maxRowsPerFile": 100000,
      "fileNamePrefix": "output_"
    }
  }
}

AzureSqlSink:

{
  "sink": {
    "type": "AzureSqlSink",
    "writeBatchSize": 10000,
    "writeBatchTimeout": "00:30:00",
    "preCopyScript": "TRUNCATE TABLE staging.TableName",
    "sqlWriterStoredProcedureName": "usp_UpsertData",
    "sqlWriterTableType": "DataTableType",
    "storedProcedureTableTypeParameterName": "DataTable",
    "tableOption": "autoCreate",
    "disableMetricsCollection": false
  }
}

WarehouseSink (Fabric):

{
  "sink": {
    "type": "WarehouseSink",
    "writeBehavior": "upsert",
    "upsertSettings": {
      "useTempDB": true,
      "keys": ["Id"],
      "interimSchemaName": "staging"
    },
    "writeBatchSize": 10000,
    "tableOption": "autoCreate"
  }
}

LakehouseTableSink:

{
  "sink": {
    "type": "LakehouseTableSink",
    "tableActionOption": "overwrite"
  }
}

Translator (Schema Mapping)

{
  "translator": {
    "type": "TabularTranslator",
    "mappings": [
      {
        "source": { "name": "CustomerID", "type": "Int32" },
        "sink": { "name": "customer_id", "type": "Int64" }
      },
      {
        "source": { "name": "CustomerName" },
        "sink": { "name": "customer_name" }
      },
      {
        "source": { "path": "$['nested']['value']" },
        "sink": { "name": "nested_value" }
      }
    ],
    "collectionReference": "$['items']",
    "mapComplexValuesToString": true
  }
}

---

ForEach Activity

Iterates over a collection and executes activities for each item.

{
  "name": "ForEach_Tables",
  "type": "ForEach",
  "dependsOn": [],
  "typeProperties": {
    "items": {
      "value": "@pipeline().parameters.TableList",
      "type": "Expression"
    },
    "isSequential": false,
    "batchCount": 20,
    "activities": [
      {
        "name": "CopyTable",
        "type": "Copy",
        "typeProperties": {
          "source": {
            "type": "AzureSqlSource",
            "sqlReaderQuery": "@concat('SELECT * FROM ', item().schemaName, '.', item().tableName)"
          },
          "sink": { "type": "ParquetSink" }
        }
      }
    ]
  }
}

Key Properties:

• isSequential: false = parallel, true = sequential
• batchCount: 1-50 (only when isSequential=false)
• Use @item() to access current iteration item

Limitations:

• Cannot nest ForEach inside ForEach
• Cannot nest Until inside ForEach
• Max 100,000 items

---

If Condition Activity

Conditional branching based on expression evaluation.

{
  "name": "IfDataExists",
  "type": "IfCondition",
  "dependsOn": [
    { "activity": "LookupCount", "dependencyConditions": ["Succeeded"] }
  ],
  "typeProperties": {
    "expression": {
      "value": "@greater(activity('LookupCount').output.firstRow.RecordCount, 0)",
      "type": "Expression"
    },
    "ifTrueActivities": [
      {
        "name": "ProcessData",
        "type": "Copy",
        "typeProperties": { }
      }
    ],
    "ifFalseActivities": [
      {
        "name": "LogNoData",
        "type": "WebActivity",
        "typeProperties": { }
      }
    ]
  }
}

Limitations:

• Cannot nest ForEach, If, Switch, Until, or Validation inside branches
• Use Execute Pipeline for complex nested logic

---

Switch Activity

Multi-way branching based on expression value.

{
  "name": "SwitchByEnvironment",
  "type": "Switch",
  "dependsOn": [],
  "typeProperties": {
    "on": {
      "value": "@pipeline().parameters.Environment",
      "type": "Expression"
    },
    "cases": [
      {
        "value": "development",
        "activities": [
          { "name": "DevProcess", "type": "Copy" }
        ]
      },
      {
        "value": "production",
        "activities": [
          { "name": "ProdProcess", "type": "Copy" }
        ]
      }
    ],
    "defaultActivities": [
      { "name": "DefaultProcess", "type": "Copy" }
    ]
  }
}

---

Until Activity

Loops until condition is true or timeout is reached.

{
  "name": "UntilComplete",
  "type": "Until",
  "dependsOn": [],
  "typeProperties": {
    "expression": {
      "value": "@equals(variables('IsComplete'), true)",
      "type": "Expression"
    },
    "timeout": "0.01:00:00",
    "activities": [
      {
        "name": "CheckStatus",
        "type": "WebActivity",
        "typeProperties": {
          "url": "@pipeline().parameters.StatusUrl",
          "method": "GET"
        }
      },
      {
        "name": "SetComplete",
        "type": "SetVariable",
        "dependsOn": [{ "activity": "CheckStatus", "dependencyConditions": ["Succeeded"] }],
        "typeProperties": {
          "variableName": "IsComplete",
          "value": "@equals(activity('CheckStatus').output.status, 'Complete')"
        }
      },
      {
        "name": "WaitBeforeRetry",
        "type": "Wait",
        "dependsOn": [{ "activity": "SetComplete", "dependencyConditions": ["Succeeded"] }],
        "typeProperties": {
          "waitTimeInSeconds": 30
        }
      }
    ]
  }
}

---

Lookup Activity

Retrieves data from a source for use in expressions.

{
  "name": "LookupConfig",
  "type": "Lookup",
  "dependsOn": [],
  "policy": {
    "timeout": "0.00:10:00",
    "retry": 2
  },
  "typeProperties": {
    "source": {
      "type": "AzureSqlSource",
      "sqlReaderQuery": "SELECT * FROM dbo.Configuration WHERE IsActive = 1"
    },
    "dataset": {
      "referenceName": "DS_AzureSql_Config",
      "type": "DatasetReference"
    },
    "firstRowOnly": false
  }
}

Output Access:

• firstRowOnly: true → @activity('LookupConfig').output.firstRow.ColumnName
• firstRowOnly: false → @activity('LookupConfig').output.value (array)

Limits:

• Max 5,000 rows
• Max 4 MB response size

---

Execute Pipeline Activity

Calls another pipeline, enabling modular design.

{
  "name": "ExecuteChildPipeline",
  "type": "ExecutePipeline",
  "dependsOn": [],
  "typeProperties": {
    "pipeline": {
      "referenceName": "PL_Child_Process",
      "type": "PipelineReference"
    },
    "waitOnCompletion": true,
    "parameters": {
      "TableName": {
        "value": "@item().tableName",
        "type": "Expression"
      },
      "ProcessDate": {
        "value": "@pipeline().parameters.ProcessDate",
        "type": "Expression"
      }
    }
  }
}

---

Web Activity

Calls REST endpoints for integration.

{
  "name": "CallRestApi",
  "type": "WebActivity",
  "dependsOn": [],
  "policy": {
    "timeout": "0.00:10:00",
    "retry": 3
  },
  "typeProperties": {
    "url": "@concat(pipeline().parameters.ApiBaseUrl, '/process')",
    "method": "POST",
    "headers": {
      "Content-Type": "application/json",
      "Authorization": "@concat('Bearer ', activity('GetToken').output.access_token)"
    },
    "body": {
      "data": "@pipeline().parameters.InputData",
      "timestamp": "@utcnow()"
    },
    "authentication": {
      "type": "MSI",
      "resource": "https://management.azure.com/"
    }
  }
}

---

Databricks Job Activity

Orchestrates Databricks Workflow Jobs.

{
  "name": "RunDatabricksJob",
  "type": "DatabricksJob",
  "dependsOn": [],
  "policy": {
    "timeout": "0.12:00:00",
    "retry": 2
  },
  "typeProperties": {
    "jobId": "123456789",
    "jobParameters": {
      "input_path": "@pipeline().parameters.InputPath",
      "output_path": "@pipeline().parameters.OutputPath",
      "process_date": "@pipeline().parameters.ProcessDate"
    }
  },
  "linkedServiceName": {
    "referenceName": "LS_Databricks",
    "type": "LinkedServiceReference"
  }
}

---

Set Variable Activity

Sets pipeline variable values.

{
  "name": "SetCounter",
  "type": "SetVariable",
  "dependsOn": [],
  "typeProperties": {
    "variableName": "Counter",
    "value": {
      "value": "@add(variables('Counter'), 1)",
      "type": "Expression"
    }
  }
}

Note: Cannot use SetVariable in parallel ForEach. Use AppendVariable or sequential mode.

---

Append Variable Activity

Appends value to an array variable.

{
  "name": "AppendResult",
  "type": "AppendVariable",
  "dependsOn": [],
  "typeProperties": {
    "variableName": "Results",
    "value": {
      "value": "@activity('ProcessItem').output",
      "type": "Expression"
    }
  }
}

---

Wait Activity

Pauses execution for specified duration.

{
  "name": "WaitForProcessing",
  "type": "Wait",
  "dependsOn": [],
  "typeProperties": {
    "waitTimeInSeconds": 60
  }
}

---

Fail Activity

Explicitly fails the pipeline with custom error.

{
  "name": "FailPipeline",
  "type": "Fail",
  "dependsOn": [
    { "activity": "Validation", "dependencyConditions": ["Failed"] }
  ],
  "typeProperties": {
    "message": "Validation failed: @{activity('Validation').output.error}",
    "errorCode": "VALIDATION_FAILED"
  }
}

---

Get Metadata Activity

Retrieves metadata about datasets or files.

{
  "name": "GetFileList",
  "type": "GetMetadata",
  "dependsOn": [],
  "typeProperties": {
    "dataset": {
      "referenceName": "DS_Blob_Folder",
      "type": "DatasetReference"
    },
    "fieldList": ["childItems", "itemName", "itemType", "lastModified", "size"],
    "storeSettings": {
      "type": "AzureBlobStorageReadSettings",
      "recursive": false
    }
  }
}

Available Fields:

• childItems, itemName, itemType
• lastModified, created, size
• exists, columnCount, structure

---

Azure ML Execute Pipeline Activity

Executes an Azure Machine Learning published pipeline. SDK v1 support ends June 2026. Migrate to batch endpoints via WebActivity.

{
  "name": "RunMLPipeline",
  "type": "AzureMLExecutePipeline",
  "dependsOn": [],
  "policy": {
    "timeout": "1.00:00:00",
    "retry": 1,
    "retryIntervalInSeconds": 60
  },
  "typeProperties": {
    "mlPipelineId": "<published-pipeline-id>",
    "experimentName": "my-experiment",
    "mlPipelineParameters": {
      "param1": "@pipeline().parameters.Value1"
    },
    "continueOnStepFailure": false
  },
  "linkedServiceName": {
    "referenceName": "LS_AzureML",
    "type": "LinkedServiceReference"
  }
}

Output:

• @activity('RunMLPipeline').output.mlPipelineRunId
• @activity('RunMLPipeline').output.status

---

Execute Data Flow Activity

Runs a Mapping Data Flow for Spark-based transformations.

{
  "name": "RunDataFlow",
  "type": "ExecuteDataFlow",
  "dependsOn": [],
  "policy": {
    "timeout": "1.00:00:00",
    "retry": 0
  },
  "typeProperties": {
    "dataFlow": {
      "referenceName": "DF_Transform",
      "type": "DataFlowReference",
      "parameters": {
        "Param1": "'value1'"
      }
    },
    "compute": {
      "coreCount": 8,
      "computeType": "General"
    },
    "staging": {
      "linkedService": {
        "referenceName": "LS_AzureBlobStorage",
        "type": "LinkedServiceReference"
      },
      "folderPath": "staging/dataflows"
    },
    "traceLevel": "Fine"
  }
}

Compute Types: General, MemoryOptimized, ComputeOptimized Core Counts: 8, 16, 32, 48, 80, 144, 272

ADF CI/CD Deployment (ARM, PrePostDeploymentScript, GitHub Actions, Azure DevOps)

Deep dive into ADF deployment: ARM template generation, the PrePostDeploymentScript.ps1 pattern (stop/start triggers around deploys, cleanup of removed resources), and complete GitHub Actions + Azure DevOps pipeline YAMLs. SKILL.md keeps deprecated features, 2025-2026 updates, doc sources, troubleshooting, and best practices; this reference holds the deployment material.

CI/CD Deployment Methods

Modern Automated Approach (Recommended)

npm Package: @microsoft/azure-data-factory-utilities

• Latest Version: 1.0.3+ (check npm for current version)
• npm URL: https://www.npmjs.com/package/@microsoft/azure-data-factory-utilities
• Node.js Requirement: Version 20.x or compatible

Key Features:

• Validates ADF resources independently of service
• Generates ARM templates programmatically
• Enables true CI/CD without manual publish button
• Supports preview mode for selective trigger management

package.json Configuration:

{
  "scripts": {
    "build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index",
    "build-preview": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index --preview"
  },
  "dependencies": {
    "@microsoft/azure-data-factory-utilities": "^1.0.3"
  }
}

Commands:

# Validate resources
npm run build validate <rootFolder> <factoryId>

# Generate ARM templates
npm run build export <rootFolder> <factoryId> [outputFolder]

# Preview mode (only stop/start modified triggers)
npm run build-preview export <rootFolder> <factoryId> [outputFolder]

Official Documentation:

• URL: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements
• Last Updated: January 2025
• Topics: Setup, configuration, build commands, CI/CD integration

Traditional Manual Approach (Legacy)

Method: Git integration + Publish button

Process: 1. Configure Git integration in ADF UI (Dev environment only) 2. Make changes in ADF Studio 3. Click "Publish" button to generate ARM templates 4. Templates saved to adf_publish branch 5. Release pipelines deploy from adf_publish branch

When to Use:

• Migrating from existing setup
• No build pipeline infrastructure
• Simple deployments without validation

Limitations:

• Requires manual publish action
• No validation until publish
• Not true CI/CD (manual step required)
• Can't validate on pull requests

Migration Path: Modern approach recommended for new implementations

ARM Template Deployment

PowerShell Deployment

Primary Command: New-AzResourceGroupDeployment

Syntax:

New-AzResourceGroupDeployment `
  -ResourceGroupName "<resource-group-name>" `
  -TemplateFile "ARMTemplateForFactory.json" `
  -TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
  -factoryName "<factory-name>" `
  -Mode Incremental `
  -Verbose

Validation:

Test-AzResourceGroupDeployment `
  -ResourceGroupName "<resource-group-name>" `
  -TemplateFile "ARMTemplateForFactory.json" `
  -TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
  -factoryName "<factory-name>"

What-If Analysis:

New-AzResourceGroupDeployment `
  -ResourceGroupName "<resource-group-name>" `
  -TemplateFile "ARMTemplateForFactory.json" `
  -TemplateParameterFile "ARMTemplateParametersForFactory.<environment>.json" `
  -factoryName "<factory-name>" `
  -WhatIf

Azure CLI Deployment

Primary Command: az deployment group create

Syntax:

az deployment group create \
  --resource-group <resource-group-name> \
  --template-file ARMTemplateForFactory.json \
  --parameters ARMTemplateParametersForFactory.<environment>.json \
  --parameters factoryName=<factory-name> \
  --mode Incremental

Validation:

az deployment group validate \
  --resource-group <resource-group-name> \
  --template-file ARMTemplateForFactory.json \
  --parameters ARMTemplateParametersForFactory.<environment>.json \
  --parameters factoryName=<factory-name>

What-If Analysis:

az deployment group what-if \
  --resource-group <resource-group-name> \
  --template-file ARMTemplateForFactory.json \
  --parameters ARMTemplateParametersForFactory.<environment>.json \
  --parameters factoryName=<factory-name>

PrePostDeploymentScript

Current Version: Ver2

Location: https://github.com/Azure/Azure-DataFactory/blob/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1

Key Improvement in Ver2:

• Turns off/on ONLY triggers that have been modified
• Ver1 stopped/started ALL triggers (slower, more disruptive)
• Compares trigger payloads to determine changes

Download Command:

# Linux/macOS/Git Bash
curl -o PrePostDeploymentScript.Ver2.ps1 https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1

# PowerShell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/Azure/Azure-DataFactory/main/SamplesV2/ContinuousIntegrationAndDelivery/PrePostDeploymentScript.Ver2.ps1" -OutFile "PrePostDeploymentScript.Ver2.ps1"

Parameters

Pre-Deployment (Stop Triggers):

./PrePostDeploymentScript.Ver2.ps1 `
  -armTemplate "<path-to-ARMTemplateForFactory.json>" `
  -ResourceGroupName "<resource-group-name>" `
  -DataFactoryName "<factory-name>" `
  -predeployment $true `
  -deleteDeployment $false

Post-Deployment (Start Triggers & Cleanup):

./PrePostDeploymentScript.Ver2.ps1 `
  -armTemplate "<path-to-ARMTemplateForFactory.json>" `
  -ResourceGroupName "<resource-group-name>" `
  -DataFactoryName "<factory-name>" `
  -predeployment $false `
  -deleteDeployment $true

PowerShell Requirements

Version: PowerShell Core (7.0+) recommended

• Azure DevOps: Use pwsh: true in AzurePowerShell@5 task
• Locally: Use pwsh command, not powershell

Modules Required:

• Az.DataFactory
• Az.Resources

Official Documentation:

• URL: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-sample-script
• Last Updated: January 2025

GitHub Actions CI/CD

Official Resources

Medium Article (Recent 2025):

• URL: https://medium.com/microsoftazure/azure-data-factory-build-and-deploy-with-new-ci-cd-flow-using-github-actions-cd46c95054e0
• Author: Jared Zagelbaum (Microsoft Azure)
• Topics: Modern CI/CD flow, npm package usage, GitHub Actions setup

Microsoft Community Hub:

• URL: https://techcommunity.microsoft.com/blog/fasttrackforazureblog/azure-data-factory-cicd-with-github-actions/3768493
• Topics: End-to-end GitHub Actions setup, workload identity federation

Community Blog (February 2025):

• URL: https://linusdata.blog/2025/03/14/automating-azure-data-factory-deployments-with-github-actions/
• Topics: Practical implementation guide, troubleshooting tips

Key GitHub Actions

Essential Actions:

• actions/checkout@v4 - Checkout repository
• actions/setup-node@v4 - Setup Node.js
• actions/upload-artifact@v4 - Publish ARM templates
• actions/download-artifact@v4 - Download ARM templates in deploy workflow
• azure/login@v2 - Authenticate to Azure
• azure/arm-deploy@v2 - Deploy ARM templates
• azure/powershell@v2 - Run PrePostDeploymentScript

Authentication Methods

Service Principal (JSON credentials):

{
  "clientId": "<GUID>",
  "clientSecret": "<STRING>",
  "subscriptionId": "<GUID>",
  "tenantId": "<GUID>"
}

Store in GitHub secret: AZURE_CREDENTIALS

Workload Identity Federation (More secure):

• No secrets stored
• Uses OIDC (OpenID Connect)
• Recommended for production
• Setup: https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure

Azure DevOps CI/CD

Official Resources

Microsoft Learn:

• URL: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-automate-azure-pipelines
• Topics: Build pipeline, release pipeline, service connections, variable groups

Community Guides:

• Adam Marczak Blog: https://marczak.io/posts/2023/02/quick-cicd-for-data-factory/
• Topics: Quick setup, best practices, folder structure

Towards Data Science:

• URL: https://towardsdatascience.com/azure-data-factory-ci-cd-made-simple-building-and-deploying-your-arm-templates-with-azure-devops-30c30595afa5
• Topics: ARM template build and deployment workflow

Key Azure DevOps Tasks

Build Pipeline Tasks:

• UseNode@1 - Install Node.js
• Npm@1 - Install packages, run build commands
• PublishPipelineArtifact@1 - Publish ARM templates

Release Pipeline Tasks:

• DownloadPipelineArtifact@2 - Download ARM templates
• AzurePowerShell@5 - Run PrePostDeploymentScript
• AzureResourceManagerTemplateDeployment@3 - Deploy ARM template

Service Connection Requirements

Permissions Needed:

• Data Factory Contributor (on all Data Factories)
• Contributor (on Resource Groups)
• Key Vault access policies (if using secrets)

Configuration:

• Project Settings → Service connections → New service connection
• Type: Azure Resource Manager
• Authentication: Service Principal (recommended) or Managed Identity

ADF Datasets - Complete JSON Reference

Azure Blob Storage

Delimited Text (CSV)

{
  "name": "DS_Blob_CSV",
  "type": "Microsoft.DataFactory/factories/datasets",
  "properties": {
    "type": "DelimitedText",
    "linkedServiceName": {
      "referenceName": "LS_AzureBlobStorage",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "AzureBlobStorageLocation",
        "container": "data",
        "folderPath": "raw/sales",
        "fileName": "sales.csv"
      },
      "columnDelimiter": ",",
      "rowDelimiter": "\n",
      "encodingName": "UTF-8",
      "escapeChar": "\\",
      "quoteChar": "\"",
      "firstRowAsHeader": true,
      "nullValue": "",
      "compressionCodec": "none"
    },
    "schema": []
  }
}

Parquet

{
  "name": "DS_Blob_Parquet",
  "properties": {
    "type": "Parquet",
    "linkedServiceName": {
      "referenceName": "LS_AzureBlobStorage",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "AzureBlobStorageLocation",
        "container": "curated",
        "folderPath": "sales/year=2025/month=01"
      },
      "compressionCodec": "snappy"
    },
    "schema": []
  }
}

JSON

{
  "name": "DS_Blob_JSON",
  "properties": {
    "type": "Json",
    "linkedServiceName": {
      "referenceName": "LS_AzureBlobStorage",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "AzureBlobStorageLocation",
        "container": "data",
        "folderPath": "events",
        "fileName": "events.json"
      },
      "encodingName": "UTF-8"
    }
  }
}

Excel

{
  "name": "DS_Blob_Excel",
  "properties": {
    "type": "Excel",
    "linkedServiceName": {
      "referenceName": "LS_AzureBlobStorage",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "AzureBlobStorageLocation",
        "container": "uploads",
        "fileName": "report.xlsx"
      },
      "sheetName": "Sheet1",
      "firstRowAsHeader": true,
      "range": "A1:Z1000"
    }
  }
}

Binary (Any File)

{
  "name": "DS_Blob_Binary",
  "properties": {
    "type": "Binary",
    "linkedServiceName": {
      "referenceName": "LS_AzureBlobStorage",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "AzureBlobStorageLocation",
        "container": "files",
        "folderPath": "uploads"
      }
    }
  }
}

---

Azure Data Lake Storage Gen2

Parquet with Partitioning

{
  "name": "DS_ADLS_Parquet_Partitioned",
  "properties": {
    "type": "Parquet",
    "linkedServiceName": {
      "referenceName": "LS_ADLS",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "AzureBlobFSLocation",
        "fileSystem": "datalake",
        "folderPath": {
          "value": "@concat('curated/sales/year=', formatDateTime(pipeline().parameters.ProcessDate, 'yyyy'), '/month=', formatDateTime(pipeline().parameters.ProcessDate, 'MM'))",
          "type": "Expression"
        }
      },
      "compressionCodec": "snappy"
    },
    "parameters": {
      "ProcessDate": { "type": "String" }
    }
  }
}

Delta Lake

{
  "name": "DS_ADLS_Delta",
  "properties": {
    "type": "Parquet",
    "linkedServiceName": {
      "referenceName": "LS_ADLS",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "AzureBlobFSLocation",
        "fileSystem": "datalake",
        "folderPath": "bronze/orders"
      }
    }
  }
}

---

Azure SQL Database

{
  "name": "DS_AzureSql_Table",
  "properties": {
    "type": "AzureSqlTable",
    "linkedServiceName": {
      "referenceName": "LS_AzureSql",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "schema": "dbo",
      "table": "Customers"
    },
    "schema": [
      { "name": "CustomerID", "type": "int" },
      { "name": "CustomerName", "type": "nvarchar" },
      { "name": "Email", "type": "nvarchar" },
      { "name": "CreatedDate", "type": "datetime2" }
    ]
  }
}

Parameterized Table

{
  "name": "DS_AzureSql_Parameterized",
  "properties": {
    "type": "AzureSqlTable",
    "linkedServiceName": {
      "referenceName": "LS_AzureSql",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "schema": {
        "value": "@dataset().SchemaName",
        "type": "Expression"
      },
      "table": {
        "value": "@dataset().TableName",
        "type": "Expression"
      }
    },
    "parameters": {
      "SchemaName": { "type": "String", "defaultValue": "dbo" },
      "TableName": { "type": "String" }
    }
  }
}

---

Azure Synapse Analytics

{
  "name": "DS_Synapse_Table",
  "properties": {
    "type": "AzureSqlDWTable",
    "linkedServiceName": {
      "referenceName": "LS_Synapse",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "schema": "staging",
      "table": "FactSales"
    }
  }
}

---

Microsoft Fabric

Lakehouse Table

{
  "name": "DS_Fabric_LakehouseTable",
  "properties": {
    "type": "LakehouseTable",
    "linkedServiceName": {
      "referenceName": "LS_FabricLakehouse",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "table": "sales_facts"
    }
  }
}

Warehouse Table

{
  "name": "DS_Fabric_WarehouseTable",
  "properties": {
    "type": "WarehouseTable",
    "linkedServiceName": {
      "referenceName": "LS_FabricWarehouse",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "schema": "dbo",
      "table": "DimCustomer"
    }
  }
}

---

REST API

{
  "name": "DS_REST_API",
  "properties": {
    "type": "RestResource",
    "linkedServiceName": {
      "referenceName": "LS_REST",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "relativeUrl": {
        "value": "@concat('/api/v2/orders?date=', dataset().QueryDate)",
        "type": "Expression"
      },
      "requestMethod": "GET",
      "additionalHeaders": {
        "Accept": "application/json"
      },
      "paginationRules": {
        "AbsoluteUrl": "$.nextLink"
      }
    },
    "parameters": {
      "QueryDate": { "type": "String" }
    }
  }
}

---

HTTP

{
  "name": "DS_HTTP_CSV",
  "properties": {
    "type": "DelimitedText",
    "linkedServiceName": {
      "referenceName": "LS_HTTP",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "HttpServerLocation",
        "relativeUrl": "/data/export.csv"
      },
      "columnDelimiter": ",",
      "firstRowAsHeader": true
    }
  }
}

---

SFTP

{
  "name": "DS_SFTP_CSV",
  "properties": {
    "type": "DelimitedText",
    "linkedServiceName": {
      "referenceName": "LS_SFTP",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "location": {
        "type": "SftpLocation",
        "folderPath": "/incoming/sales",
        "fileName": {
          "value": "@concat('sales_', formatDateTime(utcnow(), 'yyyyMMdd'), '.csv')",
          "type": "Expression"
        }
      },
      "columnDelimiter": ",",
      "firstRowAsHeader": true
    }
  }
}

---

Snowflake

{
  "name": "DS_Snowflake_Table",
  "properties": {
    "type": "SnowflakeTable",
    "linkedServiceName": {
      "referenceName": "LS_Snowflake",
      "type": "LinkedServiceReference"
    },
    "typeProperties": {
      "schema": "PUBLIC",
      "table": "ORDERS"
    }
  }
}

---

Common Patterns

Parameterized Folder Path (Date Partitioning)

{
  "typeProperties": {
    "location": {
      "type": "AzureBlobStorageLocation",
      "container": "data",
      "folderPath": {
        "value": "@concat('year=', formatDateTime(dataset().ProcessDate, 'yyyy'), '/month=', formatDateTime(dataset().ProcessDate, 'MM'), '/day=', formatDateTime(dataset().ProcessDate, 'dd'))",
        "type": "Expression"
      }
    }
  },
  "parameters": {
    "ProcessDate": { "type": "String" }
  }
}

Wildcard File Name

{
  "typeProperties": {
    "location": {
      "type": "AzureBlobStorageLocation",
      "container": "data",
      "folderPath": "raw",
      "fileName": "*.csv"
    }
  }
}

Compression Options

{
  "compressionCodec": "gzip",
  "compressionLevel": "Optimal"
}

Valid compression codecs:

• none
• gzip
• snappy (Parquet only)
• lzo
• bzip2
• deflate
• zstd
• tar
• targzip

---

Schema Definition

Explicit Schema

{
  "schema": [
    { "name": "Id", "type": "Int32" },
    { "name": "Name", "type": "String" },
    { "name": "Amount", "type": "Decimal", "precision": 18, "scale": 2 },
    { "name": "CreatedAt", "type": "DateTime" },
    { "name": "IsActive", "type": "Boolean" }
  ]
}

Import Schema from Source

Set "schema": [] and use "Import schema" in the UI, or define mappings in the Copy activity translator.

---

Dataset Parameters Usage

In pipeline Copy activity:

{
  "inputs": [
    {
      "referenceName": "DS_Blob_Parameterized",
      "type": "DatasetReference",
      "parameters": {
        "FolderPath": "@pipeline().parameters.SourceFolder",
        "FileName": "@pipeline().parameters.SourceFile"
      }
    }
  ]
}

---

Inline Datasets (Data Flows)

For Mapping Data Flows, inline datasets avoid creating separate dataset objects:

{
  "source": {
    "type": "DelimitedTextSource",
    "dataset": {
      "type": "DelimitedText",
      "linkedService": { "referenceName": "LS_Blob", "type": "LinkedServiceReference" },
      "typeProperties": {
        "location": { "type": "AzureBlobStorageLocation", "container": "data", "fileName": "*.csv" },
        "columnDelimiter": ",",
        "firstRowAsHeader": true
      }
    }
  }
}

ADF Expression Functions - Complete Reference

Expression Syntax

Expressions in ADF use @ prefix and can appear in:

• Parameter values
• Activity typeProperties
• Dynamic content fields

{
  "value": "@concat('prefix_', pipeline().parameters.Name)",
  "type": "Expression"
}

String interpolation: @{expression} within strings

"sqlReaderQuery": "SELECT * FROM dbo.@{pipeline().parameters.TableName}"

---

String Functions

concat

Combines multiple strings.

@concat('Hello', ' ', 'World')           → 'Hello World'
@concat(pipeline().parameters.Prefix, '_', item().name)

substring

Extracts part of a string.

@substring('Hello World', 0, 5)          → 'Hello'
@substring(variables('FileName'), 0, indexOf(variables('FileName'), '.'))

replace

Replaces occurrences in a string.

@replace('Hello World', 'World', 'ADF')  → 'Hello ADF'
@replace(item().path, '/', '_')

split

Splits string into array.

@split('a,b,c', ',')                     → ['a', 'b', 'c']
@split(activity('Lookup').output.firstRow.Values, ';')

join

Joins array elements into string.

@join(variables('Names'), ', ')          → 'Alice, Bob, Charlie'

toLower / toUpper

@toLower('Hello')                        → 'hello'
@toUpper('Hello')                        → 'HELLO'

trim / trimStart / trimEnd

@trim('  Hello  ')                       → 'Hello'
@trimStart('  Hello')                    → 'Hello'
@trimEnd('Hello  ')                      → 'Hello'

indexOf / lastIndexOf

@indexOf('Hello World', 'o')             → 4
@lastIndexOf('Hello World', 'o')         → 7
@indexOf('Hello', 'x')                   → -1 (not found)

startsWith / endsWith

@startsWith('Hello World', 'Hello')      → true
@endsWith('file.csv', '.csv')            → true

length

@length('Hello')                         → 5
@length(pipeline().parameters.Items)     → array length

guid

Generates a unique identifier.

@guid()                                  → 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'
@guid('N')                               → 'a1b2c3d4e5f67890abcdef1234567890'

---

Collection Functions

first / last

@first(variables('Items'))               → first element
@last(pipeline().parameters.List)        → last element

take / skip

@take(variables('Items'), 5)             → first 5 elements
@skip(variables('Items'), 10)            → elements after first 10

contains

@contains('Hello World', 'World')        → true (string)
@contains(variables('List'), 'item')     → true (array)

empty

@empty(variables('Items'))               → true if empty/null
@not(empty(activity('Lookup').output.value))

union / intersection

@union(variables('List1'), variables('List2'))
@intersection(variables('Set1'), variables('Set2'))

length (array)

@length(activity('Lookup').output.value) → number of rows
@length(pipeline().parameters.Tables)

range

Creates an array of integers.

@range(1, 10)                            → [1,2,3,4,5,6,7,8,9,10]
@range(0, length(variables('Items')))

---

Logical Functions

if

@if(equals(variables('Status'), 'Active'), 'Yes', 'No')
@if(greater(activity('Count').output.firstRow.Total, 0), 'HasData', 'Empty')

equals / not

@equals(pipeline().parameters.Env, 'prod')
@not(equals(variables('Status'), 'Failed'))

and / or

@and(greater(variables('Count'), 0), less(variables('Count'), 100))
@or(equals(item().status, 'New'), equals(item().status, 'Pending'))

greater / greaterOrEquals / less / lessOrEquals

@greater(variables('Count'), 10)
@greaterOrEquals(activity('Lookup').output.firstRow.Total, 1000)
@less(dayOfWeek(utcnow()), 6)            → is weekday?
@lessOrEquals(length(variables('Items')), 50)

coalesce

Returns first non-null value.

@coalesce(pipeline().parameters.Override, 'default')
@coalesce(activity('Lookup').output.firstRow.Value, variables('Default'))

---

Conversion Functions

int / float / string / bool

@int('42')                               → 42
@float('3.14')                           → 3.14
@string(42)                              → '42'
@bool('true')                            → true
@bool(1)                                 → true

json

Parses JSON string.

@json(activity('WebActivity').output.Response)
@json('{"name":"test"}').name            → 'test'

xml

Parses XML string.

@xml('<root><item>value</item></root>')

base64 / base64ToString

@base64('Hello')                         → 'SGVsbG8='
@base64ToString('SGVsbG8=')              → 'Hello'

decodeBase64 / encodeUriComponent / decodeUriComponent

@encodeUriComponent('hello world')       → 'hello%20world'
@decodeUriComponent('hello%20world')     → 'hello world'

array

Creates array from value.

@array('single')                         → ['single']
@array(activity('Lookup').output.firstRow)

createArray

Creates array from multiple values.

@createArray('a', 'b', 'c')              → ['a', 'b', 'c']
@createArray(item().table1, item().table2)

---

Math Functions

add / sub / mul / div / mod

@add(10, 5)                              → 15
@sub(10, 5)                              → 5
@mul(10, 5)                              → 50
@div(10, 5)                              → 2
@mod(10, 3)                              → 1

min / max

@min(10, 5, 8)                           → 5
@max(activity('Lookup').output.value)    → max in array

rand

Random number (32-bit signed integer).

@rand(-100, 100)                         → random between -100 and 100

---

Date/Time Functions

utcnow

@utcnow()                                → '2026-03-07T14:30:45.1234567Z'
@utcnow('yyyy-MM-dd')                    → '2026-03-07'
@utcnow('yyyyMMddHHmmss')                → '20260307143045'

adddays / addhours / addminutes / addseconds

@adddays(utcnow(), -1)                   → yesterday
@adddays(utcnow(), 7)                    → next week
@addhours(utcnow(), -6)                  → 6 hours ago
@addminutes(utcnow(), 30)                → 30 min from now
@addseconds(utcnow(), -3600)             → 1 hour ago

formatDateTime

@formatDateTime(utcnow(), 'yyyy-MM-dd')
@formatDateTime(utcnow(), 'yyyy/MM/dd')
@formatDateTime(utcnow(), 'yyyyMMdd')
@formatDateTime(utcnow(), 'yyyy-MM-ddTHH:mm:ss')
@formatDateTime(utcnow(), 'MMMM dd, yyyy')
@formatDateTime(utcnow(), 'dddd')        → 'Wednesday'

Format Specifiers:

startOfDay / startOfMonth / startOfHour

@startOfDay(utcnow())                    → midnight today
@startOfMonth(utcnow())                  → first of month
@startOfHour(utcnow())                   → start of current hour

dayOfMonth / dayOfWeek / dayOfYear

@dayOfMonth(utcnow())                    → 7
@dayOfWeek(utcnow())                     → 6 (0=Sunday)
@dayOfYear(utcnow())                     → 66

month / year

@month(utcnow())                         → 3
@year(utcnow())                          → 2026

ticks

Converts to ticks (100-nanosecond intervals since 1/1/0001).

@ticks(utcnow())                         → 638723523451234567

convertFromUtc / convertToUtc

@convertFromUtc(utcnow(), 'Pacific Standard Time')
@convertToUtc('2026-03-07T10:00:00', 'Eastern Standard Time')

---

Pipeline & Activity References

pipeline()

@pipeline().Pipeline                      → pipeline name
@pipeline().DataFactory                   → data factory name
@pipeline().RunId                         → current run ID
@pipeline().TriggerName                   → trigger name
@pipeline().TriggerTime                   → trigger time
@pipeline().TriggerType                   → 'Manual', 'Schedule', 'Tumbling'
@pipeline().TriggeredByPipelineName       → parent pipeline (if nested)
@pipeline().TriggeredByPipelineRunId      → parent run ID
@pipeline().GroupId                       → execution group ID

pipeline().parameters / pipeline().globalParameters

@pipeline().parameters.TableName
@pipeline().parameters.StartDate
@pipeline().globalParameters.Environment

activity()

@activity('ActivityName').output
@activity('ActivityName').output.firstRow.ColumnName
@activity('ActivityName').output.value                  → array from Lookup
@activity('ActivityName').output.rowsCopied
@activity('ActivityName').output.rowsRead
@activity('ActivityName').output.throughput
@activity('ActivityName').output.copyDuration
@activity('ActivityName').output.errors
@activity('ActivityName').status                        → 'Succeeded', 'Failed'
@activity('ActivityName').error.message
@activity('ActivityName').error.errorCode

variables()

@variables('Counter')
@variables('ResultArray')

item()

Inside ForEach loop:

@item()                                   → current item
@item().tableName
@item()['property-with-dash']

trigger()

@trigger().name
@trigger().scheduledTime
@trigger().startTime
@trigger().outputs.windowStartTime        → tumbling window
@trigger().outputs.windowEndTime          → tumbling window
@trigger().outputs.body                   → event trigger payload
@trigger().outputs.body.fileName          → blob trigger
@trigger().outputs.body.folderPath

dataset()

Inside dataset definition:

@dataset().TableName
@dataset().FolderPath

linkedService()

Inside linked service:

@linkedService().Environment

---

Common Expression Patterns

Yesterday's Date

@formatDateTime(adddays(utcnow(), -1), 'yyyy-MM-dd')

First Day of Month

@formatDateTime(startOfMonth(utcnow()), 'yyyy-MM-dd')

Last Day of Previous Month

@formatDateTime(adddays(startOfMonth(utcnow()), -1), 'yyyy-MM-dd')

Date Partition Path

@concat(
  formatDateTime(utcnow(), 'yyyy'), '/',
  formatDateTime(utcnow(), 'MM'), '/',
  formatDateTime(utcnow(), 'dd')
)

Check if Weekday

@and(greater(dayOfWeek(utcnow()), 0), less(dayOfWeek(utcnow()), 6))

Safe Property Access (with Default)

@coalesce(activity('Lookup').output.firstRow.Value, 'default')

Conditional SQL Query

@if(equals(pipeline().parameters.FullLoad, true),
  'SELECT * FROM dbo.Table',
  concat('SELECT * FROM dbo.Table WHERE Date >= ''', pipeline().parameters.LastDate, '''')
)

Dynamic Table Name

@concat(pipeline().parameters.Schema, '.', pipeline().parameters.Table)

Parse JSON Response

@json(activity('WebActivity').output.Response).data.items

File Name from Path

@substring(
  variables('FilePath'),
  add(lastIndexOf(variables('FilePath'), '/'), 1),
  sub(
    length(variables('FilePath')),
    add(lastIndexOf(variables('FilePath'), '/'), 1)
  )
)

Array to Comma-Separated String

@join(activity('Lookup').output.value, ',')

Check Array Not Empty

@not(empty(activity('GetMetadata').output.childItems))

ADF Linked Services - Complete JSON Reference

Azure Blob Storage

Managed Identity (Recommended)

{
  "name": "LS_AzureBlobStorage_MI",
  "type": "Microsoft.DataFactory/factories/linkedservices",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "serviceEndpoint": "https://mystorageaccount.blob.core.windows.net",
      "accountKind": "StorageV2"
    },
    "connectVia": {
      "referenceName": "AutoResolveIntegrationRuntime",
      "type": "IntegrationRuntimeReference"
    }
  }
}

CRITICAL: accountKind is REQUIRED for Managed Identity. Valid values: StorageV2, BlobStorage, BlockBlobStorage

Service Principal

{
  "name": "LS_AzureBlobStorage_SP",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "serviceEndpoint": "https://mystorageaccount.blob.core.windows.net",
      "accountKind": "StorageV2",
      "servicePrincipalId": "<app-id>",
      "servicePrincipalKey": {
        "type": "AzureKeyVaultSecret",
        "store": {
          "referenceName": "LS_KeyVault",
          "type": "LinkedServiceReference"
        },
        "secretName": "storage-sp-key"
      },
      "tenant": "<tenant-id>"
    }
  }
}

Connection String

{
  "name": "LS_AzureBlobStorage_CS",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "connectionString": {
        "type": "AzureKeyVaultSecret",
        "store": {
          "referenceName": "LS_KeyVault",
          "type": "LinkedServiceReference"
        },
        "secretName": "blob-connection-string"
      }
    }
  }
}

SAS URI

{
  "name": "LS_AzureBlobStorage_SAS",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "sasUri": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "blob-sas-uri"
      }
    }
  }
}

---

Azure Data Lake Storage Gen2

Managed Identity

{
  "name": "LS_ADLS_MI",
  "properties": {
    "type": "AzureBlobFS",
    "typeProperties": {
      "url": "https://mydatalake.dfs.core.windows.net"
    }
  }
}

Service Principal

{
  "name": "LS_ADLS_SP",
  "properties": {
    "type": "AzureBlobFS",
    "typeProperties": {
      "url": "https://mydatalake.dfs.core.windows.net",
      "servicePrincipalId": "<app-id>",
      "servicePrincipalKey": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "adls-sp-key"
      },
      "tenant": "<tenant-id>"
    }
  }
}

---

Azure SQL Database

Managed Identity (Recommended)

{
  "name": "LS_AzureSql_MI",
  "properties": {
    "type": "AzureSqlDatabase",
    "typeProperties": {
      "server": "myserver.database.windows.net",
      "database": "mydb"
    }
  }
}

Required SQL Setup:

CREATE USER [datafactory-name] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [datafactory-name];
ALTER ROLE db_datawriter ADD MEMBER [datafactory-name];

Connection String with Key Vault

{
  "name": "LS_AzureSql_CS",
  "properties": {
    "type": "AzureSqlDatabase",
    "typeProperties": {
      "connectionString": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "sql-connection-string"
      }
    }
  }
}

SQL Authentication

{
  "name": "LS_AzureSql_Auth",
  "properties": {
    "type": "AzureSqlDatabase",
    "typeProperties": {
      "server": "myserver.database.windows.net",
      "database": "mydb",
      "userName": "sqladmin",
      "password": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "sql-password"
      }
    }
  }
}

---

Azure Synapse Analytics

Managed Identity

{
  "name": "LS_Synapse_MI",
  "properties": {
    "type": "AzureSqlDW",
    "typeProperties": {
      "server": "mysynapse.sql.azuresynapse.net",
      "database": "mypool"
    }
  }
}

---

Microsoft Fabric Lakehouse

{
  "name": "LS_FabricLakehouse",
  "properties": {
    "type": "Lakehouse",
    "typeProperties": {
      "workspaceId": "12345678-1234-1234-1234-123456789abc",
      "artifactId": "87654321-4321-4321-4321-cba987654321"
    }
  }
}

With Service Principal:

{
  "name": "LS_FabricLakehouse_SP",
  "properties": {
    "type": "Lakehouse",
    "typeProperties": {
      "workspaceId": "12345678-1234-1234-1234-123456789abc",
      "artifactId": "87654321-4321-4321-4321-cba987654321",
      "servicePrincipalId": "<app-id>",
      "servicePrincipalKey": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "fabric-sp-key"
      },
      "tenant": "<tenant-id>"
    }
  }
}

---

Microsoft Fabric Warehouse

Managed Identity

{
  "name": "LS_FabricWarehouse_MI",
  "properties": {
    "type": "Warehouse",
    "typeProperties": {
      "endpoint": "myworkspace.datawarehouse.fabric.microsoft.com",
      "warehouse": "MyWarehouse",
      "authenticationType": "SystemAssignedManagedIdentity"
    }
  }
}

Service Principal

{
  "name": "LS_FabricWarehouse_SP",
  "properties": {
    "type": "Warehouse",
    "typeProperties": {
      "endpoint": "myworkspace.datawarehouse.fabric.microsoft.com",
      "warehouse": "MyWarehouse",
      "authenticationType": "ServicePrincipal",
      "servicePrincipalId": "<app-id>",
      "servicePrincipalKey": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "fabric-warehouse-key"
      },
      "tenant": "<tenant-id>"
    }
  }
}

---

Azure Databricks

Managed Identity (Serverless)

{
  "name": "LS_Databricks_MI",
  "properties": {
    "type": "AzureDatabricks",
    "typeProperties": {
      "domain": "https://adb-1234567890123456.7.azuredatabricks.net",
      "authentication": "MSI"
    }
  }
}

Access Token

{
  "name": "LS_Databricks_Token",
  "properties": {
    "type": "AzureDatabricks",
    "typeProperties": {
      "domain": "https://adb-1234567890123456.7.azuredatabricks.net",
      "accessToken": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "databricks-token"
      }
    }
  }
}

---

Azure Key Vault

{
  "name": "LS_KeyVault",
  "properties": {
    "type": "AzureKeyVault",
    "typeProperties": {
      "baseUrl": "https://mykeyvault.vault.azure.net"
    }
  }
}

Required Permission: ADF managed identity needs Get permission on secrets.

---

REST API

Anonymous

{
  "name": "LS_REST_Anonymous",
  "properties": {
    "type": "RestService",
    "typeProperties": {
      "url": "https://api.example.com",
      "authenticationType": "Anonymous"
    }
  }
}

Basic Authentication

{
  "name": "LS_REST_Basic",
  "properties": {
    "type": "RestService",
    "typeProperties": {
      "url": "https://api.example.com",
      "authenticationType": "Basic",
      "userName": "apiuser",
      "password": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "api-password"
      }
    }
  }
}

OAuth2 Client Credentials

{
  "name": "LS_REST_OAuth",
  "properties": {
    "type": "RestService",
    "typeProperties": {
      "url": "https://api.example.com",
      "authenticationType": "OAuth2ClientCredential",
      "tokenEndpoint": "https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
      "clientId": "<client-id>",
      "clientSecret": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "oauth-client-secret"
      },
      "scope": "https://api.example.com/.default",
      "resource": "https://api.example.com"
    }
  }
}

Managed Identity

{
  "name": "LS_REST_MI",
  "properties": {
    "type": "RestService",
    "typeProperties": {
      "url": "https://management.azure.com",
      "authenticationType": "ManagedServiceIdentity",
      "aadResourceId": "https://management.azure.com/"
    }
  }
}

---

SFTP

Password

{
  "name": "LS_SFTP_Password",
  "properties": {
    "type": "Sftp",
    "typeProperties": {
      "host": "sftp.example.com",
      "port": 22,
      "skipHostKeyValidation": false,
      "hostKeyFingerprint": "ssh-rsa 2048 xx:xx:xx...",
      "authenticationType": "Basic",
      "userName": "sftpuser",
      "password": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "sftp-password"
      }
    },
    "connectVia": {
      "referenceName": "SelfHostedIR",
      "type": "IntegrationRuntimeReference"
    }
  }
}

SSH Key

{
  "name": "LS_SFTP_Key",
  "properties": {
    "type": "Sftp",
    "typeProperties": {
      "host": "sftp.example.com",
      "port": 22,
      "authenticationType": "SshPublicKey",
      "userName": "sftpuser",
      "privateKeyPath": "/home/user/.ssh/id_rsa",
      "passPhrase": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "ssh-passphrase"
      }
    }
  }
}

---

ServiceNow V2

{
  "name": "LS_ServiceNowV2",
  "properties": {
    "type": "ServiceNowV2",
    "typeProperties": {
      "endpoint": "https://dev12345.service-now.com",
      "authenticationType": "OAuth2",
      "clientId": "<client-id>",
      "clientSecret": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "servicenow-client-secret"
      },
      "username": "service-account@company.com",
      "password": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "servicenow-password"
      },
      "grantType": "password"
    }
  }
}

---

Snowflake

Key Pair Authentication

{
  "name": "LS_Snowflake",
  "properties": {
    "type": "Snowflake",
    "typeProperties": {
      "connectionString": "jdbc:snowflake://account.snowflakecomputing.com",
      "database": "mydb",
      "warehouse": "compute_wh",
      "authenticationType": "KeyPair",
      "user": "myuser",
      "privateKey": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "snowflake-private-key"
      },
      "privateKeyPassphrase": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "snowflake-passphrase"
      }
    }
  }
}

---

PostgreSQL

{
  "name": "LS_PostgreSQL",
  "properties": {
    "type": "PostgreSql",
    "typeProperties": {
      "connectionString": "host=myserver.postgres.database.azure.com;port=5432;database=mydb;uid=myadmin",
      "password": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "postgres-password"
      },
      "enableSsl": true,
      "sslMode": "Require"
    }
  }
}

---

HTTP

{
  "name": "LS_HTTP",
  "properties": {
    "type": "HttpServer",
    "typeProperties": {
      "url": "https://api.example.com",
      "authenticationType": "Anonymous",
      "enableServerCertificateValidation": true
    }
  }
}

---

Azure File Storage

Managed Identity

{
  "name": "LS_AzureFiles_MI",
  "properties": {
    "type": "AzureFileStorage",
    "typeProperties": {
      "fileShare": "myshare",
      "accountName": "mystorageaccount",
      "authenticationType": "ManagedIdentity"
    }
  }
}

---

Azure Machine Learning

Managed Identity

{
  "name": "LS_AzureML_MI",
  "properties": {
    "type": "AzureMLService",
    "typeProperties": {
      "subscriptionId": "<subscription-id>",
      "resourceGroupName": "<resource-group>",
      "mlWorkspaceName": "<ml-workspace-name>",
      "authentication": "MSI"
    }
  }
}

Service Principal

{
  "name": "LS_AzureML_SP",
  "properties": {
    "type": "AzureMLService",
    "typeProperties": {
      "subscriptionId": "<subscription-id>",
      "resourceGroupName": "<resource-group>",
      "mlWorkspaceName": "<ml-workspace-name>",
      "servicePrincipalId": "<app-id>",
      "servicePrincipalKey": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "LS_KeyVault", "type": "LinkedServiceReference" },
        "secretName": "azureml-sp-key"
      },
      "tenant": "<tenant-id>"
    }
  }
}

Azure ML SDK v1 support ends June 2026. Migrate existing AzureMLExecutePipeline usage to batch endpoints via WebActivity. See skill adf-master:adf-ml-analytics for migration patterns.

---

Parameterized Linked Service

{
  "name": "LS_AzureSql_Parameterized",
  "properties": {
    "type": "AzureSqlDatabase",
    "typeProperties": {
      "server": "@{linkedService().ServerName}",
      "database": "@{linkedService().DatabaseName}"
    },
    "parameters": {
      "ServerName": { "type": "String" },
      "DatabaseName": { "type": "String" }
    }
  }
}

Usage in pipeline:

{
  "linkedServiceName": {
    "referenceName": "LS_AzureSql_Parameterized",
    "type": "LinkedServiceReference",
    "parameters": {
      "ServerName": "@pipeline().parameters.SqlServer",
      "DatabaseName": "@pipeline().parameters.SqlDatabase"
    }
  }
}

ADF Triggers - Complete JSON Reference

Schedule Trigger

Runs pipelines on a recurring schedule.

{
  "name": "TR_Daily_0600",
  "type": "Microsoft.DataFactory/factories/triggers",
  "properties": {
    "type": "ScheduleTrigger",
    "typeProperties": {
      "recurrence": {
        "frequency": "Day",
        "interval": 1,
        "startTime": "2025-01-01T06:00:00Z",
        "endTime": "2030-12-31T23:59:59Z",
        "timeZone": "UTC"
      }
    },
    "pipelines": [
      {
        "pipelineReference": {
          "referenceName": "PL_DailyLoad",
          "type": "PipelineReference"
        },
        "parameters": {
          "ProcessDate": "@trigger().scheduledTime"
        }
      }
    ]
  }
}

Frequency Options

Weekly Schedule

{
  "recurrence": {
    "frequency": "Week",
    "interval": 1,
    "startTime": "2025-01-01T08:00:00Z",
    "timeZone": "Eastern Standard Time",
    "schedule": {
      "weekDays": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
      "hours": [8],
      "minutes": [0]
    }
  }
}

Monthly Schedule

{
  "recurrence": {
    "frequency": "Month",
    "interval": 1,
    "startTime": "2025-01-01T00:00:00Z",
    "schedule": {
      "monthDays": [1, 15],
      "hours": [6],
      "minutes": [0]
    }
  }
}

Multiple Runs Per Day

{
  "recurrence": {
    "frequency": "Day",
    "interval": 1,
    "schedule": {
      "hours": [6, 12, 18],
      "minutes": [0, 30]
    }
  }
}

---

Tumbling Window Trigger

Processes data in fixed-time windows with built-in retry and dependency support.

Basic Tumbling Window

{
  "name": "TR_TumblingWindow_Hourly",
  "properties": {
    "type": "TumblingWindowTrigger",
    "typeProperties": {
      "frequency": "Hour",
      "interval": 1,
      "startTime": "2025-01-01T00:00:00Z",
      "endTime": "2030-12-31T23:59:59Z",
      "delay": "00:05:00",
      "maxConcurrency": 10,
      "retryPolicy": {
        "count": 3,
        "intervalInSeconds": 30
      }
    },
    "pipeline": {
      "pipelineReference": {
        "referenceName": "PL_HourlyProcess",
        "type": "PipelineReference"
      },
      "parameters": {
        "WindowStart": "@trigger().outputs.windowStartTime",
        "WindowEnd": "@trigger().outputs.windowEndTime"
      }
    }
  }
}

Key Properties:

• delay: Wait time after window ends before starting (allows late data)
• maxConcurrency: Number of parallel windows (1-50)
• Window times available via @trigger().outputs.windowStartTime/windowEndTime

Tumbling Window with Dependencies

{
  "name": "TR_TumblingWindow_Dependent",
  "properties": {
    "type": "TumblingWindowTrigger",
    "typeProperties": {
      "frequency": "Hour",
      "interval": 1,
      "startTime": "2025-01-01T00:00:00Z",
      "maxConcurrency": 5,
      "retryPolicy": {
        "count": 3,
        "intervalInSeconds": 60
      },
      "dependsOn": [
        {
          "type": "TumblingWindowTriggerDependencyReference",
          "referenceTrigger": {
            "referenceName": "TR_TumblingWindow_Upstream",
            "type": "TriggerReference"
          },
          "offset": "00:00:00",
          "size": "01:00:00"
        }
      ]
    },
    "pipeline": {
      "pipelineReference": {
        "referenceName": "PL_DependentProcess",
        "type": "PipelineReference"
      }
    }
  }
}

Self-Dependency (Sequential Processing)

{
  "dependsOn": [
    {
      "type": "SelfDependencyTumblingWindowTriggerReference",
      "offset": "-01:00:00",
      "size": "01:00:00"
    }
  ]
}

---

Event Triggers

Blob Storage Event Trigger

{
  "name": "TR_BlobCreated",
  "properties": {
    "type": "BlobEventsTrigger",
    "typeProperties": {
      "blobPathBeginsWith": "/container/raw/",
      "blobPathEndsWith": ".csv",
      "ignoreEmptyBlobs": true,
      "events": ["Microsoft.Storage.BlobCreated"],
      "scope": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>"
    },
    "pipelines": [
      {
        "pipelineReference": {
          "referenceName": "PL_ProcessNewFile",
          "type": "PipelineReference"
        },
        "parameters": {
          "FileName": "@trigger().outputs.body.fileName",
          "FolderPath": "@trigger().outputs.body.folderPath"
        }
      }
    ]
  }
}

Available Trigger Outputs:

@trigger().outputs.body.fileName
@trigger().outputs.body.folderPath
@trigger().outputs.body.uri

Custom Event Trigger (Event Grid)

{
  "name": "TR_CustomEvent",
  "properties": {
    "type": "CustomEventsTrigger",
    "typeProperties": {
      "scope": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.EventGrid/topics/<topic>",
      "events": ["DataReady", "ProcessComplete"],
      "subjectBeginsWith": "/orders/",
      "subjectEndsWith": ""
    },
    "pipelines": [
      {
        "pipelineReference": {
          "referenceName": "PL_ProcessEvent",
          "type": "PipelineReference"
        },
        "parameters": {
          "EventData": "@trigger().outputs.body.data",
          "Subject": "@trigger().outputs.body.subject"
        }
      }
    ]
  }
}

---

Storage Event Trigger (ADLS Gen2)

{
  "name": "TR_ADLSEvent",
  "properties": {
    "type": "BlobEventsTrigger",
    "typeProperties": {
      "blobPathBeginsWith": "/filesystem/landing/",
      "blobPathEndsWith": ".parquet",
      "ignoreEmptyBlobs": true,
      "events": ["Microsoft.Storage.BlobCreated"],
      "scope": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<adls-account>"
    },
    "pipelines": [
      {
        "pipelineReference": {
          "referenceName": "PL_ProcessLanding",
          "type": "PipelineReference"
        },
        "parameters": {
          "FilePath": "@concat(trigger().outputs.body.folderPath, '/', trigger().outputs.body.fileName)"
        }
      }
    ]
  }
}

---

Rerun Triggers

Tumbling Window Rerun Trigger

{
  "name": "TR_Rerun_January",
  "properties": {
    "type": "RerunTumblingWindowTrigger",
    "typeProperties": {
      "parentTrigger": {
        "referenceName": "TR_TumblingWindow_Daily",
        "type": "TriggerReference"
      },
      "requestedStartTime": "2025-01-01T00:00:00Z",
      "requestedEndTime": "2025-01-31T23:59:59Z",
      "rerunConcurrency": 5
    }
  }
}

---

Multiple Pipelines per Trigger

{
  "name": "TR_MultiplePipelines",
  "properties": {
    "type": "ScheduleTrigger",
    "typeProperties": {
      "recurrence": {
        "frequency": "Day",
        "interval": 1,
        "startTime": "2025-01-01T06:00:00Z"
      }
    },
    "pipelines": [
      {
        "pipelineReference": {
          "referenceName": "PL_ExtractData",
          "type": "PipelineReference"
        },
        "parameters": { "Source": "Sales" }
      },
      {
        "pipelineReference": {
          "referenceName": "PL_ExtractData",
          "type": "PipelineReference"
        },
        "parameters": { "Source": "Inventory" }
      },
      {
        "pipelineReference": {
          "referenceName": "PL_ExtractData",
          "type": "PipelineReference"
        },
        "parameters": { "Source": "Customers" }
      }
    ]
  }
}

---

Common Trigger Expressions

Access Trigger Properties in Pipeline

@trigger().name                          → Trigger name
@trigger().scheduledTime                 → Scheduled time (schedule trigger)
@trigger().startTime                     → Actual start time
@trigger().outputs.windowStartTime       → Window start (tumbling window)
@trigger().outputs.windowEndTime         → Window end (tumbling window)
@trigger().outputs.body.fileName         → File name (blob trigger)
@trigger().outputs.body.folderPath       → Folder path (blob trigger)

Dynamic Date from Tumbling Window

{
  "ProcessDate": {
    "value": "@formatDateTime(trigger().outputs.windowStartTime, 'yyyy-MM-dd')",
    "type": "Expression"
  }
}

---

Time Zones

Supported time zone values:

• UTC
• Eastern Standard Time
• Pacific Standard Time
• Central Standard Time
• Mountain Standard Time
• GMT Standard Time
• W. Europe Standard Time
• Tokyo Standard Time
• India Standard Time
• China Standard Time
• AUS Eastern Standard Time

Example with Time Zone:

{
  "recurrence": {
    "frequency": "Day",
    "interval": 1,
    "startTime": "2025-01-01T08:00:00",
    "timeZone": "Eastern Standard Time"
  }
}

---

Trigger States

Start/Stop via REST API:

POST /triggers/{triggerName}/start
POST /triggers/{triggerName}/stop

---

Limitations

• Schedule trigger: Max 5 executions per minute
• Tumbling window: maxConcurrency 1-50
• Event trigger: Subject max 1024 characters
• Blob trigger: Doesn't fire for folder creation
• Rerun trigger: Cannot exceed original trigger window range