Refresh Semantic Model

Refreshing Semantic Models

Trigger, monitor, validate, and troubleshoot semantic model refreshes via the Power BI Enhanced Refresh REST API and Fabric CLI.

Core Concepts

A semantic model refresh reloads data from upstream sources and/or recalculates dependent objects (calculated columns, calculated tables, measures). The scope can be the entire model, specific tables, or individual partitions.

Six refresh types are available via the REST API; a seventh (add) is TMSL-only:

*dataOnly clears dependent objects (calculated columns, calculated tables) but does not recalculate them. Follow with a calculate refresh to restore them.

For detailed descriptions, behavior with incremental refresh policies, commit modes, and parallelism options, consult `references/refresh-types.md`.

Refresh Workflow

Step 1: Resolve IDs

Extract the workspace and model GUIDs needed for API calls:

WS_ID=$(fab get "WorkspaceName.Workspace" -q "id" | tr -d '"')
MODEL_ID=$(fab get "WorkspaceName.Workspace/ModelName.SemanticModel" -q "id" | tr -d '"')

Step 2: Query Baseline Data (Pre-Refresh Validation)

Before triggering the refresh, capture a baseline snapshot to later verify that data actually changed. Execute a DAX query against the model to get current row counts or max dates:

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/executeQueries" \
  -X post -i '{
    "queries": [{"query": "EVALUATE ROW(\"RowCount\", COUNTROWS(FactSales), \"MaxDate\", MAX(FactSales[OrderDate]))"}],
    "serializerSettings": {"includeNulls": true}
  }'

Record the output. This baseline is compared after refresh to confirm new data arrived.

Step 3: Trigger the Refresh

Full model refresh (simplest):

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"full"}'

Refresh specific tables:

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{
    "type": "full",
    "objects": [{"table": "FactSales"}, {"table": "DimProduct"}]
  }'

Refresh specific partitions:

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{
    "type": "full",
    "objects": [
      {"table": "FactSales", "partition": "FactSales_2024"},
      {"table": "FactSales", "partition": "FactSales_2023"}
    ]
  }'

Data-only refresh (skip recalculation):

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"dataOnly","objects":[{"table":"FactSales"}]}'

Calculate only (no data reload):

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"calculate"}'

Clear values from a table:

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"clearValues","objects":[{"table":"StagingTable"}]}'

For the script-based approach with CLI arguments, use `scripts/refresh_model.py`.

Step 4: Monitor Status

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes?\$top=1"

Status values: Unknown, InProgress, Completed, Failed, Disabled, Cancelled

Step 5: Post-Refresh Validation

After the refresh completes, re-run the same DAX query from Step 2 and compare:

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/executeQueries" \
  -X post -i '{
    "queries": [{"query": "EVALUATE ROW(\"RowCount\", COUNTROWS(FactSales), \"MaxDate\", MAX(FactSales[OrderDate]))"}],
    "serializerSettings": {"includeNulls": true}
  }'

If data has not changed after a successful refresh:

• The upstream data source has not been updated
• The ETL pipeline (Fabric pipeline, notebook, Data Factory, or other orchestration) needs to run first
• Check the lakehouse/warehouse/SQL database to verify fresh data exists
• For Fabric lakehouses: run fab run "Workspace.Workspace/Pipeline.DataPipeline" or trigger the notebook
• The refresh only pulls what the source provides; if the source is stale, the refresh will succeed but show no new data

Step 6: Cancel (if needed)

To cancel an in-progress enhanced refresh, first retrieve the requestId from the refresh history:

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes?\$top=1"

The response includes a requestId field. Use it to cancel:

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes/<requestId>" \
  -X delete

Only works for refreshes triggered via the Enhanced API (not scheduled or portal refreshes).

Using the Refresh Script

The `scripts/refresh_model.py` script wraps the Enhanced Refresh API with CLI arguments:

# Full refresh
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID

# Refresh specific tables
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --tables Sales,Calendar

# Refresh specific partitions
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --partitions Sales:Sales_2024

# Data-only then calculate (two-phase)
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID -t dataOnly --tables FactSales
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID -t calculate

# Partial batch with parallelism
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --commit partialBatch --parallelism 4

# Skip incremental refresh policy
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --no-policy

# Check status only
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --status-only

# Poll until complete
uv run scripts/refresh_model.py -w $WS_ID -m $MODEL_ID --poll

Enhanced Refresh Options

The Enhanced Refresh API (Premium/Fabric capacity required) extends the standard refresh with:

Common Patterns

Two-Phase Refresh (Large Models)

Split data loading and recalculation for better control and failure isolation:

1. dataOnly with partialBatch to reload all tables (each committed independently) 2. calculate with transactional to recalculate everything atomically

Selective Partition Refresh

For tables with incremental refresh, refresh only specific time-range partitions rather than the entire table. To discover partition names, query the model's TMSL metadata via the XMLA endpoint using Tabular Editor, SSMS, or by exporting with the Fabric CLI:

fab export "Workspace.Workspace/Model.SemanticModel" -o /tmp/model -f

Inspect the exported TMDL table files; each partition block lists the partition name. Target specific partitions in the objects array of the refresh request.

Refresh After ETL

When orchestrating a data pipeline:

1. Run the upstream ETL (Fabric pipeline, notebook, ADF, or custom) 2. Verify fresh data in the source (lakehouse, warehouse, SQL) 3. Trigger the semantic model refresh 4. Validate with a DAX query that row counts or max dates changed 5. If unchanged, investigate the ETL output; the semantic model refresh succeeded but the source was stale

Troubleshooting

Quick reference for the most common failures. For the full troubleshooting guide with debugging workflows and detailed error tables, read `references/troubleshooting.md`.

Debugging per-table failures

When a full refresh fails, isolate the failing table by refreshing individual tables via the Enhanced Refresh API:

# Refresh dimensions first
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"full","objects":[{"table":"Customers"}]}'

# Then facts
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"full","objects":[{"table":"Invoices"}]}'

# Then recalculate
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"calculate"}'

Check the failing table's partition expression and compare source schema against the model's expected column names and types via fab export or fab table schema.

Large Model Strategies

Models over 1 GB or with refresh times exceeding an hour benefit from targeted approaches:

• Partition-level refresh: Refresh individual table partitions via the enhanced REST API or XMLA endpoint instead of the full model. Requires Premium/Fabric capacity.
• Incremental refresh: Automatically partition large tables by date; only recent data refreshes each cycle. Configure RangeStart/RangeEnd parameters in Power Query. Also supports detect-data-changes to skip unchanged partitions entirely.
• Aggregations: Pre-aggregate large fact tables at a coarser grain into an import-mode aggregation table. Detail queries fall through to DirectQuery. Reduces both refresh time and memory.
• Hybrid tables: Historical partitions in import mode; a real-time DirectQuery partition for recent data. Related tables must be Dual storage mode.
• Scale-out: Isolate refresh from query workloads by enabling semantic model scale-out on Premium capacities. A read-only replica handles queries while the primary refreshes.

Capacity Limits

Pro capacity supports only full-model standard refreshes. Enhanced refresh features (table/partition targeting, commit modes, parallelism, cancel, timeout override) require Premium or Fabric capacity.

Requirements

• Workspace contributor or higher permissions
• fab CLI authenticated: fab auth login

Additional Resources

Reference Files

• `references/refresh-types.md` -- Complete reference for all 7 refresh types, commit modes, parallelism, incremental policy interaction, status values, XMLA/TMSL details, and the two-phase refresh pattern
• `references/troubleshooting.md` -- Comprehensive troubleshooting guide: credential errors, type/schema mismatches, timeouts, capacity limits, incremental refresh issues, debugging workflows, and large model strategies

Scripts

• `scripts/refresh_model.py` -- CLI tool for triggering and monitoring refreshes with all enhanced options

Related Skills

• `semantic-model` -- Model design, build, and quality/performance review
• `lineage-analysis` -- Downstream report discovery and impact analysis
• `standardize-naming-conventions` -- Naming audit and remediation
• `fabric-cli` (fabric-cli plugin) -- Workspace and item management via fab CLI

Refresh Types Reference

Reference for all refresh types available in the Power BI Enhanced Refresh API and TMSL refresh command. These types apply to databases, tables, and partitions.

Refresh Type Summary

Detailed Descriptions

full

Reload all data from the source and recalculate all dependent objects. The most thorough but most expensive operation.

• Clears existing data before reloading
• Recalculates all calculated columns, calculated tables, and measures
• Processes all partitions in the specified scope
• For calculation partitions, recalculates the partition and all its dependents
• Use when data sources have changed structurally or after schema changes
• Use when partition states are inconsistent or in an error state

automatic

Conditionally refresh and recalculate only objects that need it. The default type when no type is specified.

• Checks the state of each object; only processes objects not in a Ready state
• If a partition already has data and is in Ready state, it is skipped
• More efficient than full for incremental scenarios
• Ideal for scheduled refreshes and general-purpose automation
• Behaves like full for objects that need processing

dataOnly

Reload data from the source without recalculating any dependent objects.

• Clears existing data and reloads from source
• Does NOT recalculate calculated columns, calculated tables, or measures
• Dependents are cleared to an unprocessed state (calculated columns become empty, calculated tables lose data); they must be recalculated with a subsequent calculate refresh
• Useful as the first step in a two-phase refresh (dataOnly + calculate)
• Reduces processing time when recalculation will happen in a separate step
• The model may be in a partially unprocessed state until the calculate step runs

calculate

Recalculate dependent objects without reloading any data from sources.

• No data source connections are made
• Recalculates calculated columns, calculated tables, and measures
• Only recalculates objects that need it (unless they are volatile formulas)
• Useful after a dataOnly refresh to recalculate in a controlled step
• Useful when only DAX logic has changed (new measures or modified expressions)

clearValues

Remove all data from the specified objects and their dependents.

• Does not reload data; simply empties the object
• Clears all dependent objects as well
• Leaves the object in an unprocessed state
• Useful to free memory before a selective reload
• Useful to reset a partition or table to empty state

add (Partition only; TMSL only)

Append new rows to an existing partition without removing current data. Not available via the REST API; requires XMLA/TMSL endpoint access (SSMS, Tabular Editor, PowerShell).

• Only valid for regular (import) partitions; not for calculation partitions
• Appends data from the partition source query
• Recalculates all dependents after appending
• Does NOT clear existing data first
• Useful for log/event tables where new data is appended
• Requires that the source query returns only the new rows to append

defragment

Clean up unused dictionary entries in column stores.

• No data is reloaded or recalculated
• Removes values from column dictionaries that no longer exist in actual data
• Reduces memory consumption after many add/remove operations
• Useful for models with frequent partition changes (incremental refresh)
• Minimal impact on query performance during execution

Two-Phase Refresh Pattern

For large models, split the refresh into two phases for better control:

Phase 1 -- Data reload:

{
  "type": "dataOnly",
  "commitMode": "partialBatch",
  "maxParallelism": 4,
  "objects": [
    {"table": "FactSales"},
    {"table": "FactInventory"}
  ]
}

Phase 2 -- Recalculation:

{
  "type": "calculate",
  "commitMode": "transactional"
}

Benefits:

• Data reload failures do not leave calculated columns in an inconsistent state
• Each phase can have different commit modes (partialBatch for data, transactional for calc)
• Better visibility into which phase failed

Commit Modes

transactional (default)

• All objects are committed atomically; either everything succeeds or nothing changes
• If any object fails, the entire operation rolls back
• Model remains in its pre-refresh state on failure
• Safer for production workloads

partialBatch

• Each object is committed individually as it completes
• If one table fails, previously committed tables retain their new data
• On failure, the model may contain a mix of old and new data
• Faster for large models with many independent tables
• applyRefreshPolicy must be false when using partialBatch

Incremental Refresh Policy Interaction

When applyRefreshPolicy is true (default) and the table has an incremental refresh policy:

Set applyRefreshPolicy: false to bypass the policy and refresh all partitions manually. This is required when using commitMode: partialBatch.

MaxParallelism

Controls the number of threads used for parallel processing.

• Default: 10
• Setting to 1 forces sequential processing
• Higher values consume more capacity resources
• For optimal parallel processing order:

1. clearValues on all objects first 2. dataOnly on all objects 3. full or calculate on all objects

Enhanced Refresh API vs Standard Refresh

Status Values

Returned by the GET /refreshes endpoint:

The extendedStatus field provides additional detail:

XMLA / TMSL Refresh (Advanced)

For direct XMLA endpoint access (SSMS, Tabular Editor, PowerShell), the TMSL refresh command accepts the same types:

{
  "refresh": {
    "type": "full",
    "objects": [
      {
        "database": "MyModel",
        "table": "FactSales",
        "partition": "Sales_2024_Q1"
      }
    ]
  }
}

TMSL also supports:

• Connection string overrides during refresh
• Query definition overrides for partitions
• Sequence commands for ordered multi-step operations
• The add type (not available via REST API)

TMSL requires a persistent XMLA connection. For long-running refreshes, prefer the REST API.

Refresh Troubleshooting

Common refresh failures, their causes, and resolutions for diagnosing refresh issues.

Credential and Authentication Errors

Data Source and Gateway Errors

Type and Schema Errors

Timeout and Size Errors

Incremental Refresh Errors

Capacity and Throttling Errors

Calculated Table / Calculated Column Errors

Debugging Workflow

1. Check the refresh history

WS_ID=$(fab get "MyWorkspace.Workspace" -q "id" | tr -d '"')
MODEL_ID=$(fab get "MyWorkspace.Workspace/MyModel.SemanticModel" -q "id" | tr -d '"')
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes?\$top=5"

Look at status and serviceExceptionJson for the specific error message and which table/partition failed.

2. Isolate the failing table

Refresh tables one at a time to find which one fails:

# Refresh individual tables via the Power BI REST API
WS_ID=$(fab get "MyWorkspace.Workspace" -q "id" | tr -d '"')
MODEL_ID=$(fab get "MyWorkspace.Workspace/MyModel.SemanticModel" -q "id" | tr -d '"')

# Dimensions first
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"Full","objects":[{"table":"Customers"}]}'

fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"Full","objects":[{"table":"Products"}]}'

# Then facts
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"Full","objects":[{"table":"Invoices"}]}'

# Then calculated tables
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" \
  -X post -i '{"type":"Calculate"}'

3. Compare source schema to model schema

Check what the source provides vs what the model expects:

# Source schema (lakehouse example)
fab table schema "MyWorkspace.Workspace/MyLakehouse.Lakehouse/Tables/invoices"

# Model column types -- inspect the TMDL definition or use fab export
fab export "MyWorkspace.Workspace/MyModel.SemanticModel" -o ./model-export -f
cat ./model-export/MyModel.SemanticModel/definition/tables/Invoices.tmdl | grep -A1 "column "

If they don't match, add type conversion steps in the partition expression.

4. Verify query folding

For incremental refresh, confirm the filter is pushed to the source:

• In Power BI Desktop: right-click the filter step in Power Query; if "View Native Query" is available and shows a WHERE clause, folding works
• At the source: use SQL Profiler or query logs to verify a single filtered query per partition

5. Check capacity state

# Is the workspace on a capacity?
fab get "MyWorkspace.Workspace" -q "capacityId"

# Is the capacity active?
fab api -A powerbi "capacities" -q "value[].{name:displayName, state:state, sku:sku}"

A workspace without a capacity (or on a suspended capacity) will fail all refresh operations.

Strategies for Large Models

For models that are too large or slow to refresh in a single operation:

Partition-level refresh

Refresh individual partitions instead of entire tables. Requires Premium/Fabric capacity with XMLA endpoint access:

# Refresh only recent partitions (via enhanced REST API)
fab api -A powerbi "groups/$WS_ID/datasets/$MODEL_ID/refreshes" -X post \
  -i '{"type":"Full","objects":[{"table":"Invoices","partition":"2024-Q4"}]}'

Incremental refresh

Automatically partition large tables by date so only recent data refreshes each cycle. Configure in Power BI Desktop with RangeStart/RangeEnd parameters. Consider:

• Store period: how much historical data to keep (e.g. 3 years)
• Refresh period: how much recent data to reload each cycle (e.g. 30 days)
• Detect data changes: skip unchanged historical partitions entirely

Aggregations

Pre-aggregate large fact tables at a coarser grain (e.g. monthly by category) into an import-mode aggregation table. Detail queries fall through to DirectQuery. Reduces both refresh time and memory.

Hybrid tables

Combine import mode (historical data) with DirectQuery (recent data) on the same table. Historical partitions import during scheduled refresh; real-time partition queries the source live. Related tables must be set to Dual storage mode.

Scale-out

Enable semantic model scale-out on Premium capacities to isolate refresh from query workloads. A read-only replica handles interactive queries while the read/write replica refreshes.