Developing Datacloud Code Extension

developing-datacloud-code-extension Skill

Overview

This skill provides a complete workflow for developing, testing, and deploying custom Python code extensions to Salesforce Data Cloud. Code extensions allow you to write Python transformations that read from and write to Data Lake Objects (DLOs) and Data Model Objects (DMOs).

When to Use

• User wants to create a new code extension project
• User needs to test a code extension locally
• User wants to scan code for required permissions
• User needs to deploy a code extension to Data Cloud
• User is working with Data Cloud transformations
• User wants to read/write DLO or DMO data programmatically

Prerequisites Check

Before executing any code extension commands, verify prerequisites:

1. SF CLI with plugin installed

   sf plugins --core | grep data-code-extension

If not installed:

   sf plugins install @salesforce/plugin-data-codeextension

2. Python 3.11

   python --version  # Should show 3.11.x

3. Data Cloud Custom Code SDK

   pip list | grep salesforce-data-customcode

If not installed:

   pip install salesforce-data-customcode

4. Docker running (for deploy only)

   docker ps

5. Authenticated org

   sf org display --target-org <org_alias> --json

Skill Workflow

Phase 1: Initialize Project

Create a new code extension project with scaffolding.

Commands:

For script-based code extensions (batch transformations):

sf data-code-extension script init --package-dir <directory>

For function-based code extensions (real-time):

sf data-code-extension function init --package-dir <directory>

Required Option:

• --package-dir, -p - Directory path where the package will be created

What it creates:

my-transform/              # Project root
├── payload/               # CRITICAL: This is what --package-dir must point to for deploy
│   ├── entrypoint.py      # Main transformation code
│   └── config.json        # Code extension configuration
├── requirements.txt       # Python dependencies
└── README.md

Directory Context During Workflow

IMPORTANT: Understanding the directory structure is critical for successful deployment.

Commands and their directory requirements:

CRITICAL: The `--package-dir` argument in deploy command MUST point to the `payload` directory, not the project root.

Phase 2: Develop Transformation

Edit payload/entrypoint.py with transformation logic.

Script Example (Batch):

from datacustomcode import Client

client = Client()

# Read from DLO
df = client.read_dlo('Employee__dll')

# Transform data (uppercase position field)
df['position_upper'] = df['position'].str.upper()

# Write to output DLO
client.write_to_dlo('Employee_Upper__dll', df, 'overwrite')

Function Example (Real-time):

from datacustomcode import FunctionClient

def transform(event, context):
    client = FunctionClient(context)
    input_data = event['data']
    output = {
        'name': input_data['name'].upper(),
        'status': 'processed'
    }
    return output

Common Operations:

• client.read_dlo('DLO_Name__dll') - Read from DLO
• client.read_dmo('DMO_Name') - Read from DMO
• client.write_to_dlo('DLO_Name__dll', df, 'overwrite') - Write to DLO
• client.write_to_dmo('DMO_Name', df, 'upsert') - Write to DMO

Phase 3: Scan for Permissions

Scan the entrypoint file to detect required permissions and generate config.json.

Command:

sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

What it detects:

• Read permissions for DLOs/DMOs
• Write permissions for DLOs/DMOs
• Python package dependencies
• Updates config.json and requirements.txt

Phase 4: Validate DLO Schema (Pre-Test Check)

CRITICAL: Before running tests locally, validate that all DLOs used in your code exist and have the expected fields.

Step 4a: Extract DLOs from config.json

After scanning, review the generated config.json to identify all DLOs:

cat payload/config.json

Step 4b: Validate Each DLO Schema

Use the `getting-datacloud-schema` skill to verify DLOs exist and check field names.

For each DLO referenced in your code:

1. Verify DLO exists:

   python3 scripts/get_dlo_schema.py <org_alias> <dlo_name>

2. Verify field names match — compare fields used in your entrypoint.py against the DLO schema.

3. Check all DLOs:

• Validate all DLOs in read permissions
• Validate all DLOs in write permissions
• Check field names match exactly (case-sensitive)
• Verify data types are compatible with operations

Step 4c: Validation Checklist

Before proceeding to run, ensure:

• [ ] All DLOs in config.json exist in target org
• [ ] All field names used in code exist in DLO schemas
• [ ] Field data types match your transformation logic
• [ ] Primary key fields are correctly identified
• [ ] Write target DLOs are created and accessible

Phase 5: Test Locally

After validating DLO schemas, run the code extension locally against your Data Cloud org.

Command:

sf data-code-extension script run --entrypoint <entrypoint_file> --target-org <org_alias> [options]

Options:

• --target-org, -o - SF CLI org alias (required)
• --config-file, -c - Custom config file path

If you get errors:

• Re-validate DLO schemas
• Check field names are exact matches
• Verify data types are compatible
• Review error messages for field/DLO issues

Phase 6: Deploy to Data Cloud

Deploy the code extension to Data Cloud for scheduled or on-demand execution.

CRITICAL: You MUST specify `--package-dir ./payload` to point to the payload directory created by init.

Command:

sf data-code-extension script deploy --target-org <org_alias> --name <name> --package-dir ./payload --package-version <version> --description <description> [options]

Required Options:

• --target-org, -o - SF CLI org alias
• --name, -n - Name for code extension deployment
• --package-dir - Path to payload directory (REQUIRED - must be ./payload when running from project root)
• --package-version - Version string (default: 0.0.1)
• --description - Description of code extension

Optional Options:

• --cpu-size - CPU size: CPU_L, CPU_XL, CPU_2XL (default), CPU_4XL
• --function-invoke-opt - Function invoke options (for function type)
• --network - Docker network (default: default)

After deployment:

• Navigate to Data Cloud in Salesforce UI
• Go to Data Transforms section
• Find your deployment by name
• Click "Run Now" to execute
• Schedule for recurring execution

Error Handling

Common Issues and Solutions

Best Practices

Development

1. Always scan before testing — run scan after code changes 2. Test locally first — use run command before deploying 3. Use version control — git commit after each successful test 4. Version your deployments — use semantic versioning (1.0.0, 1.1.0, etc.) 5. Deploy from project root with --package-dir ./payload

Performance

• CPU_L: Small datasets (< 1M records)
• CPU_2XL: Medium datasets (1M-10M records)
• CPU_4XL: Large datasets (> 10M records)

Security

1. No hardcoded credentials — use SF CLI authentication only 2. Validate input data — check for nulls and data types 3. Limit write permissions — only grant necessary DLO/DMO access

Integration with Other Skills

Use with getting-datacloud-schema skill (CRITICAL for validation):

The getting-datacloud-schema skill is required for validating DLOs before testing code extensions.

Use with Datakit Workflow: 1. Create DLO via code extension 2. Map DLO to DMO using datakit workflow 3. Use DMO in segments and activations

Command Reference

Resources

• SF CLI Plugin: https://github.com/salesforcecli/plugin-data-code-extension
• Python SDK: https://github.com/forcedotcom/datacloud-customcode-python-sdk
• Data Cloud Docs: https://help.salesforce.com/s/articleView?id=sf.c360_a_intro.htm
• Python SDK PyPI: https://pypi.org/project/salesforce-data-customcode/

Notes

• Code extensions run in isolated Python 3.11 environment
• Docker is required only for deployment, not for local testing
• Use SF CLI authentication only (no separate credential files)
• Scan command auto-detects permissions from code
• Local run uses actual Data Cloud data (not mocked)
• Deployments are versioned and can be rolled back in UI

Data Cloud Code Extension - Quick Reference

Command Cheat Sheet

Initialize Project

# Create script project
sf data-code-extension script init --package-dir <directory>

# Create function project
sf data-code-extension function init --package-dir <directory>

# Examples
sf data-code-extension script init --package-dir .
sf data-code-extension script init --package-dir my-transform

Scan for Permissions

# Basic scan
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# Preview without saving
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py --dry-run

# Custom config location
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py --config ./custom-config.json

# Skip requirements.txt
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py --no-requirements

Run Locally

# Basic run
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org <org_alias>

# With custom config
sf data-code-extension script run --entrypoint ./payload/entrypoint.py -o <org_alias> -c custom-config.json

# Examples
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe
sf data-code-extension script run --entrypoint ./payload/entrypoint.py -o afvibe

Deploy

# Minimal deployment (MUST include --package-dir ./payload)
sf data-code-extension script deploy \
  --target-org <org_alias> \
  --name <name> \
  --package-version <version> \
  --description "<description>" \
  --package-dir ./payload

# Full options
sf data-code-extension script deploy \
  --target-org <org_alias> \
  --name <name> \
  --package-version <version> \
  --description "<description>" \
  --cpu-size <CPU_L|CPU_XL|CPU_2XL|CPU_4XL> \
  --package-dir ./payload

# Examples (CRITICAL: Always include --package-dir ./payload)
sf data-code-extension script deploy \
  --target-org afvibe \
  --name Employee_Upper \
  --package-version 1.0.0 \
  --description "Uppercase employee positions" \
  --package-dir ./payload

Common Workflows

New Project from Scratch

# 1. Create directory
mkdir my-transform && cd my-transform

# 2. Initialize
sf data-code-extension script init --package-dir .

# 3. Edit payload/entrypoint.py with your transformation

# 4. Scan
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# 5. Test
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe

# 6. Deploy (MUST include --package-dir ./payload)
sf data-code-extension script deploy \
  --target-org afvibe \
  --name MyTransform \
  --package-version 1.0.0 \
  --description "My transformation" \
  --package-dir ./payload

Update Existing Code Extension

# 1. Edit payload/entrypoint.py

# 2. Re-scan
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# 3. Test
sf data-code-extension script run --entrypoint ./payload/entrypoint.py -o afvibe

# 4. Deploy with new version (include --package-dir ./payload)
sf data-code-extension script deploy \
  -o afvibe \
  -n MyTransform \
  --package-version 1.1.0 \
  --description "Updated transformation" \
  --package-dir ./payload

Python Code Patterns

Read/Write DLO

from datacustomcode import Client

client = Client()

# Read
df = client.read_dlo('Employee__dll')

# Transform
df['new_field'] = df['old_field'].str.upper()

# Write (modes: 'overwrite', 'append')
client.write_to_dlo('Output__dll', df, 'overwrite')

Read/Write DMO

# Read
df = client.read_dmo('EmployeeDMO')

# Write (modes: 'upsert', 'insert')
client.write_to_dmo('EmployeeDMO', df, 'upsert')

Multiple DLO Operations

# Read multiple
employees = client.read_dlo('Employee__dll')
departments = client.read_dlo('Department__dll')

# Join
merged = employees.merge(departments, on='dept_id')

# Write multiple
client.write_to_dlo('Enriched__dll', merged, 'overwrite')
client.write_to_dmo('EmployeeDMO', merged, 'upsert')

Data Transformations

import pandas as pd

# Filter
active = df[df['status'] == 'Active']

# Computed column
df['full_name'] = df['first'] + ' ' + df['last']

# Aggregate
summary = df.groupby('dept')['salary'].mean()

# Conditional
df['grade'] = df['position'].apply(
    lambda x: 'Senior' if 'VP' in x else 'Junior'
)

Option Reference

--cpu-size

• CPU_L - Small datasets (< 1M records)
• CPU_XL - Medium datasets (1M-5M)
• CPU_2XL - Large datasets (5M-10M) [default]
• CPU_4XL - Very large (> 10M records)

Write Modes

• overwrite - Replace all data
• append - Add to existing data
• upsert - Update or insert (DMO only)
• insert - Insert only (DMO only)

Troubleshooting Quick Fixes

# Plugin not found
sf plugins install @salesforce/plugin-data-codeextension

# Python SDK missing
pip install salesforce-data-customcode

# Verify Python version (must be 3.11.x)
python --version

# Org not connected
sf org login web --alias <org_alias>

# Config missing
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# Docker not running (for deploy)
# Start Docker Desktop

File Structure

my-project/
├── payload/
│   ├── entrypoint.py      # Main code
│   └── config.json        # Auto-generated permissions
├── requirements.txt       # Auto-generated dependencies
└── README.md

config.json Format

{
  "version": "1.0",
  "permissions": {
    "read": ["Employee__dll", "Department__dll"],
    "write": ["Enriched__dll"]
  },
  "resources": {
    "cpu_size": "CPU_2XL"
  }
}

Common Errors

Deployment Checklist

• [ ] Code written in entrypoint.py
• [ ] Scanned for permissions
• [ ] Tested locally
• [ ] Version number decided
• [ ] Description added
• [ ] CPU size chosen
• [ ] Docker running
• [ ] Org authenticated

Resources

• Plugin: https://github.com/salesforcecli/plugin-data-code-extension
• Python SDK: https://github.com/forcedotcom/datacloud-customcode-python-sdk
• Data Cloud Docs: https://help.salesforce.com/s/articleView?id=sf.c360_a_intro.htm

developing-datacloud-code-extension Skill

Overview

A skill that provides a complete workflow for developing, testing, and deploying custom Python code extensions to Salesforce Data Cloud using the SF CLI plugin.

What It Does

This skill helps you create Data Cloud Code Extensions through a complete workflow:

1. Init - Create new code extension project with scaffolding 2. Develop - Write Python transformation logic 3. Scan - Auto-detect permissions and generate config 4. Run - Test locally against Data Cloud org 5. Deploy - Package and deploy to Data Cloud

Usage

Initialize a project:

"Create a new Data Cloud code extension project called employee-transform"
"Initialize a code extension to transform employee data"

Test locally:

"Run the code extension in my-transform directory against afvibe org"
"Test the entrypoint.py file locally"

Scan for permissions:

"Scan the entrypoint.py to generate config"
"Update permissions in config.json"

Deploy:

"Deploy Employee_Upper code extension to afvibe"
"Deploy this transform with package-version 1.0.0"

Direct Command Usage

# Initialize project
sf data-code-extension script init --package-dir <directory>

# Scan for permissions
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# Test locally
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org <org_alias>

# Deploy
sf data-code-extension script deploy --target-org <org_alias> --name <name> --package-version <version> --description <description> --package-dir ./payload

Prerequisites

1. SF CLI with Plugin

   sf plugins install @salesforce/plugin-data-codeextension

2. Python 3.11

   python --version  # Must be 3.11.x

3. Data Cloud Custom Code SDK

   pip install salesforce-data-customcode

4. Docker (for deploy only)

• Docker Desktop or equivalent

5. Authenticated Org

   sf org login web --alias <org_alias>

Quick Start

Complete End-to-End Example

# 1. Create project
mkdir employee-transform && cd employee-transform
sf data-code-extension script init --package-dir .

# 2. Edit payload/entrypoint.py with your transformation

# 3. Scan for permissions
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# 4. Test locally
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe

# 5. Deploy (MUST include --package-dir ./payload)
sf data-code-extension script deploy \
  --target-org afvibe \
  --name Employee_Upper \
  --package-version 1.0.0 \
  --description "Uppercase employee positions" \
  --package-dir ./payload

Example Transformation

Read from DLO, transform, write to DLO:

from datacustomcode import Client

client = Client()

# Read employee data from DLO
employees = client.read_dlo('Employee__dll')

# Transform - uppercase position field
employees['position_upper'] = employees['position'].str.upper()

# Select output columns
output = employees[['id', 'name', 'position_upper']]

# Write to output DLO
client.write_to_dlo('Employee_Upper__dll', output, 'overwrite')

print(f"Processed {len(output)} employee records")

Project Structure

After init, you'll have:

my-transform/
├── payload/
│   ├── entrypoint.py      # Your transformation code
│   └── config.json        # Permissions and configuration
├── requirements.txt       # Python dependencies
└── README.md

Common Operations

Read/Write DLOs

# Read
df = client.read_dlo('Employee__dll')

# Write (modes: 'overwrite', 'append')
client.write_to_dlo('Employee_Upper__dll', df, 'overwrite')

Read/Write DMOs

# Read
df = client.read_dmo('EmployeeDMO')

# Write (modes: 'upsert', 'insert')
client.write_to_dmo('EmployeeDMO', df, 'upsert')

Troubleshooting

CPU Size Selection

Resources

• SF CLI Plugin: https://github.com/salesforcecli/plugin-data-code-extension
• Python SDK: https://github.com/forcedotcom/datacloud-customcode-python-sdk
• Data Cloud Docs: https://help.salesforce.com/s/articleView?id=sf.c360_a_intro.htm
• SDK on PyPI: https://pypi.org/project/salesforce-data-customcode/