Dnanexus Integration

{
  "description": "\"DNAnexus cloud genomics platform. Build apps/applets, manage data (upload/download), dxpy Python SDK, run workflows, FASTQ/BAM/VCF, for genomics pipeline development and execution.\"",
  "references": {
    "files": [
      "references/app-development.md",
      "references/configuration.md",
      "references/data-operations.md",
      "references/job-execution.md",
      "references/python-sdk.md"
    ]
  },
  "content": "### Upload and Analyze Data\r\n\r\n```python\r\nimport dxpy\r\n\r\ninput_file = dxpy.upload_local_file(\"sample.fastq\", project=\"project-xxxx\")\r\n\r\njob = dxpy.DXApplet(\"applet-xxxx\").run({\r\n    \"reads\": dxpy.dxlink(input_file.get_id())\r\n})\r\n\r\njob.wait_on_done()\r\n\r\noutput_id = job.describe()[\"output\"][\"aligned_reads\"][\"$dnanexus_link\"]\r\ndxpy.download_dxfile(output_id, \"aligned.bam\")\r\n```\r\n\r\n### Search and Download Files\r\n\r\n```python\r\nimport dxpy\r\n\r\nfiles = dxpy.find_data_objects(\r\n    classname=\"file\",\r\n    name=\"*.bam\",\r\n    properties={\"experiment\": \"exp001\"},\r\n    project=\"project-xxxx\"\r\n)\r\n\r\nfor file_result in files:\r\n    file_obj = dxpy.DXFile(file_result[\"id\"])\r\n    filename = file_obj.describe()[\"name\"]\r\n    dxpy.download_dxfile(file_result[\"id\"], filename)\r\n```\r\n\r\n### Create Simple App\r\n\r\n```python\r\n\r\n### Pattern 1: Batch Processing\r\n\r\nProcess multiple files with the same analysis:\r\n\r\n```python\r\nfiles = dxpy.find_data_objects(\r\n    classname=\"file\",\r\n    name=\"*.fastq\",\r\n    project=\"project-xxxx\"\r\n)\r\n\r\njobs = []\r\nfor file_result in files:\r\n    job = dxpy.DXApplet(\"applet-xxxx\").run({\r\n        \"input\": dxpy.dxlink(file_result[\"id\"])\r\n    })\r\n    jobs.append(job)\r\n\r\nfor job in jobs:\r\n    job.wait_on_done()\r\n```\r\n\r\n### Pattern 2: Multi-Step Pipeline\r\n\r\nChain multiple analyses together:\r\n\r\n```python\r\nqc_job = qc_applet.run({\"reads\": input_file})\r\n\r\nalign_job = align_applet.run({\r\n    \"reads\": qc_job.get_output_ref(\"filtered_reads\")\r\n})\r\n\r\nvariant_job = variant_applet.run({\r\n    \"bam\": align_job.get_output_ref(\"aligned_bam\")\r\n})\r\n```\r\n\r\n### Pattern 3: Data Organization\r\n\r\nOrganize analysis results systematically:\r\n\r\n```python\r\ndxpy.api.project_new_folder(\r\n    \"project-xxxx\",\r\n    {\"folder\": \"/experiments/exp001/results\", \"parents\": True}\r\n)",
  "name": "dnanexus-integration",
  "id": "scientific-integration-dnanexus-integration",
  "sections": {
    "Common Patterns": "result_file = dxpy.upload_local_file(\r\n    \"results.txt\",\r\n    project=\"project-xxxx\",\r\n    folder=\"/experiments/exp001/results\",\r\n    properties={\r\n        \"experiment\": \"exp001\",\r\n        \"sample\": \"sample1\",\r\n        \"analysis_date\": \"2025-10-20\"\r\n    },\r\n    tags=[\"validated\", \"published\"]\r\n)\r\n```",
    "Installation and Authentication": "### Install dxpy\r\n\r\n```bash\r\npip install dxpy\r\n```\r\n\r\n### Login to DNAnexus\r\n\r\n```bash\r\ndx login\r\n```\r\n\r\nThis authenticates your session and sets up access to projects and data.\r\n\r\n### Verify Installation\r\n\r\n```bash\r\ndx --version\r\ndx whoami\r\n```",
    "Overview": "DNAnexus is a cloud platform for biomedical data analysis and genomics. Build and deploy apps/applets, manage data objects, run workflows, and use the dxpy Python SDK for genomics pipeline development and execution.",
    "Resources": "This skill includes detailed reference documentation:\r\n\r\n### references/\r\n\r\n- **app-development.md** - Complete guide to building and deploying apps/applets\r\n- **data-operations.md** - File management, records, search, and project operations\r\n- **job-execution.md** - Running jobs, workflows, monitoring, and parallel processing\r\n- **python-sdk.md** - Comprehensive dxpy library reference with all classes and functions\r\n- **configuration.md** - dxapp.json specification and dependency management\r\n\r\nLoad these references when you need detailed information about specific operations or when working on complex tasks.",
    "Best Practices": "1. **Error Handling**: Always wrap API calls in try-except blocks\r\n2. **Resource Management**: Choose appropriate instance types for workloads\r\n3. **Data Organization**: Use consistent folder structures and metadata\r\n4. **Cost Optimization**: Archive old data, use appropriate storage classes\r\n5. **Documentation**: Include clear descriptions in dxapp.json\r\n6. **Testing**: Test apps with various input types before production use\r\n7. **Version Control**: Use semantic versioning for apps\r\n8. **Security**: Never hardcode credentials in source code\r\n9. **Logging**: Include informative log messages for debugging\r\n10. **Cleanup**: Remove temporary files and failed jobs",
    "When to Use This Skill": "This skill should be used when:\r\n- Creating, building, or modifying DNAnexus apps/applets\r\n- Uploading, downloading, searching, or organizing files and records\r\n- Running analyses, monitoring jobs, creating workflows\r\n- Writing scripts using dxpy to interact with the platform\r\n- Setting up dxapp.json, managing dependencies, using Docker\r\n- Processing FASTQ, BAM, VCF, or other bioinformatics files\r\n- Managing projects, permissions, or platform resources",
    "Getting Help": "- Official documentation: https://documentation.dnanexus.com/\r\n- API reference: http://autodoc.dnanexus.com/\r\n- GitHub repository: https://github.com/dnanexus/dx-toolkit\r\n- Support: support@dnanexus.com",
    "Quick Start Examples": "import dxpy\r\nimport subprocess\r\n\r\n@dxpy.entry_point('main')\r\ndef main(input_file, quality_threshold=30):\r\n    # Download input\r\n    dxpy.download_dxfile(input_file[\"$dnanexus_link\"], \"input.fastq\")\r\n\r\n    # Process\r\n    subprocess.check_call([\r\n        \"quality_filter\",\r\n        \"--input\", \"input.fastq\",\r\n        \"--output\", \"filtered.fastq\",\r\n        \"--threshold\", str(quality_threshold)\r\n    ])\r\n\r\n    # Upload output\r\n    output_file = dxpy.upload_local_file(\"filtered.fastq\")\r\n\r\n    return {\r\n        \"filtered_reads\": dxpy.dxlink(output_file)\r\n    }\r\n\r\ndxpy.run()\r\n```",
    "Core Capabilities": "The skill is organized into five main areas, each with detailed reference documentation:\r\n\r\n### 1. App Development\r\n\r\n**Purpose**: Create executable programs (apps/applets) that run on the DNAnexus platform.\r\n\r\n**Key Operations**:\r\n- Generate app skeleton with `dx-app-wizard`\r\n- Write Python or Bash apps with proper entry points\r\n- Handle input/output data objects\r\n- Deploy with `dx build` or `dx build --app`\r\n- Test apps on the platform\r\n\r\n**Common Use Cases**:\r\n- Bioinformatics pipelines (alignment, variant calling)\r\n- Data processing workflows\r\n- Quality control and filtering\r\n- Format conversion tools\r\n\r\n**Reference**: See `references/app-development.md` for:\r\n- Complete app structure and patterns\r\n- Python entry point decorators\r\n- Input/output handling with dxpy\r\n- Development best practices\r\n- Common issues and solutions\r\n\r\n### 2. Data Operations\r\n\r\n**Purpose**: Manage files, records, and other data objects on the platform.\r\n\r\n**Key Operations**:\r\n- Upload/download files with `dxpy.upload_local_file()` and `dxpy.download_dxfile()`\r\n- Create and manage records with metadata\r\n- Search for data objects by name, properties, or type\r\n- Clone data between projects\r\n- Manage project folders and permissions\r\n\r\n**Common Use Cases**:\r\n- Uploading sequencing data (FASTQ files)\r\n- Organizing analysis results\r\n- Searching for specific samples or experiments\r\n- Backing up data across projects\r\n- Managing reference genomes and annotations\r\n\r\n**Reference**: See `references/data-operations.md` for:\r\n- Complete file and record operations\r\n- Data object lifecycle (open/closed states)\r\n- Search and discovery patterns\r\n- Project management\r\n- Batch operations\r\n\r\n### 3. Job Execution\r\n\r\n**Purpose**: Run analyses, monitor execution, and orchestrate workflows.\r\n\r\n**Key Operations**:\r\n- Launch jobs with `applet.run()` or `app.run()`\r\n- Monitor job status and logs\r\n- Create subjobs for parallel processing\r\n- Build and run multi-step workflows\r\n- Chain jobs with output references\r\n\r\n**Common Use Cases**:\r\n- Running genomics analyses on sequencing data\r\n- Parallel processing of multiple samples\r\n- Multi-step analysis pipelines\r\n- Monitoring long-running computations\r\n- Debugging failed jobs\r\n\r\n**Reference**: See `references/job-execution.md` for:\r\n- Complete job lifecycle and states\r\n- Workflow creation and orchestration\r\n- Parallel execution patterns\r\n- Job monitoring and debugging\r\n- Resource management\r\n\r\n### 4. Python SDK (dxpy)\r\n\r\n**Purpose**: Programmatic access to DNAnexus platform through Python.\r\n\r\n**Key Operations**:\r\n- Work with data object handlers (DXFile, DXRecord, DXApplet, etc.)\r\n- Use high-level functions for common tasks\r\n- Make direct API calls for advanced operations\r\n- Create links and references between objects\r\n- Search and discover platform resources\r\n\r\n**Common Use Cases**:\r\n- Automation scripts for data management\r\n- Custom analysis pipelines\r\n- Batch processing workflows\r\n- Integration with external tools\r\n- Data migration and organization\r\n\r\n**Reference**: See `references/python-sdk.md` for:\r\n- Complete dxpy class reference\r\n- High-level utility functions\r\n- API method documentation\r\n- Error handling patterns\r\n- Common code patterns\r\n\r\n### 5. Configuration and Dependencies\r\n\r\n**Purpose**: Configure app metadata and manage dependencies.\r\n\r\n**Key Operations**:\r\n- Write dxapp.json with inputs, outputs, and run specs\r\n- Install system packages (execDepends)\r\n- Bundle custom tools and resources\r\n- Use assets for shared dependencies\r\n- Integrate Docker containers\r\n- Configure instance types and timeouts\r\n\r\n**Common Use Cases**:\r\n- Defining app input/output specifications\r\n- Installing bioinformatics tools (samtools, bwa, etc.)\r\n- Managing Python package dependencies\r\n- Using Docker images for complex environments\r\n- Selecting computational resources\r\n\r\n**Reference**: See `references/configuration.md` for:\r\n- Complete dxapp.json specification\r\n- Dependency management strategies\r\n- Docker integration patterns\r\n- Regional and resource configuration\r\n- Example configurations",
    "Workflow Decision Tree": "When working with DNAnexus, follow this decision tree:\r\n\r\n1. **Need to create a new executable?**\r\n   - Yes → Use **App Development** (references/app-development.md)\r\n   - No → Continue to step 2\r\n\r\n2. **Need to manage files or data?**\r\n   - Yes → Use **Data Operations** (references/data-operations.md)\r\n   - No → Continue to step 3\r\n\r\n3. **Need to run an analysis or workflow?**\r\n   - Yes → Use **Job Execution** (references/job-execution.md)\r\n   - No → Continue to step 4\r\n\r\n4. **Writing Python scripts for automation?**\r\n   - Yes → Use **Python SDK** (references/python-sdk.md)\r\n   - No → Continue to step 5\r\n\r\n5. **Configuring app settings or dependencies?**\r\n   - Yes → Use **Configuration** (references/configuration.md)\r\n\r\nOften you'll need multiple capabilities together (e.g., app development + configuration, or data operations + job execution)."
  }
}

---
name: dnanexus-integration
description: "DNAnexus cloud genomics platform. Build apps/applets, manage data (upload/download), dxpy Python SDK, run workflows, FASTQ/BAM/VCF, for genomics pipeline development and execution."
---

# DNAnexus Integration

## Overview

DNAnexus is a cloud platform for biomedical data analysis and genomics. Build and deploy apps/applets, manage data objects, run workflows, and use the dxpy Python SDK for genomics pipeline development and execution.

## When to Use This Skill

This skill should be used when:
- Creating, building, or modifying DNAnexus apps/applets
- Uploading, downloading, searching, or organizing files and records
- Running analyses, monitoring jobs, creating workflows
- Writing scripts using dxpy to interact with the platform
- Setting up dxapp.json, managing dependencies, using Docker
- Processing FASTQ, BAM, VCF, or other bioinformatics files
- Managing projects, permissions, or platform resources

## Core Capabilities

The skill is organized into five main areas, each with detailed reference documentation:

### 1. App Development

**Purpose**: Create executable programs (apps/applets) that run on the DNAnexus platform.

**Key Operations**:
- Generate app skeleton with `dx-app-wizard`
- Write Python or Bash apps with proper entry points
- Handle input/output data objects
- Deploy with `dx build` or `dx build --app`
- Test apps on the platform

**Common Use Cases**:
- Bioinformatics pipelines (alignment, variant calling)
- Data processing workflows
- Quality control and filtering
- Format conversion tools

**Reference**: See `references/app-development.md` for:
- Complete app structure and patterns
- Python entry point decorators
- Input/output handling with dxpy
- Development best practices
- Common issues and solutions

### 2. Data Operations

**Purpose**: Manage files, records, and other data objects on the platform.

**Key Operations**:
- Upload/download files with `dxpy.upload_local_file()` and `dxpy.download_dxfile()`
- Create and manage records with metadata
- Search for data objects by name, properties, or type
- Clone data between projects
- Manage project folders and permissions

**Common Use Cases**:
- Uploading sequencing data (FASTQ files)
- Organizing analysis results
- Searching for specific samples or experiments
- Backing up data across projects
- Managing reference genomes and annotations

**Reference**: See `references/data-operations.md` for:
- Complete file and record operations
- Data object lifecycle (open/closed states)
- Search and discovery patterns
- Project management
- Batch operations

### 3. Job Execution

**Purpose**: Run analyses, monitor execution, and orchestrate workflows.

**Key Operations**:
- Launch jobs with `applet.run()` or `app.run()`
- Monitor job status and logs
- Create subjobs for parallel processing
- Build and run multi-step workflows
- Chain jobs with output references

**Common Use Cases**:
- Running genomics analyses on sequencing data
- Parallel processing of multiple samples
- Multi-step analysis pipelines
- Monitoring long-running computations
- Debugging failed jobs

**Reference**: See `references/job-execution.md` for:
- Complete job lifecycle and states
- Workflow creation and orchestration
- Parallel execution patterns
- Job monitoring and debugging
- Resource management

### 4. Python SDK (dxpy)

**Purpose**: Programmatic access to DNAnexus platform through Python.

**Key Operations**:
- Work with data object handlers (DXFile, DXRecord, DXApplet, etc.)
- Use high-level functions for common tasks
- Make direct API calls for advanced operations
- Create links and references between objects
- Search and discover platform resources

**Common Use Cases**:
- Automation scripts for data management
- Custom analysis pipelines
- Batch processing workflows
- Integration with external tools
- Data migration and organization

**Reference**: See `references/python-sdk.md` for:
- Complete dxpy class reference
- High-level utility functions
- API method documentation
- Error handling patterns
- Common code patterns

### 5. Configuration and Dependencies

**Purpose**: Configure app metadata and manage dependencies.

**Key Operations**:
- Write dxapp.json with inputs, outputs, and run specs
- Install system packages (execDepends)
- Bundle custom tools and resources
- Use assets for shared dependencies
- Integrate Docker containers
- Configure instance types and timeouts

**Common Use Cases**:
- Defining app input/output specifications
- Installing bioinformatics tools (samtools, bwa, etc.)
- Managing Python package dependencies
- Using Docker images for complex environments
- Selecting computational resources

**Reference**: See `references/configuration.md` for:
- Complete dxapp.json specification
- Dependency management strategies
- Docker integration patterns
- Regional and resource configuration
- Example configurations

## Quick Start Examples

### Upload and Analyze Data

```python
import dxpy

# Upload input file
input_file = dxpy.upload_local_file("sample.fastq", project="project-xxxx")

# Run analysis
job = dxpy.DXApplet("applet-xxxx").run({
    "reads": dxpy.dxlink(input_file.get_id())
})

# Wait for completion
job.wait_on_done()

# Download results
output_id = job.describe()["output"]["aligned_reads"]["$dnanexus_link"]
dxpy.download_dxfile(output_id, "aligned.bam")
```

### Search and Download Files

```python
import dxpy

# Find BAM files from a specific experiment
files = dxpy.find_data_objects(
    classname="file",
    name="*.bam",
    properties={"experiment": "exp001"},
    project="project-xxxx"
)

# Download each file
for file_result in files:
    file_obj = dxpy.DXFile(file_result["id"])
    filename = file_obj.describe()["name"]
    dxpy.download_dxfile(file_result["id"], filename)
```

### Create Simple App

```python
# src/my-app.py
import dxpy
import subprocess

@dxpy.entry_point('main')
def main(input_file, quality_threshold=30):
    # Download input
    dxpy.download_dxfile(input_file["$dnanexus_link"], "input.fastq")

    # Process
    subprocess.check_call([
        "quality_filter",
        "--input", "input.fastq",
        "--output", "filtered.fastq",
        "--threshold", str(quality_threshold)
    ])

    # Upload output
    output_file = dxpy.upload_local_file("filtered.fastq")

    return {
        "filtered_reads": dxpy.dxlink(output_file)
    }

dxpy.run()
```

## Workflow Decision Tree

When working with DNAnexus, follow this decision tree:

1. **Need to create a new executable?**
   - Yes → Use **App Development** (references/app-development.md)
   - No → Continue to step 2

2. **Need to manage files or data?**
   - Yes → Use **Data Operations** (references/data-operations.md)
   - No → Continue to step 3

3. **Need to run an analysis or workflow?**
   - Yes → Use **Job Execution** (references/job-execution.md)
   - No → Continue to step 4

4. **Writing Python scripts for automation?**
   - Yes → Use **Python SDK** (references/python-sdk.md)
   - No → Continue to step 5

5. **Configuring app settings or dependencies?**
   - Yes → Use **Configuration** (references/configuration.md)

Often you'll need multiple capabilities together (e.g., app development + configuration, or data operations + job execution).

## Installation and Authentication

### Install dxpy

```bash
pip install dxpy
```

### Login to DNAnexus

```bash
dx login
```

This authenticates your session and sets up access to projects and data.

### Verify Installation

```bash
dx --version
dx whoami
```

## Common Patterns

### Pattern 1: Batch Processing

Process multiple files with the same analysis:

```python
# Find all FASTQ files
files = dxpy.find_data_objects(
    classname="file",
    name="*.fastq",
    project="project-xxxx"
)

# Launch parallel jobs
jobs = []
for file_result in files:
    job = dxpy.DXApplet("applet-xxxx").run({
        "input": dxpy.dxlink(file_result["id"])
    })
    jobs.append(job)

# Wait for all completions
for job in jobs:
    job.wait_on_done()
```

### Pattern 2: Multi-Step Pipeline

Chain multiple analyses together:

```python
# Step 1: Quality control
qc_job = qc_applet.run({"reads": input_file})

# Step 2: Alignment (uses QC output)
align_job = align_applet.run({
    "reads": qc_job.get_output_ref("filtered_reads")
})

# Step 3: Variant calling (uses alignment output)
variant_job = variant_applet.run({
    "bam": align_job.get_output_ref("aligned_bam")
})
```

### Pattern 3: Data Organization

Organize analysis results systematically:

```python
# Create organized folder structure
dxpy.api.project_new_folder(
    "project-xxxx",
    {"folder": "/experiments/exp001/results", "parents": True}
)

# Upload with metadata
result_file = dxpy.upload_local_file(
    "results.txt",
    project="project-xxxx",
    folder="/experiments/exp001/results",
    properties={
        "experiment": "exp001",
        "sample": "sample1",
        "analysis_date": "2025-10-20"
    },
    tags=["validated", "published"]
)
```

## Best Practices

1. **Error Handling**: Always wrap API calls in try-except blocks
2. **Resource Management**: Choose appropriate instance types for workloads
3. **Data Organization**: Use consistent folder structures and metadata
4. **Cost Optimization**: Archive old data, use appropriate storage classes
5. **Documentation**: Include clear descriptions in dxapp.json
6. **Testing**: Test apps with various input types before production use
7. **Version Control**: Use semantic versioning for apps
8. **Security**: Never hardcode credentials in source code
9. **Logging**: Include informative log messages for debugging
10. **Cleanup**: Remove temporary files and failed jobs

## Resources

This skill includes detailed reference documentation:

### references/

- **app-development.md** - Complete guide to building and deploying apps/applets
- **data-operations.md** - File management, records, search, and project operations
- **job-execution.md** - Running jobs, workflows, monitoring, and parallel processing
- **python-sdk.md** - Comprehensive dxpy library reference with all classes and functions
- **configuration.md** - dxapp.json specification and dependency management

Load these references when you need detailed information about specific operations or when working on complex tasks.

## Getting Help

- Official documentation: https://documentation.dnanexus.com/
- API reference: http://autodoc.dnanexus.com/
- GitHub repository: https://github.com/dnanexus/dx-toolkit
- Support: support@dnanexus.com