Cellxgene Census

{
  "description": "\"Query CZ CELLxGENE Census (61M+ cells). Filter by cell type/tissue/disease, retrieve expression data, integrate with scanpy/PyTorch, for population-scale single-cell analysis.\"",
  "references": {
    "files": [
      "references/census_schema.md",
      "references/common_patterns.md"
    ]
  },
  "content": "### 1. Opening the Census\r\n\r\nAlways use the context manager to ensure proper resource cleanup:\r\n\r\n```python\r\nimport cellxgene_census\r\n\r\nwith cellxgene_census.open_soma() as census:\r\n    # Work with census data\r\n\r\nwith cellxgene_census.open_soma(census_version=\"2023-07-25\") as census:\r\n    # Work with census data\r\n```\r\n\r\n**Key points:**\r\n- Use context manager (`with` statement) for automatic cleanup\r\n- Specify `census_version` for reproducible analyses\r\n- Default opens latest \"stable\" release\r\n\r\n### 2. Exploring Census Information\r\n\r\nBefore querying expression data, explore available datasets and metadata.\r\n\r\n**Access summary information:**\r\n```python\r\nsummary = census[\"census_info\"][\"summary\"].read().concat().to_pandas()\r\nprint(f\"Total cells: {summary['total_cell_count'][0]}\")\r\n\r\ndatasets = census[\"census_info\"][\"datasets\"].read().concat().to_pandas()\r\n\r\ncovid_datasets = datasets[datasets[\"disease\"].str.contains(\"COVID\", na=False)]\r\n```\r\n\r\n**Query cell metadata to understand available data:**\r\n```python\r\ncell_metadata = cellxgene_census.get_obs(\r\n    census,\r\n    \"homo_sapiens\",\r\n    value_filter=\"tissue_general == 'brain' and is_primary_data == True\",\r\n    column_names=[\"cell_type\"]\r\n)\r\nunique_cell_types = cell_metadata[\"cell_type\"].unique()\r\nprint(f\"Found {len(unique_cell_types)} cell types in brain\")\r\n\r\ntissue_counts = cell_metadata.groupby(\"tissue_general\").size()\r\n```\r\n\r\n**Important:** Always filter for `is_primary_data == True` to avoid counting duplicate cells unless specifically analyzing duplicates.\r\n\r\n### 3. Querying Expression Data (Small to Medium Scale)\r\n\r\nFor queries returning < 100k cells that fit in memory, use `get_anndata()`:\r\n\r\n```python\r\nadata = cellxgene_census.get_anndata(\r\n    census=census,\r\n    organism=\"Homo sapiens\",  # or \"Mus musculus\"\r\n    obs_value_filter=\"cell_type == 'B cell' and tissue_general == 'lung' and is_primary_data == True\",\r\n    obs_column_names=[\"assay\", \"disease\", \"sex\", \"donor_id\"],\r\n)\r\n\r\nadata = cellxgene_census.get_anndata(\r\n    census=census,\r\n    organism=\"Homo sapiens\",\r\n    var_value_filter=\"feature_name in ['CD4', 'CD8A', 'CD19', 'FOXP3']\",\r\n    obs_value_filter=\"cell_type == 'T cell' and disease == 'COVID-19' and is_primary_data == True\",\r\n    obs_column_names=[\"cell_type\", \"tissue_general\", \"donor_id\"],\r\n)\r\n```\r\n\r\n**Filter syntax:**\r\n- Use `obs_value_filter` for cell filtering\r\n- Use `var_value_filter` for gene filtering\r\n- Combine conditions with `and`, `or`\r\n- Use `in` for multiple values: `tissue in ['lung', 'liver']`\r\n- Select only needed columns with `obs_column_names`\r\n\r\n**Getting metadata separately:**\r\n```python\r\ncell_metadata = cellxgene_census.get_obs(\r\n    census, \"homo_sapiens\",\r\n    value_filter=\"disease == 'COVID-19' and is_primary_data == True\",\r\n    column_names=[\"cell_type\", \"tissue_general\", \"donor_id\"]\r\n)\r\n\r\ngene_metadata = cellxgene_census.get_var(\r\n    census, \"homo_sapiens\",\r\n    value_filter=\"feature_name in ['CD4', 'CD8A']\",\r\n    column_names=[\"feature_id\", \"feature_name\", \"feature_length\"]\r\n)\r\n```\r\n\r\n### 4. Large-Scale Queries (Out-of-Core Processing)\r\n\r\nFor queries exceeding available RAM, use `axis_query()` with iterative processing:\r\n\r\n```python\r\nimport tiledbsoma as soma\r\n\r\nquery = census[\"census_data\"][\"homo_sapiens\"].axis_query(\r\n    measurement_name=\"RNA\",\r\n    obs_query=soma.AxisQuery(\r\n        value_filter=\"tissue_general == 'brain' and is_primary_data == True\"\r\n    ),\r\n    var_query=soma.AxisQuery(\r\n        value_filter=\"feature_name in ['FOXP2', 'TBR1', 'SATB2']\"\r\n    )\r\n)\r\n\r\niterator = query.X(\"raw\").tables()\r\nfor batch in iterator:\r\n    # batch is a pyarrow.Table with columns:\r\n    # - soma_data: expression value\r\n    # - soma_dim_0: cell (obs) coordinate\r\n    # - soma_dim_1: gene (var) coordinate\r\n    process_batch(batch)\r\n```\r\n\r\n**Computing incremental statistics:**\r\n```python\r\nn_observations = 0\r\nsum_values = 0.0\r\n\r\niterator = query.X(\"raw\").tables()\r\nfor batch in iterator:\r\n    values = batch[\"soma_data\"].to_numpy()\r\n    n_observations += len(values)\r\n    sum_values += values.sum()\r\n\r\nmean_expression = sum_values / n_observations\r\n```\r\n\r\n### 5. Machine Learning with PyTorch\r\n\r\nFor training models, use the experimental PyTorch integration:\r\n\r\n```python\r\nfrom cellxgene_census.experimental.ml import experiment_dataloader\r\n\r\nwith cellxgene_census.open_soma() as census:\r\n    # Create dataloader\r\n    dataloader = experiment_dataloader(\r\n        census[\"census_data\"][\"homo_sapiens\"],\r\n        measurement_name=\"RNA\",\r\n        X_name=\"raw\",\r\n        obs_value_filter=\"tissue_general == 'liver' and is_primary_data == True\",\r\n        obs_column_names=[\"cell_type\"],\r\n        batch_size=128,\r\n        shuffle=True,\r\n    )\r\n\r\n    # Training loop\r\n    for epoch in range(num_epochs):\r\n        for batch in dataloader:\r\n            X = batch[\"X\"]  # Gene expression tensor\r\n            labels = batch[\"obs\"][\"cell_type\"]  # Cell type labels\r\n\r\n            # Forward pass\r\n            outputs = model(X)\r\n            loss = criterion(outputs, labels)\r\n\r\n            # Backward pass\r\n            optimizer.zero_grad()\r\n            loss.backward()\r\n            optimizer.step()\r\n```\r\n\r\n**Train/test splitting:**\r\n```python\r\nfrom cellxgene_census.experimental.ml import ExperimentDataset\r\n\r\ndataset = ExperimentDataset(\r\n    experiment_axis_query,\r\n    layer_name=\"raw\",\r\n    obs_column_names=[\"cell_type\"],\r\n    batch_size=128,\r\n)\r\n\r\ntrain_dataset, test_dataset = dataset.random_split(\r\n    split=[0.8, 0.2],\r\n    seed=42\r\n)\r\n```\r\n\r\n### 6. Integration with Scanpy\r\n\r\nSeamlessly integrate Census data with scanpy workflows:\r\n\r\n```python\r\nimport scanpy as sc\r\n\r\nadata = cellxgene_census.get_anndata(\r\n    census=census,\r\n    organism=\"Homo sapiens\",\r\n    obs_value_filter=\"cell_type == 'neuron' and tissue_general == 'cortex' and is_primary_data == True\",\r\n)\r\n\r\nsc.pp.normalize_total(adata, target_sum=1e4)\r\nsc.pp.log1p(adata)\r\nsc.pp.highly_variable_genes(adata, n_top_genes=2000)\r\n\r\nsc.pp.pca(adata, n_comps=50)\r\nsc.pp.neighbors(adata)\r\nsc.tl.umap(adata)\r\n\r\nsc.pl.umap(adata, color=[\"cell_type\", \"tissue\", \"disease\"])\r\n```\r\n\r\n### 7. Multi-Dataset Integration\r\n\r\nQuery and integrate multiple datasets:\r\n\r\n```python\r\ntissues = [\"lung\", \"liver\", \"kidney\"]\r\nadatas = []\r\n\r\nfor tissue in tissues:\r\n    adata = cellxgene_census.get_anndata(\r\n        census=census,\r\n        organism=\"Homo sapiens\",\r\n        obs_value_filter=f\"tissue_general == '{tissue}' and is_primary_data == True\",\r\n    )\r\n    adata.obs[\"tissue\"] = tissue\r\n    adatas.append(adata)\r\n\r\ncombined = adatas[0].concatenate(adatas[1:])\r\n\r\n\r\n### Always Filter for Primary Data\r\nUnless analyzing duplicates, always include `is_primary_data == True` in queries to avoid counting cells multiple times:\r\n```python\r\nobs_value_filter=\"cell_type == 'B cell' and is_primary_data == True\"\r\n```\r\n\r\n### Specify Census Version for Reproducibility\r\nAlways specify the Census version in production analyses:\r\n```python\r\ncensus = cellxgene_census.open_soma(census_version=\"2023-07-25\")\r\n```\r\n\r\n### Estimate Query Size Before Loading\r\nFor large queries, first check the number of cells to avoid memory issues:\r\n```python\r\nmetadata = cellxgene_census.get_obs(\r\n    census, \"homo_sapiens\",\r\n    value_filter=\"tissue_general == 'brain' and is_primary_data == True\",\r\n    column_names=[\"soma_joinid\"]\r\n)\r\nn_cells = len(metadata)\r\nprint(f\"Query will return {n_cells:,} cells\")\r\n\r\n```\r\n\r\n### Use tissue_general for Broader Groupings\r\nThe `tissue_general` field provides coarser categories than `tissue`, useful for cross-tissue analyses:\r\n```python\r\nobs_value_filter=\"tissue_general == 'immune system'\"\r\n\r\nobs_value_filter=\"tissue == 'peripheral blood mononuclear cell'\"\r\n```\r\n\r\n### Select Only Needed Columns\r\nMinimize data transfer by specifying only required metadata columns:\r\n```python\r\nobs_column_names=[\"cell_type\", \"tissue_general\", \"disease\"]  # Not all columns\r\n```\r\n\r\n### Check Dataset Presence for Gene-Specific Queries\r\nWhen analyzing specific genes, verify which datasets measured them:\r\n```python\r\npresence = cellxgene_census.get_presence_matrix(\r\n    census,\r\n    \"homo_sapiens\",\r\n    var_value_filter=\"feature_name in ['CD4', 'CD8A']\"\r\n)\r\n```\r\n\r\n### Two-Step Workflow: Explore Then Query\r\nFirst explore metadata to understand available data, then query expression:\r\n```python\r\nmetadata = cellxgene_census.get_obs(\r\n    census, \"homo_sapiens\",\r\n    value_filter=\"disease == 'COVID-19' and is_primary_data == True\",\r\n    column_names=[\"cell_type\", \"tissue_general\"]\r\n)\r\nprint(metadata.value_counts())",
  "name": "cellxgene-census",
  "id": "scientific-pkg-cellxgene-census",
  "sections": {
    "Reference Documentation": "This skill includes detailed reference documentation:\r\n\r\n### references/census_schema.md\r\nComprehensive documentation of:\r\n- Census data structure and organization\r\n- All available metadata fields\r\n- Value filter syntax and operators\r\n- SOMA object types\r\n- Data inclusion criteria\r\n\r\n**When to read:** When you need detailed schema information, full list of metadata fields, or complex filter syntax.\r\n\r\n### references/common_patterns.md\r\nExamples and patterns for:\r\n- Exploratory queries (metadata only)\r\n- Small-to-medium queries (AnnData)\r\n- Large queries (out-of-core processing)\r\n- PyTorch integration\r\n- Scanpy integration workflows\r\n- Multi-dataset integration\r\n- Best practices and common pitfalls\r\n\r\n**When to read:** When implementing specific query patterns, looking for code examples, or troubleshooting common issues.",
    "Available Metadata Fields": "### Cell Metadata (obs)\r\nKey fields for filtering:\r\n- `cell_type`, `cell_type_ontology_term_id`\r\n- `tissue`, `tissue_general`, `tissue_ontology_term_id`\r\n- `disease`, `disease_ontology_term_id`\r\n- `assay`, `assay_ontology_term_id`\r\n- `donor_id`, `sex`, `self_reported_ethnicity`\r\n- `development_stage`, `development_stage_ontology_term_id`\r\n- `dataset_id`\r\n- `is_primary_data` (Boolean: True = unique cell)\r\n\r\n### Gene Metadata (var)\r\n- `feature_id` (Ensembl gene ID, e.g., \"ENSG00000161798\")\r\n- `feature_name` (Gene symbol, e.g., \"FOXP2\")\r\n- `feature_length` (Gene length in base pairs)",
    "Troubleshooting": "### Query Returns Too Many Cells\r\n- Add more specific filters to reduce scope\r\n- Use `tissue` instead of `tissue_general` for finer granularity\r\n- Filter by specific `dataset_id` if known\r\n- Switch to out-of-core processing for large queries\r\n\r\n### Memory Errors\r\n- Reduce query scope with more restrictive filters\r\n- Select fewer genes with `var_value_filter`\r\n- Use out-of-core processing with `axis_query()`\r\n- Process data in batches\r\n\r\n### Duplicate Cells in Results\r\n- Always include `is_primary_data == True` in filters\r\n- Check if intentionally querying across multiple datasets\r\n\r\n### Gene Not Found\r\n- Verify gene name spelling (case-sensitive)\r\n- Try Ensembl ID with `feature_id` instead of `feature_name`\r\n- Check dataset presence matrix to see if gene was measured\r\n- Some genes may have been filtered during Census construction\r\n\r\n### Version Inconsistencies\r\n- Always specify `census_version` explicitly\r\n- Use same version across all analyses\r\n- Check release notes for version-specific changes",
    "Overview": "The CZ CELLxGENE Census provides programmatic access to a comprehensive, versioned collection of standardized single-cell genomics data from CZ CELLxGENE Discover. This skill enables efficient querying and analysis of millions of cells across thousands of datasets.\r\n\r\nThe Census includes:\r\n- **61+ million cells** from human and mouse\r\n- **Standardized metadata** (cell types, tissues, diseases, donors)\r\n- **Raw gene expression** matrices\r\n- **Pre-calculated embeddings** and statistics\r\n- **Integration with PyTorch, scanpy, and other analysis tools**",
    "Installation and Setup": "Install the Census API:\r\n```bash\r\npip install cellxgene-census\r\n```\r\n\r\nFor machine learning workflows, install additional dependencies:\r\n```bash\r\npip install cellxgene-census[experimental]\r\n```",
    "When to Use This Skill": "This skill should be used when:\r\n- Querying single-cell expression data by cell type, tissue, or disease\r\n- Exploring available single-cell datasets and metadata\r\n- Training machine learning models on single-cell data\r\n- Performing large-scale cross-dataset analyses\r\n- Integrating Census data with scanpy or other analysis frameworks\r\n- Computing statistics across millions of cells\r\n- Accessing pre-calculated embeddings or model predictions",
    "Common Use Cases": "### Use Case 1: Explore Cell Types in a Tissue\r\n```python\r\nwith cellxgene_census.open_soma() as census:\r\n    cells = cellxgene_census.get_obs(\r\n        census, \"homo_sapiens\",\r\n        value_filter=\"tissue_general == 'lung' and is_primary_data == True\",\r\n        column_names=[\"cell_type\"]\r\n    )\r\n    print(cells[\"cell_type\"].value_counts())\r\n```\r\n\r\n### Use Case 2: Query Marker Gene Expression\r\n```python\r\nwith cellxgene_census.open_soma() as census:\r\n    adata = cellxgene_census.get_anndata(\r\n        census=census,\r\n        organism=\"Homo sapiens\",\r\n        var_value_filter=\"feature_name in ['CD4', 'CD8A', 'CD19']\",\r\n        obs_value_filter=\"cell_type in ['T cell', 'B cell'] and is_primary_data == True\",\r\n    )\r\n```\r\n\r\n### Use Case 3: Train Cell Type Classifier\r\n```python\r\nfrom cellxgene_census.experimental.ml import experiment_dataloader\r\n\r\nwith cellxgene_census.open_soma() as census:\r\n    dataloader = experiment_dataloader(\r\n        census[\"census_data\"][\"homo_sapiens\"],\r\n        measurement_name=\"RNA\",\r\n        X_name=\"raw\",\r\n        obs_value_filter=\"is_primary_data == True\",\r\n        obs_column_names=[\"cell_type\"],\r\n        batch_size=128,\r\n        shuffle=True,\r\n    )\r\n\r\n    # Train model\r\n    for epoch in range(epochs):\r\n        for batch in dataloader:\r\n            # Training logic\r\n            pass\r\n```\r\n\r\n### Use Case 4: Cross-Tissue Analysis\r\n```python\r\nwith cellxgene_census.open_soma() as census:\r\n    adata = cellxgene_census.get_anndata(\r\n        census=census,\r\n        organism=\"Homo sapiens\",\r\n        obs_value_filter=\"cell_type == 'macrophage' and tissue_general in ['lung', 'liver', 'brain'] and is_primary_data == True\",\r\n    )\r\n\r\n    # Analyze macrophage differences across tissues\r\n    sc.tl.rank_genes_groups(adata, groupby=\"tissue_general\")\r\n```",
    "Key Concepts and Best Practices": "adata = cellxgene_census.get_anndata(\r\n    census=census,\r\n    organism=\"Homo sapiens\",\r\n    obs_value_filter=\"disease == 'COVID-19' and cell_type == 'T cell' and is_primary_data == True\",\r\n)\r\n```",
    "Core Workflow Patterns": "adata = cellxgene_census.get_anndata(\r\n    census=census,\r\n    organism=\"Homo sapiens\",\r\n    obs_value_filter=\"tissue_general in ['lung', 'liver', 'kidney'] and is_primary_data == True\",\r\n)\r\n```"
  }
}

---
name: cellxgene-census
description: "Query CZ CELLxGENE Census (61M+ cells). Filter by cell type/tissue/disease, retrieve expression data, integrate with scanpy/PyTorch, for population-scale single-cell analysis."
---

# CZ CELLxGENE Census

## Overview

The CZ CELLxGENE Census provides programmatic access to a comprehensive, versioned collection of standardized single-cell genomics data from CZ CELLxGENE Discover. This skill enables efficient querying and analysis of millions of cells across thousands of datasets.

The Census includes:
- **61+ million cells** from human and mouse
- **Standardized metadata** (cell types, tissues, diseases, donors)
- **Raw gene expression** matrices
- **Pre-calculated embeddings** and statistics
- **Integration with PyTorch, scanpy, and other analysis tools**

## When to Use This Skill

This skill should be used when:
- Querying single-cell expression data by cell type, tissue, or disease
- Exploring available single-cell datasets and metadata
- Training machine learning models on single-cell data
- Performing large-scale cross-dataset analyses
- Integrating Census data with scanpy or other analysis frameworks
- Computing statistics across millions of cells
- Accessing pre-calculated embeddings or model predictions

## Installation and Setup

Install the Census API:
```bash
pip install cellxgene-census
```

For machine learning workflows, install additional dependencies:
```bash
pip install cellxgene-census[experimental]
```

## Core Workflow Patterns

### 1. Opening the Census

Always use the context manager to ensure proper resource cleanup:

```python
import cellxgene_census

# Open latest stable version
with cellxgene_census.open_soma() as census:
    # Work with census data

# Open specific version for reproducibility
with cellxgene_census.open_soma(census_version="2023-07-25") as census:
    # Work with census data
```

**Key points:**
- Use context manager (`with` statement) for automatic cleanup
- Specify `census_version` for reproducible analyses
- Default opens latest "stable" release

### 2. Exploring Census Information

Before querying expression data, explore available datasets and metadata.

**Access summary information:**
```python
# Get summary statistics
summary = census["census_info"]["summary"].read().concat().to_pandas()
print(f"Total cells: {summary['total_cell_count'][0]}")

# Get all datasets
datasets = census["census_info"]["datasets"].read().concat().to_pandas()

# Filter datasets by criteria
covid_datasets = datasets[datasets["disease"].str.contains("COVID", na=False)]
```

**Query cell metadata to understand available data:**
```python
# Get unique cell types in a tissue
cell_metadata = cellxgene_census.get_obs(
    census,
    "homo_sapiens",
    value_filter="tissue_general == 'brain' and is_primary_data == True",
    column_names=["cell_type"]
)
unique_cell_types = cell_metadata["cell_type"].unique()
print(f"Found {len(unique_cell_types)} cell types in brain")

# Count cells by tissue
tissue_counts = cell_metadata.groupby("tissue_general").size()
```

**Important:** Always filter for `is_primary_data == True` to avoid counting duplicate cells unless specifically analyzing duplicates.

### 3. Querying Expression Data (Small to Medium Scale)

For queries returning < 100k cells that fit in memory, use `get_anndata()`:

```python
# Basic query with cell type and tissue filters
adata = cellxgene_census.get_anndata(
    census=census,
    organism="Homo sapiens",  # or "Mus musculus"
    obs_value_filter="cell_type == 'B cell' and tissue_general == 'lung' and is_primary_data == True",
    obs_column_names=["assay", "disease", "sex", "donor_id"],
)

# Query specific genes with multiple filters
adata = cellxgene_census.get_anndata(
    census=census,
    organism="Homo sapiens",
    var_value_filter="feature_name in ['CD4', 'CD8A', 'CD19', 'FOXP3']",
    obs_value_filter="cell_type == 'T cell' and disease == 'COVID-19' and is_primary_data == True",
    obs_column_names=["cell_type", "tissue_general", "donor_id"],
)
```

**Filter syntax:**
- Use `obs_value_filter` for cell filtering
- Use `var_value_filter` for gene filtering
- Combine conditions with `and`, `or`
- Use `in` for multiple values: `tissue in ['lung', 'liver']`
- Select only needed columns with `obs_column_names`

**Getting metadata separately:**
```python
# Query cell metadata
cell_metadata = cellxgene_census.get_obs(
    census, "homo_sapiens",
    value_filter="disease == 'COVID-19' and is_primary_data == True",
    column_names=["cell_type", "tissue_general", "donor_id"]
)

# Query gene metadata
gene_metadata = cellxgene_census.get_var(
    census, "homo_sapiens",
    value_filter="feature_name in ['CD4', 'CD8A']",
    column_names=["feature_id", "feature_name", "feature_length"]
)
```

### 4. Large-Scale Queries (Out-of-Core Processing)

For queries exceeding available RAM, use `axis_query()` with iterative processing:

```python
import tiledbsoma as soma

# Create axis query
query = census["census_data"]["homo_sapiens"].axis_query(
    measurement_name="RNA",
    obs_query=soma.AxisQuery(
        value_filter="tissue_general == 'brain' and is_primary_data == True"
    ),
    var_query=soma.AxisQuery(
        value_filter="feature_name in ['FOXP2', 'TBR1', 'SATB2']"
    )
)

# Iterate through expression matrix in chunks
iterator = query.X("raw").tables()
for batch in iterator:
    # batch is a pyarrow.Table with columns:
    # - soma_data: expression value
    # - soma_dim_0: cell (obs) coordinate
    # - soma_dim_1: gene (var) coordinate
    process_batch(batch)
```

**Computing incremental statistics:**
```python
# Example: Calculate mean expression
n_observations = 0
sum_values = 0.0

iterator = query.X("raw").tables()
for batch in iterator:
    values = batch["soma_data"].to_numpy()
    n_observations += len(values)
    sum_values += values.sum()

mean_expression = sum_values / n_observations
```

### 5. Machine Learning with PyTorch

For training models, use the experimental PyTorch integration:

```python
from cellxgene_census.experimental.ml import experiment_dataloader

with cellxgene_census.open_soma() as census:
    # Create dataloader
    dataloader = experiment_dataloader(
        census["census_data"]["homo_sapiens"],
        measurement_name="RNA",
        X_name="raw",
        obs_value_filter="tissue_general == 'liver' and is_primary_data == True",
        obs_column_names=["cell_type"],
        batch_size=128,
        shuffle=True,
    )

    # Training loop
    for epoch in range(num_epochs):
        for batch in dataloader:
            X = batch["X"]  # Gene expression tensor
            labels = batch["obs"]["cell_type"]  # Cell type labels

            # Forward pass
            outputs = model(X)
            loss = criterion(outputs, labels)

            # Backward pass
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
```

**Train/test splitting:**
```python
from cellxgene_census.experimental.ml import ExperimentDataset

# Create dataset from experiment
dataset = ExperimentDataset(
    experiment_axis_query,
    layer_name="raw",
    obs_column_names=["cell_type"],
    batch_size=128,
)

# Split into train and test
train_dataset, test_dataset = dataset.random_split(
    split=[0.8, 0.2],
    seed=42
)
```

### 6. Integration with Scanpy

Seamlessly integrate Census data with scanpy workflows:

```python
import scanpy as sc

# Load data from Census
adata = cellxgene_census.get_anndata(
    census=census,
    organism="Homo sapiens",
    obs_value_filter="cell_type == 'neuron' and tissue_general == 'cortex' and is_primary_data == True",
)

# Standard scanpy workflow
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000)

# Dimensionality reduction
sc.pp.pca(adata, n_comps=50)
sc.pp.neighbors(adata)
sc.tl.umap(adata)

# Visualization
sc.pl.umap(adata, color=["cell_type", "tissue", "disease"])
```

### 7. Multi-Dataset Integration

Query and integrate multiple datasets:

```python
# Strategy 1: Query multiple tissues separately
tissues = ["lung", "liver", "kidney"]
adatas = []

for tissue in tissues:
    adata = cellxgene_census.get_anndata(
        census=census,
        organism="Homo sapiens",
        obs_value_filter=f"tissue_general == '{tissue}' and is_primary_data == True",
    )
    adata.obs["tissue"] = tissue
    adatas.append(adata)

# Concatenate
combined = adatas[0].concatenate(adatas[1:])

# Strategy 2: Query multiple datasets directly
adata = cellxgene_census.get_anndata(
    census=census,
    organism="Homo sapiens",
    obs_value_filter="tissue_general in ['lung', 'liver', 'kidney'] and is_primary_data == True",
)
```

## Key Concepts and Best Practices

### Always Filter for Primary Data
Unless analyzing duplicates, always include `is_primary_data == True` in queries to avoid counting cells multiple times:
```python
obs_value_filter="cell_type == 'B cell' and is_primary_data == True"
```

### Specify Census Version for Reproducibility
Always specify the Census version in production analyses:
```python
census = cellxgene_census.open_soma(census_version="2023-07-25")
```

### Estimate Query Size Before Loading
For large queries, first check the number of cells to avoid memory issues:
```python
# Get cell count
metadata = cellxgene_census.get_obs(
    census, "homo_sapiens",
    value_filter="tissue_general == 'brain' and is_primary_data == True",
    column_names=["soma_joinid"]
)
n_cells = len(metadata)
print(f"Query will return {n_cells:,} cells")

# If too large (>100k), use out-of-core processing
```

### Use tissue_general for Broader Groupings
The `tissue_general` field provides coarser categories than `tissue`, useful for cross-tissue analyses:
```python
# Broader grouping
obs_value_filter="tissue_general == 'immune system'"

# Specific tissue
obs_value_filter="tissue == 'peripheral blood mononuclear cell'"
```

### Select Only Needed Columns
Minimize data transfer by specifying only required metadata columns:
```python
obs_column_names=["cell_type", "tissue_general", "disease"]  # Not all columns
```

### Check Dataset Presence for Gene-Specific Queries
When analyzing specific genes, verify which datasets measured them:
```python
presence = cellxgene_census.get_presence_matrix(
    census,
    "homo_sapiens",
    var_value_filter="feature_name in ['CD4', 'CD8A']"
)
```

### Two-Step Workflow: Explore Then Query
First explore metadata to understand available data, then query expression:
```python
# Step 1: Explore what's available
metadata = cellxgene_census.get_obs(
    census, "homo_sapiens",
    value_filter="disease == 'COVID-19' and is_primary_data == True",
    column_names=["cell_type", "tissue_general"]
)
print(metadata.value_counts())

# Step 2: Query based on findings
adata = cellxgene_census.get_anndata(
    census=census,
    organism="Homo sapiens",
    obs_value_filter="disease == 'COVID-19' and cell_type == 'T cell' and is_primary_data == True",
)
```

## Available Metadata Fields

### Cell Metadata (obs)
Key fields for filtering:
- `cell_type`, `cell_type_ontology_term_id`
- `tissue`, `tissue_general`, `tissue_ontology_term_id`
- `disease`, `disease_ontology_term_id`
- `assay`, `assay_ontology_term_id`
- `donor_id`, `sex`, `self_reported_ethnicity`
- `development_stage`, `development_stage_ontology_term_id`
- `dataset_id`
- `is_primary_data` (Boolean: True = unique cell)

### Gene Metadata (var)
- `feature_id` (Ensembl gene ID, e.g., "ENSG00000161798")
- `feature_name` (Gene symbol, e.g., "FOXP2")
- `feature_length` (Gene length in base pairs)

## Reference Documentation

This skill includes detailed reference documentation:

### references/census_schema.md
Comprehensive documentation of:
- Census data structure and organization
- All available metadata fields
- Value filter syntax and operators
- SOMA object types
- Data inclusion criteria

**When to read:** When you need detailed schema information, full list of metadata fields, or complex filter syntax.

### references/common_patterns.md
Examples and patterns for:
- Exploratory queries (metadata only)
- Small-to-medium queries (AnnData)
- Large queries (out-of-core processing)
- PyTorch integration
- Scanpy integration workflows
- Multi-dataset integration
- Best practices and common pitfalls

**When to read:** When implementing specific query patterns, looking for code examples, or troubleshooting common issues.

## Common Use Cases

### Use Case 1: Explore Cell Types in a Tissue
```python
with cellxgene_census.open_soma() as census:
    cells = cellxgene_census.get_obs(
        census, "homo_sapiens",
        value_filter="tissue_general == 'lung' and is_primary_data == True",
        column_names=["cell_type"]
    )
    print(cells["cell_type"].value_counts())
```

### Use Case 2: Query Marker Gene Expression
```python
with cellxgene_census.open_soma() as census:
    adata = cellxgene_census.get_anndata(
        census=census,
        organism="Homo sapiens",
        var_value_filter="feature_name in ['CD4', 'CD8A', 'CD19']",
        obs_value_filter="cell_type in ['T cell', 'B cell'] and is_primary_data == True",
    )
```

### Use Case 3: Train Cell Type Classifier
```python
from cellxgene_census.experimental.ml import experiment_dataloader

with cellxgene_census.open_soma() as census:
    dataloader = experiment_dataloader(
        census["census_data"]["homo_sapiens"],
        measurement_name="RNA",
        X_name="raw",
        obs_value_filter="is_primary_data == True",
        obs_column_names=["cell_type"],
        batch_size=128,
        shuffle=True,
    )

    # Train model
    for epoch in range(epochs):
        for batch in dataloader:
            # Training logic
            pass
```

### Use Case 4: Cross-Tissue Analysis
```python
with cellxgene_census.open_soma() as census:
    adata = cellxgene_census.get_anndata(
        census=census,
        organism="Homo sapiens",
        obs_value_filter="cell_type == 'macrophage' and tissue_general in ['lung', 'liver', 'brain'] and is_primary_data == True",
    )

    # Analyze macrophage differences across tissues
    sc.tl.rank_genes_groups(adata, groupby="tissue_general")
```

## Troubleshooting

### Query Returns Too Many Cells
- Add more specific filters to reduce scope
- Use `tissue` instead of `tissue_general` for finer granularity
- Filter by specific `dataset_id` if known
- Switch to out-of-core processing for large queries

### Memory Errors
- Reduce query scope with more restrictive filters
- Select fewer genes with `var_value_filter`
- Use out-of-core processing with `axis_query()`
- Process data in batches

### Duplicate Cells in Results
- Always include `is_primary_data == True` in filters
- Check if intentionally querying across multiple datasets

### Gene Not Found
- Verify gene name spelling (case-sensitive)
- Try Ensembl ID with `feature_id` instead of `feature_name`
- Check dataset presence matrix to see if gene was measured
- Some genes may have been filtered during Census construction

### Version Inconsistencies
- Always specify `census_version` explicitly
- Use same version across all analyses
- Check release notes for version-specific changes