Hybrid Cloud Outboxes

Hybrid Cloud Outboxes

Sentry uses a transactional outbox pattern for eventually consistent operations. When a model changes, an outbox row is written inside the same database transaction. After the transaction commits, the outbox is drained — firing a signal that triggers side effects such as RPC calls, tombstone propagation, or audit logging.

The most common use case is cross-silo data replication: a model saved in the Cell silo produces a CellOutbox that, when processed, replicates data to the Control silo (or vice versa via ControlOutbox). But the pattern is general — outboxes work for any operation that should happen reliably after a transaction commits, even within a single silo.

There are two outbox types corresponding to the two directions of flow:

• `CellOutbox` — written in a Cell silo, processed in the Cell silo to push data toward Control (via RPC calls in signal receivers).
• `ControlOutbox` — written in the Control silo, processed in the Control silo to push data toward one or more Cell silos. Each ControlOutbox row targets a specific cell_name.

Critical Constraints

Outboxes MUST be written in the same transaction as the data change.

The mixin classes (ReplicatedCellModel, ReplicatedControlModel) enforce this automatically via prepare_outboxes(). If you write outboxes manually, always use outbox_context(transaction.atomic(...)).

Handlers MUST be idempotent.

Outboxes can be retried on failure and are coalesced — the handler may receive only the latest version of a change, or be called multiple times for the same change.

`drain_shard()` MUST NOT run inside a transaction.

It acquires SELECT FOR UPDATE locks and processes messages one at a time. Calling it inside a transaction will deadlock or hold locks for too long.

Only the latest payload survives coalescing.

Multiple outbox writes for the same (scope, shard_identifier, category, object_identifier) are coalesced — only the row with the highest ID is processed. Never rely on every intermediate payload being delivered.

Every `OutboxCategory` must be registered to exactly one `OutboxScope`.

An assertion at import time enforces this. A category registered to zero or multiple scopes causes an import crash.

Bulk operations must use the producing manager.

Use MyModel.objects.bulk_create() / bulk_update() / bulk_delete() from CellOutboxProducingManager or ControlOutboxProducingManager. Raw querysets bypass outbox creation.

Snowflake ID models cannot use `bulk_create`.

The producing manager pre-allocates IDs via SELECT nextval(...), which conflicts with snowflake ID generation. Use individual save() calls instead.

Step 1: Determine What You Need

Step 2: Add Outbox Replication to a New Model

2.1 Choose the Mixin

2.2 ReplicatedCellModel Template

Use this when a Cell model needs to replicate data to the Control silo.

from sentry.backup.scopes import RelocationScope
from sentry.db.models import (
    FlexibleForeignKey,
    Model,
    cell_silo_model,
    sane_repr,
)
from sentry.db.models.manager.base_query_set import BaseQuerySet
from sentry.hybridcloud.outbox.base import ReplicatedCellModel, CellOutboxProducingManager
from sentry.hybridcloud.outbox.category import OutboxCategory


class MyModelManager(CellOutboxProducingManager["MyModel"]):
    """Manager that ensures bulk operations create outboxes."""
    pass


@cell_silo_model
class MyModel(ReplicatedCellModel):
    __relocation_scope__ = RelocationScope.Organization

    # Required: the OutboxCategory for this model (must already be registered)
    category = OutboxCategory.MY_MODEL_UPDATE

    # Use the producing manager for bulk operation support
    objects: ClassVar[MyModelManager] = MyModelManager()

    # Model fields...
    organization = FlexibleForeignKey("sentry.Organization")
    name = models.CharField(max_length=128)

    class Meta:
        app_label = "sentry"
        db_table = "sentry_mymodel"

    def payload_for_update(self) -> dict[str, Any] | None:
        """
        Optional: include data needed by the deletion handler.
        Keep payloads minimal — only data that cannot be recovered
        after the row is deleted. Payloads are coalesced (only the
        latest survives).
        """
        return None  # Override if needed

    @classmethod
    def handle_async_deletion(
        cls,
        identifier: int,
        shard_identifier: int,
        payload: Mapping[str, Any] | None,
    ) -> None:
        """
        Called when this object has been deleted (row no longer exists).
        Clean up cross-silo resources. Must be idempotent.
        """
        my_mapping_service.delete(
            my_model_id=identifier,
            organization_id=shard_identifier,
        )

    def handle_async_replication(self, shard_identifier: int) -> None:
        """
        Called when this object has been created or updated.
        Replicate to the control silo via RPC. Must be idempotent.
        """
        my_mapping_service.upsert(
            my_model_id=self.id,
            organization_id=shard_identifier,
            mapping=RpcMyModelMapping.from_orm(self),
        )

2.3 ReplicatedControlModel Template

Use this when a Control model needs to replicate data to Cell silo(s). The key difference: Control outboxes fan out to one or more cells, so the model must declare which cells to target.

from sentry.db.models import control_silo_model
from sentry.hybridcloud.outbox.base import ReplicatedControlModel, ControlOutboxProducingManager
from sentry.hybridcloud.outbox.category import OutboxCategory


class MyControlModelManager(ControlOutboxProducingManager["MyControlModel"]):
    pass


@control_silo_model
class MyControlModel(ReplicatedControlModel):
    __relocation_scope__ = RelocationScope.Global

    category = OutboxCategory.MY_CONTROL_MODEL_UPDATE

    objects: ClassVar[MyControlModelManager] = MyControlModelManager()

    # Model fields...
    organization = FlexibleForeignKey("sentry.Organization")
    user = FlexibleForeignKey("sentry.User")

    class Meta:
        app_label = "sentry"
        db_table = "sentry_mycontrolmodel"

    def outbox_cell_names(self) -> Collection[str]:
        """
        Which cells should receive outboxes for this change.
        Default implementation checks organization_id then user_id.
        Override for custom logic (e.g., all cells, specific cells).
        """
        # Default: auto-detects from organization_id or user_id attributes.
        # Override only if the default doesn't work for your model.
        return super().outbox_cell_names()

    @classmethod
    def handle_async_deletion(
        cls,
        identifier: int,
        cell_name: str,
        shard_identifier: int,
        payload: Mapping[str, Any] | None,
    ) -> None:
        """Note: receives cell_name — one call per target cell."""
        pass

    def handle_async_replication(self, cell_name: str, shard_identifier: int) -> None:
        """Note: receives cell_name — one call per target cell."""
        pass

2.4 Wire Up the Category Connection

The mixin classes auto-connect signal receivers via OutboxCategory.connect_cell_model_updates() (or connect_control_model_updates()). This happens at class definition time when the category class variable is set. The connection dispatches to your handle_async_replication and handle_async_deletion methods automatically.

No manual signal receiver is needed for replicated models — the mixin handles it. Manual receivers are only needed for categories that don't map to a replicated model (see Step 4).

If your OutboxCategory doesn't exist yet, create it first (Step 3).

Step 3: Add a New OutboxCategory

Every outbox message type needs an OutboxCategory enum value registered to exactly one OutboxScope.

Quick steps:

1. Add a new value to the OutboxCategory enum in src/sentry/hybridcloud/outbox/category.py 2. Register it under the appropriate OutboxScope (determines the shard key) 3. If using model mixins, set category = OutboxCategory.MY_CATEGORY on the model

Load references/category-and-scope.md for the full scope-to-category mapping, how to pick a scope, and registration mechanics.

Step 4: Write a Manual Signal Receiver

Use manual receivers when the outbox category is not tied to a ReplicatedCellModel or ReplicatedControlModel. Common cases:

• Payload-only operations (audit logs, IP events) that carry all data in the payload
• Actions triggered by a model change but not replicating that model directly
• Cross-silo signal forwarding (SEND_SIGNAL, RESET_IDP_FLAGS)
• Complex multi-step operations requiring custom dispatch logic

Load references/signal-receivers.md for copy-paste receiver templates, the maybe_process_tombstone pattern, and placement rules.

Step 5: Migrate an Existing Model to Use Outboxes

When adding outbox replication to a model that already has data in production:

5.1 Code Changes (Non-Breaking)

1. Change the model's base class to ReplicatedCellModel or ReplicatedControlModel 2. Add the category class variable 3. Add a producing manager (CellOutboxProducingManager / ControlOutboxProducingManager) 4. Implement handle_async_replication and handle_async_deletion 5. If needed, add payload_for_update() for deletion recovery data 6. Create the OutboxCategory if it doesn't exist (Step 3)

These changes are non-breaking: new model saves will create outboxes, but existing rows have no outboxes yet.

5.2 Backfill Existing Data

Existing rows need outboxes created retroactively. Set replication_version = 2 (or higher) on the model class and configure the backfill system — see Step 6.

Step 6: Set Up a Backfill

The backfill system creates outboxes for existing model rows that predate the outbox integration. It processes rows in batches, tracked via Redis state.

Load references/backfill.md for the replication_version mechanism, option key format, Redis state tracking, and SaaS vs self-hosted rollout procedures.

Step 7: Test Outbox-Based Replication

For detailed outbox test templates and copy-paste patterns, invoke the hybrid-cloud-test-gen skill.

The guidance below covers what to test; hybrid-cloud-test-gen covers how to generate the test code.

7.1 Core Test Utilities

`outbox_runner()` — the primary test tool. Context manager that drains all pending outboxes synchronously after the wrapped code succeeds:

from sentry.testutils.outbox import outbox_runner

with outbox_runner():
    my_model.save()
# All outboxes drained — cross-silo effects have happened

It runs up to 10 drain iterations (raises OutboxRecursionLimitError if exceeded). Works with TestCase — no TransactionTestCase needed for standard outbox tests.

`outbox_context(flush=False)` — creates outbox records without processing them. Use to verify outbox creation independently of processing:

from sentry.hybridcloud.models.outbox import outbox_context

with outbox_context(flush=False):
    MyModel(id=10).outbox_for_update().save()

assert CellOutbox.objects.count() == 1

`assume_test_silo_mode` / `assume_test_silo_mode_of` — switch silo context within a test to query cross-silo models:

from sentry.testutils.silo import assume_test_silo_mode_of

with assume_test_silo_mode_of(MyMapping):
    assert MyMapping.objects.filter(my_model_id=obj.id).exists()

7.2 What to Test

Outbox creation — verify saving/deleting the model creates outbox rows with correct scope, category, and identifiers:

def test_outbox_created_on_save(self):
    with outbox_context(flush=False):
        obj = MyModel(id=10, organization_id=1)
        obj.outbox_for_update().save()

    outbox = CellOutbox.objects.first()
    assert outbox.category == OutboxCategory.MY_MODEL_UPDATE.value
    assert outbox.shard_scope == OutboxScope.ORGANIZATION_SCOPE.value
    assert outbox.shard_identifier == 1

Replication propagates — verify the full round-trip: save model -> drain outboxes -> cross-silo effect:

def test_replication_creates_mapping(self):
    org = self.create_organization()
    with outbox_runner():
        obj = MyModel.objects.create(organization=org, name="test")

    with assume_test_silo_mode_of(MyMapping):
        mapping = MyMapping.objects.get(my_model_id=obj.id)
        assert mapping.name == "test"

Deletion and tombstone — verify deleting the model triggers handle_async_deletion and cleans up cross-silo resources:

def test_delete_cleans_up_mapping(self):
    org = self.create_organization()
    with outbox_runner():
        obj = MyModel.objects.create(organization=org, name="test")

    with outbox_runner():
        obj.delete()

    with assume_test_silo_mode_of(MyMapping):
        assert not MyMapping.objects.filter(my_model_id=obj.id).exists()

Idempotency — verify draining the same shard twice produces no duplicates or errors:

def test_idempotent_replication(self):
    with outbox_runner():
        obj = MyModel.objects.create(organization=org, name="test")

    with assume_test_silo_mode_of(MyMapping):
        count_after_first = MyMapping.objects.count()

    with outbox_runner():
        pass  # Drain again — should be a no-op

    with assume_test_silo_mode_of(MyMapping):
        assert MyMapping.objects.count() == count_after_first

7.3 Silo Test Decorators

• Use `@cell_silo_test` for tests focused on CellOutbox creation
• Use `@control_silo_test` for tests focused on ControlOutbox creation
• Use `@all_silo_test` for end-to-end replication tests that exercise both silos
• Only use `TransactionTestCase` for threading/concurrency tests (e.g., threading.Barrier), not for standard outbox drain tests

7.4 Common Pitfalls

• Factory calls (self.create_organization(), etc.) must NEVER be wrapped in assume_test_silo_mode. Factories handle silo mode internally.
• `outbox_runner()` clears outboxes on exit. If you need to inspect outbox state, use outbox_context(flush=False) instead.
• If an outbox handler creates more outboxes (cascading), outbox_runner handles this automatically (up to 10 iterations).

Step 8: Debug Stuck Outboxes

Load references/debugging.md for the full processing pipeline walkthrough, shard inspection methods, backoff schedule, kill switches, and useful SQL/metrics queries.

Step 9: Verify (Pre-flight Checklist)

Before submitting your PR, verify:

• [ ] Model inherits from ReplicatedCellModel or ReplicatedControlModel (or uses manual receivers)
• [ ] category class variable is set to the correct OutboxCategory
• [ ] OutboxCategory is registered to exactly one OutboxScope
• [ ] The chosen OutboxScope matches the model's shard key (organization_id, user_id, etc.)
• [ ] handle_async_replication is idempotent (safe to call multiple times)
• [ ] handle_async_deletion is idempotent and handles the case where the row is already gone
• [ ] payload_for_update() includes only data needed for deletion recovery (not rapidly-changing fields)
• [ ] Producing manager (CellOutboxProducingManager / ControlOutboxProducingManager) is set on the model
• [ ] Bulk operations go through the producing manager, not raw querysets
• [ ] ReplicatedControlModel has correct outbox_cell_names() implementation
• [ ] Tests verify outbox creation (scope, category, identifiers)
• [ ] Tests verify end-to-end replication (save -> drain -> cross-silo effect)
• [ ] Tests verify deletion propagation (delete -> drain -> cleanup)
• [ ] Tests verify idempotency (drain twice -> no duplicates)
• [ ] If migrating an existing model, replication_version is bumped and backfill is configured

Outbox Backfill Reference

Overview

When a model is migrated to use outboxes (or its replication logic changes), existing rows need outboxes created retroactively. The backfill system handles this incrementally, processing rows in batches with cursor position tracked in Redis and version gating controlled by the sentry options system.

Source file: src/sentry/hybridcloud/tasks/backfill_outboxes.py

replication_version Mechanism

Every CellOutboxProducingModel and ControlOutboxProducingModel has a class variable:

replication_version: int = 1  # Default

Two systems work together to control backfills:

1. Sentry options — gate the effective replication version (controls _whether_ a backfill runs) 2. Redis cursor — track backfill progress as (lower_bound_id, current_version) (controls _where_ a backfill resumes)

Version Resolution via Options

find_replication_version() determines the effective target version:

def find_replication_version(model, force_synchronous=False) -> int:
    coded_version = model.replication_version
    if force_synchronous:
        return coded_version
    model_key = f"outbox_replication.{model._meta.db_table}.replication_version"
    return min(options.get(model_key), coded_version)

The effective version is min(option_value, coded_version). This means:

• If the option is not set or set lower than the code, the backfill won't advance to the new version
• If the option is set equal to or higher than the code, the coded version is used
• If force_synchronous=True (self-hosted), the option is bypassed entirely

Cursor Tracking via Redis

Redis tracks (lower_bound_id, current_version) per model table:

# Key format:
f"outbox_backfill.{model._meta.db_table}"

# Value: JSON-encoded tuple of (lower_bound_id, current_version)

_chunk_processing_batch() compares the Redis cursor's version against the options-resolved target_version:

• If version > target_version: backfill already complete, skip
• If version < target_version: new version detected, reset cursor to 0 and start fresh
• If version == target_version: continue from where we left off

To trigger a backfill: Bump replication_version on the model class:

class MyModel(ReplicatedCellModel):
    replication_version = 2  # Was 1; bumping triggers backfill

SaaS vs Self-Hosted Rollout

SaaS (Gradual Rollout via Options)

The option key format is:

f"outbox_replication.{model._meta.db_table}.replication_version"

# Example for OrganizationMember:
"outbox_replication.sentry_organizationmember.replication_version"

Rollout procedure:

1. Merge the code change with bumped replication_version 2. At this point, min(option_value, coded_version) still returns the old version — no backfill runs yet 3. Set the option to the new version value in the Sentry options system 4. Now min(option_value, coded_version) returns the new version — backfill starts on the next enqueue_outbox_jobs cycle 5. Monitor via Redis cursor state and task metrics

This two-step process allows deploying code first, then enabling the backfill separately — useful for coordinating with other changes or rolling back quickly by lowering the option.

Self-Hosted (Synchronous)

On self-hosted instances, backfills run synchronously during sentry upgrade via the run_outbox_replications_for_self_hosted function (connected to the post_upgrade signal). This function:

1. Calls backfill_outboxes_for(force_synchronous=True) — bypasses options, uses model.replication_version directly 2. Drains all pending outbox shards 3. Ensures the instance is fully caught up after every upgrade

Redis Cursor State Transitions

1. Initial: (0, 1) — no backfill has run (created on first get_processing_state call) 2. In progress: (last_processed_id + 1, target_version) — backfill is processing rows 3. Complete: (0, replication_version + 1) — all rows processed, version advanced past target 4. New version detected: cursor resets to (0, new_target_version) and starts from the beginning

Batch Processing

OUTBOX_BACKFILLS_PER_MINUTE = 10_000

Each batch (via process_outbox_backfill_batch):

1. Calls _chunk_processing_batch to determine the ID range (low, up) for this batch 2. For each instance in model.objects.filter(id__gte=low, id__lte=up):

• Cell models: inst.outbox_for_update().save() inside outbox_context(flush=False)
• Control models: saves all inst.outboxes_for_update() inside outbox_context(flush=False)

3. If no more rows: sets cursor to (0, replication_version + 1) (marks complete) 4. Otherwise: advances cursor to (up + 1, version)

Rate is limited by OUTBOX_BACKFILLS_PER_MINUTE adjusted by the count of already-scheduled outboxes. The backfill_outboxes_for function iterates all registered models and processes batches until the rate limit is reached.

Monitoring a Backfill

Check Redis Cursor State

from sentry.hybridcloud.tasks.backfill_outboxes import get_processing_state

lower_bound, version = get_processing_state("sentry_mymodel")
# lower_bound > 0 means backfill is in progress
# version == model.replication_version + 1 means backfill is complete

Check Option Value

from sentry import options

# See what version the option is gating to:
options.get("outbox_replication.sentry_mymodel.replication_version")

Check Outbox Queue Depth

-- Cell outboxes for a specific category
SELECT count(*) FROM sentry_regionoutbox
WHERE category = <category_value>;

-- Top shards by depth
SELECT shard_scope, shard_identifier, count(*) as depth
FROM sentry_regionoutbox
GROUP BY shard_scope, shard_identifier
ORDER BY depth DESC
LIMIT 10;

Metrics

• backfill_outboxes.low_bound — gauge of the current cursor position per table
• backfill_outboxes.backfilled — counter of rows backfilled per cycle
• outbox.saved — counter incremented each time an outbox is saved
• outbox.processed — counter incremented each time a coalesced outbox is processed
• outbox.processing_lag — histogram of time from outbox creation to processing

OutboxCategory and OutboxScope Reference

Overview

Every outbox message has a category (what kind of change) and a scope (how it's sharded). Categories are members of the OutboxCategory IntEnum; scopes are members of OutboxScope. Each category must be registered to exactly one scope — an assertion at import time enforces this.

Source file: src/sentry/hybridcloud/outbox/category.py

Scope-to-Category Mapping

Scope to category mappings can be found in src/sentry/hybridcloud/outbox/category.py

When selecting a scope to use, consider which other operations the target outbox depends on.

Retired Categories and Scopes

Categories and scopes should never be deleted. If a category is to be retired, simply add an inline comment denoting it as no longer in use.

If a scope is to be retired, remove all categories from its nested definition, and denote that it's no longer in use with a comment above the list.

Sharding Pitfalls

Understanding how shards interact with processing is critical to choosing the right scope. Getting it wrong causes subtle, hard-to-diagnose production issues.

Head-of-Line Blocking

A shard is processed sequentially — every category sharing the same (scope, shard_identifier) sits in one queue. If a handler for one category fails, all other categories in that shard enter backoff together. The entire shard's scheduled_for is bumped, not just the failing message's.

Example: ORGANIZATION_SCOPE groups ~21 categories per org. If the AUTH_PROVIDER_UPDATE handler crashes for org 42, then ORGANIZATION_MEMBER_UPDATE, PROJECT_UPDATE, and all other org-42 categories are blocked until the backoff expires and the failing handler either succeeds or is fixed.

This is why high-volume or failure-prone operations sometimes get their own dedicated scope (e.g., AUDIT_LOG_SCOPE and USER_IP_SCOPE are separate from ORGANIZATION_SCOPE and USER_SCOPE respectively) — isolating them prevents their failures from blocking unrelated replication work.

Harmful Coalescing

Outboxes with the same (scope, shard_identifier, category, object_identifier) are coalesced: only the row with the highest ID is processed, all others are deleted. This is correct for "latest state wins" replication (model sync) but destructive for event-style data where every occurrence matters.

Bad: Using a single category for audit log events with object_identifier = org_id. Multiple audit events for the same org would coalesce to just the latest one — losing audit history.

Good: AUDIT_LOG_EVENT uses its own scope and carries all data in the payload. Each event gets a unique object_identifier (or the coalescing is harmless because the payload is self-contained).

Rule: If every individual outbox message matters (not just the latest), either ensure object_identifier is unique per message, or use a payload-only pattern where coalescing the envelope is harmless because the signal receiver reads the payload, not the DB row.

Hot Shards

A "hot shard" is a single (scope, shard_identifier) with a disproportionate number of pending outboxes. Since one shard is processed sequentially, a hot shard becomes a bottleneck.

Causes:

• A large org with frequent updates across many categories in ORGANIZATION_SCOPE
• A backfill that generates thousands of outboxes for a single shard
• A handler that's slow (network calls, large queries), causing the shard to grow faster than it drains

Mitigation: The system has should_skip_shard() kill switches for disabling specific org/user shards, and the get_shard_depths_descending() method helps identify hot shards. But the best fix is choosing a scope with the right granularity — see "When to Create a New Scope" below.

Wrong Shard Key

If your model's natural grouping doesn't match the scope's shard key, you get either unnecessary contention or broken ordering guarantees.

Example: Putting an integration-scoped model under ORGANIZATION_SCOPE means all integration changes for an org share a shard with org member updates, project updates, etc. — contention with no benefit. Worse, if the model doesn't have an organization_id at all, infer_identifiers() will fail at runtime.

When to Create a New Category

Always create a new category when:

• You have a new model inheriting from ReplicatedCellModel or ReplicatedControlModel
• You have a new type of event/signal that needs outbox delivery
• The handler logic is distinct from all existing categories

Do not reuse an existing category for a different model or operation. Categories map 1:1 to signal receivers — reusing means both models' changes trigger the same handler.

When to Create a New Scope vs Reuse an Existing One

Reuse an existing scope when:

• Your model naturally keys on the same identifier (e.g., has organization_id → use ORGANIZATION_SCOPE)
• Head-of-line blocking with the other categories in that scope is acceptable (i.e., your handler is reliable and fast)
• Coalescing with the existing shard granularity makes sense for your data

Create a new scope when:

• Your model's natural key doesn't match any existing scope (e.g., keyed on integration_id before INTEGRATION_SCOPE existed)
• Your handler is high-volume or failure-prone, and blocking other categories is unacceptable
• Your operation is event-style (every message matters) and you need isolation from "latest state wins" categories
• You need a different shard key granularity (e.g., per-token rather than per-org)

Examples of good scope isolation decisions:

• AUDIT_LOG_SCOPE — high-volume, every event matters, failures shouldn't block org replication
• USER_IP_SCOPE — very high-volume fire-and-forget, isolates from user profile replication
• PROVISION_SCOPE — rare but critical, isolates from general org updates to avoid head-of-line blocking during provisioning
• API_TOKEN_SCOPE — tokens aren't org-scoped or user-scoped in a way that fits existing scopes

Rule of thumb: Start with an existing scope that matches your shard key. Only create a new scope if you have a concrete concern about head-of-line blocking, harmful coalescing, or hot shards. Unnecessary scope proliferation adds operational complexity (more shards to monitor, more code paths to maintain).

How to Pick a Scope

Rules:

1. If your model has an organization_id (or IS an Organization), use ORGANIZATION_SCOPE 2. If your model has a user_id (or IS a User) and no org context, use USER_SCOPE 3. If your model has an integration_id, use INTEGRATION_SCOPE 4. If your model has an api_application_id or is a SentryApp, use APP_SCOPE 5. If none of the above fit, or you have a concrete isolation concern (see above), create a new scope

The infer_identifiers() function in category.py auto-detects shard_identifier and object_identifier from model attributes based on the scope. Check its implementation to understand what field names it looks for.

Registration Mechanics

Adding a New Category

1. Add a new member to OutboxCategory with the next available integer value 2. Add the category to the appropriate OutboxScope member's scope_categories() call 3. The scope_categories() helper asserts no category is registered twice

# In OutboxCategory enum:
MY_NEW_CATEGORY = 45  # Next available value

# In OutboxScope enum, add to the appropriate scope:
ORGANIZATION_SCOPE = scope_categories(0, {
    OutboxCategory.ORGANIZATION_UPDATE,
    # ... existing categories ...
    OutboxCategory.MY_NEW_CATEGORY,  # Add here
})

Adding a New Scope

# In OutboxScope enum:
MY_NEW_SCOPE = scope_categories(13, {  # Next available integer
    OutboxCategory.MY_NEW_CATEGORY,
})

Then update infer_identifiers() to handle the new scope — add a branch that maps the scope to the correct model attribute for shard_identifier.

Retiring a Category

Categories that are no longer in use should:

1. Keep their enum value (never reuse integer values) 2. Add a # no longer in use comment 3. Stay in their OutboxScope registration (removing causes assertion failures for in-flight outboxes)

Identifier Inference

OutboxCategory.infer_identifiers(scope, model) auto-detects identifiers by scope:

If inference fails (model doesn't have the expected attribute), pass shard_identifier explicitly to outbox_for_update().

Debugging Stuck Outboxes

Processing Pipeline

Understanding the pipeline helps locate where things break:

1. Model save/delete writes outbox row inside outbox_context(transaction.atomic(...)) 2. On commit: if flush=True, drain_shard() runs synchronously for that shard 3. Periodic task: enqueue_outbox_jobs (cell) / enqueue_outbox_jobs_control (control) runs on a cron schedule 4. `schedule_batch` partitions the ID range into CONCURRENCY=5 chunks and spawns drain_outbox_shards tasks 5. `drain_outbox_shards` calls process_outbox_batch which:

• Calls find_scheduled_shards(lo, hi) to find shards with scheduled_for <= now
• Calls prepare_next_from_shard(shard) to lock the first message and bump backoff
• Calls shard_outbox.drain_shard(flush_all=True) to process the shard

6. `drain_shard` loops: process_shard() (lock) -> process() -> process_coalesced() -> send_signal() 7. Signal receiver fires for the OutboxCategory, executing the handler logic (RPC calls, tombstones, etc.) 8. On success: coalesced outbox rows are deleted in batches of 50

Backoff Schedule

When processing fails, prepare_next_from_shard bumps scheduled_for using exponential backoff:

Attempt 1: now + 2 * last_delay  (initial delay ~seconds)
Attempt 2: now + 4 * last_delay
Attempt 3: now + 8 * last_delay
...
Maximum: 1 hour between retries

The backoff is computed as:

def next_schedule(self, now):
    return now + min((self.last_delay() * 2), datetime.timedelta(hours=1))

Where last_delay() is scheduled_for - scheduled_from (time since last attempt).

Constructing Diagnostic SQL Queries

When debugging stuck outboxes, you'll often need to generate SQL for a developer to run against production PostgreSQL. Follow these rules to construct the correct query.

Choosing the Correct Table

How to determine direction: Look at the model that changed.

• If the source model is decorated @cell_silo_model (or inherits ReplicatedCellModel), it writes to sentry_regionoutbox
• If the source model is decorated @control_silo_model (or inherits ReplicatedControlModel), it writes to sentry_controloutbox

Column Reference

Both tables share these columns:

sentry_controloutbox has one additional column:

Resolving Enum Values

Before constructing a query, resolve the integer values for the category and scope from src/sentry/hybridcloud/outbox/category.py. Read the file to get the exact values. For example:

• OutboxCategory.ORGANIZATION_MEMBER_UPDATE = 3
• OutboxScope.ORGANIZATION_SCOPE = 0

Always include the resolved enum names as SQL comments so the developer knows what the magic numbers mean.

Query Templates

When generating SQL for a developer, print the query to the terminal so they can copy-paste it into a production psql session. Always include:

1. A comment header explaining what the query does 2. Comments mapping integer values to their enum names 3. Reasonable LIMIT clauses to avoid overwhelming output

Find stuck shards (cell)

-- Find cell outbox shards stuck in backoff
-- shard_scope: 0 = ORGANIZATION_SCOPE, 1 = USER_SCOPE, etc.
-- category: see OutboxCategory enum in category.py
SELECT
    shard_scope,
    shard_identifier,
    category,
    count(*) AS depth,
    min(scheduled_for) AS next_attempt,
    min(date_added) AS oldest_message,
    max(date_added) AS newest_message
FROM sentry_regionoutbox
WHERE scheduled_for > NOW()
GROUP BY shard_scope, shard_identifier, category
ORDER BY depth DESC
LIMIT 20;

Find stuck shards (control)

-- Find control outbox shards stuck in backoff
SELECT
    region_name,
    shard_scope,
    shard_identifier,
    category,
    count(*) AS depth,
    min(scheduled_for) AS next_attempt
FROM sentry_controloutbox
WHERE scheduled_for > NOW()
GROUP BY region_name, shard_scope, shard_identifier, category
ORDER BY depth DESC
LIMIT 20;

Inspect a specific shard

-- Inspect messages in a specific shard (most recent first)
-- Replace <scope>, <shard_id> with actual values
SELECT
    id,
    category,
    object_identifier,
    payload,
    scheduled_from,
    scheduled_for,
    date_added
FROM sentry_regionoutbox
WHERE shard_scope = <scope>  -- e.g., 0 = ORGANIZATION_SCOPE
  AND shard_identifier = <shard_id>  -- e.g., the organization_id
ORDER BY id DESC
LIMIT 50;

Check depth for a specific category

-- Count pending outboxes for a specific category
-- category: <N> = <CATEGORY_NAME>
SELECT count(*) AS pending
FROM sentry_regionoutbox
WHERE category = <N>;

Find outboxes for a specific object

-- Find all outboxes for a specific model instance
-- category: <N> = <CATEGORY_NAME>
SELECT
    id,
    shard_scope,
    shard_identifier,
    payload,
    scheduled_from,
    scheduled_for,
    date_added
FROM sentry_regionoutbox
WHERE category = <N>
  AND object_identifier = <object_id>
ORDER BY id DESC
LIMIT 20;

Top shards by depth (overall health check)

-- Top 10 deepest shards across all scopes/categories
SELECT
    shard_scope,
    shard_identifier,
    count(*) AS depth
FROM sentry_regionoutbox
GROUP BY shard_scope, shard_identifier
ORDER BY depth DESC
LIMIT 10;

Agent Instructions for SQL Generation

When a developer asks you to debug stuck outboxes:

1. Determine the table: Ask which model or direction is involved, or infer from context. Use sentry_regionoutbox for cell models, sentry_controloutbox for control models. 2. Resolve enum values: Read src/sentry/hybridcloud/outbox/category.py to get the integer values for the relevant OutboxCategory and OutboxScope. 3. Construct the query: Use the templates above, substituting resolved values. Always add comments with the human-readable enum names. 4. Print to terminal: Output the final SQL so the developer can copy it. Do NOT attempt to run it — you don't have production database access. 5. Explain what to look for: Tell the developer what the results mean (e.g., "if scheduled_for is far in the future, the shard is in exponential backoff after repeated failures").

Kill Switches

Disable Specific Shards

The should_skip_shard() method checks these options:

# Skip specific organization shards (cell outboxes)
"hybrid_cloud.authentication.disabled_organization_shards": [org_id_1, org_id_2]

# Skip specific user shards (cell/control outboxes)
"hybrid_cloud.authentication.disabled_user_shards": [user_id_1, user_id_2]

When a shard is skipped, its outboxes remain in the table but are not processed until the option is removed.

Disable Backfills

Set the option value lower than the code's replication_version to prevent a backfill from running:

# If model.replication_version = 3, setting this to 2 prevents the v3 backfill:
"outbox_replication.sentry_mymodel.replication_version": 2

See references/backfill.md for details on how find_replication_version() uses min(option_value, coded_version).

Useful Metrics

Check Shard Depths Programmatically

For local debugging or in a Django shell:

from sentry.hybridcloud.models.outbox import CellOutbox, ControlOutbox

# Top 10 deepest cell shards
for shard in CellOutbox.get_shard_depths_descending(limit=10):
    print(f"Scope={shard['shard_scope']} ID={shard['shard_identifier']} Depth={shard['depth']}")

Common Debugging Scenarios

Outbox Rows Accumulating But Not Processing

1. Check if enqueue_outbox_jobs task is running (Taskbroker / cron) 2. Check if drain_outbox_shards tasks are being spawned (check Taskbroker queue) 3. Check if specific shards are disabled via kill switches 4. Check if all shards are in backoff (scheduled_for > now()) 5. Check if the signal handler is crashing or raising any exceptions

Handler Raising Exceptions

1. In tests: OutboxFlushError wraps the original exception with the outbox details 2. In production: errors are captured to Sentry — search for the outbox category name 3. Check the signal receiver code for the OutboxCategory value in the stuck outbox

Data Replicated But Stale

1. Outboxes are coalesced — intermediate updates are skipped 2. Check that the handler reads from the DB (not the payload) for current data 3. If using payload_for_update(), ensure the payload contains only immutable or slowly-changing data

Test Outbox Issues

• `OutboxFlushError`: The signal receiver raised an exception during outbox_runner(). Read the nested exception.
• `OutboxRecursionLimitError`: More than 10 drain iterations — likely an outbox handler that creates more outboxes in an infinite loop.
• Outbox not created: Ensure the model inherits from the right mixin and the manager is a producing manager. Raw QuerySet.update() / QuerySet.delete() bypass outbox creation.

Signal Receiver Reference

Overview

Manual signal receivers are used for OutboxCategory values that are not tied to a ReplicatedCellModel or ReplicatedControlModel. The model mixins auto-connect receivers via connect_cell_model_updates() / connect_control_model_updates() — you only write manual receivers for categories with custom dispatch logic.

Source files:

• src/sentry/receivers/outbox/cell.py — cell outbox receivers
• src/sentry/receivers/outbox/control.py — control outbox receivers
• src/sentry/receivers/outbox/__init__.py — maybe_process_tombstone helper

Placement Rules

• Cell outbox receivers go in src/sentry/receivers/outbox/cell.py (or a new file under src/sentry/receivers/outbox/)
• Control outbox receivers go in src/sentry/receivers/outbox/control.py (or a new file under src/sentry/receivers/outbox/)
• Receivers must be imported at startup to register. Check that the receiver module is imported in src/sentry/receivers/__init__.py or a file that is.

Cell Outbox Receivers

Cell outbox signals fire with these keyword arguments:

• sender: OutboxCategory enum value
• payload: dict | None — the JSON payload from the outbox
• object_identifier: int — the ID of the source object
• shard_identifier: int — the shard key (e.g., organization_id)
• shard_scope: int — the OutboxScope value

Template: Payload-Only Receiver

For categories that carry all data in the payload (no DB lookup needed):

from django.dispatch import receiver
from sentry.hybridcloud.outbox.signals import process_cell_outbox
from sentry.hybridcloud.outbox.category import OutboxCategory


@receiver(process_cell_outbox, sender=OutboxCategory.MY_CATEGORY)
def process_my_category(payload: Any, **kwds: Any) -> None:
    if payload is not None:
        my_rpc_service.do_something(data=MyRpcData(**payload))

Template: Tombstone-Check Receiver

For categories tied to a model where you need to detect create/update vs delete:

from django.dispatch import receiver
from sentry.hybridcloud.outbox.signals import process_cell_outbox
from sentry.hybridcloud.outbox.category import OutboxCategory
from sentry.receivers.outbox import maybe_process_tombstone


@receiver(process_cell_outbox, sender=OutboxCategory.MY_CATEGORY)
def process_my_category(object_identifier: int, **kwds: Any) -> None:
    if (instance := maybe_process_tombstone(MyModel, object_identifier)) is None:
        return  # Object was deleted — tombstone recorded
    # Object exists — replicate
    my_rpc_service.sync(model_id=instance.id, data=serialize(instance))

Template: Payload + Tombstone Receiver

When you need both the payload and a tombstone check:

@receiver(process_cell_outbox, sender=OutboxCategory.MY_CATEGORY)
def process_my_category(object_identifier: int, payload: Any, **kwds: Any) -> None:
    if (instance := maybe_process_tombstone(MyModel, object_identifier)) is None:
        return
    if payload and "extra_field" in payload:
        my_rpc_service.sync_with_extra(
            model_id=instance.id,
            extra_field=payload["extra_field"],
        )

Control Outbox Receivers

Control outbox signals include an additional cell_name argument:

• sender: OutboxCategory enum value
• payload: dict | None
• object_identifier: int
• shard_identifier: int
• cell_name: str — the target cell
• shard_scope: int
• date_added: datetime
• scheduled_for: datetime

Template: Control Tombstone-Check Receiver

from django.dispatch import receiver
from sentry.hybridcloud.outbox.signals import process_control_outbox
from sentry.hybridcloud.outbox.category import OutboxCategory
from sentry.receivers.outbox import maybe_process_tombstone


@receiver(process_control_outbox, sender=OutboxCategory.MY_CATEGORY)
def process_my_category(object_identifier: int, cell_name: str, **kwds: Any) -> None:
    if (instance := maybe_process_tombstone(
        MyModel, object_identifier, cell_name=cell_name
    )) is None:
        return
    # Replicate to the specific cell
    my_cell_service.sync(cell_name=cell_name, data=serialize(instance))

Template: Control Pure-RPC Receiver

For categories where the receiver makes an RPC call without looking up a model:

@receiver(process_control_outbox, sender=OutboxCategory.MY_CATEGORY)
def process_my_category(
    payload: Mapping[str, Any], shard_identifier: int, **kwds: Any
) -> None:
    my_cell_service.do_something(
        organization_id=shard_identifier,
        data=payload["data"],
    )

maybe_process_tombstone Pattern

def maybe_process_tombstone(
    model: type[T],
    object_identifier: int,
    cell_name: str | None = None,
) -> T | None:

This function:

1. Queries model.objects.filter(id=object_identifier).last() 2. If found: returns the instance (for replication) 3. If not found: records a tombstone via cell_tombstone_service or control_tombstone_service and returns None

The tombstone system drives HybridCloudForeignKey cascade deletes across silos. When an object is deleted from one silo, the tombstone propagated to the other silo triggers cleanup of dependent records.

When to use: Any receiver that needs to distinguish between "object was created/updated" and "object was deleted". Not needed for payload-only categories (audit logs, IP events) where the payload carries all necessary data.

`cell_name` parameter: Pass cell_name for control outbox receivers (tombstone goes to the cell). Omit for cell outbox receivers (tombstone goes to control).